To use the token approach, you insert a stream tokenizer between your input-stream reader and your program. You do not need a buffered reader.
To create a stream tokenizer connected to an input-stream reader, you use an
expression that creates an instance of the StreamTokenizer
class
for that input-stream reader. In the following statement, for example, you
declare a StreamTokenizer
variable, and create a
StreamTokenizer
instance from a InputStreamReader
instance:
Input-stream reader instance --* | v StreamTokenizer tokens = new StreamTokenizer(reader);
StreamTokenizer
instances are called either stream tokenizers
or just tokenizers.