|
|
|
|
|
|
|
Whenever the token is a whitespace-delimited
string, that token is assigned to the sval instance variable of the
tokenizer. For example, if movie names are included in the file, each word
in those movie names becomes a value of the sval instance variable:
*-- First token *--------------------*
| | tokens |
| | sval: "Apocalypse |
v *--------------------*
-----------
"Apocalypse Now" 4 7 ...
*-- Second token *--------------------*
| | tokens |
| | sval: Now" |
v *--------------------*
----
"Apocalypse Now" 4 7 ...