|
|
|
|
|
|
|
As an illustration of other features of tokenizers, suppose that you have a file that supplies not only movie titles and rating information, but also, optionally, names of poster files, which you learn to use in Chapter 44:
"Apocalypse Now" 4 7 3 "apnow.jpg" "Bedtime for Bonzo" 8 8 7
An ordinary tokenizer treats all carriage returns as though they were spaces; it treats double quotation marks as ordinary characters. Accordingly, you need to learn how to alter a tokenizer such that it will recognize carriage returns. You also need to learn how to use double quotation marks to delimit strings that contain embedded spaces.