The argument chars may be a single character, a list of characters, or a string. Each character in chars is treated as though
tok:char-groupwas called with that character alone.
The argument chars-proc must be a procedure of one argument, a list of characters. After
tokenizehas finished accumulating the characters for a token, it calls chars-proc with the list of characters. The value returned is the token which
The argument group may be an exact integer or a procedure of one character argument. The following discussion concerns the treatment which the tokenizing routine,
tokenize, will accord to characters on the basis of their groups.
When group is a non-zero integer, characters whose group number is equal to or exactly one less than group will continue to accumulate. Any other character causes the accumulation to stop (until a new token is to be read).
The group of zero is special. These characters are ignored when parsed pending a token, and stop the accumulation of token characters when the accumulation has already begun. Whitespace characters are usually put in group 0.
If group is a procedure, then, when triggerd by the occurence of an initial (no accumulation) chars character, this procedure will be repeatedly called with each successive character from the input stream until the group procedure returns a non-false value.
The following convenient constants are provided for use with
Is the string consisting of all upper-case letters ("ABCDEFGHIJKLMNOPQRSTUVWXYZ").
Is the string consisting of all lower-case letters ("abcdefghijklmnopqrstuvwxyz").