`class TokenScanner`

This class divides a string into individual tokens. The typical use of the TokenScanner class is illustrated by the following pattern, which reads the tokens in the string variable input:

   TokenScanner scanner;
   scanner.setInput(input);
   while (scanner.hasMoreTokens()) {
      string token = scanner.nextToken();
      . . . process the token . . .
   }

The TokenScanner class exports several additional methods that give clients more control over its behavior. Those methods are described individually in the documentation.

Constructor
TokenScanner() TokenScanner(str) TokenScanner(infile)	Initializes a scanner object.
Methods
setInput(str) setInput(infile)	Sets the token stream for this scanner to the specified string or input stream.
hasMoreTokens()	Returns `true` if there are additional tokens for this scanner to read.
nextToken()	Returns the next token from this scanner.
saveToken(token)	Pushes the specified token back into this scanner's input stream.
ignoreWhitespace()()	Tells the scanner to ignore whitespace characters.
ignoreComments()	Tells the scanner to ignore comments.
scanNumbers()	Controls how the scanner treats tokens that begin with a digit.
scanStrings()	Controls how the scanner treats tokens enclosed in quotation marks.
addWordCharacters(str)	Adds the characters in `str` to the set of characters legal in a word.
addOperator(op)	Defines a new multicharacter operator.
getPosition()	Returns the current position of the scanner in the input stream.
isWordCharacter(ch)	Returns `true` if the character is valid in a word.
verifyToken(expected)	Reads the next token and makes sure it matches the string `expected`.
getTokenType(token)	Returns the type of this token.

Constructor detail

TokenScanner();
TokenScanner(string str);
TokenScanner(istream & infile);

Initializes a scanner object. The initial token stream comes from the specified string or input stream, if supplied. The default constructor creates a scanner with an empty token stream.

Usage:

TokenScanner scanner;
TokenScanner scanner(str);
TokenScanner scanner(infile);

Method detail

void setInput(string str);
void setInput(istream & infile);

Sets the token stream for this scanner to the specified string or input stream. Any previous token stream is discarded.

Usage:

scanner.setInput(str);
scanner.setInput(infile);

bool hasMoreTokens();

Returns true if there are additional tokens for this scanner to read.

Usage:

if (scanner.hasMoreTokens()) . . .

string nextToken();

Returns the next token from this scanner. If nextToken is called when no tokens are available, it returns the empty string.

Usage:

token = scanner.nextToken();

void saveToken(string token);

Pushes the specified token back into this scanner's input stream. On the next call to nextToken, the scanner will return the saved token without reading any additional characters from the token stream.

Usage:

scanner.saveToken(token);

void ignoreWhitespace();

Tells the scanner to ignore whitespace characters. By default, the nextToken method treats whitespace characters (typically spaces and tabs) just like any other punctuation mark and returns them as single-character tokens. Calling

   scanner.ignoreWhitespace();

changes this behavior so that the scanner ignore whitespace characters.

Usage:

scanner.ignoreWhitespace();

void ignoreComments();

Tells the scanner to ignore comments. The scanner package recognizes both the slash-star and slash-slash comment format from the C-based family of languages. Calling

   scanner.ignoreComments();

sets the parser to ignore comments.

Usage:

scanner.ignoreComments();

void scanNumbers();

Controls how the scanner treats tokens that begin with a digit. By default, the nextToken method treats numbers and letters identically and therefore does not provide any special processing for numbers. Calling

   scanner.scanNumbers();

changes this behavior so that nextToken returns the longest substring that can be interpreted as a real number.

Usage:

scanner.scanNumbers();

void scanStrings();

Controls how the scanner treats tokens enclosed in quotation marks. By default, quotation marks (either single or double) are treated just like any other punctuation character. Calling

   scanner.scanStrings();

changes this assumption so that nextToken returns a single token consisting of all characters through the matching quotation mark. The quotation marks are returned as part of the scanned token so that clients can differentiate strings from other token types.

Usage:

scanner.scanStrings();

void addWordCharacters(string str);

Adds the characters in str to the set of characters legal in a word. For example, calling addWordCharacters("_") adds the underscore to the set of word characters.

Usage:

scanner.addWordCharacters(str);

void addOperator(string op);

Defines a new multicharacter operator. Whenever you call nextToken when the input stream contains operator characters, the scanner returns the longest possible operator string that can be read at that point.

Usage:

scanner.addOperator(op);

int getPosition();

Returns the current position of the scanner in the input stream. If saveToken has been called, this position corresponds to the beginning of the saved token. If saveToken is called more than once, the position is unavailable.

Usage:

int pos = scanner.getPosition();

bool isWordCharacter(char ch);

Returns true if the character is valid in a word.

Usage:

if (scanner.isWordCharacter(ch)) . . .

void verifyToken(string expected);

Reads the next token and makes sure it matches the string expected. If it does not, verifyToken throws an error.

Usage:

scanner.verifyToken(expected);

TokenType getTokenType(string token);

Returns the type of this token. This type will match one of the following enumerated type constants: EOF, SEPARATOR, WORD, NUMBER, STRING, or OPERATOR.

Usage:

TokenType type = scanner.getTokenType(token);