PerCederberg.Grammatica.Parser
Class Tokenizer

System.Object
   |
   +--Tokenizer

class Tokenizer

extends System.Object

A character stream tokenizer. This class groups the characters read from the stream together into tokens ("words"). The grouping is controlled by token patterns that contain either a fixed string to search for, or a regular expression. If the stream of characters don't match any of the token patterns, a parse exception is thrown.

Tokenizer

public Tokenizer( TextReader input );

Creates a new tokenizer for the specified input stream.

Parameters:

input - the input stream to read

AddPattern

public void AddPattern( TokenPattern pattern );

Adds a new token pattern to the tokenizer. The pattern will be added last in the list, choosing a previous token pattern in case two matches the same string.

Parameters:

pattern - the pattern to add

Throws:

ParserCreationException - if the pattern couldn't be added to the tokenizer

GetCurrentColumn

public int GetCurrentColumn();

Returns the current column number. This number will be the column number of the next token returned.

Returns:

the current column number

GetCurrentLine

public int GetCurrentLine();

Returns the current line number. This number will be the line number of the next token returned.

Returns:

the current line number

GetPatternDescription

public string GetPatternDescription( int id );

Returns a description of the token pattern with the specified id.

Parameters:

id - the token pattern id

Returns:

the token pattern description, or null if not present

GetUseTokenList

public bool GetUseTokenList();

Checks if the token list feature is used. The token list feature makes all tokens (including ignored tokens) link to each other in a linked list. By default the token list feature is not used.

Returns:

true if the token list feature is used, or false otherwise

Since:

1.4

See Also:

setUseTokenList, Token.getPreviousToken, Token.getNextToken

Next

public Token Next();

Finds the next token on the stream. This method will return null when end of file has been reached. It will return a parse exception if no token matched the input stream, or if a token pattern with the error flag set matched. Any tokens matching a token pattern with the ignore flag set will be silently ignored and the next token will be returned.

Returns:

the next token found, or null if end of file was encountered

Throws:

ParseException - if the input stream couldn't be read or parsed correctly

SetUseTokenList

public void SetUseTokenList( bool useTokenList );

Sets the token list feature flag. The token list feature makes all tokens (including ignored tokens) link to each other in a linked list when active. By default the token list feature is not used.

Parameters:

useTokenList - the token list feature flag

Since:

1.4

See Also:

getUseTokenList, Token.getPreviousToken, Token.getNextToken

ToString

public override string ToString();

Returns a string representation of this object. The returned string will contain the details of all the token patterns contained in this tokenizer.

Returns:

a detailed string representation