Token is the smallest element of a mikroPascal PRO for ARM program, meaningful to the compiler. The parser separates tokens from the input stream by creating the longest token possible using the input characters in a left–to–right scan.

mikroPascal PRO for ARM recognizes the following kinds of tokens:

• keywords
• identifiers
• constants
• operators
• punctuators (also known as separators)

Token Extraction Example

Here is an example of token extraction. Take a look at the following example code sequence:

end_flag := 0;

First, note that end_flag would be parsed as a single identifier, rather than as the keyword end followed by the identifier _flag.

The compiler would parse it as the following four tokens:

end_flag  // variable identifier
:=        // assignment operator
0         // literal
;         // statement terminator

Note that := parses as one token (the longest token possible), not as token : followed by token =.

Want more examples and libraries? 
Find them on