Token is the smallest element of the Pascal program that compiler can recognize. The parser separates tokens from the input stream by creating the longest token possible using the input characters in a left–to–right scan.

mikroPascal PRO for PIC recognizes the following kinds of tokens:

• keywords
• identifiers
• constants
• operators
• punctuators (also known as separators)

Token Extraction Example

Here is an example of token extraction. Take a look at the following example code sequence:

end_flag := 0;

First, note that end_flag would be parsed as a single identifier, rather than as the keyword end followed by the identifier _flag.

The compiler would parse it as the following four tokens:

end_flag  // variable identifier
:=        // assignment operator
0         // literal
;         // statement terminator

Note that := parses as one token (the longest token possible), not as token : followed by token =.

Want more examples and libraries? 
Find them on