L2 Lexical Analysis

if (i == j)
    z = 0;
else
    z = 1;

\tif (i == j)\n\t\tz = 0;\n\telse\n\t\tz = 1;

Token Class(or Class)

• In English

  • Noun, Verb, Adjective...

• In a programming language:

  • Identifier, keywords,'(', ')', Numbers

• Token classes correspond to sets of strings.

• Identifier:

  • strings of lettters or digits, starting with a letter.

• Integer:

  • a non-empty string of digits.

• Keyword

  • "else" or "if" or "begin" or...

• Whitespace:

  • a non-empty sequence of blanks, newlines, and tabs.

• Classify program substrings according to role.

• Communicate tokens to the parser.

\tif (i == j)\n\t\tz = 0;\n\telse\n\t\tz = 1;

• Operator
• Whitespace
• Keywords
• Identifiers
• Numbers
• '(', ')', ; , =

WKWIWOWI)WIW=N...

An implementation must do two things:

1. Recognize substrings corresponding to tokens
   • The lexemes
2. Identify the token class of each lexeme
   • <token class, lexeme> is called a token.