1. Lesson Objectives Aims Key Words
Be able to understand Lexical Analysis
Define the terms Lexical Analysis, Syntax, Symbol and Token
Key Words

2. Stages of Compilation

3. Lexical Analysis
Lexical analysis is the act of “parsing” source code
Parsing means to read in sections and “make sense” of it
The outcome is a “tokenised” version of the code and a symbol table

4. Parse each line (work out what is a variable, operator, function name, key word etc)
Tokenise each unique element (tokens for key words etc will already be defined)
Token stream

5. Symbol Table
Token
Identifier
Key word/Type
001 IF
002
THEN
003
MyName
VARIABLE
Any time an identifier is found in the code, the token will be put in its place
Tokens are unique and not duplicated
The symbol table should store:
an index or identifier
the name of the token
data type
scope and restrictions.

6. Why
Compilers/computers do not need human readable identifiers
Long identifiers and key words take up more space in memory
Identifying each unique token ensures efficient compilation

7. Lexical analysis
In order to do this, code must first be stripped of unnecessary data
Spaces
Comments
During this task, delimiters and end of line symbols etc are used to break code down in to its “atomic” elements
An atomic piece of code is one single key word, operator or name

8. Errors generated at this stage:
Things that don’t match a key word
Or identifiers
Or a literal value

9. Things it does NOT do… Lexical analysis does NOT “read” the code
No syntax errors are generated at this stage
It is checking for patterns:
DIMMM Jiggle AS Interrrrger
This would pass lexical analysis – but is clearly syntactically incorrect
Why would it pass lexical analysis?

10. How does this work?
Lexical analysis uses regular expressions (pattern matching) to try and find code that is acceptable
Regular expressions can be quite complex, and make use of a set of symbols that have individual meanings…

11. Regular Expression Symbols
Some meanings:
Symbol
Meaning []
A set | OR *
0 or more ?
0 or 1 $
End of line ^
NOT

12. Example

13. Tokenise this line of code: IF A>5 THEN A = A * 2
Task
Tokenise this line of code:
IF A>5 THEN A = A * 2
Make a symbol table
Identify each unique key word
Output a token stream
Token
Type
Identifier/Value 1 IF -

15. Review/Success Criteria
You should know:
What lexical analysis is
What lexical analysis does
How
What it cannot do!