Lesson Objectives Aims Key Words Be able to understand Lexical Analysis Define the terms Lexical Analysis, Syntax, Symbol and Token Key Words

Stages of Compilation

Lexical Analysis Lexical analysis is the act of “parsing” source code Parsing means to read in sections and “make sense” of it The outcome is a “tokenised” version of the code and a symbol table

Parse each line (work out what is a variable, operator, function name, key word etc) Tokenise each unique element (tokens for key words etc will already be defined) Token stream

Symbol Table Token Identifier Key word/Type 001 IF 002 THEN 003 MyName VARIABLE Any time an identifier is found in the code, the token will be put in its place Tokens are unique and not duplicated The symbol table should store: an index or identifier the name of the token data type scope and restrictions.

Why Compilers/computers do not need human readable identifiers Long identifiers and key words take up more space in memory Identifying each unique token ensures efficient compilation

Lexical analysis In order to do this, code must first be stripped of unnecessary data Spaces Comments During this task, delimiters and end of line symbols etc are used to break code down in to its “atomic” elements An atomic piece of code is one single key word, operator or name

Errors generated at this stage: Things that don’t match a key word Or identifiers Or a literal value

Things it does NOT do… Lexical analysis does NOT “read” the code No syntax errors are generated at this stage It is checking for patterns: DIMMM Jiggle AS Interrrrger This would pass lexical analysis – but is clearly syntactically incorrect Why would it pass lexical analysis?

How does this work? Lexical analysis uses regular expressions (pattern matching) to try and find code that is acceptable Regular expressions can be quite complex, and make use of a set of symbols that have individual meanings…

Regular Expression Symbols Some meanings: Symbol Meaning [] A set | OR * 0 or more ? 0 or 1 $ End of line ^ NOT

Example

Tokenise this line of code: IF A>5 THEN A = A * 2 Task Tokenise this line of code: IF A>5 THEN A = A * 2 Make a symbol table Identify each unique key word Output a token stream Token Type Identifier/Value 1 IF -

1 2 3 4 5 2 6 2 7 8

Review/Success Criteria You should know: What lexical analysis is What lexical analysis does How What it cannot do!