1. Recognition of Tokens

2. Recognition of Tokens A grammar for branching statements
stmt  if expr then stmt | if expr then stmt else stmt | 
expr  term relop term | term
term  id | number

3. Example Patterns for tokens in the grammar
digit  [0-9] digits  digit+ number  digits (. digits)? (E [+|-]? digits )? id  letter (letter |digit)* if  if then  then else  else relop  < | > | <= | >= | = | < >
ws  (blank | tab | newline)+

4. Tokens, their patterns, and attribute values
Lexemes
Token name
Attribute value
Any ws if
then
else
Any id
Any number
<
<= =
<>
>
>= - id
number
relop
Pointer to table entry LT LE EQ NE GT GE

5. Example C=a+b*5 <id, pointer to symbol table entry>
<relop, EQ>
<assign_op, ->
<multi_op, ->
<num, pointer to symbol table entry>

6. Transition Diagrams
Nodes: states, conditions that could occur during the process of scanning the input looking for a lexeme that matches one of several patterns
Edges: directed from state to state
Labeled by a symbol or set of symbols
Deterministic: there’s never more than one edge out of a given state with a given symbol among its labels
Certain states are accepting or final: a lexeme has been found
Double circle
If it’s necessary to retract the forward pointer, we shall additionally place a * near that accepting state
Start state, or initial state, is indicated by an edge, labeled “start”, entering from nowhere
start
< = 1 2
return(relop, LE)
> 3
return(relop, NE)
other * 4
return(relop, LT)

7. Example Transition Diagram for relop
start
< = 1 2
return(relop, LE)
> 3 =
return(relop, NE)
other * 4
return(relop, LT) 5
>
return(relop, EQ) = 6 7
return(relop, GE)
other * 8
return(relop, GT)

8. Recognition of Reserved Words and Identifiers
Problem: keywords look like identifiers
Solution:
Install the reserved words in the symbol table initially
Create separate transition diagrams for each keyword

9. Examples for Identifiers and Keywords*
start
letter
other 9 10 11
return(getToken(), installID())
letter or digit *

10. Completion of the Running Example – Unsigned Numbers
3.14
314

11. Transition Diagram for Whitespace
delim *
start
delim
other 22 23 24
delim -> blank | tab | newline

12. Transition Diagram 1 2 3 4 5 7 8 6 start < = > * other1 2 3 4 5 7 8 6
start
< =
> *
other
return(relop, EQ) 9 10 11
start
letter
other 12 13 20
start
digit
other * 14 15 21 . 17 18 19 16
+ or - E 22 23 24
start
delim
other *
Transition Diagram

13. C code to find next start state

14. C Code for Lexical analyzers

15. Finite Automata Finite automata are recognizers Two kinds:
They simply say “yes” or “no” about each input string
Two kinds:
Nondeterministic finite automata (NFA)
No restrictions on the labels of the edges
Deterministic finite automata (DFA)
For each state, and for each symbol, there’s exactly one edge with that symbol leaving that state

16. Nondeterministic Finite Automata
NFA consists of
A finite set of states S
A set of input symbol , the input alphabet
A transition function that gives, for each state, and for each symbol in ∪{} a set of states
A state s0 from S (the start state or initial state)
A set of states F, a subset of S (the accepting states, or final states)

17. NFA can be represented by a transition graph
There’s an edge labeled a from state s to state t iff t is one of the next states for state s and input a
It’s similar to a transition diagram except:
The same symbol can label edges from one state to several different states
An edge may be labeled by , in addition to symbols from the input alphabet

18. An Example NFA: (a|b)*abb
Transition Tables
Transition Graph a
State a b 
{0, 1}
{0}  1
{2} 2
{3} 3 1 3 a 2 b
start

19. Example NFA: aa*|bb* a a 1 3 
start b 2 4  b

20. Deterministic Finite Automata
DFA is a special case of an NFA where:
There are no moves on input 
For each state s and input symbol a, there’s exactly one edge out of s labeled s
Every regular expression and every NFA can be converted to a DFA accepting the same language

21. Example DFA accepting (a|b)*abb
start a b b 1 2 3 a a a

22. Construction of an NFA from a Regular Expression (Thomson’s algorithm)
Basis:
For expression , construct the NFA
For subexpression a in , construct the NFA
start  i f
start a i f

23. NFA for the concatenation of two regular expressions N(s).N(t)
start
N(s)
N(t) i f
abb a b b
start 1 2 3

24. NFA for the union of two regular expressions r=N(s)|N(t) 
start i f  
N(t) a
a|b 1 2  
start 5 b   3 4 

25. NFA for the closure of a regular expression N(s)*
start 
N(s)  i f  
(a|b)* a 2 3  
start   1 6 7 b   4 5 

26. NFA for (a|b)*abb#         a 2 3 start a b b # 1 6 7 8 9 10 111 6 7 8 9 10 11 b   4 5 