Recognition of Tokens

Recognition of Tokens A grammar for branching statements stmt  if expr then stmt | if expr then stmt else stmt |  expr  term relop term | term term  id | number

Example Patterns for tokens in the grammar digit  [0-9] digits  digit+ number  digits (. digits)? (E [+|-]? digits )? id  letter (letter |digit)* if  if then  then else  else relop  < | > | <= | >= | = | < > ws  (blank | tab | newline)+

Tokens, their patterns, and attribute values Lexemes Token name Attribute value Any ws if then else Any id Any number < <= = <> > >= - id number relop Pointer to table entry LT LE EQ NE GT GE

Example C=a+b*5 <id, pointer to symbol table entry> <relop, EQ> <assign_op, -> <multi_op, -> <num, pointer to symbol table entry>

Transition Diagrams Nodes: states, conditions that could occur during the process of scanning the input looking for a lexeme that matches one of several patterns Edges: directed from state to state Labeled by a symbol or set of symbols Deterministic: there’s never more than one edge out of a given state with a given symbol among its labels Certain states are accepting or final: a lexeme has been found Double circle If it’s necessary to retract the forward pointer, we shall additionally place a * near that accepting state Start state, or initial state, is indicated by an edge, labeled “start”, entering from nowhere start < = 1 2 return(relop, LE) > 3 return(relop, NE) other * 4 return(relop, LT)

Example Transition Diagram for relop start < = 1 2 return(relop, LE) > 3 = return(relop, NE) other * 4 return(relop, LT) 5 > return(relop, EQ) = 6 7 return(relop, GE) other * 8 return(relop, GT)

Recognition of Reserved Words and Identifiers Problem: keywords look like identifiers Solution: Install the reserved words in the symbol table initially Create separate transition diagrams for each keyword

Examples for Identifiers and Keywords * start letter other 9 10 11 return(getToken(), installID()) letter or digit *

Completion of the Running Example – Unsigned Numbers 3.14 314

Transition Diagram for Whitespace delim * start delim other 22 23 24 delim -> blank | tab | newline

Transition Diagram 1 2 3 4 5 7 8 6 start < = > * other 1 2 3 4 5 7 8 6 start < = > * other return(relop, EQ) 9 10 11 start letter other 12 13 20 start digit other * 14 15 21 . 17 18 19 16 + or - E 22 23 24 start delim other * Transition Diagram

C code to find next start state

C Code for Lexical analyzers

Finite Automata Finite automata are recognizers Two kinds: They simply say “yes” or “no” about each input string Two kinds: Nondeterministic finite automata (NFA) No restrictions on the labels of the edges Deterministic finite automata (DFA) For each state, and for each symbol, there’s exactly one edge with that symbol leaving that state

Nondeterministic Finite Automata NFA consists of A finite set of states S A set of input symbol , the input alphabet A transition function that gives, for each state, and for each symbol in ∪{} a set of states A state s0 from S (the start state or initial state) A set of states F, a subset of S (the accepting states, or final states)

NFA can be represented by a transition graph There’s an edge labeled a from state s to state t iff t is one of the next states for state s and input a It’s similar to a transition diagram except: The same symbol can label edges from one state to several different states An edge may be labeled by , in addition to symbols from the input alphabet

An Example NFA: (a|b)*abb Transition Tables Transition Graph a State a b  {0, 1} {0}  1 {2} 2 {3} 3 1 3 a 2 b start

Example NFA: aa*|bb* a a 1 3  start b 2 4  b

Deterministic Finite Automata DFA is a special case of an NFA where: There are no moves on input  For each state s and input symbol a, there’s exactly one edge out of s labeled s Every regular expression and every NFA can be converted to a DFA accepting the same language

Example DFA accepting (a|b)*abb start a b b 1 2 3 a a a

Construction of an NFA from a Regular Expression (Thomson’s algorithm) Basis: For expression , construct the NFA For subexpression a in , construct the NFA start  i f start a i f

NFA for the concatenation of two regular expressions N(s).N(t) start N(s) N(t) i f abb a b b start 1 2 3

NFA for the union of two regular expressions r=N(s)|N(t)   start i f   N(t) a a|b 1 2   start 5 b   3 4 

NFA for the closure of a regular expression N(s)*  start  N(s)  i f   (a|b)* a 2 3   start   1 6 7 b   4 5 

NFA for (a|b)*abb#         a 2 3 start a b b # 1 6 7 8 9 10 11 1 6 7 8 9 10 11 b   4 5 