1. Other Issues - § 3.9 – Not Discussed
More advanced algorithm construction – regular expression to DFA directly

2. Final Notes : R.E. to NFA Construction
So, an NFA may be simulated by algorithm, when NFA is constructed using Previous techniques
Algorithm run time is proportional to |N| * |x| where |N| is the number of states and |x| is the length of input
Alternatively, we can construct DFA from NFA and use the resulting Dtran to recognize input:
space required
O(|r|)
O(|r|*|x|)
O(|x|)
O(2|r|)
DFA
NFA
time to simulate
where |r| is the length of the regular expression.

3. Pulling Together Concepts
Designing Lexical Analyzer Generator
Reg. Expr.  NFA construction
NFA  DFA conversion
DFA simulation for lexical analyzer
Recall Lex Structure
Pattern Action
… …
Each pattern recognizes lexemes
Each pattern described by regular expression
(a | b)*abb 
e.g. 
(abc)*ab 
etc.
Recognizer!

4. Lex Specification  Lexical Analyzer
Let P1, P2, … , Pn be Lex patterns
(regular expressions for valid tokens in prog. lang.)
Construct N(P1), N(P2), … N(Pn)
Note: accepting state of N(Pi) will be marked by Pi
Construct NFA:
N(P1)
Lex applies conversion algorithm to construct DFA that is equivalent!  
N(P2) 
N(Pn)

5. Pictorially Lex Specification Lex Compiler Transition Table
(a) Lex Compiler
lexeme
input buffer FA
Simulator
Transition Table
(b) Schematic lexical analyzer

6. Example P1 : a P2 : abb P3 : a*b+ 3 patterns NFA’s : start 1 b a 2 3 45 8 7 6 P1 P2 P3

7. Example – continued (2) Combined NFA :  1 a P1 2  a b b start P2 3 4P2 3 4 5 6 a b  P3 7 8 b
Examples a a b a {0,1,3,7} {2,4,7} {7} {8} death pattern matched: P P
a b b {0,1,3,7} {2,4,7} {5,8} {6,8} pattern matched: P P3 P2,P3 
break tie in favor of P2

8. Example – continued (3)
Alternatively Construct DFA: (keep track of correspondence between patterns and new accepting states) P2
{8} -
{6,8} P3
{5,8}
none
{7} P1
{2,4,7}
{0,1,3,7}
Pattern b a
STATE
Input Symbol
break tie in favor of P2

9. Minimizing the Number of States of DFA
Construct initial partition  of S with two groups: accepting/ non-accepting.
(Construct new )For each group G of  do begin
Partition G into subgroups such that two states s,t of G are in the same subgroup iff for all symbols a states s,t have transitions on a to states of the same group of .
Replace G in new by the set of all these subgroups.
Compare new and . If equal, final:=  then proceed to 4, else set  := new and goto 2.
Aggregate states belonging in the groups of final

10. example a a A a F B a b b a D C b b b a a
A,C,D
B,F b
Minimized DFA: b

11. Using LEX Lex Program Structure:
declarations %% translation rules %% auxiliary procedures
Name the file e.g. test.lex Then, “lex test.lex” produces the file “lex.yy.c” (a C-program)

12. LEX %{
/* definitions of all constants LT, LE, EQ, NE, GT, GE, IF, THEN, ELSE, ... */ %}
......
letter [A-Za-z]
digit [0-9]
id {letter}({letter}|{digit})* %%
if { return(IF);}
then { return(THEN);}
{id} { yylval = install_id(); return(ID); }
install_id()
{ /* procedure to install the lexeme to the ST */
C declarations
declarations
Rules
Auxiliary

13. Example of a Lex Program
int num_lines = 0, num_chars = 0; %%
\n {++num_lines; ++num_chars;} {++num_chars;}
main( argc, argv ) int argc; char **argv;
{ argv, --argc; /* skip over program name */ if ( argc > 0 ) yyin = fopen( argv[0], "r" ); else yyin = stdin; yylex(); printf( "# of lines = %d, # of chars = %d\n", num_lines, num_chars ); }

14. Another Example %{ #include <stdio.h> %} WS [ \t\n]* %%
[ ] printf("NUMBER\n");
[a-zA-Z][a-zA-Z0-9]* printf("WORD\n");
{WS} /* do nothing */
printf(“UNKNOWN\n“);
main( argc, argv )
int argc; char **argv;
{ ++argv, --argc; if ( argc > 0 ) yyin = fopen( argv[0], "r" );
else yyin = stdin;
yylex(); }

15. Concluding Remarks Looking Ahead:
Focused on Lexical Analysis Process, Including
- Regular Expressions
Finite Automaton
Conversion
Lex
Interplay among all these various aspects of lexical analysis
Looking Ahead:
The next step in the compilation process is Parsing:
Top-down vs. Bottom-up
- Relationship to Language Theory