Chapter 4: Syntax Analysis Part 2: Top-Down Parsing Prof. Steven A. Demurjian Computer Science & Engineering Department The University of Connecticut 371 Fairfield Way, Unit 2155 Storrs, CT 06269-3155 steve@engr.uconn.edu http://www.engr.uconn.edu/~steve (860) 486 - 4818 Material for course thanks to: Laurent Michel Aggelos Kiayias Robert LeBarre
Overview of Top-Down Parsing Motivating Parsing and Lookaheads LL(1) Grammars Top-Down Parsing Recursive Descent Parsing Predictive Parsing Non-Recursive Table-Driven Parsing Underling Concepts for Top-Down and Bottom-Up Parsing Let is a string of grammar symbols (T and NT) Let A be a non-terminal and S the start NT Constructing the Table for Table-Driven Parsing Error Recovery
Motivation of Parsing Source We have a grammar Product We want a parsing algorithm Idea Synthesize the algorithm from the grammar Problem How are we going to use the grammar ??? Hint... We can look at the beginning of the input...
Basic Intuition Use a sliding window over input to Lookahead Benefit Reveal the input Slowly A little bit at a time 1 token at a time Maybe 2 at a time ? But systematically Left to Right What kind of derivation can use tokens in this way? The Input
Predictive Parsing Uses a Lookahead Technique Use a window of lookahead Guide a LEFTMOST derivation with the window content Helps to select the right production So...How to guide ? Match Grammar expr → assignment_expr ; | new_variable_decl_expr ; new_variable_decl_expr → id := id new id ( func_call ) ; func_call → id . id ()
Example Consider the grammar And the input fragment Which production should we start with ? Why? stmt → if expr then stmt else stmt | while expr do stmt | for ( expr ; expr ; expr) stmt | { stmtList } | lvalue = expr lvalue → id expr → id | integer while x do { if x then ....
Key Idea Use the production body To know what can start a production Selecting a production Pick the rule that can start with the symbol in the lookahead window! Using the production What should we do with chosen production? Matching Production while expr do stmt Input while x do { if x then ....
Matching When input matches Eat the matched symbols In the production In the input Move the window further down Deal with the rest of the production How? while expr do stmt Production Input while x do { if x then ....
No Matching? What does it mean? Example below In production: NON-TERMINAL expr In window: TERMINAL x [an identifier token] What should we do ? while expr do stmt Production Input while x do { if x then ....
Dealing with non-terminals Simple... The non-terminal is defined somewhere The non-terminal is defined by a set of productions Corollary Pop the non-terminal Use the lookahead to choose a production for NT. Push and Recur
On the Example Recall the grammar and the current state stmt → if expr then stmt else stmt .... lvalue → id expr → id | integer 2. Choose production for expr result: expr → id while expr do stmt Production Input while x do { if x then .... 1. Pop result: expr 3. Push & Recur... result: while id do stmt Production Input while x do { if x then ....
Gain? Recall The topmost symbol ( id ) The lookahead x [an instance of id] Gain? The top symbol and lookahead match! So.... Recur Recursive call will match them and pop them Keep on going
Final outcome What can happen in the end ? We “eat” (match) the entire input Meaning ? We get “stuck” at some point What does it mean (to be stuck) How can we get stuck ?
Negative outcome We can get stuck Lookahead window and topmost non-terminal yield... An empty prediction! This is the “classic” syntax error Expecting xyz A list of tokens predicting the productions of the current non-terminal Got abc “abc” is the actual content of the lookahead window Syntax error at file:line. Expecting xyz got abc
Overall Top-Down Algorithm Data structures A stack Holds the symbols to treat A lookahead window to choose a prediction Algorithm Startup Initialize stack to start symbol (a non-terminal) Initialize lookahead window at start of token stream Recursive Process Find out if front of window and top of stack match If match Consume the symbols No match Pop / Select / Push
How Could Algorithm Work? Grammar: E TE’ E’ +TE’ | T id Input : id + id $ Derivation: E TE’ idE’ id +TE’ id +idE’ id +id’ Processing Stack: Contains the Derivation plus $ E $ T E’ $ id E’ $ E’ $ + T E’ $ T E’ $ id E’ $ E’ $ $
Are We Done ? Almost.... A Few Remaining issues... How do we get the predictions from the grammar ? What should we do if the same symbol predicts 2 or more rules ? How can we implement the Algorithm/Process? How large should the lookahead window be ? Why is this called “top-down” ? What can be automated about all this ?
The Lookahead Window How large ? Very good question. The art of counting 1,2, a lot. Conclusion Usually, 1 token is plenty Occasionally, 2 tokens may be needed Beyond 2 is wild In theory Any finite constant will do 1,2,3....,k
Lookahead Size and Languages With k tokens of lookahead Some languages can be parsed (this way) Some languages cannot be parsed What about k+1 tokens ? More languages can be parsed.... So the set of languages recognized with k is a subset of the set of languages recognized with k+1! We have a hierarchy! Still... In practice LL(1) should be enough.
LL(1) Grammars for Top-Down Parsing L : Scan input from Left to Right L : Construct a Leftmost Derivation 1 : Use “1” input symbol as lookahead in conjunction with stack to decide on the parsing action LL(1) grammars have no multiply-defined entries in the parsing table. Properties of LL(1) grammars: Grammar can’t be ambiguous or left recursive Grammar is LL(1) when A 1. & do not derive strings starting with the same terminal a 2. Either or can derive , but not both. Note: It may not be possible for a grammar to be manipulated into an LL(1) grammar
Top-Down Parsing Identify a leftmost derivation for an input string Why ? By always replacing the leftmost non-terminal symbol via a production rule, we are guaranteed of developing a parse tree in a left-to-right fashion consistent with scanning the input. A aBc adDc adec (scan a, scan d, scan e, scan c - accept!) Recursive-descent parsing concepts Predictive parsing Recursive / Brute force technique non-recursive / table driven Error recovery Implementation
Recursive Descent Parsing Concepts General category of Parsing Top-Down Choose production rule based on input symbol May require backtracking to correct a wrong choice. Example: S c A d A ab | a input: cad cad S c d A a b Problem: backtrack cad S c d A S cad S c d A a b cad S c d A a cad c d A a
Predictive Parsing : Recursive To eliminate backtracking, grammar must have: no left recursion apply left factoring remove -moves If so, we can utilize current input symbol in conjunction with non-terminals to be expanded to uniquely determine the next action Utilize transition diagrams (TDs): For each non-terminal of the grammar: Create an initial and final state If A X1X2…Xn is a production, add path with edges X1, X2, … , Xn TDs can be algorithmized into Program
Transition Diagrams (TDs) Unlike lexical equivalents, each edge represents a token Transition implies: if token, choose edge, call proc Recall earlier grammar and its associated TDs E TE’ E’ + TE’ | T FT’ T’ * FT’ | F ( E ) | id 2 1 T E’ E: How are transition diagrams used ? Are -moves a problem ? Can we simplify transition diagrams ? Why is simplification critical ? 3 6 + 4 T E’: 5 E’ 7 9 8 F T’ T: 10 13 * 11 F T’: 12 T’ 14 17 ( 15 E F: 16 ) id
How are Transition Diagrams Used ? main() { TD_E(); } TD_E’() { token = get_token(); if token = ‘+’ then { TD_T(); TD_E’(); } } What happened to -moves? NOTE: not all error conditions have been represented. TD_F() { token = get_token(); if token = ‘(’ then { TD_E(); match(‘)’); } else if token.value <> id then {error + EXIT} ... } TD_E() { TD_T(); TD_E’(); } TD_T() { TD_F(); TD_T’(); } TD_E’() { token = get_token(); if token = ‘*’ then { TD_F(); TD_T’(); } }
How can Transition Diagrams be Simplified ? 6 E’ E’: 5 3 + 4 T
How can Transition Diagrams be Simplified ? (2) 6 E’ E’: 5 3 + 4 T E’: 5 3 + 4 T 6
How can Transition Diagrams be Simplified ? (3) 6 E’ E’: 5 3 + 4 T E’: 3 + 4 T 6 E’: 5 3 + 4 T 6
How can Transition Diagrams be Simplified ? (4) 6 E’ E’: 5 3 + 4 T E’: 3 + 4 T 6 E’: 5 3 + 4 T 6 2 1 E’ T E:
How can Transition Diagrams be Simplified ? (5) 6 E’ E’: 5 3 + 4 T E’: 3 + 4 T 6 E’: 5 3 + 4 T 6 2 1 E’ T E: T E: 3 + 4 6
How can Transition Diagrams be Simplified ? (6) 5 3 + 4 T E’: 3 + 4 T 6 E’: 5 3 + 4 T 6 2 1 E’ T E: + T E: 3 6 T E: 3 + 4 6 How ?
Additional Transition Diagram Simplifications Similar steps for T and T’ Simplified Transition diagrams: * F 7 T: 10 13 Why is simplification important ? How does code change? T’: 10 * 11 F 13 14 17 ( 15 E F: 16 ) id
Motivating Table-Driven Parsing 1. Left to right scan input 2. Find leftmost derivation Grammar: E TE’ E’ +TE’ | T id Terminator Input : id + id $ Derivation: E TE’ idE’ id +TE’ id +idE’ id +id’ Processing Stack: Contains the Derivation plus $ E $ T E’ $ id E’ $ E’ $ + T E’ $ T E’ $ id E’ $ E’ $ $
Non-Recursive / Table Driven + b $ Y X Z Input Predictive Parsing Program Stack Output Parsing Table M[A,a] (String + terminator) NT + T symbols of CFG What actions parser should take based on stack / input Empty stack symbol General parser behavior: X : top of stack a : current input 1. When X=a = $ halt, accept, success 2. When X=a $ , POP X off stack, advance input, go to 1. 3. When X is a non-terminal, examine M[X,a] if it is an error call recovery routine if M[X,a] = {X UVW}, POP X, PUSH W,V,U in reverse order DO NOT expend any input
Algorithm for Non-Recursive Parsing Set ip to point to the first symbol of w$; repeat let X be the top stack symbol and a the symbol pointed to by ip; if X is terminal or $ then if X=a then pop X from the stack and advance ip else error() else /* X is a non-terminal */ if M[X,a] = XY1Y2…Yk then begin pop X from stack; push Yk, Yk-1, … , Y1 onto stack, with Y1 on top output the production XY1Y2…Yk end until X=$ /* stack is empty */ Input pointer May also execute other code based on the production used
Example Our well-worn example ! Table M E TE’ E’ + TE’ | T FT’ T’ * FT’ | F ( E ) | id Our well-worn example ! Table M Non-terminal INPUT SYMBOL id + * ( ) $ E E’ T T’ F ETE’ TFT’ Fid E’+TE’ T’ T’*FT’ F(E) E’
Trace of Example Expend Input STACK INPUT OUTPUT $E $E’T $E’T’F $E’T’id $E’T’ $E’ $E’T+ $E’T’F* $ id + id * id$ + id * id$ id * id$ * id$ id$ $ E TE’ T FT’ F id T’ E’ +TE’ T’ *FT’ E’ Expend Input
Leftmost Derivation for the Example The leftmost derivation for the example is as follows: E TE’ FT’E’ id T’E’ id E’ id + TE’ id + FT’E’ id + id T’E’ id + id * FT’E’ id + id * id T’E’ id + id * id E’ id + id * id
What’s the Missing Puzzle Piece ? Constructing the Parsing Table M ! 1st : Calculate First & Follow for Grammar 2nd: Apply Construction Algorithm for Parsing Table Conceptual Perspective: First: Let be a string of grammar symbols. First() are the first terminals that can appear in in any possible derivation. NOTE: If , then is First( ). Follow: Let A be a non-terminal. Follow(A) is the set of terminals that can appear directly to the right of A in some sentential form. (S Aa, for some and ). NOTE: If S A, then $ is Follow(A). * * *
Conceptually – What is First? Calculated for Each Non-Terminal of the Grammar First (E), First (E’), First (T), First (T’), First(F) What is the First terminal that the non-terminal turns into? Consider Grammar Below: What is First(F)? What is First (T’)? What is First (E’)? What is First (T)? What is First (E)? { ( , id } { * } { + } {( , id } {( , id } E TE’ E’ + TE’ | T FT’ T’ * FT’ | F ( E ) | id
Conceptually – What is First? So far: What is Unique about E’ and T’ Rules? E’ T’ Add in epsilon to First calculations Now, what is First (T)? First(T) = First (FT’) = First(F) + First(T’) if in First(F) So Finally, what is First (E)? First(E) = First (TE’) = First(T) + First(E’) if in First(T) First(F) = { ( , id } First(E’) = { + } First(T’) = { * } First(F) = { ( , id } First(E’) = { +, } First(T’) = { *, } First(T) = {( , id } First(E) = {( , id }
Computing First(X) : All Grammar Symbols 1. If X is a terminal, First(X) = {X} 2. If X is a production rule, add to First(X) 3. If X is a non-terminal, and X Y1Y2…Yk is a production rule Place First(Y1) in First(X) if Y1 , Place First(Y2) in First(X) - if in First(Y2) if Y2 , Place First(Y3) in First(X) - if in First(Y2) & First(Y3) … if Yk-1 , Place First(Yk) in First(X) - if in First(Y2) to First(Yk-1) NOTE: As soon as Yi , Stop. May repeat 1, 2, and 3, above for each Yj * * * *
Computing First(X) : All Grammar Symbols - continued Informally, suppose we want to compute First(X1 X2 … Xn ) = First (X1) “+” First(X2) if is in First(X1) “+” First(X3) if is in First(X2) “+” … First(Xn) if is in First(Xn-1) Note 1: Only add to First(X1 X2 … Xn) if is in First(Xi) for all i Note 2: For First(X1), if X1 Z1 Z2 … Zm , then we need to compute First(Z1 Z2 … Zm) !
Conceptually: What is First (E, T, …) in Derivation? The leftmost derivation for the example is as follows: INPUT: id + id * id $ E $ TE’ FT’E’ id T’E’ id E’ id + TE’ id + FT’E’ id + id T’E’ id + id * FT’E’ id + id * id T’E’ id + id * id E’ id + id * id $ First(E,F,T) = { (, id } First(E’) = { +, } First(T’) = { *, }
Example Computing First for: E TE’ E’ + TE’ | T FT’ T’ * FT’ | F ( E ) | id First(TE’) First(T) “+” First(E’) First(E) * Not First(E’) since T First(T) First(F) “+” First(T’) First(F) * Not First(T’) since F First((E)) “+” First(id) “(“ and “id” Overall: First(E) = { ( , id } = First(F) First(E’) = { + , } First(T’) = { * , } First(T) First(F) = { ( , id }
Example 2 Given the production rules: Verify that S i E t SS’ | a S’ eS | E b Verify that First(S) = { i, a } First(S’) = { e, } First(E) = { b }
Conceptually – What is Follow? Again, Calculated for Each Non-Terminal of the Grammar Follow (E), Follow (E’), Follow (T), Follow (T’), Follow (F) What is first terminal that follows Non-Terminal in a derivation? Consider Grammar Below: Look at where Non-Terminal appears on the Right Hand Side of Each Rule E TE’ E’ + TE’ | T FT’ T’ * FT’ | F ( E ) | id
Conceptually – What is Follow? Examine each Non-Terminal appears on the Right Hand Side of Each Rule What is Follow(E)? Rule: F ( E ) So - Follow(E) = { )} What is Follow(T)? Rules: E TE’ and E’ +TE’ So - Follow(T)={+} = First(E’) w/o What is Follow(F)? Rules: T FT’ and T’ *FT’ So - Follow(F) ={*}= First(T’) w/o First(E) = { ( , id } First(T) = { ( , id } First(F) = { ( , id } First(E’) = { + , } First(T’) = { * , }
Conceptually – What is Follow? Are we Done Yet? Some other Things to Consider Assume that the Input Stream has a final token “$” So for the Start Symbol E, $ Follows any possible input so Follow(E) = { ), $} Consider Follow(T) in E TE’ where First(E’) contains Since E TE’ and in First(E’) Place Follow(E) into Follow(T) So Follow(T)= { +, ), $ } First(E) = { ( , id } First(T) = { ( , id } First(F) = { ( , id } First(E’) = { + , } First(T’) = { * , }
Let’s Explore in Detail Consider the Derivation: Input id$ E$ TE’ $ FT’E’$ idT’E’ $ idE’$ id$ What is in Follow(T) if T’ and E’ ? Whatever Follows E! In Rule, if E’ , E TE’ becomes E T and whatever follows T is what follows E! So what do we add in? E TE’ E’ + TE’ | T FT’ T’ * FT’ | F ( E ) | id First(E,F,T) = { (, id } First(E’) = { +, } First(T’) = { *, } Follow(E) = { ), $} Follow(F) = { * } Follow(T) = { +, ) }
Computing Follow(A) : All Non-Terminals 1. Place $ in Follow(S), where S is the start symbol and $ signals end of input 2. If there is a production A B, then everything in First() is in Follow(B) except for . 3. If A B is a production, or A B and (First() contains ), then everything in Follow(A) is in Follow(B) (Whatever followed A must follow B, since nothing follows B from the production rule) * We’ll calculate Follow for two grammars.
Conceptually: What is Follow in Derivation? The leftmost derivation for the example is as follows: INPUT: id + id * id $ E$ TE’ $ FT’E’$ id T’E’ $ id E’$ id + TE’$ id + FT’E’$ id + id T’E’$ id + id * FT’E’$ id + id * id T’E’ $ id + id * id E’$ id + id * id $ What Follow(E)? What Follow(F)? First(T’) E TE’ E’ + TE’ | T FT’ T’ * FT’ | F ( E ) | id First(E,F,T) = { (, id } First(E’) = { +, } First(T’) = { *, } Follow(E) = { ), $} Follow(F) = { * } Follow(T) = { +, }
Let’s Explore in Detail Consider the Derivation: Input id$ E$ TE’ $ FT’E’$ idT’E’ $ idE’$ id$ What is in Follow(T) if T’ and E’ ? Whatever Follows E! In Rule, if E’ , E TE’ becomes E T and whatever follows T is what follows E! So what do we add in? E TE’ E’ + TE’ | T FT’ T’ * FT’ | F ( E ) | id First(E,F,T) = { (, id } First(E’) = { +, } First(T’) = { *, } Follow(E) = { ), $} Follow(F) = { * } Follow(T) = { +, ) }
Example Compute Follow for: E TE’ E’ + TE’ | T FT’ T’ * FT’ | F ( E ) | id First(E) = { ( , id } First(T) = { ( , id } First(F) = { ( , id } First(E’) = { + , } First(T’) = { * , } Follow(E) - contains $ since E is the start symbol. Also, since F (E) then First(“)”) is in Follow(E). Thus Follow(E) = { ) , $ } Follow(E’) : E TE’ implies Follow(E) is in Follow(E’), and Follow(E’) = { ) , $ } E’ at end of rule, whatever follows E follows E’ Follow(T) : E TE’ implies put in First(E’). Since E’ , put in Follow(E). Since E’ +TE’ , Put in First(E’), and since E’ , put in Follow(E’). Thus Follow(T) = { +, ), $ }. Follow(T’) Follow(F) * * You do these !
Example Compute Follow for: E TE’ E’ + TE’ | T FT’ T’ * FT’ | F ( E ) | id Follow(T’) Follow(F) First(E,F,T) = { (, id } First(E’) = { +, } First(T’) = { *, } Follow(E) = { ), $ } Follow(E’) = { ), $ } Follow(T) = { +, ), }
Computing Follow : 2nd Example Recall: S i E t SS’ | a S’ eS | E b First(S) = { i, a } First(S’) = { e, } First(E) = { b } Follow(S) – Contains $, since S is start symbol Since S i E t SS’ , put in First(S’) – not Since S’ , Put in Follow(S) Since S’ eS, put in Follow(S’) So…. Follow(S) = { e, $ } Follow(S’) = Follow(S) HOW? Follow(E) = { t } *
First & Follow – Examine the Derivation Consider the following derivation: E TE’ FT’E’ ( E ) T’E’ ( TE’ ) T’E’ ( FT’E’ ) T’E’ ( id T’E’ ) T’E’ ( id E’ ) T’E’ ( id ) T’E’ ( id ) * FT’E’ ( id ) * id T’E’ ( id ) * id E’ ( id ) * id + TE’ * ( id ) * id + FT’E’ ( id ) * id + T’E’ ( id ) * id + id$
First & Follow – Examine the Derivation Consider the following derivation: First(E) = { ( , id } What’s First for each non-terminal ? E TE’ FT’E’ ( E ) T’E’ ( TE’ ) T’E’ ( FT’E’ ) T’E’ ( id T’E’ ) T’E’ ( id E’ ) T’E’ ( id ) T’E’ ( id ) * FT’E’ ( id ) * id T’E’ ( id ) * id E’ ( id ) * id + TE’ * ( id ) * id + FT’E’ ( id ) * id + T’E’ ( id ) * id + id$
First & Follow – Examine the Derivation Consider the following derivation: First(T) = { ( , id } What’s First for each non-terminal ? E TE’ FT’E’ ( E ) T’E’ ( TE’ ) T’E’ ( FT’E’ ) T’E’ ( id T’E’ ) T’E’ ( id E’ ) T’E’ ( id ) T’E’ ( id ) * FT’E’ ( id ) * id T’E’ ( id ) * id E’ ( id ) * id + TE’ * ( id ) * id + FT’E’ ( id ) * id + T’E’ ( id ) * id + id$
First & Follow – Examine the Derivation Consider the following derivation: First(T’) = { * , } What’s First for each non-terminal ? E TE’ FT’E’ ( E ) T’E’ ( TE’ ) T’E’ ( FT’E’ ) T’E’ ( id T’E’ ) T’E’ ( id E’ ) T’E’ ( id ) T’E’ ( id ) * FT’E’ ( id ) * id T’E’ ( id ) * id E’ ( id ) * id + TE’ T’ * ( id ) * id + FT’E’ ( id ) * id + T’E’ ( id ) * id + id$
First & Follow – Examine the Derivation Consider the following derivation: First(E’) = { + , } What’s First for each non-terminal ? E TE’ FT’E’ ( E ) T’E’ ( TE’ ) T’E’ ( FT’E’ ) T’E’ ( id T’E’ ) T’E’ ( id E’ ) T’E’ ( id ) T’E’ ( id ) * FT’E’ ( id ) * id T’E’ ( id ) * id E’ ( id ) * id + TE’ * ( id ) * id + FT’E’ ( id ) * id + T’E’ ( id ) * id + id$ E’
First & Follow – Examine the Derivation Consider the following derivation: First(F) = { ( , id } What’s First for each non-terminal ? You do First(F) ! E TE’ FT’E’ ( E ) T’E’ ( TE’ ) T’E’ ( FT’E’ ) T’E’ ( id T’E’ ) T’E’ ( id E’ ) T’E’ ( id ) T’E’ ( id ) * FT’E’ ( id ) * id T’E’ ( id ) * id E’ ( id ) * id + TE’ * ( id ) * id + FT’E’ ( id ) * id + T’E’ ( id ) * id + id$
First & Follow – Examine the Derivation Consider the following derivation: What’s First for each non-terminal ? Still needs your First(F) E TE’ FT’E’ ( E ) T’E’ ( TE’ ) T’E’ ( FT’E’ ) T’E’ ( id T’E’ ) T’E’ ( id E’ ) T’E’ ( id ) T’E’ ( id ) * FT’E’ ( id ) * id T’E’ ( id ) * id E’ ( id ) * id + TE’ T’ * ( id ) * id + FT’E’ ( id ) * id + T’E’ ( id ) * id + id$ E’
First & Follow – Examine the Derivation Consider the following derivation: Follow(E) = { ( , id } What’s Follow for each non-terminal ? E TE’ FT’E’ ( E ) T’E’ ( TE’ ) T’E’ ( FT’E’ ) T’E’ ( id T’E’ ) T’E’ ( id E’ ) T’E’ ( id ) T’E’ ( id ) * FT’E’ ( id ) * id T’E’ ( id ) * id E’ ( id ) * id + TE’ * ( id ) * id + FT’E’ ( id ) * id + T’E’ ( id ) * id + id$
First & Follow – Examine the Derivation Consider the following derivation: Follow(T) = { + , ) , $ } What’s Follow for each non-terminal ? E TE’ FT’E’ ( E ) T’E’ ( TE’ ) T’E’ ( FT’E’ ) T’E’ ( id T’E’ ) T’E’ ( id E’ ) T’E’ ( id ) T’E’ ( id ) * FT’E’ ( id ) * id T’E’ ( id ) * id E’ ( id ) * id + TE’ This “+” in Follow(T) comes from the First(E’) * ( id ) * id + FT’E’ ( id ) * id + T’E’ ( id ) * id + id$
First & Follow – Examine the Derivation Consider the following derivation: Follow(T’) = { + , ) , $ } What’s Follow for each non-terminal ? E TE’ FT’E’ ( E ) T’E’ ( TE’ ) T’E’ ( FT’E’ ) T’E’ ( id T’E’ ) T’E’ ( id E’ ) T’E’ ( id ) T’E’ ( id ) * FT’E’ ( id ) * id T’E’ ( id ) * id E’ ( id ) * id + TE’ T’ * ( id ) * id + FT’E’ ( id ) * id + T’E’ ( id ) * id + id$
First & Follow – Examine the Derivation Consider the following derivation: Follow(E’) = { ) , $ } What’s Follow for each non-terminal ? E TE’ FT’E’ ( E ) T’E’ ( TE’ ) T’E’ ( FT’E’ ) T’E’ ( id T’E’ ) T’E’ ( id E’ ) T’E’ ( id ) T’E’ ( id ) * FT’E’ ( id ) * id T’E’ ( id ) * id E’ ( id ) * id + TE’ * ( id ) * id + FT’E’ ( id ) * id + T’E’ ( id ) * id + id$ E’
First & Follow – Examine the Derivation Consider the following derivation: Follow(F) = { +, *, ) , $ } You do Follow(F) ! What’s Follow for each non-terminal ? E TE’ FT’E’ ( E ) T’E’ ( TE’ ) T’E’ ( FT’E’ ) T’E’ ( id T’E’ ) T’E’ ( id E’ ) T’E’ ( id ) T’E’ ( id ) * FT’E’ ( id ) * id T’E’ ( id ) * id E’ ( id ) * id + TE’ * ( id ) * id + FT’E’ ( id ) * id + T’E’ ( id ) * id + id$
First & Follow – Examine the Derivation Consider the following derivation: Still needs your Follow(F) What’s Follow for each non-terminal ? E TE’ FT’E’ ( E ) T’E’ ( TE’ ) T’E’ ( FT’E’ ) T’E’ ( id T’E’ ) T’E’ ( id E’ ) T’E’ ( id ) T’E’ ( id ) * FT’E’ ( id ) * id T’E’ ( id ) * id E’ ( id ) * id + TE’ T’ * ( id ) * id + FT’E’ ( id ) * id + T’E’ ( id ) * id + id$ E’
First & Follow – Examine the Derivation Consider the following derivation: Still needs your First(F) and Follow(F) What’s First for each non-terminal ? What’s Follow for each non-terminal ? E TE’ FT’E’ ( E ) T’E’ ( TE’ ) T’E’ ( FT’E’ ) T’E’ ( id T’E’ ) T’E’ ( id E’ ) T’E’ ( id ) T’E’ ( id ) * FT’E’ ( id ) * id T’E’ ( id ) * id E’ ( id ) * id + TE’ T’ * ( id ) * id + FT’E’ ( id ) * id + T’E’ ( id ) * id + id$ E’
In Class Exercise One – First and Follow A B A’ A’ o B A’ A’ B D B ( C ) C D C’ C’ a D C’ C’ D x D n x First(A) = { x, n, ( } First(A’) = { o, } First(B) = { x, n, ( } First(C) = {x, n } First(C’) = { a, } First(D) = {x, n } Follow(A) = { $ } Follow(A’) = { $ } Follow(B) = { $, o} Follow(C) = { ) } Follow(C’) = { ) } Follow(D) = {$, a, o}
In Class Exercise Two – First and Follow S ( L ) | E b | a L L , S | S | c E [ E ] | L c | E d S ( L ) | E b | a L c L’ | ( L ) L’ | E b L’ | a L’ L’ , S L’ | E [ E ] E’ | c L’ c E’ | ( L ) L’ c E’ | a L’ c E’ E’ d E’ | b L’ c E’ | First(S) = { (, a, [, c } First(L) = {(, a, [, c } First(E) = {(, a, [, c } First(E’) = {b, d, } First(L’) = { “,”, } Follow(S) = { $ } Follow(L) = { c, ‘,’, )} Follow(E)= { b, d, ] } Follow(L’)= {c, ‘,’, )} Follow(E’)= { b, d, ]}
Motivation Behind First & Follow Is used to indicate the relationship between non-terminals (in the stack) and input symbols (in input stream) Example: If A , and a is in First(), then when a=input, replace with . ( a is one of first symbols of , so when A is on the stack and a is input, POP A and PUSH . Follow: Is used when First has a conflict, to resolve choices. When or , then what follows A dictates the next choice to be made. * Example: If A , and b is in Follow(A ), then when A , and if b is an input character, then we expand A with , which will eventually expand to , of which b follows! ( Above . Here First( ) contains .) * *
Constructing Top-Down Parsing Table Algorithm: Repeat Steps 2 & 3 for each rule A Terminal a in First()? Add A to M[A, a ] 3.1 in First()? Add A to M[A, a ] for all terminals b in Follow(A). 3.2 in First() and $ in Follow(A)? Add A to M[A, $ ] 4. All undefined entries are errors.
Step 2:Each rule A -- First() E TE’ E’ + TE’ | T FT’ T’ * FT’ | F ( E ) | id First(E,F,T) = { (, id } First(E’) = { +, } First(T’) = { *, } Follow(E,E’) = { ), $} Follow(F) = { *, +, ), } Follow(T,T’) = { +, ), } F ( E ) F id E TE’ T FT’ E’ + TE’ T’ * FT’ E’ T’ INPUT SYMBOL Non-terminal id + * ( ) $ E ETE’ ETE’ Table M E’ E’+TE’ E’ E’ T TFT’ TFT’ T’ T’ T’*FT’ T’ T’ F Fid F(E)
Constructing Top-Down Parsing Table Example 2 E TE’ E’ + TE’ | T FT’ T’ * FT’ | F ( E ) | id First(E,F,T) = { (, id } First(E’) = { +, } First(T’) = { *, } Follow(E,E’) = { ), $} Follow(F) = { *, +, ), } Follow(T,T’) = { +, ), } Expression Example: E TE’ : First(TE’) = First(T) = { (, id } M[E, ( ] : E TE’ M[E, id ] : E TE’ (by rule 2) E’ +TE’ : First(+TE’) = + : M[E’, +] : E’ +TE’ (by rule 3) E’ : in First( ) T’ : in First( ) by rule 2 M[E’, )] : E’ (3.1) M[T’, +] : T’ (3.1) M[E’, $] : E’ (3.2) M[T’, )] : T’ (3.1) (Due to Follow(E’) M[T’, $] : T’ (3.2) (Due to Follow(T’)
Example Our well-worn example ! Table M E TE’ E’ + TE’ | T FT’ T’ * FT’ | F ( E ) | id Our well-worn example ! Table M Non-terminal INPUT SYMBOL id + * ( ) $ E E’ T T’ F ETE’ TFT’ Fid E’+TE’ T’ T’*FT’ F(E) E’
Constructing Top-Down Parsing Table Example 2 S i E t SS’ | a S’ eS | E b First(S) = { i, a } First(S’) = { e, } First(E) = { b } Follow(S) = { e, $ } Follow(S’) = { e, $ } Follow(E) = { t } S i E t SS’ S a E b First(i E t SS’)={i} First(a) = {a} First(b) = {b} S’ eS S First(eS) = {e} First() = {} Follow(S’) = { e, $ } Non-terminal INPUT SYMBOL a b e i t $ S S a S iEtSS’ S’ S’ S’ eS S E E b
First & Follow – Derivation to Parsing Table Consider the following derivation: What are implications ? ( id ) * id + id$ (input) M - Table E TE’ FT’E’ ( E ) T’E’ 1. E TE’ and ( in First(E) 2. TFT’ and ( in First(T) 3. F (E) and ( in First(F) 4. E’ and ) in Follow(E’) 5. Since $ in Follow(T’), T’ 6. Since $ in Follow(E’), E’ 1. M [ E, ( ] 2. M [ T, ( ] 3. M [ F, ( ] ( TE’ ) T’E’ ( FT’E’ ) T’E’ ( id T’E’ ) T’E’ ( id E’ ) T’E’ ( id ) T’E’ ( id ) * FT’E’ 4. M [ E’, ) ] ( id ) * id T’E’ ( id ) * id E’ ( id ) * id + TE’ * ( id ) * id + FT’E’ ( id ) * id + T’E’ ( id ) * id + id$ 5. M [ T’, $ ] 6. M [ E’, $ ]
Detailed Example Step 1 Compute Follow First T → F T’ T’ → * F T’ → F → ( E ) → Id S → E $ E → T E’ E’ → + T E’ Overall: First(S) = { } First(E) = { ( , id } = First(F) First(E’) = { + , } First(T’) = { * , } First(T) First(F) = { ( , id } Follow(E) = Follow(E’) = { ), $ } Follow(T) = Follow(T’) = {+, ), $ } Follow(F) = {+, *, ), $ }
Detailed Example Step 2 Build the parser table Step 3 Input: Id + Id * Id $ T → F T’ T’ → * F T’ → F → ( E ) → Id S → E $ E → T E’ E’ → + T E’ Input Symbols NT Id + * ( ) $ S S → E$ E E → TE’ E →TE’ E’ E’ →+TE’ E’ → T T → FT’ T →FT’ T’ T’ → T’ →*FT’ F F → Id F → (E) Parser Table
Parsing Process Over Time Id + Id * Id $ Input Symbols NT Id + * ( ) $ S S → E$ E E → TE’ E →TE’ E’ E’ →+TE’ E’ → T T → FT’ T →FT’ T’ T’ → T’ →*FT’ F F → Id F → (E)
Parsing Process Over Time Id + Id * Id $ Input Symbols NT Id + * ( ) $ S S → E$ E E → TE’ E →TE’ E’ E’ →+TE’ E’ → T T → FT’ T →FT’ T’ T’ → T’ →*FT’ F F → Id F → (E)
Parsing Process Over Time Id + Id * Id $ Input Symbols NT Id + * ( ) $ S S → E$ E E → TE’ E →TE’ E’ E’ →+TE’ E’ → T T → FT’ T →FT’ T’ T’ → T’ →*FT’ F F → Id F → (E)
Parsing Process Over Time Id + Id * Id $ Input Symbols NT Id + * ( ) $ S S → E$ E E → TE’ E →TE’ E’ E’ →+TE’ E’ → T T → FT’ T →FT’ T’ T’ → T’ →*FT’ F F → Id F → (E)
Parsing Process Over Time Id + Id * Id $ Input Symbols NT Id + * ( ) $ S S → E$ E E → TE’ E →TE’ E’ E’ →+TE’ E’ → T T → FT’ T →FT’ T’ T’ → T’ →*FT’ F F → Id F → (E)
Parsing Process Over Time Id + Id * Id $ Input Symbols NT Id + * ( ) $ S S → E$ E E → TE’ E →TE’ E’ E’ →+TE’ E’ → T T → FT’ T →FT’ T’ T’ → T’ →*FT’ F F → Id F → (E)
Parsing Process Over Time Id + Id * Id $ Input Symbols NT Id + * ( ) $ S S → E$ E E → TE’ E →TE’ E’ E’ →+TE’ E’ → T T → FT’ T →FT’ T’ T’ → T’ →*FT’ F F → Id F → (E)
Parsing Process Over Time Id + Id * Id $ Input Symbols NT Id + * ( ) $ S S → E$ E E → TE’ E →TE’ E’ E’ →+TE’ E’ → T T → FT’ T →FT’ T’ T’ → T’ →*FT’ F F → Id F → (E)
Parsing Process Over Time Id + Id * Id $ Input Symbols NT Id + * ( ) $ S S → E$ E E → TE’ E →TE’ E’ E’ →+TE’ E’ → T T → FT’ T →FT’ T’ T’ → T’ →*FT’ F F → Id F → (E)
Parsing Process Over Time Id + Id * Id $ Input Symbols NT Id + * ( ) $ S S → E$ E E → TE’ E →TE’ E’ E’ →+TE’ E’ → T T → FT’ T →FT’ T’ T’ → T’ →*FT’ F F → Id F → (E)
Parsing Process Over Time Id + Id * Id $ Input Symbols NT Id + * ( ) $ S S → E$ E E → TE’ E →TE’ E’ E’ →+TE’ E’ → T T → FT’ T →FT’ T’ T’ → T’ →*FT’ F F → Id F → (E)
Parsing Process Over Time Id + Id * Id $ Input Symbols NT Id + * ( ) $ S S → E$ E E → TE’ E →TE’ E’ E’ →+TE’ E’ → T T → FT’ T →FT’ T’ T’ → T’ →*FT’ F F → Id F → (E)
Parsing Process Over Time Id + Id * Id $ Input Symbols NT Id + * ( ) $ S S → E$ E E → TE’ E →TE’ E’ E’ →+TE’ E’ → T T → FT’ T →FT’ T’ T’ → T’ →*FT’ F F → Id F → (E)
Parsing Process Over Time Id + Id * Id $ Input Symbols NT Id + * ( ) $ S S → E$ E E → TE’ E →TE’ E’ E’ →+TE’ E’ → T T → FT’ T →FT’ T’ T’ → T’ →*FT’ F F → Id F → (E)
Parsing Process Over Time Id + Id * Id $ Input Symbols NT Id + * ( ) $ S S → E$ E E → TE’ E →TE’ E’ E’ →+TE’ E’ → T T → FT’ T →FT’ T’ T’ → T’ →*FT’ F F → Id F → (E)
Parsing Process Over Time Id + Id * Id $ Input Symbols NT Id + * ( ) $ S S → E$ E E → TE’ E →TE’ E’ E’ →+TE’ E’ → T T → FT’ T →FT’ T’ T’ → T’ →*FT’ F F → Id F → (E)
Parsing Process Over Time Id + Id * Id $ Input Symbols NT Id + * ( ) $ S S → E$ E E → TE’ E →TE’ E’ E’ →+TE’ E’ → T T → FT’ T →FT’ T’ T’ → T’ →*FT’ F F → Id F → (E)
Parsing Process Over Time Id + Id * Id $ Input Symbols NT Id + * ( ) $ S S → E$ E E → TE’ E →TE’ E’ E’ →+TE’ E’ → T T → FT’ T →FT’ T’ T’ → T’ →*FT’ F F → Id F → (E)
Parsing Process Over Time Id + Id * Id $ Input Symbols NT Id + * ( ) $ S S → E$ E E → TE’ E →TE’ E’ E’ →+TE’ E’ → T T → FT’ T →FT’ T’ T’ → T’ →*FT’ F F → Id F → (E)
Parsing Process Over Time Id + Id * Id $ Input Symbols NT Id + * ( ) $ S S → E$ E E → TE’ E →TE’ E’ E’ →+TE’ E’ → T T → FT’ T →FT’ T’ T’ → T’ →*FT’ F F → Id F → (E)
LL(1) Grammars for Top-Down Parsing L : Scan input from Left to Right L : Construct a Leftmost Derivation 1 : Use “1” input symbol as lookahead in conjunction with stack to decide on the parsing action LL(1) grammars have no multiply-defined entries in the parsing table. Properties of LL(1) grammars: Grammar can’t be ambiguous or left recursive Grammar is LL(1) when A 1. & do not derive strings starting with the same terminal a 2. Either or can derive , but not both. Note: It may not be possible for a grammar to be manipulated into an LL(1) grammar
Predictive Parsing Program Error Recovery When Do Errors Occur? Recall Predictive Parser Function: a + b $ Y X Z Input Predictive Parsing Program Stack Output Parsing Table M[A,a] If X is a terminal and it doesn’t match input. If M[ X, Input ] is empty – No allowable actions Consider two recovery techniques: A. Panic Mode B. Phase-level Recovery
Panic Mode Recovery Augment parsing table with action that attempts to realign / synchronize token stream with the expected input. Suppose : A on top of stack doesn’t mesh with current input symbol 1. Use Follow(A) to remove input tokens – sync (discard) 2. Use First(A) to determine when to restart parsing 3. Incorporate higher level language concepts (begin/end, while, repeat/until) to sync actions we don’t skip tokens unnecessarily. Other actions: 4. When A , use it to manipulate stack to postpone error detection 5. Use non-matching terminal on stack as token that is inserted into input.
Revised Parsing Table / Example INPUT SYMBOL Non-terminal id + * ( ) $ E ETE’ ETE’ synch synch E’ E’+TE’ E’ E’ T TFT’ synch TFT’ synch synch T’ T’ T’*FT’ T’ T’ F Fid synch synch F(E) synch synch Skip input symbol From Follow sets. Pop stack entry – T or NT
Skip & Synch Meaning Skip Synch Messages Discard input symbol Synch Pop top of stack Messages Constructed based on lookahead an non-terminal Example NT = F Lookahead = + Expecting a FACTOR. Got + for a Term. So a factor is missing.
Revised Parsing Table / Example(2) $E’T $E’T’F $E’T’id $E’T’ $E’T’F* $E’ $E’T+ $ ) id * + id$ id * + id$ * + id$ + id$ id$ STACK INPUT Remark error, skip ) id is in First(E) error, M[F,+] = synch F has been popped
Phase-Level Recovery Fill in blanks entries of parsing table with error handling routines These routines modify stack and / or input stream issue error message Problems: Modifying stack has to be done with care, so as to not create possibility of derivations that aren’t in language Infinite loops must be avoided Can be used in conjunction with panic mode to have more complete error handling
How Would You Implement TD Parser Stack – Easy to handle. Write ADT to manipulate its contents Input Stream – Responsibility of lexical analyzer Key Issue – How is parsing table implemented ? One approach: Assign unique IDS Non-terminal INPUT SYMBOL id + * ( ) $ E E’ T T’ F ETE’ TFT’ Fid E’+TE’ T’ T’*FT’ F(E) E’ synch All rules have unique IDs Also for blanks which handle errors Ditto for synch actions
Revised Parsing Table: INPUT SYMBOL Non-terminal id + * ( ) $ E 1 18 19 1 9 10 E’ 20 2 21 22 3 3 T 4 11 23 4 12 13 T’ 24 6 5 25 6 6 F 8 14 15 7 16 17 1 ETE’ 2 E’+TE’ 3 E’ 4 TFT’ 5 T’*FT’ 6 T’ 7 F(E) 8 Fid 9 – 17 : Sync Actions 18 – 25 : Error Handlers
Revised Parsing Table: (2) Each # ( or set of #s) corresponds to a procedure that: Uses Stack ADT Gets Tokens Prints Error Messages Prints Diagnostic Messages Handles Errors
How is Parser Constructed ? One large CASE statement: state = M[ top(s), current_token ] switch (state) { case 1: proc_E_TE’( ) ; break ; … case 8: proc_F_id( ) ; case 9: proc_sync_9( ) ; case 17: proc_sync_17( ) ; case 18: case 25: } Combine put in another switch Some sync actions may be same Some error handlers may be similar Procs to handle errors
In Class Exercise Three– First and Follow S A A | A B A’ | z A’ A’ A B A’ | B x A’ y B’ | y B’ B’ A’ y B’ | First(S) = {x, y, z , } First(A) = {x, y, z} First(A’) = {x, y, z , } First(B) = {x, y} First(B’) = {x, y, z , } Follow(S) = { $ } Follow(A) = {x, y, z, $} Follow(A’) = {x, y, z, d} Follow(B) = {x, y, z, $} Follow(B’) = {x, y, z, $ }
Final Comments – Top-Down Parsing So far, We’ve examined grammars and language theory and its relationship to parsing Key concepts: Rewriting grammar into an acceptable form Examined Top-Down parsing: Brute Force : Transition diagrams & recursion Elegant : Table driven We’ve identified its shortcomings: Not all grammars can be made LL(1) ! Bottom-Up Parsing – Next Up!