1. Compiler Principles Fall 2014-2015 Compiler Principles Lecture 4: Parsing part 3 Roman Manevich Ben-Gurion University

2. Tentative syllabus Front End Scanning Top-down Parsing (LL) Bottom-up Parsing (LR) Attribute Grammars Intermediate Representation Lowering Optimizations Local Optimizations Dataflow Analysis Loop Optimizations Code Generation Register Allocation Instruction Selection 2 mid-termexam

3. Previously 3 Top-down parsing – Recursive descent – Handling conflicts – LL(k) via pushdown automata

4. Agenda 4 Shift-reduce (LR) parsing model Building the LR parsing table Types of conflicts

5. Shift-reduce parsing 5

6. Some terminology The opposite of derivation is called reduction – Let A  α be a production rule – Let βAµ be a sentential form – A reduction replaces α with A: βαµ  βAµ A handle is a substring that is reduced during a series of steps in a rightmost derivation 6

7. Using shift and reduce to parse E  E + (E) E  i 7 actionInputStack shift1 + (2) + (3) reduce+ (2) + (3)1 shift+ (2) + (3)E shift (2) + (3)E + shift2) + (3)E + ( reduce) + (3)E + (2 shift) + (3)E + (E reduce+ (3)E + (E) shift+ (3)E shift(3)E + shift3)E + ( reduce)E + (3 shift)E + (E reduceE + (E) acceptE On each step we either: - shift a symbol from the input to the stack, or - reduce symbols on the stack

8. How will the parser know what to do? A state will keep the info gathered so far A stack will maintain formerly reduced handles and partially reduced handles A table will tell it “what to do” based on – Current state, – Symbol on top of stack, and – k-next tokens (k≥0) 8

9. Model of an LR parser 9 LR Parsing program Stack $id+ + Output Parser table Input State

10. States and LR(0) items The state will “remember” the potential derivation rules given the part that was already identified For example, if we have already identified E then the state will remember the two alternatives: (1) E → E * B, (2) E → E + B Actually, we will also remember where we are in each of them: (1) E → E ● * B, (2) E → E ● + B A derivation rule with a location marker is called an LR(0) item The state is actually a set of LR(0) items – For example: q 13 = { E → E ● * B, E → E ● + B} 10 E → E * B | E + B | B B → 0 | 1

11. Intuition We gather the input token by token until we find a right-hand side of a rule and then we replace it with the nonterminal on the left side Going over a token and remembering it in the stack is a shift Each shift moves to a state that remembers what we’ve seen so far A reduce replaces a string in the stack with the nonterminal that derives it 11

12. Why do we need the stack? E  E + (E) E  i 12 actionInputStack shift1 + (2) + (3) reduce+ (2) + (3)1 shift+ (2) + (3)E shift (2) + (3)E + shift2) + (3)E + ( reduce) + (3)E + (2 shift) + (3)E + (E reduce+ (3)E + (E) shift+ (3)E shift(3)E + shift3)E + ( reduce)E + (3 shift)E + (E reduceE + (E) acceptE Suppose so far we have discovered E → 1 and gather information on “E +” In the given grammar this can only mean E → E + ● (E) Suppose state q represents this possibility Now, the next token is (, and we need to ignore q for a minute, and work on E → 2 to obtain E+(E) Therefore, we push q to the stack, and after identifying E, we pop it to continue

13. LR parser stack 13 LR Parsing program 5 T 2 + 7 id 0 Stack $id+ + Output state symbol gotoaction Input State

14. LR parsing table state terminals non-terminals shift/reduce actions goto part 0 1...... snsn rkrk shift state nreduce by rule k gmgm goto state m acc accept error 14

15. LR(0) parser table example 15 (1) S  E $ (2) E  T (3) E  E + T (4) T  id (5) T  ( E ) gotoactionSTATE TE$)(+id g6g1s7s50 s2s31 acc2 g4s7s53 r3 4 r4 5 r2 6 g6g8s7s57 s9s38 r5 9 Always entire row of rk Always entire row of shift and gotos (possibly accept)

16. LR parser moves 16

17. Shift move 17 LR Parsing program q...... Stack $…t… Output gotoaction Input If action[q, t] = sn then push t, push n current state n is the next state

18. Result of shift 18 LR Parsing program n t q...... Stack $…t… Output gotoaction Input If action[q, t] = sn then push t, push n

19. Reduce move If action[q n, t] = rk Production: (k) A  σ 1 … σ n Top of stack looks like q 1 σ 1 … q n σ n 1.Pop q n σ n … q 1 σ 1 2.If goto[q, A] = q’ then push A, push q’ 19 LR Parsing program qnqn … q … Stack $…t… Output gotoaction Input 2*n Rule k

20. Result of reduce move If action[q n, t] = rk Production: (k) A  σ 1 … σ n Top of stack looks like q 1 σ 1 … q n σ n 1.Pop q n σ n … q 1 σ 1 2.If goto[q, A] = q’ then push A, push q’ 20 LR Parsing program Stack Output gotoaction q’ A q … $…t… Input

21. Accept move 21 LR Parsing program q...... Stack $t… Output gotoaction Input If action[q, t] = accept parsing completed

22. Error move 22 LR Parsing program q...... Stack $…t… Output gotoaction Input If action[q, t] = error parsing discovered a syntactic error

23. Example of Shift-reduce parser run 23

24. Parsing id+id$ 24 gotoactionS TE$)(+id g6g1s7s50 s2s31 acc2 g4s7s53 r3 4 r4 5 r2 6 g6g8s7s57 s9s38 r5 9 (1) S  E $ (2) E  T (3) E  E + T (4) T  id (5) T  ( E ) StackInputAction 0id + id $? Initialize with state 0

25. Parsing id+id$ 25 gotoactionS TE$)(+id g6g1s7s50 s2s31 acc2 g4s7s53 r3 4 r4 5 r2 6 g6g8s7s57 s9s38 r5 9 (1) S  E $ (2) E  T (3) E  E + T (4) T  id (5) T  ( E ) StackInputAction 0id + id $s5

26. Parsing id+id$ 26 StackInputAction 0id + id $s5 0 id 5+ id $r4 gotoactionS TE$)(+id g6g1s7s50 s2s31 acc2 g4s7s53 r3 4 r4 5 r2 6 g6g8s7s57 s9s38 r5 9 (1) S  E $ (2) E  T (3) E  E + T (4) T  id (5) T  ( E )

27. Parsing id+id$ 27 StackInputAction 0id + id $s5 0 id 5+ id $r4 gotoactionS TE$)(+id g6g1s7s50 s2s31 acc2 g4s7s53 r3 4 r4 5 r2 6 g6g8s7s57 s9s38 r5 9 (1) S  E $ (2) E  T (3) E  E + T (4) T  id (5) T  ( E ) pop id 5

28. Parsing id+id$ 28 StackInputAction 0id + id $s5 0 id 5+ id $r4 gotoactionS TE$)(+id g6g1s7s50 s2s31 acc2 g4s7s53 r3 4 r4 5 r2 6 g6g8s7s57 s9s38 r5 9 (1) S  E $ (2) E  T (3) E  E + T (4) T  id (5) T  ( E ) push T 6

29. Parsing id+id$ 29 StackInputAction 0id + id $s5 0 id 5+ id $r4 0 T 6+ id $r2 gotoactionS TE$)(+id g6g1s7s50 s2s31 acc2 g4s7s53 r3 4 r4 5 r2 6 g6g8s7s57 s9s38 r5 9 (1) S  E $ (2) E  T (3) E  E + T (4) T  id (5) T  ( E )

30. Parsing id+id$ 30 StackInputAction 0id + id $s5 0 id 5+ id $r4 0 T 6+ id $r2 0 E 1+ id $s3 gotoactionS TE$)(+id g6g1s7s50 s2s31 acc2 g4s7s53 r3 4 r4 5 r2 6 g6g8s7s57 s9s38 r5 9 (1) S  E $ (2) E  T (3) E  E + T (4) T  id (5) T  ( E )

31. Parsing id+id$ 31 StackInputAction 0id + id $s5 0 id 5+ id $r4 0 T 6+ id $r2 0 E 1+ id $s3 0 E 1 + 3id $s5 gotoactionS TE$)(+id g6g1s7s50 s2s31 acc2 g4s7s53 r3 4 r4 5 r2 6 g6g8s7s57 s9s38 r5 9 (1) S  E $ (2) E  T (3) E  E + T (4) T  id (5) T  ( E )

32. Parsing id+id$ 32 StackInputAction 0id + id $s5 0 id 5+ id $r4 0 T 6+ id $r2 0 E 1+ id $s3 0 E 1 + 3id $s5 0 E 1 + 3 id 5$r4 gotoactionS TE$)(+id g6g1s7s50 s2s31 acc2 g4s7s53 r3 4 r4 5 r2 6 g6g8s7s57 s9s38 r5 9 (1) S  E $ (2) E  T (3) E  E + T (4) T  id (5) T  ( E )

33. Parsing id+id$ 33 StackInputAction 0id + id $s5 0 id 5+ id $r4 0 T 6+ id $r2 0 E 1+ id $s3 0 E 1 + 3id $s5 0 E 1 + 3 id 5$r4 0 E 1 + 3 T 4$r3 gotoactionS TE$)(+id g6g1s7s50 s2s31 acc2 g4s7s53 r3 4 r4 5 r2 6 g6g8s7s57 s9s38 r5 9 (1) S  E $ (2) E  T (3) E  E + T (4) T  id (5) T  ( E )

34. Parsing id+id$ 34 StackInputAction 0id + id $s5 0 id 5+ id $r4 0 T 6+ id $r2 0 E 1+ id $s3 0 E 1 + 3id $s5 0 E 1 + 3 id 5$r4 0 E 1 + 3 T 4$r3 0 E 1$s2 gotoactionS TE$)(+id g6g1s7s50 s2s31 acc2 g4s7s53 r3 4 r4 5 r2 6 g6g8s7s57 s9s38 r5 9 (1) S  E $ (2) E  T (3) E  E + T (4) T  id (5) T  ( E )

35. Parsing id+id$ 35 StackInputAction 0id + id $s5 0 id 5+ id $r4 0 T 6+ id $r2 0 E 1+ id $s3 0 E 1 + 3id $s5 0 E 1 + 3 id 5$r4 0 E 1 + 3 T 4$r3 0 E 1$s2 0 E 1 $ 2acc gotoactionS TE$)(+id g6g1s7s50 s2s31 acc2 g4s7s53 r3 4 r4 5 r2 6 g6g8s7s57 s9s38 r5 9 (1) S  E $ (2) E  T (3) E  E + T (4) T  id (5) T  ( E )

36. Constructing an LR(0) parsing table 36

37. Overall process 1.Construct a (determinized) transition diagram from LR(0) items 2.If there are conflicts – stop – Grammar is not LR(0) 3.Otherwise, fill table entries from diagram 37

38. LR(0) item N  α  β Already matched To be matched Input Hypothesis about αβ being a possible handle, so far we’ve matched α, expecting to see β 38

39. LR(0) items N  α  β Shift Item N  αβ  Reduce Item 39

40. LR(0) items enumeration example All items can be obtained by placing a dot at every position for every production: (1) S  E $ (2) E  T (3) E  E + T (4) T  id (5) T  ( E ) 1: S   E$ 2: S  E  $ 3: S  E $  4: E   T 5: E  T  6: E   E + T 7: E  E  + T 8: E  E +  T 9: E  E + T  10: T   id 11: T  id  12: T   (E) 13: T  (  E) 14: T  (E  ) 15: T  (E)  Grammar LR(0) items 40

41. Operations for transition diagram construction Initial = {S’   S$} For an item set I solve: Closure(I) = Closure(I) + {X   µ is in grammar| N  α  Xβ in I} Goto(I, σ) = { N  ασ  β | N  α  σβ in I} – σ is either a terminal or nonterminal 41

42. Initial example Initial = { S   E $ } 42 (1) S  E $ (2) E  T (3) E  E + T (4) T  id (5) T  ( E ) Grammar

43. Closure example Initial = { S   E $ } Closure({ S   E $ }) = S   E $ E   T E   E + T T   id T   ( E ) 43 (1) S  E $ (2) E  T (3) E  E + T (4) T  id (5) T  ( E ) Grammar

44. Goto example Initial = { S   E $ } Closure({ S   E $ }) = S   E $ E   T E   E + T T   id T   ( E ) Goto({S   E $, E   E + T, T   id}, E) = {S  E  $, E  E  + T} 44 (1) S  E $ (2) E  T (3) E  E + T (4) T  id (5) T  ( E ) Grammar

45. Constructing the transition diagram 1.Start with state 0 containing item Closure({ S   E $ }) 2.Repeat until no new states are discovered – For every state p containing item set Ip, and symbol N, compute state q containing item set Iq = Closure(Goto(Ip, N)) 45 Why does it terminate?

46. LR(0) automaton example 46 (1) S  E $ (2) E  T (3) E  E + T (4) T  id (5) T  ( E ) S   E$ E   T E   E + T T   id T   (E) T  (  E) E   T E   E + T T   id T   (E) E  E + T  T  (E)  S  E$  S  E  $ E  E  + T E  E+  T T   id T   (E) T  id  T  (E  ) E  E  +T E  T  q0q0 q1q1 q2q2 q3q3 q4q4 q5q5 q6q6 q7q7 q8q8 q9q9 T ( id E + $ T ) + E T ( i (

47. LR(0) automaton construction example 47 (1) S  E $ (2) E  T (3) E  E + T (4) T  id (5) T  ( E ) S   E$ q0q0 Initialize

48. LR(0) automaton construction example 48 (1) S  E $ (2) E  T (3) E  E + T (4) T  id (5) T  ( E ) S   E$ E   T E   E + T T   id T   (E) q0q0 apply Closure

49. LR(0) automaton construction example 49 (1) S  E $ (2) E  T (3) E  E + T (4) T  id (5) T  ( E ) S   E$ E   T E   E + T T   id T   (E) q0q0 E  T  q6q6 T T  (  E) E   T E   E + T T   id T   (E) ( T  id  q5q5 id S  E  $ E  E  + T q1q1 E

50. LR(0) automaton construction example 50 (1) S  E $ (2) E  T (3) E  E + T (4) T  id (5) T  ( E ) S   E$ E   T E   E + T T   id T   (E) T  (  E) E   T E   E + T T   id T   (E) E  E + T  T  (E)  S  E$  S  E  $ E  E  + T E  E+  T T   id T   (E) T  id  T  (E  ) E  E  +T E  T  q0q0 q1q1 q2q2 q3q3 q4q4 q5q5 q6q6 q7q7 q8q8 q9q9 T ( id E + $ T ) + E T ( i ( terminal transition corresponds to shift action in parse table non-terminal transition corresponds to goto action in parse table a single reduce item corresponds to reduce action

51. LR(0) conflicts 51

52. Conflicts Can construct a diagram for every grammar but some may introduce conflicts shift-reduce conflict: an item set contains at least one shift item and one reduce item reduce-reduce conflict: an item set contains two reduce items 52 What about shift-shift conflicts?

53. Shift-reduce conflict example S  E $ E  T E  E + T T  id T  ( E ) T  id[E] S   E$ E   T E   E + T T   id T   (E) T   id[E] T  id  T  id  [E] q0q0 q5q5 T ( id E Shift/reduce conflict … … … 53

54. Reduce-reduce conflict example S  E $ E  T E  V E  E + T T  id V  id T  ( E ) S   E$ E   T E   V E   E + T T   id V   id T   (E) T   i[E] T  id  V  id  q0q0 q5q5 T ( id E reduce/reduce conflict … … … 54

55. LR(0) conflicts Any grammar with an  -rule cannot be LR(0) Inherent shift/reduce conflict – A   – reduce item – P  α  Aβ – shift item – A   can always be predicted from P  α  Aβ Similar to FIRST-FOLLOW conflicts in LL(1) parsing – Similar solution 55

56. LR parsing variants 56

57. LR variants LR(0) – what we’ve seen so far SLR(0) – Removes infeasible reduce actions via FOLLOW set reasoning LR(1) – LR(0) with one lookahead token in items LALR(1) – LR(1) with merging of states with same LR(0) component 57

58. SLR parsing 58

59. SRL parsing A handle should not be reduced to a non- terminal N if the lookahead is a token that cannot follow N A reduce item N  α  is applicable only when the lookahead is in FOLLOW(N) – If b is not in FOLLOW(N) we just proved there is no terminating derivation S =>* βNb and thus it is safe to remove the reduce item from the conflicted state Differs from LR(0) only on the ACTION table – Now a row in the parsing table may contain both shift actions and reduce actions and we need to consult the current token to decide which one to take 59

60. SLR action table Stateid+()[]$ 0shift 1 2accept 3shift 4E  E+T 5T  id r5, s6 T  id 6ETETETETETET 7shift 8 9T  (E) vs. stateaction q0shift q1shift q2 q3shift q4E  E+T q5T  id q6ETET q7shift q8shift q9TETE SLR – use 1 token look-aheadLR(0) – no look-ahead … as before… T  id T  id[E] Lookahead token from the input 60 [ is not in FOLLOW(T)

61. Next lecture: SLR/LR(1)/LALR(1)/Parser generation