1. Review 1.Lexical Analysis 2.Syntax Analysis 3.Semantic Analysis 4.Code Generation 5.Code Optimization

2. Syntax Analysis Often called parsing Groups tokens of source program into grammatical phrases that are used by the compiler to check for correct syntax and to help in generating code Creation of a hierarchical structure called a syntax tree –Tree helps us determine if program is syntactically correct –Also aids in the translation of source program to target language

3. Grammar Example San FranciscoSeattle  Lexical  Ogay orthnay eightay undrendhay ilesmay  Syntactical  Miles go hundred north eight  Logical  Go eight hundred miles (facing south)  Run-Time  Go eight hundred miles (facing West)

4. Grammars – Defining the Language Rules Terminals (tokens) Non-terminals - Syntactic variable. Contains groups of tokens that define some part of the language –Example: expression, if statement, etc Start symbol - A special non-terminal (i.e. a program) Productions –The manner in which terminals and non-terminals are combined to form statements –A non-terminal in LHS and a string of terminals and non- terminals in RHS

5. Example Variable Declaration –A type followed by one or more comma separated identifiers that end with a semi-colon. -> ->, and are non-terminals The comma is a terminal The two lines are called productions

6. Grammars We will be defining a grammar for the entire JO99 programming language. We will then have JCUP produce for us a parser that detects if the JO99 program is syntactically correct. In addition, the parser will create for us a tree that represents the program and allows us to be able to do other things like semantic checks and code generation.

7. Example Simple arithmetic expressions with + and * –8.2 + 35.6 –8.32 + 86 * 45.3 –(6.001 + 6.004) * (6.035 * -(6.042 + 6.046)) Terminals (or tokens) –num for all the numbers –‘+’, ‘-’, ‘*’, ‘(‘, ‘)’ What is the grammar for all possible expressions?

8. Example   ( )  -  num  +  *

9. Categories of Parsers ()

10. –L - parse from left to right –R - parse from right to left ()

11. Categories of Parsers –L - leftmost derivation –R - rightmost derivation ()

12. Categories of Parsers –Number of look ahead characters ()

13. Categories of Parsers –Examples: LL(0) – Parse Left to Right, Derive the tree using a leftmost derivation (top down), no look ahead characters LR(1) – Parse Left to Right, Derive the tree using a rightmost derivation (bottom up), 1 look ahead character. –Each category of parsing handles a different type of language. –We will be learning about LR(k) parsers and will implement an LR(k) parser.

14. Why LR(k)? Virtually all programming language grammars can be parsed using a LR(k) technique Most general parsing method for programming grammars Can build a very efficient parser engine given just the syntax rules of the language. Can detect a syntactic error as soon as it is possible to do so Because its so general, programs have been written (JCUP) that produce the parser instead of writing it from scratch.

15. LR(k) Parser implementation Sometimes called a Shift-Reduce Parser Parse from left to right (get the tokens from left to right) Bottom up parsing (same as rightmost derivation)

16. Actions of a Shift-Reduce Parser Parse Tree

17. Actions of a Shift-Reduce Parser Parse Tree Parse Tree

18. Actions of a Shift-Reduce Parser Parse Tree Parse Tree

19. Actions of a Shift-Reduce Parser Parse Tree Parse Tree

20. Actions of a Shift-Reduce Parser Parse Tree Parse Tree

21. Actions of a Shift-Reduce Parser Parse Tree Parse Tree

22. Actions of a Shift-Reduce Parser Parse Tree Parse Tree

23. Actions of a Shift-Reduce Parser Parse Tree Parse Tree

24. Actions of a Shift-Reduce Parser Parse Tree

25. Actions of a Shift-Reduce Parser Parse Tree

26. Actions of a Shift-Reduce Parser Parse Tree

27. Actions of a Shift-Reduce Parser Parse Tree

28. Actions of a Shift-Reduce Parser How do we build this tree? As productions are recognized, a portion of the tree is created. This portion of the tree will be needed later to build bigger portions of the tree and therefore must be saved for future use. This requires a stack. The stack plus the next token read from the source program determines the action.

29. Actions of a Shift-Reduce Parser Stack Current Symbol stack Parser Action Parser Engine

30. Actions of a Shift-Reduce Parser Shift –Shift the current element into top of the stack –Move the current pointer (next token) Reduce –Apply a production (we recognize a part of the program) –Top of the stack should match the RHS of the grammar –Remove those symbols from the stack –Add the LHS non-terminal Accept –End of stream reached and stack only has the start symbol Reject –End of stream reached but stack has more than the start symbol

31. Shift-Reduce Parser Example *(+num)

32. Shift-Reduce Parser Example *(+num)   ( )  -  num  +  -  *

33. Shift-Reduce Parser Example *(+num)   ( )  -  num  +  -  *

34. Shift-Reduce Parser Example *(+num)   ( )  -  num  +  -  * num SHIFT

35. Shift-Reduce Parser Example *(+num)   ( )  -  num  +  -  * num REDUCE

36. Shift-Reduce Parser Example *(+num) num   ( )  -  num  +  -  * REDUCE

37. Shift-Reduce Parser Example (+num) num*   ( )  -  num  +  -  * * SHIFT

38. Shift-Reduce Parser Example (+num) num*   ( )  -  num  +  -  * * REDUCE

39. Shift-Reduce Parser Example (+num) num*   ( )  -  num  +  -  * REDUCE

40. Shift-Reduce Parser Example +num) num*(   ( )  -  num  +  -  * ( SHIFT

41. Shift-Reduce Parser Example *+num) num(   ( )  -  num  +  -  * ( num SHIFT

42. Shift-Reduce Parser Example *+num) num(   ( )  -  num  +  -  * ( num REDUCE

43. Shift-Reduce Parser Example *+num) num(   ( )  -  num  +  -  * ( REDUCE

44. Shift-Reduce Parser Example *num) num( +   ( )  -  num  +  -  * ( + SHIFT

45. Shift-Reduce Parser Example *num) num( +   ( )  -  num  +  -  * ( + REDUCE

46. Shift-Reduce Parser Example *num) num( +   ( )  -  num  +  -  * ( REDUCE

47. Shift-Reduce Parser Example *) num( +   ( )  -  num  +  -  * ( num SHIFT

48. Shift-Reduce Parser Example *) num( +   ( )  -  num  +  -  * ( num REDUCE

49. Shift-Reduce Parser Example *) num( +   ( )  -  num  +  -  * ( REDUCE

50. Shift-Reduce Parser Example *) num( +   ( )  -  num  +  -  * ( REDUCE

51. Shift-Reduce Parser Example *) num( +   ( )  -  num  +  -  * ( REDUCE

52. Shift-Reduce Parser Example *) num( +   ( )  -  num  +  -  * ( ) SHIFT

53. Shift-Reduce Parser Example *) num( +   ( )  -  num  +  -  * ( ) REDUCE

54. Shift-Reduce Parser Example *) num( +   ( )  -  num  +  -  * REDUCE

55. Shift-Reduce Parser Example *) num( + REDUCE   ( )  -  num  +  -  *

56. Shift-Reduce Parser Example *) num( + REDUCE   ( )  -  num  +  -  *

57. Shift-Reduce Parser Example *) num( + ACCEPT   ( )  -  num  +  -  *

58. What does the parser engine do? If the top symbols of the stack match the RHS of a production then do the reduction –Pop the RHS from the top of the stack –Push the LHS symbol onto the stack If no production is found do the shift – Push the current input into the stack If the input is empty –Accept if only the start symbol is on the stack –Reject otherwise Parser Engine

59. This is not that simple! Many choices of reductions if there are multiple RHS. Which LHS do we put on stack in its place? Choice between shift and reduce –Stack matches a RHS –But that may not be the right match –May need to shift an input onto stack and later find a different reduction

60. Shift-Reduce Parser Example   ( )  -  num  +  -  * Change in the Grammar

61. Shift-Reduce Parser Example   ( )  -  num  +  -  * Change in the Grammar

62. Shift-Reduce Parser Example   ( )  -  num  +  -  * Change in the Grammar

63. Shift-Reduce Parser Example   ( )  -  num  +  -  * Change in the Grammar

64. Shift-Reduce Parser Example -num   ( )  -  num  +  -  *

65. Shift-Reduce Parser Example -num   ( )  -  num  +  -  * num

66. Shift-Reduce Parser Example -num   ( )  -  num  +  -  * num SHIFT

67. num Shift-Reduce Parser Example -num   ( )  -  num  +  -  * num REDUCE

68. Shift-Reduce Parser Example -num   ( )  -  num  +  -  * num REDUCE

69. - Shift-Reduce Parser Example num   ( )  -  num  +  -  * num - SHIFT

70. - Shift-Reduce Parser Example num   ( )  -  num  +  -  * num - We have a choice!!! REDUCE

71. - Shift-Reduce Parser Example num   ( )  -  num  +  -  * num - But not the right thing to do!! REDUCE

72. Shift-Reduce Parser Example num   ( )  -  num  +  -  * num - REDUCE But not the right thing to do!!

73. num Shift-Reduce Parser Example num   ( )  -  num  +  -  * num - But not the right thing to do!! SHIFT

74. num Shift-Reduce Parser Example num   ( )  -  num  +  -  * num - But not the right thing to do!! REDUCE

75. Shift-Reduce Parser Example num   ( )  -  num  +  -  * num - But not the right thing to do!! REDUCE

76. Shift-Reduce Parser Example num   ( )  -  num  +  -  * num - But not the right thing to do!! No more actions!!! ERROR

77. Shift-Reduce Parser Example But this is perfectly valid input for the grammar We chose the wrong production and thus the wrong LHS Lets see what happens when we choose the right production and the right LHS

78. - Shift-Reduce Parser Example num   ( )  -  num  +  -  * num - We have a choice REDUCE The step before we went wrong

79. - Shift-Reduce Parser Example num   ( )  -  num  +  -  * num - Use the other production REDUCE

80. Shift-Reduce Parser Example num   ( )  -  num  +  -  * num - REDUCE Use the other production

81. num Shift-Reduce Parser Example num   ( )  -  num  +  -  * num - SHIFT

82. num Shift-Reduce Parser Example num   ( )  -  num  +  -  * num - REDUCE

83. Shift-Reduce Parser Example num   ( )  -  num  +  -  * num - REDUCE

84. Shift-Reduce Parser Example num   ( )  -  num  +  -  * num - REDUCE

85. Shift-Reduce Parser Example num   ( )  -  num  +  -  * num - ACCEPT

86. Shift-Reduce Parser Parser Engine is far more complicated that it appears. Requires it to know all possible productions that would match the top of the stack and the given input symbol.

87. Constructing a LR(k) Parser What is in the parse engine –decide between shift and reduce –decide on the right reduction

88. Constructing a LR(k) Parser Create a DFA –Encodes all the possible states that the parser can be in –DFA state transition occurs on terminals and non- terminals Create a Parser Table – From the DFA create a transition table that stores what action should be taken for the current state and current input character Maintain a stack of states in parallel with the stack of symbols

89. LR(k) Parser Engine Current Symbol Parser Action LR(k) Parser Engine Symbol Stack State Stack

90. Parser Tables Look-up [top of state stack] [ input symbol] in the parser table Carry-out the described action

91. Parser Tables Shift to sn –Push input token into the symbol stack –Push sn into state stack –Advance to next input symbol

92. Parser Tables Reduce (n) –Pop both stacks as many times as the number of symbols on the RHS of rule n –Push LHS of rule n into symbol stack –Lookup [top of the state stack][top of symbol stack] –Push that state (in goto k) into state stack

93. Parser Tables Accept –Stop parsing and report success

94. Parser Tables Error –Stop parsing and report failure

95. LR example The grammar  $(1)  ( )(2)  ( )(3)

96. Question The grammar  $(1)  ( )(2)  ( )(3)

97. Parser Table in Action The grammar  $(1)  ( )(2)  ( )(3)

98. Parser Table in Action Parser Table The grammar  $(1)  ( )(2)  ( )(3)

99. Parser Table in Action Parser Table The grammar  $(1)  ( )(2)  ( )(3) $

100. Parser Table in Action Parser Table The grammar  $(1)  ( )(2)  ( )(3) s0 $

101. Parser Table in Action Parser Table The grammar  $(1)  ( )(2)  ( )(3) s0 $ )(()$

102. Parser Table in Action Parser Table The grammar  $(1)  ( )(2)  ( )(3) s0 $ )()($

103. Parser Table in Action Parser Table The grammar  $(1)  ( )(2)  ( )(3) s0 $ )()($

104. Parser Table in Action Parser Table The grammar  $(1)  ( )(2)  ( )(3) s0 $ )()($

105. ( s2 ( Parser Table in Action Parser Table The grammar  $(1)  ( )(2)  ( )(3) s0 $ ))($

106. s2 ( Parser Table in Action Parser Table The grammar  $(1)  ( )(2)  ( )(3) s0 $ ))(($

107. s2 ( Parser Table in Action Parser Table The grammar  $(1)  ( )(2)  ( )(3) s0 $ ))(($

108. s2 ( Parser Table in Action Parser Table The grammar  $(1)  ( )(2)  ( )(3) s0 $ ))(($

109. s2 ( ( Parser Table in Action Parser Table The grammar  $(1)  ( )(2)  ( )(3) s0 $ ))(($

110. s2 ( ( Parser Table in Action Parser Table The grammar  $(1)  ( )(2)  ( )(3) s0 $ )(()$

111. s2 ( ( Parser Table in Action Parser Table The grammar  $(1)  ( )(2)  ( )(3) s0 $ )(()$

112. s2 ( ( Parser Table in Action Parser Table The grammar  $(1)  ( )(2)  ( )(3) s0 $ )(()$

113. s5 ) s2 ( ( Parser Table in Action Parser Table The grammar  $(1)  ( )(2)  ( )(3) s0 $ )(()$

114. s5 ) s2 ( ( Parser Table in Action Parser Table The grammar  $(1)  ( )(2)  ( )(3) s0 $ )(()$

115. s5 ) s2 ( ( Parser Table in Action Parser Table The grammar  $(1)  ( )(2)  ( )(3) s0 $ )(()$

116. s5 ) s2 ( ( Parser Table in Action Parser Table The grammar  $(1)  ( )(2)  ( )(3) s0 $ )(()$

117. s5 ) s2 ( ( Parser Table in Action Parser Table The grammar  $(1)  ( )(2)  ( )(3) s0 $ )(()$

118. s5 ) s2 ( ( Parser Table in Action Parser Table The grammar  $(1)  ( )(2)  ( )(3) s0 $ )(() s5 ) s2 ( $

119. X ( Parser Table in Action Parser Table The grammar  $(1)  ( )(2)  ( )(3) s0 $ )(()$

120. X s2 ( Parser Table in Action Parser Table The grammar  $(1)  ( )(2)  ( )(3) s0 $ )(()$

121. X s2 ( Parser Table in Action Parser Table The grammar  $(1)  ( )(2)  ( )(3) s0 $ )(()$

122. s3 X s2 ( Parser Table in Action Parser Table The grammar  $(1)  ( )(2)  ( )(3) s0 $ )(()$

123. s3 X s2 ( Parser Table in Action Parser Table The grammar  $(1)  ( )(2)  ( )(3) s0 $ )(()$

124. s3 X s2 ( Parser Table in Action Parser Table The grammar  $(1)  ( )(2)  ( )(3) s0 $ )(()$

125. s3 X s2 ( Parser Table in Action Parser Table The grammar  $(1)  ( )(2)  ( )(3) s0 $ )(()$

126. s4 ) s3 X s2 ( Parser Table in Action Parser Table The grammar  $(1)  ( )(2)  ( )(3) s0 $ )(()$

127. s4 ) s3 X s2 ( Parser Table in Action Parser Table The grammar  $(1)  ( )(2)  ( )(3) s0 $ )(()$

128. s4 ) s3 X s2 ( Parser Table in Action Parser Table The grammar  $(1)  ( )(2)  ( )(3) s0 $ )(()$

129. s4 ) s3 X s2 ( Parser Table in Action Parser Table The grammar  $(1)  ( )(2)  ( )(3) s0 $ )(()$

130. s4 ) s3 X s2 ( Parser Table in Action Parser Table The grammar  $(1)  ( )(2)  ( )(3) s0 $ )(()$ s4 ) s3 X s2 (

131. X Parser Table in Action Parser Table The grammar  $(1)  ( )(2)  ( )(3) s0 $ )(()$

132. X Parser Table in Action Parser Table The grammar  $(1)  ( )(2)  ( )(3) s0 $ )(()$

133. X Parser Table in Action Parser Table The grammar  $(1)  ( )(2)  ( )(3) s0 $ )(()$

134. s1 X Parser Table in Action Parser Table The grammar  $(1)  ( )(2)  ( )(3) s0 $ )(()$

135. s1 X Parser Table in Action Parser Table The grammar  $(1)  ( )(2)  ( )(3) s0 $ )(()$

136. s1 X Parser Table in Action Parser Table The grammar  $(1)  ( )(2)  ( )(3) s0 $ )(()$

137. s1 X Parser Table in Action The grammar  $(1)  ( )(2)  ( )(3) s0 $ )(()$ Accept

138. Parser Table The table (DFA) and the stacks are called Push Down Automaton (PDA). There is an algorithm for converting a grammar to a PDA. Parser generators – given a grammar, produce a parser table with a stack (PDA) We will not study how this is done because its somewhat complicated. JCUP will convert our grammar into a shift-reduce parser.