1. Courtesy Costas Buch - RPI1 Simplifications of Context-Free Grammars

2. Courtesy Costas Buch - RPI2 A Substitution Rule Substitute Equivalent grammar

3. Courtesy Costas Buch - RPI3 A Substitution Rule Equivalent grammar Substitute

4. Courtesy Costas Buch - RPI4 In general: Substitute equivalent grammar

5. Courtesy Costas Buch - RPI5 Nullable Variables Nullable Variable:

6. Courtesy Costas Buch - RPI6 Removing Nullable Variables Example Grammar: Nullable variable

7. Courtesy Costas Buch - RPI7 Substitute Final Grammar

8. Courtesy Costas Buch - RPI8 Unit-Productions Unit Production: (a single variable in both sides)

9. Courtesy Costas Buch - RPI9 Removing Unit Productions Observation: Is removed immediately

10. Courtesy Costas Buch - RPI10 Example Grammar:

11. Courtesy Costas Buch - RPI11 Substitute

12. Courtesy Costas Buch - RPI12 Remove

13. Courtesy Costas Buch - RPI13 Substitute

14. Courtesy Costas Buch - RPI14 Remove repeated productions Final grammar

15. Courtesy Costas Buch - RPI15 Useless Productions Some derivations never terminate... Useless Production

16. Courtesy Costas Buch - RPI16 Another grammar: Not reachable from S Useless Production

17. Courtesy Costas Buch - RPI17 In general: if then variable is useful otherwise, variable is useless contains only terminals

18. Courtesy Costas Buch - RPI18 A production is useless if any of its variables is useless Productions useless Variables useless

19. Courtesy Costas Buch - RPI19 Removing Useless Productions Example Grammar:

20. Courtesy Costas Buch - RPI20 First: find all variables that can produce strings with only terminals Round 1: Round 2:

21. Courtesy Costas Buch - RPI21 Keep only the variables that produce terminal symbols: (the rest variables are useless) Remove useless productions

22. Courtesy Costas Buch - RPI22 Second: Find all variables reachable from Use a Dependency Graph not reachable

23. Courtesy Costas Buch - RPI23 Keep only the variables reachable from S Final Grammar (the rest variables are useless) Remove useless productions

24. Courtesy Costas Buch - RPI24 Removing All Step 1: Remove Nullable Variables Step 2: Remove Unit-Productions Step 3: Remove Useless Variables

25. Courtesy Costas Buch - RPI25 Normal Forms for Context-free Grammars

26. Courtesy Costas Buch - RPI26 Chomsky Normal Form Each productions has form: variable or terminal

27. Courtesy Costas Buch - RPI27 Examples: Not Chomsky Normal Form Chomsky Normal Form

28. Courtesy Costas Buch - RPI28 Convertion to Chomsky Normal Form Example: Not Chomsky Normal Form

29. Courtesy Costas Buch - RPI29 Introduce variables for terminals:

30. Courtesy Costas Buch - RPI30 Introduce intermediate variable:

31. Courtesy Costas Buch - RPI31 Introduce intermediate variable:

32. Courtesy Costas Buch - RPI32 Final grammar in Chomsky Normal Form: Initial grammar

33. Courtesy Costas Buch - RPI33 From any context-free grammar (which doesn’t produce ) not in Chomsky Normal Form we can obtain: An equivalent grammar in Chomsky Normal Form In general:

34. Courtesy Costas Buch - RPI34 The Procedure First remove: Nullable variables Unit productions

35. Courtesy Costas Buch - RPI35 Then, for every symbol : In productions: replace with Add production New variable:

36. Courtesy Costas Buch - RPI36 Replace any production with New intermediate variables:

37. Courtesy Costas Buch - RPI37 Theorem: For any context-free grammar (which doesn’t produce ) there is an equivalent grammar in Chomsky Normal Form

38. Courtesy Costas Buch - RPI38 Observations Chomsky normal forms are good for parsing and proving theorems It is very easy to find the Chomsky normal form for any context-free grammar

39. Courtesy Costas Buch - RPI39 Greinbach Normal Form All productions have form: symbolvariables

40. Courtesy Costas Buch - RPI40 Examples: Greinbach Normal Form Not Greinbach Normal Form

41. Courtesy Costas Buch - RPI41 Conversion to Greinbach Normal Form: Greinbach Normal Form

42. Courtesy Costas Buch - RPI42 Theorem: For any context-free grammar (which doesn’t produce ) there is an equivalent grammar in Greinbach Normal Form

43. Courtesy Costas Buch - RPI43 Observations Greinbach normal forms are very good for parsing It is hard to find the Greinbach normal form of any context-free grammar

44. Courtesy Costas Buch - RPI44 Compilers

45. Courtesy Costas Buch - RPI45 Compiler Program v = 5; if (v>5) x = 12 + v; while (x !=3) { x = x - 3; v = 10; }...... Add v,v,0 cmp v,5 jmplt ELSE THEN: add x, 12,v ELSE: WHILE: cmp x,3... Machine Code

46. Courtesy Costas Buch - RPI46 Lex

47. Courtesy Costas Buch - RPI47 Lex: a lexical analyzer A Lex program recognizes strings For each kind of string found the lex program takes an action

48. Courtesy Costas Buch - RPI48 Var = 12 + 9; if (test > 20) temp = 0; else while (a < 20) temp++; Lex program Identifier: Var Operand: = Integer: 12 Operand: + Integer: 9 Semicolumn: ; Keyword: if Parenthesis: ( Identifier: test.... Input Output

49. Courtesy Costas Buch - RPI49 In Lex strings are described with regular expressions “if” “then” “+” “-” “=“ /* operators */ /* keywords */ Lex program Regular expressions

50. Courtesy Costas Buch - RPI50 (0|1|2|3|4|5|6|7|8|9)+ /* integers */ /* identifiers */ Regular expressions (a|b|..|z|A|B|...|Z)+ Lex program

51. Courtesy Costas Buch - RPI51 integers [0-9]+(0|1|2|3|4|5|6|7|8|9)+

52. Courtesy Costas Buch - RPI52 (a|b|..|z|A|B|...|Z)+ [a-zA-Z]+ identifiers

53. Courtesy Costas Buch - RPI53 Each regular expression has an associated action (in C code) Examples: \n Regular expressionAction linenum++; [a-zA-Z]+ printf(“identifier”); [0-9]+ prinf(“integer”);

54. Courtesy Costas Buch - RPI54 Default action: ECHO; Prints the string identified to the output

55. Courtesy Costas Buch - RPI55 A small lex program % [a-zA-Z]+printf(“Identifier\n”); [0-9]+printf(“Integer\n”); [ \t\n] ; /*skip spaces*/

56. Courtesy Costas Buch - RPI56 1234 test var 566 78 9800 Input Output Integer Identifier Integer

57. Courtesy Costas Buch - RPI57 % [a-zA-Z]+ printf(“Identifier\n”); [0-9]+ prinf(“Integer\n”); [ \t] ; /*skip spaces*/. printf(“Error in line: %d\n”, linenum); Another program %{ int linenum = 1; %} \nlinenum++;

58. Courtesy Costas Buch - RPI58 1234 test var 566 78 9800 + temp Input Output Integer Identifier Integer Error in line: 3 Identifier

59. Courtesy Costas Buch - RPI59 Lex matches the longest input string “if” “ifend” Regular Expressions Input: ifend if Matches: “ifend” “if” Example:

60. Courtesy Costas Buch - RPI60 Internal Structure of Lex Lex Regular expressions NFADFA Minimal DFA The final states of the DFA are associated with actions

61. Courtesy Costas Buch - RPI61 Lexical analyzer parser Compiler program machine code input output

62. Courtesy Costas Buch - RPI62 A parser knows the grammar of the programming language

63. Courtesy Costas Buch - RPI63 Parser PROGRAM STMT_LIST STMT_LIST STMT; STMT_LIST | STMT; STMT EXPR | IF_STMT | WHILE_STMT | { STMT_LIST } EXPR EXPR + EXPR | EXPR - EXPR | ID IF_STMT if (EXPR) then STMT | if (EXPR) then STMT else STMT WHILE_STMT while (EXPR) do STMT

64. Courtesy Costas Buch - RPI64 The parser finds the derivation of a particular input 10 + 2 * 5 Parser E -> E + E | E * E | INT E => E + E => E + E * E => 10 + E*E => 10 + 2 * E => 10 + 2 * 5 input derivation

65. Courtesy Costas Buch - RPI65 10 E 25 E => E + E => E + E * E => 10 + E*E => 10 + 2 * E => 10 + 2 * 5 derivation derivation tree EE EE + *

66. Courtesy Costas Buch - RPI66 10 E 25 derivation tree EE EE + * mult a, 2, 5 add b, 10, a machine code

67. Courtesy Costas Buch - RPI67 Parsing

68. Courtesy Costas Buch - RPI68 grammar Parser input string derivation

69. Courtesy Costas Buch - RPI69 Example: Parser derivation input ?

70. Courtesy Costas Buch - RPI70 Exhaustive Search Phase 1: All possible derivations of length 1 Find derivation of

71. Courtesy Costas Buch - RPI71

72. Courtesy Costas Buch - RPI72 Phase 2 Phase 1

73. Courtesy Costas Buch - RPI73 Phase 2 Phase 3

74. Courtesy Costas Buch - RPI74 Final result of exhaustive search Parser derivation input (top-down parsing)

75. Courtesy Costas Buch - RPI75 Time complexity of exhaustive search Suppose there are no productions of the form Number of phases for string :

76. Courtesy Costas Buch - RPI76 Time for phase 1: possible derivations For grammar with rules

77. Courtesy Costas Buch - RPI77 Time for phase 2: possible derivations

78. Courtesy Costas Buch - RPI78 Time for phase : possible derivations

79. Courtesy Costas Buch - RPI79 Total time needed for string : Extremely bad!!! phase 1 phase 2 phase 2|w|

80. Courtesy Costas Buch - RPI80 There exist faster algorithms for specialized grammars S-grammar: symbolstring of variables appears once Pair

81. Courtesy Costas Buch - RPI81 S-grammar example: Each string has a unique derivation

82. Courtesy Costas Buch - RPI82 In the exhaustive search parsing there is only one choice in each phase For S-grammars: Total time for parsing string : Time for a phase:

83. Courtesy Costas Buch - RPI83 For general context-free grammars: There exists a parsing algorithm that parses a string in time