Fall 2004COMP 3351 Simplifications of Context-Free Grammars

Fall 2004COMP 3352 A Substitution Rule Substitute Equivalent grammar

Fall 2004COMP 3353 A Substitution Rule Equivalent grammar Substitute

Fall 2004COMP 3354 In general: Substitute equivalent grammar

Fall 2004COMP 3355 Nullable Variables Nullable Variable:

Fall 2004COMP 3356 Removing Nullable Variables Example Grammar: Nullable variable

Fall 2004COMP 3357 Substitute Final Grammar

Fall 2004COMP 3358 Unit-Productions Unit Production: (a single variable in both sides)

Fall 2004COMP 3359 Removing Unit Productions Observation: Is removed immediately

Fall 2004COMP Example Grammar:

Fall 2004COMP Substitute

Fall 2004COMP Remove

Fall 2004COMP Substitute

Fall 2004COMP Remove repeated productions Final grammar

Fall 2004COMP Useless Productions Some derivations never terminate... Useless Production

Fall 2004COMP Another grammar: Not reachable from S Useless Production

Fall 2004COMP In general: if then variable is useful otherwise, variable is useless contains only terminals

Fall 2004COMP A production is useless if any of its variables is useless Productions useless Variables useless

Fall 2004COMP Removing Useless Productions Example Grammar:

Fall 2004COMP First: find all variables that can produce strings with only terminals Round 1: Round 2:

Fall 2004COMP Keep only the variables that produce terminal symbols: (the rest variables are useless) Remove useless productions

Fall 2004COMP Second: Find all variables reachable from Use a Dependency Graph not reachable

Fall 2004COMP Keep only the variables reachable from S Final Grammar (the rest variables are useless) Remove useless productions

Fall 2004COMP Removing All Step 1: Remove -productions Step 2: Remove Unit-productions Step 3: Remove Useless productions

Fall 2004COMP Normal Forms for Context-free Grammars

Fall 2004COMP Chomsky Normal Form Each productions has form: variable or terminal

Fall 2004COMP Examples: Not Chomsky Normal Form Chomsky Normal Form

Fall 2004COMP Conversion to Chomsky Normal Form Example: Not Chomsky Normal Form

Fall 2004COMP Introduce variables for terminals:

Fall 2004COMP Introduce intermediate variable:

Fall 2004COMP Introduce intermediate variable:

Fall 2004COMP Final grammar in Chomsky Normal Form: Initial grammar

Fall 2004COMP From any context-free grammar (which doesn’t produce ) not in Chomsky Normal Form we can obtain: An equivalent grammar in Chomsky Normal Form In general:

Fall 2004COMP The Procedure First remove: Nullable variables Unit productions

Fall 2004COMP Then, for every symbol : In productions: replace with Add production New variable:

Fall 2004COMP Replace any production with New intermediate variables:

Fall 2004COMP Theorem: For any context-free grammar (which doesn’t produce ) there is an equivalent grammar in Chomsky Normal Form

Fall 2004COMP Observations Chomsky normal forms are good for parsing and proving theorems It is very easy to find the Chomsky normal form for any context-free grammar

Fall 2004COMP Greibach Normal Form All productions have form: symbolvariables

Fall 2004COMP Examples: Greibach Normal Form Not Greibach Normal Form

Fall 2004COMP Conversion to Greibach Normal Form: Greibach Normal Form

Fall 2004COMP Theorem: For any context-free grammar (which doesn’t produce ) there is an equivalent grammar in Greibach Normal Form

Fall 2004COMP Observations Greibach normal forms are very good for parsing It is hard to find the Greibach normal form of any context-free grammar

Fall 2004COMP The CYK Parser

Fall 2004COMP The CYK Membership Algorithm Input: Grammar in Chomsky Normal Form String Output: find if

Fall 2004COMP The Algorithm Grammar : String : Input example:

Fall 2004COMP 33547

Fall 2004COMP 33548

Fall 2004COMP 33549

Fall 2004COMP 33550

Fall 2004COMP Therefore: Time Complexity: The CYK algorithm can be easily converted to a parser (bottom up parser) Observation: