1. Programming Language Concepts (CIS 635) Elsa L Gunter 4303 GITC NJIT, www.cs.njit.edu/~elsa/635 www.cs.njit.edu/~elsa/635

2. Copyright 2002 Elsa L. Gunter Sample Grammar ::= | + | - ::= | * | / ::= | ( )

3. Copyright 2002 Elsa L. Gunter Tokens as SML Datatypes + - * / ( ) Becomes an SML datatype datatype token = Id_token of string | Left_parenthesis | Right_parenthesis | Times_token | Divide_token | Plus_token | Minus_token

4. Copyright 2002 Elsa L. Gunter Parsing Token Streams We will create three mutually recursive parsing functions: expr : (token option * (unit -> token option) -> (bool * (token option * (unit -> token option) term : (token option * (unit -> token option) -> (bool * (token option * (unit -> token option) factor : (token option * (unit -> token option) -> (bool * (token option * (unit -> token option)

5. Copyright 2002 Elsa L. Gunter ::= [( + | - ) ] fun expr tokens = (case term tokens of ( true, tokens_after_term) => (case tokens_after_term of ( SOME Plus_token, tokens_after_plus) => Parsing an Expression

6. Copyright 2002 Elsa L. Gunter ::= + fun expr tokens = (case term tokens of ( true, tokens_after_term) => (case tokens_after_term of ( SOME Plus_token, tokens_after_plus) => Parsing a Plus Expression

7. Copyright 2002 Elsa L. Gunter ::= + (case expr (tokens_after_plus(), tokens_after_plus) of ( true, tokens_after_expr) => ( true, tokens_after_expr) Parsing a Plus Expression

8. Copyright 2002 Elsa L. Gunter ::= + (case expr (tokens_after_plus(), tokens_after_plus) of ( true, tokens_after_expr) => ( true, tokens_after_expr) Parsing a Plus Expression

9. Copyright 2002 Elsa L. Gunter ::= + (case expr (tokens_after_plus(), tokens_after_plus) of ( true, tokens_after_expr) => ( true, tokens_after_expr) Parsing a Plus Expression

10. Copyright 2002 Elsa L. Gunter ::= + | ( false,rem_tokens) => ( false, rem_tokens)) Code for Minus_token is almost identical What If No Expression After Plus

11. Copyright 2002 Elsa L. Gunter ::= | _ => ( true, tokens_after_term)) What If No Plus or Minus

12. Copyright 2002 Elsa L. Gunter expr> ::= [( + | - ) ] | ( false, rem_tokens) => ( false, rem_tokens)) Code for term is same as for expr except for replacing addition with multiplication and subtraction with division What if No Term

13. Copyright 2002 Elsa L. Gunter ::= and factor (SOME (Id_token id_name), tokens) = ( true, (tokens(), tokens)) Parsing Factor as Id

14. Copyright 2002 Elsa L. Gunter ::= ( ) | factor (SOME Left_parenthesis, tokens) = (case expr (tokens(), tokens) of ( true, tokens_after_expr) => Parsing Factor as Parenthesized Expression

15. Copyright 2002 Elsa L. Gunter ::= ( ) (case tokens_after_expr of ( SOME Right_parenthesis, tokens_after_rparen ) => ( true, (tokens_after_rparen(), tokens_after_rparen)) Parsing Factor as Parenthesized Expression

16. Copyright 2002 Elsa L. Gunter What if No Right Parenthesis ::= ( ) | _ => ( false, tokens_after_expr))

17. Copyright 2002 Elsa L. Gunter ::= ( ) | ( false, rem_tokens) => ( false, rem_tokens)) What If No Expression After Left Parenthesis

18. Copyright 2002 Elsa L. Gunter What If No Id or Left Parenthesis ::= | ( ) | factor tokens = ( false, tokens)

19. Copyright 2002 Elsa L. Gunter Parsing - in C Assume global variable currentToken that holds the latest token removed from token stream Assume subroutine lex( ) to analyze the character stream, find the next token at the head of that stream and update currentToken with that token Assume subroutine error( ) to raise an exception

20. Copyright 2002 Elsa L. Gunter Parsing expr – in C ::= [( + | - ) ] void expr ( ) { term ( ); if (nextToken == PLUS_CODE) { lex ( ); expr ( ); } else if (nextToken == MINUS_CODE) { lex ( ); expr ( );}

21. Copyright 2002 Elsa L. Gunter SML Code fun expr tokens = (case term tokens of ( true, tokens_after_term) => (case tokens_after_term of (SOME Plus_token,tokens_after_plus) => (case expr (tokens_after_plus(), tokens_after_plus) of ( true, tokens_after_expr) => ( true, tokens_after_expr)

22. Copyright 2002 Elsa L. Gunter Parsing expr – in C (optimized) ::= [( + | - ) ] void expr ( ) { term( ); while (nextToken == PLUS_CODE || nextToken == MINUS_CODE) { lex ( ); term ( ); }

23. Copyright 2002 Elsa L. Gunter Parsing factor – in C ::= void factor ( ) { if (nextToken = ID_CODE) lex ( );

24. Copyright 2002 Elsa L. Gunter ::= and factor (SOME (Id_token id_name), tokens) = ( true, (tokens(), tokens)) Parsing Factor as Id

25. Copyright 2002 Elsa L. Gunter Parsing factor – in C ::= ( ) else if (nextToken == LEFT_PAREN_CODE) { lex ( ); expr ( ); if (nextToken == RIGHT_PAREN_CODE) lex;

26. Copyright 2002 Elsa L. Gunter Comparable SML Code | factor (SOME Left_parenthesis, tokens) = (case expr (tokens(), tokens) of ( true, tokens_after_expr) => (case tokens_after_expr of ( SOME Right_parenthesis, tokens_after_rparen ) => ( true, (tokens_after_rparen(), tokens_after_rparen))

27. Copyright 2002 Elsa L. Gunter Parsing factor – in C else error ( ); /* Right parenthesis missing */ } else error ( ); /* Neither nor ( was found at start */ }

28. Copyright 2002 Elsa L. Gunter Error cases in SML (* No right parenthesis *) | _ => ( false, tokens_after_expr)) (* No expression found *) | ( false, rem_tokens) => ( false, rem_tokens)) (* Neither nor left parenthesis found *) | factor tokens = ( false, tokens)

29. Copyright 2002 Elsa L. Gunter Lexers – Simple Parsers Lexers are parsers driven by regular grammars Use character codes and arithmetic comparisons rather than case analysis to determine syntactic category for each character Often some semantic action must be taken –Compute a number or build a string and record it in a symbol table

30. Copyright 2002 Elsa L. Gunter Example = | = 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 fun digit c = (case Char.ord c of n => if n >= Char.ord #”0” andalso n <= Char.ord #”9” then SOME (n – Char.ord #”0”) else NONE)

31. Copyright 2002 Elsa L. Gunter Example fun pos [] = (NONE,[]) | pos (chars as ch::rem_chars) = (case digit ch of NONE => (NONE, chars) | SOME n => (case pos rem_chars of (NONE, more_chars) => (SOME (10,n), more_chars) | (SOME (p,m), more_chars) => (SOME (10*p,(p*n)+m), more_chars)))

32. Copyright 2002 Elsa L. Gunter Problems for Recursive- Descent Parsing Left Recursion: A ::= Aw translates to a subroutine that loops forever Indirect Left Recursion: A ::= Bw B ::= Av causes the same problem

33. Copyright 2002 Elsa L. Gunter Problems for Recursive- Descent Parsing Parser must always be able to choose the next action based only only the next very next token Pairwise disjointedness Test: Can we always determine which rule (in the non-extended BNF) to choose based on just the first token

34. Copyright 2002 Elsa L. Gunter Pairwise Disjointedness Test For each rule A ::= y Calculate FIRST (y) = {a | y =>* aw}  {  | if y =>*  } For each pair of rules A ::= y and A ::= z, require FIRST(y)  FIRST(z) = { } Test too strong: Can’t handle ::= [ ( + | - ) ]

35. Copyright 2002 Elsa L. Gunter Example Grammar: ::= a b ::= b | b ::= a | a FIRST ( b) = {b} Rules for not pairwise disjoint