1. Lexical and Syntax Analysis
Chapter 4
Lexical and Syntax Analysis

2. Chapter 4 Lexical and Syntax Analysis
Contents:
Introduction
Lexical Analysis
The Parsing Problem
Recursive-Descent Parsing
Bottom-Up Parsing

3. Introduction
Language implementation systems must analyze source code, regardless of the specific implementation approach
Nearly all syntax analysis is based on a formal description of the syntax of the source language (BNF)

4. Advantages of Using BNF to Describe Syntax
Provides a clear and concise syntax description
The parser can be based directly on the BNF
Parsers based on BNF are easy to maintain

5. Syntax Analysis
The syntax analysis portion of a language processor nearly always consists of two parts:
A low-level part called a lexical analyzer (mathematically, a finite automaton based on a regular grammar)
A high-level part called a syntax analyzer, or parser (mathematically, a push-down automaton based on a context-free grammar, or BNF)

6. Reasons to Separate Lexical and Syntax Analysis
Simplicity - less complex approaches can be used for lexical analysis; separating them simplifies the parser
Efficiency - separation allows optimization of the lexical analyzer
Portability - parts of the lexical analyzer may not be portable, but the parser always is portable

7. Lexical Analysis Lexical Analysis and Compiling process Program Parse
Syntactic analysis
Lexical analysis
Token Stream
Parse
Semantic analysis
Parse
Code generation
Machine code
Optimization

8. Lexical analysis is the process of converting a sequence of characters into a sequence of tokens.
A program or function which performs lexical analysis is called a lexical analyzer, lexer or scanner.
A lexer often exists as a single function which is called by a parser or another function.

9. Lexical Analysis (Detailed description)
A lexical analyzer is a pattern matcher for character strings
A lexical analyzer is a “front-end” for the parser
Identifies substrings of the source program that belong together - lexemes
Lexemes match a character pattern, which is associated with a lexical category called a token
Sum = oldsum – value / 100;
Lexeme  sum, = , oldsum, -, value, / , 100, ;
Token  IDENTIFIER(IDENT), ASSIGN_OP, IDENT, SUBTRACT_OP, IDENT, DIVISION_OP, INT_LIT, SEMICOLON
sum is a lexeme; its token may be IDENT

10. Lexical Analysis (continued)
The lexical analyzer is usually a function that is called by the parser when it needs the next token
Three approaches to build a lexical analyzer:
1- Write a formal description of the tokens and use a software tool that constructs table-driven lexical analyzers given such a description.

11. 2- Design a state diagram that describes the tokens and write a program that implements the state diagram
3- Design a state diagram that describes the tokens and hand-construct a table-driven implementation of the state diagram
We only discuss approach 2
State Diagram Design
A naive state diagram would have a transition from every state on every character in the source language - such a diagram would be very large!

12. Lexical Analysis (contd.)
In many cases, transitions can be combined to simplify the state diagram
When recognizing an identifier, all uppercase and lowercase letters are equivalent
Use a character class that includes all letters
When recognizing an integer literal, all digits are equivalent - use a digit class

13. Lexical Analysis (contd.)
Reserved words and identifiers can be recognized together (rather than having a part of the diagram for each reserved word)
Use a table lookup to determine whether a possible identifier is in fact a reserved word

14. Lexical Analysis (contd.)
Convenient utility subprograms:
getChar - gets the next character of input, puts it in nextChar, determines its class and puts the class in charClass
addChar - puts the character from nextChar into the place the lexeme is being accumulated, lexeme
lookup - determines whether the string in lexeme is a reserved word (returns a code)

15. State Diagram
A state diagram to recognize names, reserved words, and integer literals

16. Lexical Analysis (continued)
Implementation (assume initialization): int lex() { switch (charClass) { case LETTER: /* Parse identifiers and reserved words */ addChar(); getChar(); while ( charClass == LETTER || charClass == DIGIT) { } return lookup(lexeme); break;

17. Lexical Analysis (continued)
case DIGIT: /* Parse integer literals */ addChar(); getChar(); while ( charClass == DIGIT ) { } return INT_LIT; break; } /* End of switch */ } /* End of function lex */

18. The Parsing Problem
Parsing, or, more formally, syntactic analysis, is the process of analyzing a text, made of a sequence of tokens to determine its grammatical structure with respect to a given formal grammar.
Goals of the parser, given an input program:
Find all syntax errors; for each errror, produce an appropriate diagnostic message and recover quickly
Produce the parse tree, or at least a trace of the parse tree, for the program

19. The Parsing Problem (continued)
The task of the parser is essentially to determine if and how the input can be derived from the start symbol of the grammar. This can be done in essentially two ways:
Two categories of parsers
1- Top down - produce the parse tree, beginning at the root
Order is that of a leftmost derivation
Traces or builds the parse tree in preorder
2- Bottom up - produce the parse tree, beginning at the leaves
Order is that of the reverse of a rightmost derivation
Useful parsers look only one token ahead in the input

20. The Parsing Problem (contd.)
Top-down Parsers
Top-down parsing can be viewed as an attempt to find left-most derivations of an input-stream by searching for parse trees using a top-down expansion of the given formal grammar rules.
In other words, given a sentential form, xA , the parser must choose the correct A-rule to get the next sentential form in the leftmost derivation, using only the first token produced by A
The most common top-down parsing algorithms:
Recursive descent - a coded implementation
LL parsers (Left-to-right, Leftmost derivation) - table driven implementation

21. The Parsing Problem (contd.)
Bottom-up parsers
A bottom-up parser can start with the input and attempt to rewrite it to the start symbol. Intuitively, the parser attempts to locate the most basic elements, then the elements containing these, and so on. Another term used for this type of parser is Shift-Reduce parsing.
In other words, given a right sentential form, , determine what substring of  is the right-hand side of the rule in the grammar that must be reduced to produce the previous sentential form in the right derivation
The most common bottom-up parsing algorithms are in the LR (Left-to-right, Rightmost derivation) family

22. The Parsing Problem (contd.)
The Complexity of Parsing
Parsers that work for any unambiguous grammar are complex and inefficient ( O(n3), where n is the length of the input )
Compilers use parsers that only work for a subset of all unambiguous grammars, but do it in linear time ( O(n), where n is the length of the input )

23. Recursive-Descent Parsing
There is a subprogram for each non-terminal in the grammar, which can parse sentences that can be generated by that non-terminal
EBNF is ideally suited for being the basis for a recursive-descent parser, because EBNF minimizes the number of non-terminals
A grammar for simple expressions:
<expr>  <term> {(+ | -) <term>}
<term>  <factor> {(* | /) <factor>}
<factor>  id | ( <expr> )

24. Recursive-Descent Parsing (contd.)
Assume we have a lexical analyzer named lex, which puts the next token code in nextToken
The coding process when there is only one RHS:
For each terminal symbol in the RHS, compare it with the next input token; if they match, continue, else there is an error
For each non-terminal symbol in the RHS, call its associated parsing subprogram

25. Recursive-Descent Parsing (contd.)
/* Function expr Parses strings in the language generated by the rule: <expr> → <term> {(+ | -) <term>} */ void expr() { /* Parse the first term */ term(); /* non-terminal is function call */ /* As long as the next token is + or -, call lex to get the next token, and parse the next term */ while (nextToken == PLUS_CODE || nextToken == MINUS_CODE){ lex(); term(); } }

26. Recursive-Descent Parsing (cont.)
This particular routine does not detect errors
Convention: Every parsing routine leaves the next token in nextToken
A non-terminal that has more than one RHS requires an initial process to determine which RHS it is to parse
The correct RHS is chosen on the basis of the next token of input (the lookahead)
The next token is compared with the first token that can be generated by each RHS until a match is found
If no match is found, it is a syntax error

27. Recursive-Descent Parsing (contd.)
/* Function factor Parses strings in the language generated by the rule: <factor> -> id | (<expr>) */ void factor() { if (nextToken) == ID_CODE) /* Determine which RHS */ lex(); /* For the RHS id, just call lex */ /* If the RHS is (<expr>) – call lex to pass over the left parenthesis, call expr, and check for the right parenthesis */ else if (nextToken == LEFT_PAREN_CODE) { lex(); expr(); /* non-terminal is function call */ if (nextToken == RIGHT_PAREN_CODE) lex(); else error(); } /* End of else if (nextToken == ... */ else error(); /* Neither RHS matches */ }

28. Top Down Parsing (Contd.)
The LL Parser
An LL parser is a top-down parser that parses the input from Left to right, and constructs a Leftmost derivation of the sentence. The class of grammars which are parsable in this way is known as the LL grammars.
An LL parser is called an LL(k) parser if it uses k tokens of lookahead when parsing a sentence. If such a parser exists for a certain grammar and it can parse sentences of this grammar without backtracking then it is called an LL(k) grammar.
LL(1) grammars are very popular because the corresponding LL parsers only need to look at the next token to make their parsing decisions

29. The LL Parser (Contd.) LL(1) Parser
The parser works on strings from a particular context-free grammar.
The parser consists of
an input buffer, holding the input string (built from the grammar)
a stack on which to store the terminals and non-terminals from the grammar yet to be parsed
a parsing table which tells it what (if any) grammar rule to apply given the symbols on top of its stack and the next input token
The parser applies the rule found in the table by matching the top-most symbol on the stack (row) with the current symbol in the input stream (column).
When the parser starts, the stack already contains two symbols:
where '$' is a special terminal to indicate the bottom of the stack and the end of the input stream, and 'S' is the start symbol of the grammar. The parser will attempt to rewrite the contents of this stack to what it sees on the input stream. However, it only keeps on the stack what still needs to be rewritten.
[ S, $ ]

30. The LL Parser (Contd.) Example:
To explain its workings we will consider the following small grammar:
S → F
S → ( S + F )
F → a
and parse the following input:
( a + a )
The parsing table for this grammar looks as follows ( a + ) $ S 2 1 - F 3

31. The LL Parser Example (Contd.)
Parsing procedure
In each step, the parser reads the next-available symbol from the input stream, and the top-most symbol from the stack. If the input symbol and the stack-top symbol match, the parser discards them both, leaving only the unmatched symbols in the input stream and on the stack.
Thus, in its first step, the parser reads the input symbol '(' and the stack-top symbol 'S'. The parsing table instruction comes from the column headed by the input symbol '(' and the row headed by the stack-top symbol 'S'; this cell contains '2', which instructs the parser to apply rule (2). The parser has to rewrite 'S' to '( S + F )' on the stack and write the rule number 2 to the output. The stack then becomes:
[ ( , S, +, F, ), $ ]

32. The LL Parser Example (Contd.)
Since the '(' from the input stream did not match the top-most symbol, 'S', from the stack, it was not removed, and remains the next-available input symbol for the following step.
In the second step, the parser removes the '(' from its input stream and from its stack, since they match. The stack now becomes:
Now the parser has an 'a' on its input stream and an 'S' as its stack top. The parsing table instructs it to apply rule (1) from the grammar and write the rule number 1 to the output stream. The stack becomes:
The parser now has an 'a' on its input stream and an 'F' as its stack top. The parsing table instructs it to apply rule (3) from the grammar and write the rule number 3 to the output stream. The stack becomes:
[ S, +, F, ), $ ]
[ F, +, F, ), $ ]
[ a, +, F, ), $ ]

33. The LL Parser Example (Contd.)
In the next two steps the parser reads the 'a' and '+' from the input stream and, since they match the next two items on the stack, also removes them from the stack. This results in:
In the next three steps the parser will replace 'F' on the stack by 'a', write the rule number 3 to the output stream and remove the 'a' and ')' from both the stack and the input stream. The parser thus ends with '$' on both its stack and its input stream.
In this case the parser will report that it has accepted the input string and write the following list of rule numbers to the output stream:
[ 2, 1, 3, 3 ]
This is indeed a list of rules for a leftmost derivation of the input string, which is:
S → ( S + F ) → ( F + F ) → ( a + F ) → ( a + a )
[ F, ), $ ]

34. The LL Parser
Remarks
As can be seen from the example the parser performs three types of steps depending on whether the top of the stack is a nonterminal, a terminal or the special symbol $:
If the top is a nonterminal then it looks up in the parsing table on the basis of this nonterminal and the symbol on the input stream which rule of the grammar it should use to replace it with on the stack. The number of the rule is written to the output stream. If the parsing table indicates that there is no such rule then it reports an error and stops.
If the top is a terminal then it compares it to the symbol on the input stream and if they are equal they are both removed. If they are not equal the parser reports an error and stops.
If the top is $ and on the input stream there is also a $ then the parser reports that it has successfully parsed the input, otherwise it reports an error. In both cases the parser will stop.
These steps are repeated until the parser stops, and then it will have either completely parsed the input and written a leftmost derivation to the output stream or it will have reported an error.

35. Bottom-up Parsing
A bottom-up parser starts with the input and attempt to rewrite it to the start symbol. The name comes from the concept of a parse tree, in which the most fundamental units are at the bottom, and structures composed of them are in successively higher layers, until at the top of the tree a single unit, or start symbol, comprises all of the information being analyzed.
In other words, bottom-up parsing is a parsing method that works by identifying terminal symbols first, and combines them successively to produce nonterminals.
The most common bottom-up parsers are the shift-reduce parsers.

36. Action table Shift-Reduce Parsers
These parsers examine the input tokens and either shift (push) them onto a stack or reduce elements at the top of the stack, replacing a right-hand side by a left-hand side.
Action table
Often an action (or parse) table is constructed which helps the parser determine what to do next. The following is a description of what can be held in an action table.
Actions
Shift - push token onto stack
Reduce - remove handle from stack and push on corresponding nonterminal
Accept - recognize sentence when stack contains only the distinguished symbol and input is empty
Error - happens when none of the above is possible; means original input was not a sentence

37. Algorithm: Shift-reduce parsing
1. Start with the sentence to be parsed as the initial sentential form
2. Until the sentential form is the start symbol do:
2.1. Scan through the input until we recognise something that corresponds to the RHS of one of the production rules (this is called a handle)
2.2. Apply a production rule in reverse; i.e., replace the RHS of the rule which appears in the sentential form with the LHS of the rule (an action known as a reduction)
A shift-reduce parser is most commonly implemented using a stack, where we proceed as follows:
start with an empty stack
a "shift" action corresponds to pushing the current input symbol onto the stack
a "reduce" action occurs when we have a handle on top of the stack. To perform the reduction, we pop the handle off the stack and replace it with the nonterminal on the LHS of the corresponding rule.

38. Bottom-up Parsing Example
Let us take the following grammar:
(1). Sentence --> NounPhrase VerbPhrase
(2). NounPhrase --> Article Noun
(3). VerbPhrase --> Verb | Adverb Verb
(4). Article --> the | a | ...
(5). Verb --> jumps | sings | ...
(6). Noun --> dog | cat | ...
And the input sequence:
the dog jumps
Let us apply the Shift – Reduce Parsing algorithm to this input sequence and see how the Shift-Reduce Parsers work.

39. Bottom-up Parsing Example Contd.
‘the dog jumps’ (input Sequence) Then the bottom up parsing using Shift-Reduce Algorithm is: Stack Input Sequence () (the dog jumps) (the) (dog jumps) SHIFT word onto stack (Art) (dog jumps) REDUCE using grammar rule (4) (Art dog) (jumps) SHIFT.. (Art Noun) (jumps) REDUCE using grammar rule (6) (NounPhrase) (jumps) REDUCE using grammar rule (2) (NounPhrase jumps) () SHIFT (NounPhrase Verb) () REDUCE using grammar rule (5) (NounPhrase VerbPhrase) () REDUCE using grammar rule (3) (Sentence) () SUCCESS using grammar rule (1)

40. Summary Syntax analysis is a common part of language implementation
A lexical analyzer is a pattern matcher that isolates small-scale parts of a program
Detects syntax errors
Produces a parse tree
A recursive-descent parser is an LL parser
Parsing problem for bottom-up parsers: find the substring of current sentential form
The shift-reduce parsers is the most common bottom-up parsing approach