Muhammad Idrees, Lecturer University of Lahore 1 Top-Down Parsing Top down parsing can be viewed as an attempt to find a leftmost derivation for an input string. Equivalently, it can be viewed as an attempt to construct a parse tree for the input starting form the root and creating nodes of the parse tree in preorder.

Muhammad Idrees, Lecturer University of Lahore 2 Top-Down Parsing(cont’d) We now consider a general form of top down parsing, called recursive descent, that may involve backtracking, that is making repeated scans of the input.However, backtracking parsers are not seen frequently

Muhammad Idrees, Lecturer University of Lahore 3 Top-Down Parsing(cont’d) One reason is that backtracking is rarely needed to parse programming language constructs. In situations like natural language parsing, backtracking is still not very efficient, and tabular methods such as the dynamic programming algorithm or method of Earley are preferred.

Muhammad Idrees, Lecturer University of Lahore 4 Example. Consider the grammar, S cAd A ab|a and the input string w = cad. To construct a parse tree for this string top down, we initially create a tree consisting of a single node labeled S.

Muhammad Idrees, Lecturer University of Lahore 5 Example (cont’d) An input pointer points to “c”, the first symbol of w. We then use the first production for S to expand the tree and obtain the tree of the Fig.(a) S Fig(a) c Ad

Muhammad Idrees, Lecturer University of Lahore 6 Example (cont’d) The leftmost leaf, labeled “c”, matches the first symbol of w,So we now advance the input pointer to “a”,the second symbol of w, and consider the next leaf, labeled A. We can then expand A using the first alternative for A to obtain the tree of the fig(b).

Muhammad Idrees, Lecturer University of Lahore 7 Example (cont’d) S c A d a b Fig(b)

Muhammad Idrees, Lecturer University of Lahore 8 Example (cont’d) We now have a match for the second input symbol so we advance the input pointer to “d”, the third input symbol and compare “d” against the next leaf, labeled “b”. Since “b” does not match “d”, we report failure and go back to A to see whether there is another alternative for A that we have not tried but that might produce a match.

Muhammad Idrees, Lecturer University of Lahore 9 Example (cont’d) S c Ad a Fig(c)

Muhammad Idrees, Lecturer University of Lahore 10 Example (cont’d) In going back to A, we must reset the input pointer to position 2, the position it had when we first came to A, which means that the procedure for A must store the input pointer in a local variable. We now try the second alternative for A to obtain the tree of the fig(c). The leaf “a” matches the second symbol w and the leaf “d” matches the third symbol. Since we have produced a parse tree for w, we halt and announce successful completion of parsing.

Muhammad Idrees, Lecturer University of Lahore 11 Top-down Parsing To find a leftmost derivation for an input string Construct a parse tree from the root Example S  cAd A  ab | a Input w = cad S cAd S cAd S cAd ab a

Muhammad Idrees, Lecturer University of Lahore 12 Example input: cad cad S cd A S cd A ab S cd A ab Problem: backtrack cad S cd A a S cd A a Example: S  c A d A  ab | a

Muhammad Idrees, Lecturer University of Lahore 13 Parsing – Top-Down & Predictive Top-Down Parsing  Parse tree / derivation of a token string occurs in a top down fashion. For Example, Consider : type  simple |  id | array [ simple ] of type simple  integer | char | num dotdot num Suppose input is : array [ num dotdot num ] of integer Parsing would begin with type  ??? Start symbol

Muhammad Idrees, Lecturer University of Lahore 14 Top-Down Parse type]simpleof [ array type ]simpleof[array type num dotdot Input : array [ num dotdot num ] of integer Lookahead symbol type ? Input : array [ num dotdot num ] of integer Lookahead symbol type  simple |  id | array [ simple ] of type simple  integer | char | num dotdot num Start symbol

Muhammad Idrees, Lecturer University of Lahore 15 Top-Down Parse Input : array [ num dotdot num ] of integer type]simpleof[array type num dotdotsimple type]simpleof[array type num dotdotsimple integer Lookahead symbol type  simple |  id | array [ simple ] of type simple  integer | char | num dotdot num Start symbol

Muhammad Idrees, Lecturer University of Lahore 16 Recursive Descent or Predictive Parsing Parser Operates by Attempting to Match Tokens in the Input Stream Utilize both Grammar and Input Below to Motivate Code for Algorithm array [ num dotdot num ] of integer type  simple |  id | array [ simple ] of type simple  integer | char | num dotdot num procedure match ( t : token ) ; begin if lookahead = t then lookahead : = nexttoken else error end ;

Muhammad Idrees, Lecturer University of Lahore 17 Top-down algorithm (continued) procedure simple ; begin if lookahead = integer then match ( integer ); else if lookahead = char then match ( char ); else if lookahead = num then begin match (num); match (dotdot); match (num) end else error end ; type  simple |  id | array [ simple ] of type simple  integer | char | num dotdot num

Muhammad Idrees, Lecturer University of Lahore 18 Top-Down Algorithm (Continued) procedure type ; begin if lookahead is in { integer, char, num } then simple else if lookahead = ‘  ’ then begin match (‘  ’ ) ; match( id ) end else if lookahead = array then begin match( array ); match(‘[‘); simple; match(‘]’); match(of); type end else error end ; type  simple |  id | array [ simple ] of type simple  integer | char | num dotdot num

Muhammad Idrees, Lecturer University of Lahore 19 Tracing Input: array [ num dotdot num ] of integer To initialize the parser: set global variable : lookahead = array call procedure: type Procedure call to type with lookahead = array results in the actions: match( array ); match(‘[‘); simple; match(‘]’); match(of); type Procedure call to simple with lookahead = num results in the actions: match (num); match (dotdot); match (num) Procedure call to type with lookahead = integer results in the actions: simple Procedure call to simple with lookahead = integer results in the actions: match ( integer ) type  simple |  id | array [ simple ] of type simple  integer | char | num dotdot num

Muhammad Idrees, Lecturer University of Lahore 20 Compiler Phases – Front End Semantic Action Checking Scanner Parser Intermediate Representation Intermediate Representation Semantic Error Request Token Get Token Start

Muhammad Idrees, Lecturer University of Lahore 21 Big Picture  Parsing: Matching code we are translating to rules of a grammar. Building a representation of the code.  Scanning: An abstraction that simplifies the parsing process by converting the raw text input into a stream of known objects called tokens.  Grammar dictates syntactic rules of a language i.e, how a legal sentence in a language could be formed  Lexical rules of a language dictate how a legal word in a language is formed by concatenating alphabet of the language.

Muhammad Idrees, Lecturer University of Lahore 22 Overall Operation Parser is in control of the overall operation Demands scanner to produce a token Scanner reads input file into token buffer & forms a token Token is returned to parser Parser attempts to match the token Failure: Syntax Error! Success: – Does nothing and returns to get next token or – Takes Semantic Action

Muhammad Idrees, Lecturer University of Lahore 23 Overall Operation Semantic Action: Lookup variable name – If found okay – If not: Put in symbol table If semantic checks succeed, do code- generation Return to get next token No more tokens? Done!

Muhammad Idrees, Lecturer University of Lahore 24 Tokenization Input File Token Buffer  What does the Token Buffer contain? – Token being identified  Why a two-way ( ) street? – Characters can be read – and unread – Termination of a token

Muhammad Idrees, Lecturer University of Lahore 25 Example main() m

Muhammad Idrees, Lecturer University of Lahore 26 Example main() am

Muhammad Idrees, Lecturer University of Lahore 27 Example main() iam

Muhammad Idrees, Lecturer University of Lahore 28 Example main() niam

Muhammad Idrees, Lecturer University of Lahore 29 Example main() (niam

Muhammad Idrees, Lecturer University of Lahore 30 Example main() niam Keyword : main

Muhammad Idrees, Lecturer University of Lahore 31 Overall Operation Parser is in control of the overall operation Demands scanner to produce a token Scanner reads input file into token buffer & forms a token Token is returned to parser Parser attempts to match the token Failure: Syntax Error! Success: – Does nothing and returns to get next token OR – Takes Semantic Action

Muhammad Idrees, Lecturer University of Lahore 32 Overall Operation Semantic Action: Lookup variable name – If found okay – If not: Put in symbol table If semantic checks succeed, do code- generation Return to get next token No more tokens? Done!

Muhammad Idrees, Lecturer University of Lahore 33 Grammar Rules  MAIN OPENPAR CLOSEPAR  NULL  VAR , VAR  NULL  CURLYOPEN CURLYCLOSE  VAR ;  VAR = ;  VAR  +  -  INT  FLOAT

Muhammad Idrees, Lecturer University of Lahore 34 Demo main() { int a,b; a = b; } Parser Scanner Token Buffer

Muhammad Idrees, Lecturer University of Lahore 35 Demo main() { int a,b; a = b; } Parser Scanner "Please, get me the next token" Token Buffer

Muhammad Idrees, Lecturer University of Lahore 36 Demo main() { int a,b; a = b; } Parser Scanner m

Muhammad Idrees, Lecturer University of Lahore 37 Demo main() { int a,b; a = b; } Parser Scanner am

Muhammad Idrees, Lecturer University of Lahore 38 Demo main() { int a,b; a = b; } Parser Scanner iam

Muhammad Idrees, Lecturer University of Lahore 39 Demo main() { int a,b; a = b; } Parser Scanner niam

Muhammad Idrees, Lecturer University of Lahore 40 Demo main() { int a,b; a = b; } Parser Scanner (niam

Muhammad Idrees, Lecturer University of Lahore 41 Demo main() { int a,b; a = b; } Parser Scanner niam

Muhammad Idrees, Lecturer University of Lahore 42 Demo main() { int a,b; a = b; } Parser Scanner Token: main Token Buffer

Muhammad Idrees, Lecturer University of Lahore 43 Demo main() { int a,b; a = b; } Parser Scanner "I recognize this" Token Buffer

Muhammad Idrees, Lecturer University of Lahore 44 Parsing (Matching) Start matching using a rule When match takes place at a certain position, move further (get next token & repeat the process) If expansion needs to be done, choose appropriate rule (How to decide which rule to choose?) If no rule found, declare error If several rules found the grammar (set of rules) is ambiguous Grammar ambiguous? Language ambiguous?

Muhammad Idrees, Lecturer University of Lahore 45 Scanning & Parsing Combined main() { int a,b; a = b; } Parser Scanner "Please, get me the next token"

Muhammad Idrees, Lecturer University of Lahore 46 Scanning & Parsing Combined main() { int a,b; a = b; } Parser Scanner Token: MAIN  MAIN OPENPAR CLOSEPAR

Muhammad Idrees, Lecturer University of Lahore 47 Scanning & Parsing Combined main() { int a,b; a = b; } Parser Scanner "Please, get me the next token"  MAIN OPENPAR CLOSEPAR

Muhammad Idrees, Lecturer University of Lahore 48 Scanning & Parsing Combined main() { int a,b; a = b; } Parser Scanner Token: OPENPAR  MAIN OPENPAR CLOSEPAR

Muhammad Idrees, Lecturer University of Lahore 49 Scanning & Parsing Combined main() { int a,b; a = b; } Parser Scanner Token: CLOSEPAR  MAIN OPENPAR CLOSEPAR  NULL

Muhammad Idrees, Lecturer University of Lahore 50 Scanning & Parsing Combined main() { int a,b; a = b; } Parser Scanner Token: CLOSEPAR  MAIN OPENPAR CLOSEPAR  NULL

Muhammad Idrees, Lecturer University of Lahore 51 Scanning & Parsing Combined main() { int a,b; a = b; } Parser Scanner Token: CLOSEPAR  MAIN OPENPAR CLOSEPAR

Muhammad Idrees, Lecturer University of Lahore 52 Scanning & Parsing Combined main() { int a,b; a = b; } Parser Scanner Token: CURLYOPEN  MAIN OPENPAR CLOSEPAR  CURLYOPEN CURLYCLOSE

Muhammad Idrees, Lecturer University of Lahore 53 Scanning & Parsing Combined main() { int a,b; a = b; } Parser Scanner Token: INT  MAIN OPENPAR CLOSEPAR  CURLYOPEN CURLYCLOSE  VAR ;  INT

Muhammad Idrees, Lecturer University of Lahore 54 Scanning & Parsing Combined main() { int a,b; a = b; } Parser Scanner Token: INT  MAIN OPENPAR CLOSEPAR  CURLYOPEN CURLYCLOSE  VAR ;  INT

Muhammad Idrees, Lecturer University of Lahore 55 Scanning & Parsing Combined main() { int a,b; a = b; } Parser Scanner Token: INT  MAIN OPENPAR CLOSEPAR  CURLYOPEN CURLYCLOSE  VAR ;  INT

Muhammad Idrees, Lecturer University of Lahore 56 Scanning & Parsing Combined main() { int a,b; a = b; } Parser Scanner Token: VAR  MAIN OPENPAR CLOSEPAR  CURLYOPEN CURLYCLOSE  VAR ; , VAR  NULL

Muhammad Idrees, Lecturer University of Lahore 57 Scanning & Parsing Combined main() { int a,b; a = b; } Parser Scanner Token: ',' [COMMA]  MAIN OPENPAR CLOSEPAR  CURLYOPEN CURLYCLOSE  VAR ; , VAR  NULL

Muhammad Idrees, Lecturer University of Lahore 58 Scanning & Parsing Combined main() { int a,b; a = b; } Parser Scanner Token: VAR  MAIN OPENPAR CLOSEPAR  CURLYOPEN CURLYCLOSE  VAR ; , VAR  NULL

Muhammad Idrees, Lecturer University of Lahore 59 Scanning & Parsing Combined main() { int a,b; a = b; } Parser Scanner Token: ';'  MAIN OPENPAR CLOSEPAR  CURLYOPEN CURLYCLOSE  VAR ; , VAR  NULL

Muhammad Idrees, Lecturer University of Lahore 60 Scanning & Parsing Combined main() { int a,b; a = b; } Parser Scanner Token: ';'  MAIN OPENPAR CLOSEPAR  CURLYOPEN CURLYCLOSE  VAR ; , VAR  NULL

Muhammad Idrees, Lecturer University of Lahore 61 Scanning & Parsing Combined main() { int a,b; a = b; } Parser Scanner Token: ';'  MAIN OPENPAR CLOSEPAR  CURLYOPEN CURLYCLOSE  VAR ; , VAR  NULL

Muhammad Idrees, Lecturer University of Lahore 62 Scanning & Parsing Combined main() { int a,b; a = b; } Parser Scanner Token: ';'  MAIN OPENPAR CLOSEPAR  CURLYOPEN CURLYCLOSE  VAR ; , VAR  NULL

Muhammad Idrees, Lecturer University of Lahore 63 Scanning & Parsing Combined main() { int a,b; a = b; } Parser Scanner Token: ';'  MAIN OPENPAR CLOSEPAR  CURLYOPEN CURLYCLOSE  VAR ;

Muhammad Idrees, Lecturer University of Lahore 64 Scanning & Parsing Combined main() { int a,b; a = b; } Parser Scanner Token: ';'  MAIN OPENPAR CLOSEPAR  CURLYOPEN CURLYCLOSE  VAR ;

Muhammad Idrees, Lecturer University of Lahore 65 Scanning & Parsing Combined main() { int a,b; a = b; } Parser Scanner Token: VAR  MAIN OPENPAR CLOSEPAR  CURLYOPEN CURLYCLOSE  VAR = ;  VAR

Muhammad Idrees, Lecturer University of Lahore 66 Scanning & Parsing Combined main() { int a,b; a = b; } Parser Scanner Token: '='  MAIN OPENPAR CLOSEPAR  CURLYOPEN CURLYCLOSE  VAR = ;  VAR

Muhammad Idrees, Lecturer University of Lahore 67 Scanning & Parsing Combined main() { int a,b; a = b; } Parser Scanner Token: VAR  MAIN OPENPAR CLOSEPAR  CURLYOPEN CURLYCLOSE  VAR = ;  VAR

Muhammad Idrees, Lecturer University of Lahore 68 Scanning & Parsing Combined main() { int a,b; a = b; } Parser Scanner Token: VAR  MAIN OPENPAR CLOSEPAR  CURLYOPEN CURLYCLOSE  VAR = ;  VAR

Muhammad Idrees, Lecturer University of Lahore 69 Scanning & Parsing Combined main() { int a,b; a = b; } Parser Scanner Token: VAR  MAIN OPENPAR CLOSEPAR  CURLYOPEN CURLYCLOSE  VAR = ;  VAR

Muhammad Idrees, Lecturer University of Lahore 70 Scanning & Parsing Combined main() { int a,b; a = b; } Parser Scanner Token: ';'  MAIN OPENPAR CLOSEPAR  CURLYOPEN CURLYCLOSE  VAR = ;

Muhammad Idrees, Lecturer University of Lahore 71 Scanning & Parsing Combined main() { int a,b; a = b; } Parser Scanner Token: ';'  MAIN OPENPAR CLOSEPAR  CURLYOPEN CURLYCLOSE  VAR = ;

Muhammad Idrees, Lecturer University of Lahore 72 Scanning & Parsing Combined main() { int a,b; a = b; } Parser Scanner Token: CURLYCLOSE  MAIN OPENPAR CLOSEPAR  CURLYOPEN CURLYCLOSE