Lesson 10 CDT301 – Compiler Theory, Spring 2011 Teacher: Linus Källberg

2 Outline Flex Bison Abstract syntax trees

FLEX 3

Flex Tool for automatic generation of scanners Open-source version of Lex Takes regular expressions as input Outputs a C (or C++) file for the scanner 4

Flex 5 Regexps mylexer.l int yylex() … mylexer.c FlexC compiler … mylexer.obj

The input file to Flex Definitions % Rules % User code 6

The definitions section Macro definitions: –Specify a letter: letter [A-Za-z] –Specify a delimiter: delimiter [,:;.] –Specify a digit: digit [0-9] –Specify an identifier: id letter(letter|digit)* 7

The definitions section User code: %{ #include int a_nice_global_variable = 0; int my_favourite_function(void) {return 42;} %} 8

The rules section Rule = regexp + C code Longest matching pattern is used If two equally long patterns match, the first one in the file is used Examples: =|>=?| )?{ return RELOP; } {id}{ return ID; } 9

The regexp language of Flex ?Previous regexp is optional {}Macro expansion (defined in the definitions section).Matches any character that is not end of line $Matches the end of a line ^Matches the beginning of a line []Matches any enclosed character 10

The [] syntax Similar to | but more powerful Example: digit[ ] is the same as digit0|1|2|3|4|5|6|7|8|9 Special characters inside the brackets: – and ^ digit [0-9] letter [A-Za-z] non_digit [^0-9] 11

The user code section Only C code valid here Will be copied unchanged to the generated C file 12

The generated scanner By default, a function called yylex() is defined –Works similar to your GetNextToken() from lab 1 –The name can be changed with options Some globals are defined as well (can be changed into local variables with options): yyinThe file to read from yytextThe matched lexeme (char*) yylengThe length of yytext yylinenoLine number of the match 13

The yywrap() function Called upon end-of-file Should be supplied by the user Suppressed with %option noyywrap or --noyywrap 14

Scanner states in Flex Affects what tokens should be recognized Example from the language ALF: { fref 32 DEADC0DE }<- Identifier { hex_val DEADC0DE }<- Hex constant 15

Scanner states in Flex Declare state: %x READ_HEX Use the state to make rules conditional: hex_val{ BEGIN(READ_HEX); return HEX_VAL_KW; } [a-zA-Z_][a-zA-Z0-9_]*{ return ID; } [0-9a-fA-F]+{ BEGIN(INITIAL); return NUM; } 16

Online resources 17

BISON 18

Bison Tool for automatic generation of parsers Open-source alternative to Yacc Takes an SDT scheme as input Outputs C (or C++) source code for an LALR parser Commonly used together with Flex 19

Bison 20 SDT scheme myparser.y int parse() … myparser.c Bison C compiler … myparser.obj Token definitions myparser.h

The input file to Bison Definitions % SDT scheme % User code 21

Definitions section Define tokens Define operator precedence Define operator associativity Define the types of grammar symbol attributes Write C code between %{ and %} Issue certain commands to Bison 22

Token definition Normal case: %token IDENTIFIER %token WHILE Token, precedence, associativity, and type: %left RELOP %left MINUSOP PLUSOP %right NOTOP Enables use of ambiguous grammars! 23

Defining types Just enter the type inside <> before the list of tokens: %left RELOP %left MULOP %right NOTOP UNOP %token ID STRING Or the same for non-terminals: %type stmnt expr actuals exprs 24

The variable yylval Used by the lexical analyzer to store token attributes Default type is int May be given another type(s) using %union: %union { int Operator; char *String; NODE_TYPE Node; } The type (member name) is then used like this: %token ID STRING 25

Code provided by the user yyerror(char* msg) –Function called on syntax errors yylex() –Function called to get the next token 26

Options to Bison Given on the command line or in the grammar file --defines or %defines: Output a C header file with definitions useful to a scanner –Tokens (#defines) and the type on yylval %error-verbose: More detailed error messages --name-prefix or %name-prefix: Change the default “yy” prefix on all names %define api.pure: Do not use globals --verbose or %verbose: Write detailed information to extra output file 27

Translation scheme section decl: BASIC_TYPE idents ';' ; idents: idents ',' ident | ident ; ident: ID ; 28

Semantic actions Written in C Executed when the production is used in a reduction $$, $1, $2, etc. refer to the attributes of the grammar symbols –Can be used as regular C variables –$$ refer to the attribute of the head, $1 to the attribute of the first symbol in the body, etc. E : E '+' T { $$ = $1 + $3; } ; 29

Using ambiguous grammars in Bison Default actions: –Reduce/reduce: choose first rule in file –Shift/reduce: always shift With explicit precedence and associativity: –Shift/reduce: Compare prec/ass of rule with that of lookahead token 30

The %expect declaration To suppress shift/reduce warnings: %expect n where n is the exact nr of conflicts 31

Contextual precedence Same token might have different precedence depending on context: expr → expr – expr | expr * expr | – expr | id 32 StackInput … – expr* expr …

Contextual precedence Define dummy token: %left '-' %left '*' %left UMINUS Use the %prec modifier: expr → – expr %prec UMINUS 33

Examples of parser configurations StackInputAction … if (cond) stmtelse …shift StackInputAction … expr + expr* …shift StackInputAction … expr * expr+ …red. expr → expr * expr StackInputAction … expr * expr* …red. expr → expr * expr 34

Online resources 35

ABSTRACT SYNTAX TREES 36

Abstract syntax trees “AST” or just “syntax tree” 37 E E E a + E E b5 * + * a b5

Syntax trees vs. parse trees Parse trees: Interior nodes are nonterminals, leaves are terminals Rarely constructed as an explicit data structure Represents the concrete syntax Syntax trees: Interior nodes are “operators”, leaves are operands Commonly constructed as an explicit data structure Represents the abstract syntax 38

Why syntax trees? Simplifies subsequent analyses Independent on the parsing strategy Makes it easier to add new analysis passes without having to modify the parser More compact representation than parse trees 39

Syntax tree example if (a < 1) b = 2 + 3; else { c = d * 4; e(f, 5); } 40 if < = a = c call e f * 1 b + 23 d4 null 5

Exercise (1) Draw an abstract syntax tree for the statement while (i < 100) { x = 2 * x; i = i + 1; } 41

Constructing a syntax tree in Bison expr: expr '+' expr{ $$ = createOpNode($1, '+',$3); } | expr '*' expr{ $$ = createOpNode($1, '*',$3); } | ID{ $$ = createIdNode($1.name); } ; 42

Constructing a syntax tree in Bison stmt : RETURN expr ';'{ $$ = mReturn($2, $1); } ; stmts : stmts stmt { $$ = connectStmts($1, $2); } | { $$ = NULL; } ; 43

Conclusion Flex generates C source code for a scanner given a set of regular expressions Bison generates C source code for a bottom- up parser given a syntax-directed translation scheme Building syntax trees simplifies subsequent analyses of the program Syntax trees can be built in semantic actions 44

Next time Syntax-directed definitions and translation schemes Semantic analysis and type analysis 45