1. Compiler Tools Lex/Yacc – Flex & Bison

2. Compiler Front End (from Engineering a Compiler) Scanner (Lexical Analyzer) Maps stream of characters into words  Basic unit of syntax  x = x + y ; becomes The actual words are its lexeme Its part of speech (or syntactic category ) is called its token type Scanner discards white space & (often) comments Source code Scanner Intermediate Representation Parser Errors tokens Speed is an issue in scanning  use a specialized recognizer

3. The Front End (from Engineering a Compiler) Parser Checks stream of classified words (parts of speech) for grammatical correctness Determines if code is syntactically well-formed Guides checking at deeper levels than syntax Builds an IR representation of the code Parsing is harder than scanning. Better to put more rules in scanner (whitespace etc). Source code Scanner IR Parser Errors tokens

4. The Big Picture Language syntax is specified with parts of speech, not words Syntax checking matches parts of speech against a grammar 1. goal  expr 2. expr  expr op term 3. | term 4. term  number 5. | id 6. op  + 7. | – S = goal T = { number, id, +, - } N = { goal, expr, term, op } P = { 1, 2, 3, 4, 5, 6, 7} No words here! Parts of speech, not words!

5. Why study lexical analysis? We want to avoid writing scanners by hand Finite automata are used in other applications: grep, website filtering, various “find” commands Goals:  To simplify specification & implementation of scanners  To understand the underlying techniques and technologies Scanner Generator specifications source codeparts of speech & words tables or code Specifications written as “regular expressions” Represent words as indices into a global table

6. Finite Automata Formally a finite automata is a five-tuple(S,  ,  , s 0, S F ) where S is the set of states, including error state S e. S must be finite.  is the alphabet or character set used by recognizer. Typically union of edge labels (transitions between states).  (s,c) is a function that encodes transitions (i.e., character c in   changes to state s in S. ) s 0 is the designated start state S F is the set of final states, drawn with double circle in transition diagram

7. Finite Automata Finite automata to recognize fee and fie: S = {s 0, s 1, s 2, s 3, s 4, s 5, s e }  = {f, e, i}  (s,c) set of transitions shown above s 0 = s 0 S F = { s 3, s 5 } Set of words accepted by a finite automata F forms a language L(F). Can also be described by regular expressions. S0S0 S4S4 S1S1 f S3S3 S5S5 S2S2 e i e e

8. Finite Automata Quick Exercise Draw a finite automata that can recognize CU | CSU | CSM | DU (drawing included below for reference) S0S0 S4S4 S1S1 f S3S3 S5S5 S2S2 e i e e

9. RE vs CFG* Context-free grammars are strictly more powerful than regular expressions.  Any language that can be generated using regular expressions can be generated by a context-free grammar.  There are languages that can be generated by a context- free grammar that cannot be generated by any regular expression. As a corollary, CFGs are strictly more powerful than DFAs and NDFAs. The proof is in two parts:  Given a regular expression R, we can generate a CFG G such that L(R) == L(G).  We can define a grammar G for which there there is no FA F such that L(F) == L(G). * from http://www.cs.rochester.edu/~nelson/courses/csc_173/grammars/cfg.html

10. Example No finite automata (and therefore no regular expression) can recognize the set of strings consisting of some number of 0s followed by the same number of 1s. {0^n 1^n|n>=1} --> 0 1 What does this mean for us? Recognizing tokens can be done with a regular expression, but we need a CFG to recognize sentences in our language.

11. Regular Expression Examples digit: [0-9] int with at least 1 digit: [0-9]+ int that can have 0 digits: [0-9]* What about float?  [0-9]*\.[0-9]+ // literal., at least 1 digit after.  – what about 0 or 2?  ([0-9]+)| ([0-9]*\.[0-9]+) // combine int and float, notice use of (),  - what about unary -?  -?(([0-9]+)| ([0-9]*\.[0-9]+))

12. Regular Expressions in Lex* The characters that form regular expressions include: . matches any single character except newline  * matches zero or more copies of preceding expression  [] a character class that matches any character within the brackets. If first character is ^ will match any character except those within brackets. A dash can be used for character range, e.g., [0-4] is equivalent to [01234]. more in book…  Examples:  a.[0]* would match ab, ac, aa, ab0, ac0, aa0, ab00, ac00, aa00, …  a[bc] would match ab or ac  a[bc]* would match a, ab, ac, abb, acc, abc, acb, … * from lex & yacc by Levine, Mason & Brown

13. Regular Expressions in Lex* The characters that form regular expressions also include:  ^ matches beginning of line as first character of expression (also negation within [], as listed above).  $ matches end of line as last character of expression  {} indicates how many times previous pattern is allowed to match, e.g., A{1,3} matches one to three occurrences of A.  \ used to escape metacharacters, e.g., \* is literal asterisk, \” is a literal quote, \{ is literal open brace, etc.  Examples:  ^I[ab] matches Ia or Ib at the beginning of a line  I[ab]$ matches Ia or Ib at the end of a line, e.g., not Iaj .\.. matches a.b, c.d etc. * from lex & yacc by Levine, Mason & Brown

14. Regular Expressions, continued  + matches one or more occurrences of preceding expression, e.g., [0-9]+ matches “1” “11” or “1234” but not empty string  ? matches zero or one occurrence of preceding expression, e.g., -?[0-9]+ matches signed number with optional leading minus sign  | matches either preceding or following expression, e.g., cow|pig|sheep matches any of the three words  “…” interprets everything inside quotation marks literally  () Groups a series of regular expressions into a new regular expression, e.g., (01) becomes character sequence 01. Useful when building up complex patterns with *, + and |.  Examples:  a[0-9]+(dog|cat) matches a0cat, a21dog, etc. but NOT acat

15. More Regular Expression Examples What’s a regular expression for matching quotes?  Try: \”.*\”  won’t work for lines with multiple quoted parts, such as: “mine” and “yours” because lex matches largest possible pattern.  \”[^”\n]*[“\n]  will work by excluding “ (forces lex to stop as soon as “ is reached). The \n keeps a quoted string from exceeding one line.

16. Flex – Fast Lexical Analyzer FLEX scanner (program to recognize patterns in text) regular expressions & C-code rules lex.yy.c contains yylex() compile executable – analyzes and executes input Here’s where we’ll put the regular expressions to good use! (Scanner generator)

17. Flex input file 3 sections definitions % rules % user code

18. Definition Section Examples name definition DIGIT [0-9] ID [a-z][a-z0-9]* A subsequent reference to {DIGIT}+"."{DIGIT}* is identical to: ([0-9])+"."([0-9])*

19. C Code Can include C-code in definitions %{ /* This is a comment inside the definition */ #include // may need headers #include // for printf in BB #include // for exit(0) in BB %}

20. Rules The rules section of the flex input contains a series of rules of the form: pattern action In the definitions and rules sections, any indented text or text enclosed in %{ and %} is copied verbatim to the output (with the %{ %}'s removed). The %{ %}'s must appear unindented on lines by themselves.

21. Example: Simple Pascal-like recognizer Definitions section: /* scanner for a toy Pascal-like language */ %{ /* need for the call to atof() below */ #include %} DIGIT [0-9] ID [a-z][a-z0-9]* Remember these are on a line by themselves, unindented! } Lines inserted as-is into resulting code } Definitions that can be used in rules section

22. Example continued Rules section: % {DIGIT}+ { printf("An integer: %s (%d)\n", yytext, atoi(yytext ));} {DIGIT}+"."{DIGIT}* {printf("A float: %s (%g)\n", yytext, atof(yytext));} if|then|begin|end|procedure|function {printf("A keyword: %s\n", yytext);} {ID} { printf( "An identifier: %s\n", yytext ); } "+"|"-"|"*"|"/" { printf( "An operator: %s\n", yytext ); } "{"[^}\n]*"}" /* eat up one-line comments */ [ \t\n]+ /* eat up whitespace */. { printf( "Unrecognized character: %s\n", yytext ); } pattern action text that matched the pattern (a char*)

23. Example continued User code (required for flex, in library for lex) % yywrap() {} // needed to link, unless libfl.a is available // OR put %option noyywrap at the top of a flex file. int main(int argc, char ** argv ) { ++argv, --argc; /* skip over program name */ if ( argc > 0 ) yyin = fopen( argv[0], "r" ); else yyin = stdin; yylex(); } lexer function produced by lex lex input file

24. Flex exercise #1 1. Download pascal.l 2. From a command prompt (Start->Run->cmd): Flex -opascal.c -L pascal.l NOTE: without –o option output file will be called lex.yy.c -L option suppresses #lines that cause problems with some compilers (e.g. DevC++) 3. Compile and execute pascal.c (batch on Blackboard) gcc –opascal.exe –Lc:\progra~1\gnuwin32\lib pascal.c –lfl -ly 4. Execute program. Type in digits, ids, keywords etc. End program with Ctrl-Z

25. Flex exercise #2 Copy words.l (from lex & yacc) Use flex then compile and execute What does it do? Extend the example with 1 new part of speech. Recognize lexemes R0-R9 as register names Recognize complex numbers, including for example -3+4i, +5-6i, +7i, 8i, -12i, but not 3++4i (hint: print newline before displaying your complex number, lexer may display 3+ and then recognize +4i)

26. Lex techniques Hardcoding lists not very effective. Often use symbol table. Example in lec & yacc, not covered in class but see me if you’re interested.

27. Bison – like Yacc (yet another compiler compiler) Context-free Grammar in BNF form, LALR(1)* Bison Bison parser (c program) group tokens according to grammar rules Bison parser provides yyparse You must provide: the lexical analyzer (e.g., flex) an error-handling routine named yyerror a main routine that calls yyparse *LookAhead Left Recursive

28. Bison Parser Same sections as flex (yacc came first): definitions, rules, C-Code We’ll discuss rules first, then definitions and C-Code

29. Bison Parser – Rule Section Consider CFG => ID = Would be written in bison “rules” section as: statement: NAME ‘=‘ expression | expression { printf("= %d\n", $1); } ; expression: NUMBER ‘+’ NUMBER { $$ = $1 + $3; } | NUMBER ‘-’ NUMBER { $$ = $1 + $3; } | NUMBER { $$ = $1; } ; Use : between lhs and rhs, place ; at end. What are $$? next slide… white space ; at end NOTE: The first rule in statement won’t be operational yet…

30. More on bison Rules and Actions $1, $3 refer to RHS values. $$ sets value of LHS. In expression, $$ = $1 + $3 means it sets the value of lhs (expression) to NUMBER ($1) + NUMBER ($3) A rule action is executed when the parser reduces that rule (will have recognized both NUMBER symbols) lexer should have returned a value via yylval (next slide) statement: NAME ‘=‘ expression | expression { printf("= %d\n", $1); } ; expression: NUMBER ‘+’ NUMBER { $$ = $1 + $3; } | NUMBER ‘-’ NUMBER { $$ = $1 - $3; } ; $$ $1 $2 $3 when is this executed?

31. Coordinating flex and bison Example to return int value: [0-9]+{ yylval = atoi(yytext); return NUMBER;} returns recognized token sets value for use in actions This one just returns the numeric value of the string stored in yytext atoi is C function to convert string to integer In prior flex examples we just returned tokens, not values Also need to skip whitespace, return symbols [ \t];/* ignore white space */ \nreturn 0;/* logical EOF */. return yytext[0];

32. Bison Rule Details Unlike flex, bison doesn’t care about line boundaries, so add white space for readability Symbol on lhs of first rule is start symbol, can override with %start declaration in definition section Symbols in bison have values, must be “declared” as some type  YYSTYPE determines type  Default for all values is int  We’ll be using different types for YYSTYPE in the SimpleCalc exercises

33. Bison Parser – Definition Section Definition Section  Tokens used in grammar should be defined. Example rule: expression: NUMBER ‘+’ NUMBER { $$ = $1 + $3; } The token NUMBER should be defined. Later we’ll see cases where expression should also be defined, and how to define tokens with other data types. %token must be lowercase, e.g.,: %token NUMBER  From the tokens that are defined, Bison will create an appropriate header file  Single quoted characters can be used as tokens without declaring them, e.g., ‘+’, ‘=‘ etc.

34. Lex - Definition Section Must include the header created by bison Must declare yylval as extern %{ #include "simpleCalc.tab.h extern int yylval; #include %}

35. Bison Parser – C Section At a minimum, provide yyerror and main routines yyerror(char *errmsg) { fprintf(stderr, "%s\n", errmsg); } main() { yyparse(); }

36. Bison Intro Exercise Download SimpleCalc.y, SimpleCalc.l and mbison.bat Create calculator executable  mbison simpleCalc FYI, mbison includes these steps:  bison -d simpleCalc.y  flex -L -osimpleCalc.c simpleCalc.l  gcc -c simpleCalc.c  gcc -c simpleCalc.tab.c  gcc -Lc:\progra~1\gnuwin32\lib simpleCalc.o simpleCalc.tab.o -osimpleCalc.exe -lfl –ly Test with valid sentences (e.g., 3+6-4) and invalid sentences.

37. Understanding simpleCalc %{ #include "simpleCalc.tab.h" extern int yylval; %} % [0-9]+{ yylval = atoi(yytext); return NUMBER; } [ \t]; /* ignore white space */ \nreturn 0;/* logical EOF */.return yytext[0]; % /*---------------------------------------*/ /* 5. Other C code that we need. */ yyerror(char *errmsg) { fprintf(stderr, "%s\n", errmsg); } main() { yyparse(); } #ifndef YYTOKENTYPE # define YYTOKENTYPE /* Put the tokens into the symbol table, so that GDB and other debuggers know about them. */ enum yytokentype { NAME = 258, NUMBER = 259 }; #endif /* Tokens. */ #define NAME 258 #define NUMBER 259 simpleCalc.tab.h simpleCalc.l Explanation: When the lexer recognizes a number [0-9]+ it returns the token NUMBER and sets yylval to the corresponding integer value. When the lexer sees a carriage return it returns 0. If it sees a space or tab it ignores it. When it sees any other character it returns that character (the first character in the yytext buffer). If the yyparse recognizes it – good! Otherwise the parser can generate an error.

38. Understanding simpleCalc, continued %token NAME NUMBER % statement:NAME '=' expression |expression{ printf("= %d\n", $1); } ; expression:expression '+' NUMBER { $$ = $1 + $3; } |expression '-' NUMBER{ $$ = $1 - $3; } |NUMBER{ $$ = $1; } ; Explanation Execute simpleCalc and enter expression 1+2 main program calls yyparse. This calls lex to recognize 1 as a NUMBER (puts 1 in yylval), sets $$ = $1 Calls lex which returns +, matches ‘+’ in first expression rhs Calls lex to recognize 2 as a NUMBER (puts 2 in yylval) Recognize expression + NUMBER and “reduce” this rule, does action {$$ = $1 + $3}. Recognizes expression as a statement, so it does the printf action.

39. SimpleCalc Do simpleCalc exercise #1

40. Adding other variable types* YYSTYPE determines the data type of the values returned by the lexer. If lexer returns different types depending on what is read, include a union: %union { // C feature, allows one memory area to char cval; // be interpreted in different ways. char *sval; // For bison, will be used with yylval int ival; } The union will be placed at the top of your.y file (in the definitions section) Tokens and non-terminals should be defined using the union * relates to SimpleCalc exercise 2

41. Adding other variable types - Example Definitions in simpleCalc.y: %union { float fval; int ival; } %token NUMBER %token FNUMBER %type expression Use union in rules in simpleCalc.l: {DIGIT}+ { yylval.ival = atoi(yytext); return NUMBER;}

42. Processing lexemes in flex* Sometimes you want to modify a lexeme before it is passed to bison. This can be done by putting a function call in the flex rules Example: to convert input to lower case  put a prototype for your function in the definition section (above first %)  write the function definition in the C-code section (bottom of file)  call your function when the token is recognized. Use strdup to pass the value to bison. * relates to SimpleCalc exercise 3

43. Example continued %{ #include “example.tab.h“ void make_lower(char *text_in); %} % [a-zA-Z]+ {make_lower(yytext); yylval.sval = strdup(yytext); return KEYWORD; } % void make_lower(char *text_in) { int i; for (i=0; i<strlen(yytext); ++i) yytext[i]=tolower(yytext[i]); } need prototype here function code in C section function call to process text make duplicate using strdup return token type

44. Adding actions to rules * For more complex processing, functions can be added to bison. Remember to add a prototype at the top, and the function at the bottom * relates to SimpleCalc exercise 4

45. Processing more than one line * To process more than one line, ensure the \n is simply ignored Use a recursive rule to allow multiple inputs * relates to SimpleCalc exercise 4

46. Example todo.l %{ #include "todo.tab.h" %} % buy | write | read | ride | eat | go { yylval.sval = strdup(yytext); return VERB; } groceries | clothes | dinner | book | bicycle | home { yylval.sval = strdup(yytext); return NOUN; } [ \t\n\r]+ ;/* ignore white space */ [a-zA-Z]+ { yylval.sval = strdup(yytext); return NAME; }. { return yytext[0]; } > return 0; % int yyerror(char *errmsg) { fprintf(stderr, "error is: %s\n", errmsg); } main() { yyparse(); } All one file – two columns used to fit one slide.

47. Example todo.y // Recognizes "to-do" list in form name { verb noun verb noun } %union { char *sval; } %{ void end(char*); %} %token VERB NOUN NAME %type todoList person task entry % entry: person '{' todoList '}' { end($1); } ; person: NAME { printf("Greetings, %s\n", $1); $$ = $1;} ; todoList: todoList task | task ; task: VERB NOUN { printf("You need to %s %s\n", $1, $2); } ; % void end(char* name) { printf("That is all for now, %s!\n", name); exit(0); }; All one file – two columns used to fit one slide.

48. SimpleCalc Do remaining simpleCalc exercises

49. Using files with Bison The standard file for Bison is yyin. The following code can be used to take an optional command-line argument: int main(argc, argv) int argc; char **argv; { FILE *file; if (argc == 2) { file = fopen(argv[1], "r"); if (!file) { fprintf(stderr, “Couldn't open %s\n", argv[1]); exit(1); } yyin = file; }

50. More details (if you’re curious) Running flex creates simpleCalc.c. This creates the following case statement (I added the printf statements: case 1: YY_RULE_SETUP printf("returning number value %d\n", atoi(yytext)); { yylval = atoi(yytext); return NUMBER; } YY_BREAK case 2: YY_RULE_SETUP printf("ignoring white space\n"); ;/* ignore white space */ YY_BREAK case 3: YY_RULE_SETUP printf("recognized eof\n"); return 0;/* logical EOF */ YY_BREAK case 4: YY_RULE_SETUP printf("returning other character %c\n", yytext[0]); return yytext[0]; YY_BREAK

51. Continuing more detail Running bison creates simpleCalc.tab.c switch (yyn) { case 3: #line 4 "simpleCalc.y" { printf("= %d\n", (yyvsp[0])); ;} break; case 4: #line 7 "simpleCalc.y" { (yyval) = (yyvsp[-2]) + (yyvsp[0]); ;} break; case 5: #line 8 "simpleCalc.y" { (yyval) = (yyvsp[-2]) - (yyvsp[0]); ;} break; case 6: #line 9 "simpleCalc.y" { (yyval) = (yyvsp[0]); ;} break; NOTE: I added extra printf statements to each case, which is what you can see in the trace. Notice use of stack pointer sp for $values

52. Continuing more detail In exercise 2 you define a union. This gets translated to code within SimpleCalc.tab.h: #if ! defined (YYSTYPE) && ! defined (YYSTYPE_IS_DECLARED) #line 1 "simpleCalcEx2.y" typedef union YYSTYPE { float fval; int ival; } YYSTYPE; extern YYSTYPE yylval; This is what makes your yylval return part of the union

53. Continuing more detail Symbols you define in bison’s CFG are added to a symbol table: static const char *const yytname[] = { "$end", "error", "$undefined", "NUMBER", "FNUMBER", "NAME", "'='", "'+'", "'*'", "'('", "')'", "$accept", "statement", "expression", "term", "factor", 0 };

54. Continuing more detail New rules make use of union: switch (yyn) { case 3: #line 15 "simpleCalcEx2.y" { printf("= %f\n", (yyvsp[0].fval)); ;} break; case 4: #line 18 "simpleCalcEx2.y" { (yyval.fval) = (yyvsp[-2].fval) + (yyvsp[0].fval); ;} break; case 5: #line 19 "simpleCalcEx2.y" { (yyval.fval) = (yyvsp[0].fval); ;} break; expression is defined as, so is NUMBER

55. Summary of steps (from online manual) The actual language-design process using Bison, from grammar specification to a working compiler or interpreter, has these parts: 1. Formally specify the grammar in a form recognized by Bison (i.e., machine-readable BNF). For each grammatical rule in the language, describe the action that is to be taken when an instance of that rule is recognized. The action is described by a sequence of C statements. 2. Write a lexical analyzer to process input and pass tokens to the parser. 3. Write a controlling function (main) that calls the Bison-produced parser. 4. Write error-reporting routines.