Compiler Construction Lexical Analysis Rina Zviel-Girshin and Ohad Shacham School of Computer Science Tel-Aviv University

2 Generic compiler structure Executable code exe Source text txt Semantic Representation Backend (synthesis) Compiler Frontend (analysis)

3 Lexical Analysis converts characters to tokens converts characters to tokens class Quicksort { int[] a; int partition(int low, int high) { int pivot = a[low];... } 1: CLASS 1: CLASS_ID(Quicksort) 1: LCBR 2: INT 2: LB 2: RB 2: ID(a)... 2: SEMI

4 Lexical Analysis Tokens Tokens ID – _size, _num ID – _size, _num Num – 7, 5, 9, Num – 7, 5, 9, COMMA –, COMMA –, SEMI – ; SEMI – ; … Non tokens Non tokens Comment – // Comment – // Whitespace Whitespace Macro Macro …

5 Problem Input Input Program text Program text Tokens specification Tokens specification Output Output Sequence of tokens Sequence of tokens class Quicksort { int[] a; int partition(int low, int high) { int pivot = a[low];... } 1: CLASS 1: CLASS_ID(Quicksort) 1: LCBR 2: INT 2: LB 2: RB 2: ID(a)... 2: SEMI

6 Solution Write a lexical analyzer Write a lexical analyzer Token nextToken() { char c ; loop: c = getchar(); switch (c){ case ` `:goto loop ; case `;`: return SemiColumn; case `+`: c = getchar() ; switch (c) { case `+': return PlusPlus ; case '=’ return PlusEqual; default: ungetc(c); return Plus; } case `<`: case `w`: … }

7 Solution’s Problem A lot of work A lot of work Corner cases Corner cases Error prune Error prune Hard to debug Hard to debug Exhausting Exhausting Boring Boring Hard to reuse Hard to reuse Switch parser’s code between people Switch parser’s code between people …. ….

8 Scanner generator: history LEX LEX a lexical analyzer generator, written by Lesk and Schmidt at Bell Labs in 1975 for the UNIX operating system; a lexical analyzer generator, written by Lesk and Schmidt at Bell Labs in 1975 for the UNIX operating system; It now exists for many operating systems; It now exists for many operating systems; LEX produces a scanner which is a C program; LEX produces a scanner which is a C program; LEX accepts regular expressions and allows actions (i.e., code to executed) to be associated with each regular expression. LEX accepts regular expressions and allows actions (i.e., code to executed) to be associated with each regular expression. JLex JLex Lex that generates a scanner written in Java; Lex that generates a scanner written in Java; Itself is also implemented in Java. Itself is also implemented in Java. There are many similar tools, for every programming language There are many similar tools, for every programming language

9 Overall picture Tokens Scanner generator NFA RE Java scanner program String stream DFA Minimize DFA Simulate DFA

10 JFlex Off the shelf lexical analysis generator Off the shelf lexical analysis generator Input Input scanner specification file scanner specification file Output Output Lexical analyzer written in Java Lexical analyzer written in Java JFlexjavac IC.lex Lexical analyzer IC text tokens Lexer.java

11 JFlex Simple Simple Good for reuse Good for reuse Easy to understand Easy to understand Many developers and users debugged the generators Many developers and users debugged the generators "+" { return new symbol (sym.PLUS); } "boolean" { return new symbol (sym.BOOLEAN); } “int" { return new symbol (sym.INT); } "null" {return new symbol (sym.NULL);} "while" {return new symbol (sym.WHILE);} "=" {return new symbol (sym.ASSIGN);}…

12 JFlex Spec File User code Copied directly to Java file Copied directly to Java file % JFlex directives Define macros, state names Define macros, state names % Lexical analysis rules How to break input to tokens How to break input to tokens Action when token matched Action when token matched Possible source of javac errors down the road DIGIT= [0-9] LETTER= [a-zA-Z] YYINITIAL {LETTER} ({LETTER}|{DIGIT})*

13 User code package IC.Parser; import IC.Parser.Token; … any scanner-helper Java code …

14 JFlex Directives Control JFlex internals Control JFlex internals  %line switches line counting on  %char switches character counting on  %class class-name changes default name  %cup CUP compatibility mode  %type token-class-name  %public Makes generated class public (package by default)  %function read-token-method  %scanerror exception-type-name

15 JFlex Directives State definitions State definitions %state state-name %state state-name %state STRING %state STRING Macro definitions Macro definitions macro-name = regex macro-name = regex

16 Regular Expression r $r $r $r $ match reg. exp. r at end of a line. any character except the newline "..."string {name} macro expansion * zero or more repetitions + one or more repetitions ? zero or one repetitions (...) grouping within regular expressions a|ba|ba|ba|b match a or b [...] class of characters - any one character enclosed in brackets a–ba–ba–ba–b range of characters [^…] negated class – any one not enclosed in brackets

17 Example macros ALPHA=[A-Za-z_]DIGIT=[0-9]ALPHA_NUMERIC={ALPHA}|{DIGIT}IDENT={ALPHA}({ALPHA_NUMERIC})*NUMBER=({DIGIT})+NUMBER=[0-9]+

18 Rules [states] regexp {action as Java code} Priorities Priorities Longest match Longest match Order in the lex file Order in the lex file Rules should match all inputs!!! Rules should match all inputs!!! Breaks Input to Tokens Invokes when regexp matches break breakdown int identifier or integer ? The regexp should be evaluated ?

19 Rules Examples DIGIT}+ { return new Symbol(sym.NUMBER, yytext(), yyline); } "-" { return new Symbol(sym.MINUS, yytext(), yyline); } [a-zA-Z] ([a-zA-Z0-9]) * { return new Symbol(sym.ID, yytext(), yyline); }

20 Rules – Action Action Action Java code Java code Can use special methods and vars Can use special methods and vars yyline yyline yytext() yytext() Returns a token for a token Returns a token for a token Eats chars for non tokens Eats chars for non tokens

21 Rules – State State State Which regexp should be evaluated? Which regexp should be evaluated? yybegin(stateX) yybegin(stateX) jumps to stateX jumps to stateX YYINITIAL YYINITIAL JFlex’s initial state JFlex’s initial state

22 Rules – State "//" { yybegin(COMMENTS); } [^\n] { } [\n] { yybegin(YYINITIAL); } YYINITIALCOMMENTS ‘ // ’ \n ^\n

23 Lines Count Example import java_cup.runtime.Symbol; % %cup %{ private int lineCounter = 0; %} %eofval{ System.out.println("line number=" + lineCounter); return new Symbol(sym.EOF); %eofval} NEWLINE=\n % {NEWLINE} { lineCounter++; } [^{NEWLINE}] { }

24 Lines Count Example JFlex javac lineCount.lex Lexical analyzer text tokens Yylex.java Main.java JFlex and JavaCup must be on CLASSPATH sym.java java JFlex.Main lineCount.lex javac *.java

25 Test Bed import java.io.*; public class Main { public static void main(String[] args) { Symbol currToken; try { FileReader txtFile = new FileReader(args[0]); Yylex scanner = new Yylex(txtFile); do { currToken = scanner.next_token(); // do something with currToken } while (currToken.sym != sym.EOF); } catch (Exception e) { throw new RuntimeException("IO Error (brutal exit)” + e.toString()); }

26 Common Pitfalls Classpath Classpath Path to executable Path to executable Define environment variables Define environment variables JAVA_HOME JAVA_HOME CLASSPATH CLASSPATH

27 JFlex directives to use %cup (integrate with cup) %line (count lines) %type Token (pass type Token) %class Lexer (gen. scanner class)

28 Structure JFlex javac IC.lex Lexical analyzer test.ic tokens Lexer.java sym.java Token.java LexicalError.java Compiler.java