1. Lexical Analysis: Regular Expressions CS 471 September 3, 2007

2. CS 471 – Fall 2007 1 A program that translates a program in one language to another language the essential interface between applications & architectures Typically lowers the level of abstraction analyzes and reasons about the program & architecture We expect the program to be optimized i.e., better than the original ideally exploiting architectural strengths and hiding weaknesses Last Time … Compiler High-Level Programming Languages Machine Code Error Messages

3. CS 471 – Fall 2007 2 Phases of a Compiler Lexical analyzer Syntax analyzer Semantic analyzer Intermediate code generator Code optimizer Code generator Source program Target program Lexical Analyzer Group sequence of characters into lexemes – smallest meaningful entity in a language (keywords, identifiers, constants) Characters read from a file are buffered – helps decrease latency due to i/o. Lexical analyzer manages the buffer Makes use of the theory of regular languages and finite state machines Lex and Flex are tools that construct lexical analyzers from regular expression specifications

4. CS 471 – Fall 2007 3 Phases of a Compiler Lexical analyzer Syntax analyzer Semantic analyzer Intermediate code generator Code optimizer Code generator Source program Target program Parser Convert a linear structure – sequence of tokens – to a hierarchical tree-like structure – an AST The parser imposes the syntax rules of the language Work should be linear in the size of the input (else unusable)  type consistency cannot be checked in this phase Deterministic context free languages and pushdown automata for the basis Bison and yacc allow a user to construct parsers from CFG specifications

5. CS 471 – Fall 2007 4 Phases of a Compiler Lexical analyzer Syntax analyzer Semantic analyzer Intermediate code generator Code optimizer Code generator Source program Target program Semantic Analysis Calculates the program’s “meaning” Rules of the language are checked (variable declaration, type checking) Type checking also needed for code generation (code gen for a + b depends on the type of a and b)

6. CS 471 – Fall 2007 5 Phases of a Compiler Lexical analyzer Syntax analyzer Semantic analyzer Intermediate code generator Code optimizer Code generator Source program Target program Intermediate Code Generation Makes it easy to port compiler to other architectures (e.g. Pentium to MIPS) Can also be the basis for interpreters (such as in Java) Enables optimizations that are not machine specific

7. CS 471 – Fall 2007 6 Phases of a Compiler Lexical analyzer Syntax analyzer Semantic analyzer Intermediate code generator Code optimizer Code generator Source program Target program Intermediate Code Optimization Constant propagation, dead code elimination, common sub-expression elimination, strength reduction, etc. Based on dataflow analysis – properties that are independent of execution paths

8. CS 471 – Fall 2007 7 Phases of a Compiler Lexical analyzer Syntax analyzer Semantic analyzer Intermediate code generator Code optimizer Code generator Source program Target program Native Code Generation Intermediate code is translated into native code Register allocation, instruction selection Native Code Optimization Peephole optimizations – small window is optimized at a time

9. CS 471 – Fall 2007 8 Administration 1. PA1 on website (from book): C Primer 2. Questionnaire Results…

10. CS 471 – Fall 2007 9 A Quick Linux Primer Tar – archiving program Gzip/Bzip2 – compression Cvs/Svn – version control Ssh – secure shell Scp/Sftp – secure copy Make/Scons – build/run utility Other useful tools: –Man! –Which –Locate –Diff (or sdiff)

11. CS 471 – Fall 2007 10 Makefiles Target: source file(s) command proj1 main.oio.odata.o main.cio.hio.cdata.hdata.c

12. CS 471 – Fall 2007 11 First Step: Lexical Analysis (Tokenizing) Breaking the program down into words or “tokens” Input: stream of characters Output: stream of names, keywords, punctuation marks Side effect: Discards white space, comments Source code: if (b==0) a = “Hi”; Token Stream: Lexical Analysis Parsing

13. CS 471 – Fall 2007 12 Lexical Tokens Identifiers: x y11 elsex _i00 Keywords: if else while break Integers: 2 1000 -500 5L Floating point: 2.0 0.00020.02 1.1e5 0.e-10 Symbols: + * { } ++ = Strings: “x” “He said, \“Are you?\”” Comments: /** don’t change this **/

14. CS 471 – Fall 2007 13 Lexical Tokens float match0(char *s) /* find a zero */ { if (!strncmp(s, “0.0”, 3)) return 0.; } FLOAT ID(match0) _______ CHAR STAR ID(s) RPAREN LBRACE IF LPAREN BANG _______ LPAREN ID(s) COMMA STRING(0.0) ______ NUM(3) RPAREN RPAREN RETURN REAL(0.0) ______ RBRACE EOF

15. CS 471 – Fall 2007 14 Ad-hoc Lexer Hand-write code to generate tokens How to read identifier tokens? Token readIdentifier( ) { String id = “”; while (true) { char c = input.read(); if (!identifierChar(c)) return new Token(ID, id, lineNumber); id = id + String(c); }

16. CS 471 – Fall 2007 15 Problems Don’t know what kind of token we are going to read from seeing first character –if token begins with “i’’ is it an identifier? –if token begins with “2” is it an integer? constant? –interleaved tokenizer code is hard to write correctly, harder to maintain More principled approach: lexer generator that generates efficient tokenizer automatically (e.g., lex, flex)

17. CS 471 – Fall 2007 16 Issues How to describe tokens unambiguously 2.e0 20.e-01 2.0000 “” “x” “\\” “\”\’” How to break text down into tokens if (x == 0) a = x<<1; if (x == 0) a = x<1; How to tokenize efficiently –tokens may have similar prefixes –want to look at each character ~1 time

18. CS 471 – Fall 2007 17 How To Describe Tokens Programming language tokens can be described using regular expressions A regular expression R describes some set of strings L(R) L(R) is the “language” defined by R – L(abc) = { abc } – L(hello|goodbye) = {hello, goodbye} – L([1-9][0-9]*) = _______________ Idea: define each kind of token using RE

19. CS 471 – Fall 2007 18 Regular expressions Language – set of strings String – finite sequence of symbols Symbols – taken from a finite alphabet Specify languages using regular expressions Symbolaone instance of a Epsilon  empty string AlternationR | Sstring from either L(R) or L(S) ConcatenationR ∙ Sstring from L(R) followed by L(S) RepetitionR*

20. CS 471 – Fall 2007 19 Convenient Shorthand [abcd] one of the listed characters (a | b | c | d) [b-g] [bcdefg] [b-gM-Qkr] [^ab]anything but one of the listed chars [^a-f] M?Zero or one M M+One or more M M* “a.+*”literally a.+*.Any single character (except \n)

21. CS 471 – Fall 2007 20 Examples Regular Expression Strings in L(R) digit = [0-9] “0” “1” “2” “3” … posint = digit+ “8” “412” … int = -? posint “-42” “1024” … real = int (ε | (. posint)) “-1.56” “12” “1.0” [a-zA-Z_][a-zA-Z0-9_]* C identifiers Lexer generators support abbreviations – cannot be recursive

22. CS 471 – Fall 2007 21 More Examples Whitespace: Integers: Hex numbers: Valid UVa User Ids: Loop keywords in C:

23. CS 471 – Fall 2007 22 Breaking up Text elsex = 0; REs alone not enough: need rules for choosing Most languages: longest matching token wins –even if a shorter token is only way Exception: early FORTRAN (totally whitespace- insensitive) Ties in length resolved by prioritizing tokens RE’s + priorities + longest-matching token rule = lexer definition else x = 0;

24. CS 471 – Fall 2007 23 Lexer Generator Specification Input to lexer generator: –list of regular expressions in priority order –associated action for each RE (generates appropriate kind of token, other bookkeeping) Output: –program that reads an input stream and breaks it up into tokens according to the REs. (Or reports lexical error -- “Unexpected character” )

25. CS 471 – Fall 2007 24 Lex: A Lexical Analyzer Generator Lex produces a C program from a lexical specification http://www.epaperpress.com/lexandyacc/index.html % DIGITS [0-9]+ ALPHA [A-Za-z] CHARACTER {ALPHA}|_ IDENTIFIER {ALPHA}({CHARACTER}|{DIGITS})* % if{return IF; } {IDENTIFIER}{return ID; } {DIGITS}{return NUM; } ([0-9]+”.”[0-9]*)|([0-9]*”.”[0-9]+){return ____; }.{error(); }

26. CS 471 – Fall 2007 25 Summary Lexical analyzer converts a text stream to tokens Ad-hoc lexers hard to get right, maintain For most languages, legal tokens conveniently, precisely defined using regular expressions Lexer generators generate lexer code automatically from token RE’s, precedence Next time: –Theory: finite automata –Practice: how lexer generators work Read Chapter 2 Start PA1 if you haven’t already!