1. A brief [f]lex tutorial Saumya Debray The University of Arizona Tucson, AZ 85721

2. A quick tutorial on Lex2 flex (and lex): Overview Scanner generators:  Helps write programs whose control flow is directed by instances of regular expressions in the input stream. Input: a set of regular expressions + actions Output: C code implementing a scanner: function: yylex() file: lex.yy.c flex (or lex)

3. A quick tutorial on Lex3 Using flex lex input spec (regexps + actions) file: lex.yy.c yylex() { … } driver code compiler flex user supplies main() {…} or parser() {…}

4. A quick tutorial on Lex4 flex: input format An input file has the following structure: definitions % rules % user code optional required Shortest possible legal flex input: %

5. A quick tutorial on Lex5 Definitions A series of:  name definitions, each of the form name definition e.g.: DIGIT [0-9] CommentStart "/*" ID [a-zA-Z][a-zA-Z0-9]*  start conditions  stuff to be copied verbatim into the flex output (e.g., declarations, #includes): – enclosed in %{ … }%, or – indented

6. A quick tutorial on Lex6 Rules The rules portion of the input contains a sequence of rules. Each rule has the form pattern action where :  pattern describes a pattern to be matched on the input  pattern must be un-indented  action must begin on the same line.

7. A quick tutorial on Lex7 Patterns Essentially, extended regular expressions.  Syntax: similar to grep (see man page)  > to match “end of file”  Character classes: – [:alpha:], [:digit:], [:alnum:], [:space:], etc. (see man page)  {name} where name was defined earlier. “start conditions” can be used to specify that a pattern match only in specific situations.

8. A quick tutorial on Lex8 Example %{ #include %} dgt [0-9] % {dgt}+ return atoi(yytext); % void main() { int val, total = 0, n = 0; while ( (val = yylex()) > 0 ) { total += val; n++; } if (n > 0) printf(“ave = %d\n”, total/n); } A flex program to read a file of (positive) integers and compute the average:

9. A quick tutorial on Lex9 Example %{ #include %} dgt [0-9] % {dgt}+ return atoi(yytext); % void main() { int val, total = 0, n = 0; while ( (val = yylex()) > 0 ) { total += val; n++; } if (n > 0) printf(“ave = %d\n”, total/n); } A flex program to read a file of (positive) integers and compute the average: Definition for a digit (could have used builtin definition [:digit:] instead) Rule to match a number and return its value to the calling routine Driver code (could instead have been in a separate file) definitions rules user code

10. A quick tutorial on Lex10 Example %{ #include %} dgt [0-9] % {dgt}+ return atoi(yytext); % void main() { int val, total = 0, n = 0; while ( (val = yylex()) > 0 ) { total += val; n++; } if (n > 0) printf(“ave = %d\n”, total/n); } A flex program to read a file of (positive) integers and compute the average: definitions rules user code defining and using a name

11. A quick tutorial on Lex11 Example %{ #include %} dgt [0-9] % {dgt}+ return atoi(yytext); % void main() { int val, total = 0, n = 0; while ( (val = yylex()) > 0 ) { total += val; n++; } if (n > 0) printf(“ave = %d\n”, total/n); } A flex program to read a file of (positive) integers and compute the average: definitions rules user code defining and using a name char * yytext; a buffer that holds the input characters that actually match the pattern

12. A quick tutorial on Lex12 Example %{ #include %} dgt [0-9] % {dgt}+ return atoi(yytext); % void main() { int val, total = 0, n = 0; while ( (val = yylex()) > 0 ) { total += val; n++; } if (n > 0) printf(“ave = %d\n”, total/n); } A flex program to read a file of (positive) integers and compute the average: definitions rules user code defining and using a name char * yytext; a buffer that holds the input characters that actually match the pattern Invoking the scanner: yylex() Each time yylex() is called, the scanner continues processing the input from where it last left off. Returns 0 on end-of-file.

13. Avoiding compiler warnings If compiled using “gcc –Wall” the previous flex file will generate compiler warnings: lex.yy.c: … : warning: `yyunput’ defined but not used lex.yy.c: … : warning: `input’ defined but not used These can be removed using ‘%option’ declarations in the first part of the flex input file: %option nounput %option noinput A quick tutorial on Lex13

14. A quick tutorial on Lex14 Matching the Input When more than one pattern can match the input, the scanner behaves as follows:  the longest match is chosen;  if multiple rules match, the rule listed first in the flex input file is chosen;  if no rule matches, the default is to copy the next character to stdout. The text that matched (the “token”) is copied to a buffer yytext.

15. A quick tutorial on Lex15 Matching the Input (cont’d) Pattern to match C-style comments: /* … */ "/*"(. |\n)*"*/" Input: #include /* definitions */ int main(int argc, char * argv[ ]) { if (argc <= 1) { printf(“Error!\n”); /* no arguments */ } printf(“%d args given\n”, argc); return 0; }

16. A quick tutorial on Lex16 Matching the Input (cont’d) Pattern to match C-style comments: /* … */ "/*"(. |\n)*"*/" Input: #include /* definitions */ int main(int argc, char * argv[ ]) { if (argc <= 1) { printf(“Error!\n”); /* no arguments */ } printf(“%d args given\n”, argc); return 0; } longest match:

17. A quick tutorial on Lex17 Matching the Input (cont’d) Pattern to match C-style comments: /* … */ "/*"(. |\n)*"*/" Input: #include /* definitions */ int main(int argc, char * argv[ ]) { if (argc <= 1) { printf(“Error!\n”); /* no arguments */ } printf(“%d args given\n”, argc); return 0; } longest match: Matched text shown in blue

18. A quick tutorial on Lex18 Start Conditions Used to activate rules conditionally.  Any rule prefixed with will be activated only when the scanner is in start condition S. Declaring a start condition S:  in the definition section: %x S – “%x” specifies “exclusive start conditions” – flex also supports “inclusive start conditions” (“%s”), see man pages. Putting the scanner into start condition S:  action: BEGIN(S)

19. A quick tutorial on Lex19 Start Conditions (cont’d) Example:  [^"]* { …match string body… } – [^"] matches any character other than " – The rule is activated only if the scanner is in the start condition STRING. INITIAL refers to the original state where no start conditions are active. matches all start conditions.

20. A quick tutorial on Lex20 Using Start Conditions Start conditions let us explicitly simulate finite state machines. This lets us get around the “longest match” problem for C- style comments. / * non-* * * / non -{ /,* } flex input: %x S1, S2, S3 % "/" BEGIN(S1); "*" BEGIN(S2); [^*] ; /* stay in S2 */ "*" BEGIN(S3); "*" ; /* stay in S3 */ [^*/] BEGIN(S2); "/" BEGIN(INITIAL); S1S2S3 FSA for C comments :

21. A quick tutorial on Lex21 Using Start Conditions Start conditions let us explicitly simulate finite state machines. This lets us get around the “longest match” problem for C- style comments. / * non-* * * / non -{ /,* } flex input: %x S1, S2, S3 % "/" BEGIN(S1); "*" BEGIN(S2); [^*] ; /* stay in S2 */ "*" BEGIN(S3); "*“ ; /* stay in S3 */ [^*/] BEGIN(S2); "/" BEGIN(INITIAL); S1S2S3 FSA for C comments :

22. A quick tutorial on Lex22 Using Start Conditions Start conditions let us explicitly simulate finite state machines. This lets us get around the “longest match” problem for C- style comments. / * non-* * * / non -{ /,* } flex input: %x S1, S2, S3 % "/" BEGIN(S1); "*" BEGIN(S2); [^*] ; /* stay in S2 */ "*" BEGIN(S3); "*“ ; /* stay in S3 */ [^*/] BEGIN(S2); "/" BEGIN(INITIAL); S1S2S3 FSA for C comments :

23. A quick tutorial on Lex23 Using Start Conditions Start conditions let us explicitly simulate finite state machines. This lets us get around the “longest match” problem for C- style comments. / * non-* * * / non -{ /,* } flex input: %x S1, S2, S3 % "/" BEGIN(S1); "*" BEGIN(S2); [^*] ; /* stay in S2 */ "*" BEGIN(S3); "*“ ; /* stay in S3 */ [^*/] BEGIN(S2); "/" BEGIN(INITIAL); S1S2S3 FSA for C comments :

24. A quick tutorial on Lex24 Using Start Conditions Start conditions let us explicitly simulate finite state machines. This lets us get around the “longest match” problem for C- style comments. / * non-* * * / non -{ /,* } flex input: %x S1, S2, S3 % "/" BEGIN(S1); "*" BEGIN(S2); [^*] ; /* stay in S2 */ "*" BEGIN(S3); "*“ ; /* stay in S3 */ [^*/] BEGIN(S2); "/" BEGIN(INITIAL); S1S2S3 FSA for C comments :

25. A quick tutorial on Lex25 Using Start Conditions Start conditions let us explicitly simulate finite state machines. This lets us get around the “longest match” problem for C- style comments. / * non-* * * / non -{ /,* } flex input: %x S1, S2, S3 % "/" BEGIN(S1); "*" BEGIN(S2); [^*] ; /* stay in S2 */ "*" BEGIN(S3); "*“ ; /* stay in S3 */ [^*/] BEGIN(S2); "/" BEGIN(INITIAL); S1S2S3 FSA for C comments :

26. A quick tutorial on Lex26 Using Start Conditions Start conditions let us explicitly simulate finite state machines. This lets us get around the “longest match” problem for C- style comments. / * non-* * * / non -{ /,* } flex input: %x S1, S2, S3 % "/" BEGIN(S1); "*" BEGIN(S2); [^*] ; /* stay in S2 */ "*" BEGIN(S3); "*“ ; /* stay in S3 */ [^*/] BEGIN(S2); "/" BEGIN(INITIAL); S1S2S3 FSA for C comments :

27. A quick tutorial on Lex27 Using Start Conditions Start conditions let us explicitly simulate finite state machines. This lets us get around the “longest match” problem for C- style comments. / * non-* * * / non -{ /,* } flex input: %x S1, S2, S3 % "/" BEGIN(S1); "*" BEGIN(S2); [^*] ; /* stay in S2 */ "*" BEGIN(S3); "*“ ; /* stay in S3 */ [^*/] BEGIN(S2); "/" BEGIN(INITIAL); S1S2S3 FSA for C comments :

28. A quick tutorial on Lex28 Putting it all together Scanner implemented as a function int yylex();  return value indicates type of token found (encoded as a +ve integer);  the actual string matched is available in yytext. Scanner and parser need to agree on token type encodings  let yacc generate the token type encodings – yacc places these in a file y.tab.h  use “#include y.tab.h” in the definitions section of the flex input file. When compiling, link in the flex library using “-lfl”