1 Using Lex

Flex – Lexical Analyzer Generator A language for specifying lexical analyzers Flex compilerlex.yy.clang.l C compiler -lfl a.outlex.yy.c a.outtokenssource code

3 The Structure of a Lex Program (Definition section) % (Rules section) % (User subroutines section)

Flex Programs %{ auxiliary declarations %} regular definitions % translation rules % auxiliary procedures

5 %{ /* * this sample demonstrates (very) simple recognition: * a verb/not a verb. */ %} % [\t ]+ /* ignore white space */ ; is | am | are | were | was | be | being | been | do | does | did | will | would | should | can | could | has | have | had | go { printf("%s: is a verb\n", yytext); } [a-zA-Z]+ { printf("%s: is not a verb\n", yytext); }.|\n { ECHO; /* normal default anyway */ } % main() { yylex(); } Example 1-1: Word recognizer ch1-02.l

6 The definition section Lex copies the material between “%{“ and “%}” directly to the generated C file, so you may write any valid C codes here

7 Rules section Each rule is made up of two parts –A pattern –An action E.g. [\t ]+ /* ignore white space */ ;

8 Rules section (Cont’d) E.g. is | am | are | were | was | be | being | been | do | does | did | will | would | should | can | could | has | have | had | go { printf("%s: is a verb\n", yytext); }

9 Rules section (Cont’d) E.g. [a-zA-Z]+ { printf("%s: is not a verb\n", yytext); }.|\n { ECHO; /* normal default anyway */ } Lex had a set of simple disambiguating rules: 1.Lex patterns only match a given input character or string once 2.Lex executes the action for the longest possible match for the current input

10 User subroutines section It can consists of any legal C code Lex copies it to the C file after the end of the Lex generated code % main() { yylex(); }

11 Regular Expressions Regular expressions used by Lex catenation. the class of all characters except newline. * 0 or more [] character class ^ not (left most) $ 以 newline 為跟隨字元 ab$ {} macroexpansion of a symbol ex: {Digit}+ \ escape character + once or more ? Zero time or once | or “…” … to be taken / 以甚麼為跟隨字元 ab/cd () used for more complex expressions - range

Functions and Variables yylex() a function implementing the lexical analyzer and returning the token matched yytext a global pointer variable pointing to the lexeme matched yyleng a global variable giving the length of the lexeme matched yylval an external global variable storing the attribute of the token

Functions and Variables yywrap() –If yywrap() returns false (zero), then it is assumed that the function has gone ahead and set up yyin to point to another input file, and scanning continues. If it returns true (non-zero), then the scanner terminates, returning 0 to its caller. –If you do not supply your own version of yywrap(), then you must either use % option noyywrap or…. 13

14 Examples of Regular Expressions [0-9] [0-9]+ [0-9]* -?[0-9]+ [0-9]*\.[0-9]+ ([0-9]+)|([0-9]*\.[0-9]+) -?(([0-9]+)|([0-9]*\.[0-9]+)) [eE][-+]?[0-9]+ -?(([0-9]+)|([0-9]*\.[0-9]+))([eE][-+]?[0-9]+)?)

15 Example 2-1 % [\n\t ] ; -?(([0-9]+)|([0-9]*\.[0-9]+)([eE][-+]?[0-9]+)?) { printf("number\n"); }. ECHO; % main() { yylex(); }

16 A Word Counting Program The definition section %{ unsigned charCount = 0, wordCount = 0, lineCount = 0; %} word [^ \t\n]+ eol \n

17 A Word Counting Program (Cont’d) The rules section {word} { wordCount++; charCount += yyleng; } {eol} { charCount++; lineCount++; }. charCount++;

18 A Word Counting Program (Cont’d) The user subroutines section main(argc,argv) int argc; char **argv; { if (argc > 1) { FILE *file; file = fopen(argv[1], "r"); if (!file) { fprintf(stderr,"could not open %s\n",argv[1]); exit(1); } yyin = file; } yylex(); printf("%d %d %d\n",charCount, wordCount, lineCount); return 0; }

stripquotes No tossing mechanism provided. int frompos, topos, numquotes=2; For (frompos=1;frompos<yyleng;frompos++) { –yytext[topos++]=yytext[frompos]; –if(yytext[frompos]==‘”’ && yytext[frompos+1]==‘”’) {frompos++;numqotes++;}} yyleng-=numquotes; yytext[yyleng]=‘\0’; 19

Micro scanner auxiliary declarations –%option noyywrap –%{ –#ifndef DEFINE –#define DEFINE 1 –#include – #include "token.h" –#endif – #define YYLMAX 33 – char token_buffer[YYLMAX]; – FILE *out_fd, *status_fd; – extern void check_id(char *); – extern void list_token_type(token); – extern void get_file_name(char *, char *, char *); –%} 20

Micro scanner regular definitions –letter [a-zA-Z] –digit [0-9] –KEYWORD "BEGIN"|"begin"|"END"|"end"|"READ"|"read"|"W RITE"|"write"|"SCANEOF" –literal {digit}+ –IDENTIFIER {letter}+[{letter}|{digit}|"_"]* –special_char "+"|"-"|"*"|":="|"("|")"|";"|"," –comment “--”.*[\n] 21

Micro scanner translation rules –% –{KEYWORD} {list_token_type(check_reserved(yytext));} –{literal} {list_token_type(INTLITERAL) –{IDENTIFIER} {list_token_type(ID);} –{special_char}{list_token_type(check_special_char(yytext) );} –[\n] –[ \t] –{comment} –. {lexical_error(yytext);} 22

Micro scanner auxiliary procedures –% –main(int argc,char *argv[]) – { – if ((yyin=fopen(argv[1],"r"))==NULL) – { – printf("ERROR:file open error!!\n"); – exit(0); – } – yylex(); – fclose(yyin); –} 23

Micro scanner check_reserved – 確認是否為保留字 check_special_char – 確認是否為 +-*(),;:= list_token_type – 顯示 token 型態 24

Example begin read (A,B); A := (B+3)*A; B := A*3-B; write (A,B,3*4); end SCANEOF 25

Example BeGIN READ LPAREN ID COMMA ID RPAREN SEMICOLON ID ASSIGNOP LPAREN ID PLUSOP INTLITERAL RPAREN ID COMMA INTLITERAL MULTOP INTLITERAL RPAREN SEMICOLON END SCANEOF 26 MULTOP ID SEMICOLON ID ASSIGNOP ID MULTOP INTLITERAL MINUSOP ID SEMICOLON WRITE LPAREN ID COMMA