Sung-Dong Kim, School of Computer Engineering, Hansung University LEX Sung-Dong Kim, School of Computer Engineering, Hansung University

LEX 1975, Lesk Input: regular expression + action code (tiny.l) Output: C program (lex.yy.c or lexyy.c) Procedure yylex Table-driven implementation of a DFA Compilation: cc –o scanner lex.yy.c -lm Lex Scanner (C code) RE + action (2017-1) Compiler

LEX Convention (1) Metacharacters Quotes: actual characters Backslash For not metacharacters: “if”, if For metacharacters: “(” Backslash \(\* = “\*” \n, \t (aa|bb)(a|b)*c? = (“aa”|“bb”)(“a”|“b”)* “c”? (2017-1) Compiler

LEX Convention (2) [...] : any one of them Hyphen [abxz]: any one of the characters a, b, x, z (aa|bb)(ab)*c? Hyphen Ranges of characters [0-9] (2017-1) Compiler

LEX Convention (3) . ^ Represents a set of characters Any character except a newline ^ Complementary sets [^0-9abc]: any character that is not a digit and is not one of the letter a, b, c (2017-1) Compiler

LEX Convention (4) Square bracket Most of the meta-characters lose their special status [-+] == (“+”|“-”) [+-]: from “+”, all characters [.”?]: any of the three characters ., ”, ? [\^\\]: ^ or \ (2017-1) Compiler

LEX Convention (5) Curly bracket Names of regular expressions nat = [0-9]+ signedNat = (“+”|“-”)? nat nat [0-9]+ signedNat (“+”|“-”)? {nat} (2017-1) Compiler

Format of LEX Input (1) Input file = regular expression + C code Definitions Any C code that must be inserted to any function - %{…}% Names of regular expressions Rules Regular expressions + C code (action) Auxiliary routines (optional) C code + main program (if needed) (2017-1) Compiler

Format of LEX Input (2) Layout {definitions} %% {rules} {auxiliary routines} (2017-1) Compiler

Example 1: scanner that adds line numbers to text %{ /* a Lex program that adds line numbers to lines of text, printing the new text to the standard output */ #include <stdio.h> int lineno = 1; %} line .*\n %% {line} {printf(“%5d %s”,lineno++,yytext); } main() { yylex(); return 0; } (2017-1) Compiler

Example 2: prints the count of # of replacements %{ /* a Lex program that changes all numbers from decimal to hexadecimal notation, printing a summary statistic stderr */ #include <stdlib.h> #include <stdio.h> int count = 0; %} digit [0-9] number {digit}+ %% {number} { int n = atoi(yytext); printf(“%x”, n); if (n > 9) count++; } (2017-1) Compiler

fprintf(stderr, “number of replacements = %d”, count); return 0; } main() { yylex(); fprintf(stderr, “number of replacements = %d”, count); return 0; } (2017-1) Compiler

Example 3: prints all input lines that begin or end with the ‘a’ %{ /* Selects only lines that end or begin with the letter ‘a’. Deletes everything else. */ #include <stdio.h> %} ends_with_a .*a\n begins_with_a a.*\n %% {ends_with_a} ECHO; {begins_with_a} ECHO; .*\n ; main() { yylex(); return 0; } (2017-1) Compiler

Summary (1) Ambiguity resolution The principles of longest substring Substring with equal length: first-match first-serve No match: copy the next character to the output and continue (2017-1) Compiler

Summary (2) Insertion of C Code %{ … %}: exact copy Auxiliary procedure section: exact copy at the end Any code following a RE (action): at the appropriate place in yylex (2017-1) Compiler

Lex Internal Names lex.yy.c: Lex output file name or lexyy.c yylex: Lex scanning routine yytext: String matched on current action yyin: Lex input file (default: stdin) yyout: Lex output file (default: stdout) input: Lex buffered input routine ECHO: Lex default action (print yytext to yyout) (2017-1) Compiler

LEX for TINY %{ #include “globals.h” #include “util.h” #include “scan.h” /* lexeme of identifier or reserved word */ char tokenString[MAXTOKENLEN+1]; %} digit [0-9] number {digit}+ letter [a-zA-Z] identifier {letter}+ newline \n whitespace [ \t] %% (2017-1) Compiler

“repeat” { return REPEAT; } “until” { return UNTIL; } “if” { return IF; } “then” { return THEN; } “else” { return ELSE; } “end” { return END; } “repeat” { return REPEAT; } “until” { return UNTIL; } “read” { return READ; } “write” { return WRITE; } “:=” { return ASSIGN; } “=” { return EQ; } “<” { return LT; } “+” { return PLUS; } “-” { return MINUS; } “*” { return TIMES; } “/” { return OVER; } “(” { return LPAREN; } “)” { return RPAREN; } “;” { return SEMI; } (2017-1) Compiler

{number} { return NUM; } {identifier} { return ID; } {newline} { lineno++; } {whitespace} { /* skip whitespace */ } “{” { char c; do { c = input(); if (c == ‘\n’) lineno++; } while (c != ‘}’); } . { return ERROR; } %% (2017-1) Compiler

TokenType getToken(void) { static int firstTime = TRUE; TokenType currentToken; if (firstTime) { firstTime = FALSE; lineno++; yyin = source; yyout = listing; } currentToken = yylex(); strncpy(tokenString, yytext, MAXTOKENLEN); if (TraceScan) { fprintf(listing, “\t%d: “, lineno); printToken(currentToken, tokenString); return currentToken; (2017-1) Compiler

참고 교재 Lex & Yacc 2nd Edition, John R. Levine, Tony Mason, Doug Brown, O'Reilly,1992 예제 http://myweb.stedwards.edu/laurab/cosc4342/lex- examples.html (2017-1) Compiler

Exercises 입력파일에 있는 단어의 개수를 구하는 프로그램을 위한 lex input 파일을 작성하시오. (2017-1) Compiler