SCRIBE SUBMISSION GROUP 8 Date: 7/8/2013 By – IKHAR SUSHRUT MEGHSHYAM 11CS10017 Lexical Analyser Constructing Tokens State-Transition Diagram S-T Diagrams of Operators, Variables, Digits Index Topic Covered : Detecting lexemes from a given set of patterns/stream of chars

Lexical Analyser BLACK BOX Construct Token Pattern Regular Expression Regular Language Stream of Characters Lexeme Pattern To Parser Describe Patterns using regular expression For a specific pattern we can define a regular expression corresponding to regular language

Construct Tokens for specified set of patterns Tokens for some patterns: 1. Keywords : if, else, while, then, do 2. Operators :>,>=,,. = 3.Variables : start with letter followed by letters/digits/underscores 4.Numbers :Whole numbers & Floating point numbers 5.Whitespaces :tab/newline/whitespace No tokens will be created

STATE - TRANSITION DIAGRAM S-T diagram is a directed graph consisting states as set of nodes and directed edges corresponding to transitions from one state to another. start a a b a I N P U T Start state Final states ∑ = {a, b} For an input string X, If final state is reached then X is accepted by the machine M defined over the alphabet ∑ L(M) denotes the set of all accepted strings by machine M

S-T Diagrams for some patterns 1. S-T Diagram for ‘while’ start \0 h w i e l backtracking Token : 2. S-T Diagram for ‘digits’ start. digit other symbol * * * digit : [0-9] digits : {digit}*

3. S-T Diagram for ‘operators’ start =< \0 Other symbol => \0 Other symbol * * =

3. S-T Diagram for ‘variables’ start else letters letters/digits/underscore * backtracking To distinguish between keywords and variables we try to maximize the length of the lexeme. We apply “parallel simulation” for all the above S-T diagrams and determine the token for which lexeme is of maximum length letter : [A-Za-z] letters : {letter}* digit : [0-9] digits : {digit}* underscore : _ + €