1. Syntactic sugar causes cancer of the semicolon.
CSCE 330 Programming Language Structures Chapter 3: Lexical and Syntactic Analysis (based mainly on Tucker and Noonan; Watt and Brown)
Fall 2010
Marco Valtorta
Syntactic sugar causes cancer of the semicolon.
A.Perlis

2. Contents 3.1 Chomsky Hierarchy 3.2 Lexical Analysis
3.3 Syntactic Analysis

3. 3.1 Chomsky Hierarchy Regular grammar -- least powerful
Context-free grammar (BNF)
Context-sensitive grammar
Unrestricted grammar

4. Regular Grammar Simplest; least powerful Equivalent to:
Regular expression
Finite-state automaton
Right regular grammar:   T*, B  N
A →  B
A → 

5. Example
Integer → 0 Integer | 1 Integer | ... | 9 Integer | | 1 | ... | 9

6. Regular Grammars Left regular grammar: equivalent
Used in construction of tokenizers (scanners, lexers)
Less powerful than context-free grammars
Not a regular language
{ aⁿ bⁿ | n ≥ 1 }
i.e., cannot balance: ( ), { }, begin end

7. Context-free Grammars
BNF a stylized form of CFG
Equivalent to a pushdown automaton
For a wide class of unambiguous CFGs, there are table-driven, linear time parsers

8. Context-Sensitive Grammars
Production:
α → β |α| ≤ |β|
α, β  (N  T)*
i.e., left-hand side can be composed of strings of terminals and nonterminals

9. Undecidable Properties of CSGs
Given a string  and grammar G:   L(G)
L(G) is non-empty
Defn: Undecidable means that you cannot write a computer program that is guaranteed to halt to decide the question for all   L(G).

10. Unrestricted Grammar Equivalent to: Turing machine von Neumann machine
C++, Java
That is, can compute any computable function.

11. Contents 3.1 Chomsky Hierarchy 3.2 Lexical Analysis
3.3 Syntactic Analysis

12. Lexical Analysis Purpose: transform program representation
Input: printable Ascii characters
Output: tokens
Discard: whitespace, comments
Defn: A token is a logically cohesive sequence of characters representing a single symbol.

13. Example Tokens Identifiers Literals: 123, 5.67, 'x', true
Keywords: bool char ...
Operators: + - * / ...
Punctuation: ; , ( ) { }

14. Other Sequences Whitespace: space tab Comments
// any-char* end-of-line
End-of-line
End-of-file

15. Why a Separate Phase? Simpler, faster machine model than parser
75% of time spent in lexer for non-optimizing compiler
Differences in character sets
End of line convention differs

16. Regular Expressions RegExpr Meaning x a character x
\x an escaped character, e.g., \n
{ name } a reference to a name
M | N M or N
M N M followed by N
M* zero or more occurrences of M

17. RegExpr Meaning
M+ One or more occurrences of M
M? Zero or one occurrence of M
[aeiou] the set of vowels
[0-9] the set of digits
. Any single character

18. Clite Lexical Syntax Category Definition anyChar [ -~] Letter [a-zA-Z]
Digit [0-9]
Whitespace [ \t]
Eol \n
Eof \004

19. Category Definition
Keyword bool | char | else | false | float |if | int | main | true | while
Identifier {Letter}({Letter} | {Digit})*
integerLit {Digit}+
floatLit {Digit}+\.{Digit}+
charLit ‘{anyChar}’

20. Category Definition
Operator = | || | && | == | != | < | <= | > | >= | + | - | * | / |! | [ | ]
Separator ; | . | { | } | ( | )
Comment // ({anyChar} | {Whitespace})* {eol}

21. Generators Input: usually regular expression
Output: table (slow), code
C/C++: Lex, Flex
Java: JLex

22. Finite State Automata Set of states: representation – graph nodes
Input alphabet + unique end symbol
State transition function
Labelled (using alphabet) arcs in graph
Unique start state
One or more final states

23. Deterministic FSA
Defn: A finite state automaton is deterministic if for each state and each input symbol, there is at most one outgoing arc from the state labeled with the input symbol.

24. A Finite State Automaton for Identifiers

25. Definitions
A configuration on an FSA consists of a state and the remaining input.
A move consists of traversing the arc exiting the state that corresponds to the leftmost input symbol, thereby consuming it. If no such arc, then:
If no input and state is final, then accept.
Otherwise, error.

26. An input is accepted if, starting with the start state, the automaton consumes all the input and halts in a final state.

27. Example (S, a2i$) ├ (I, 2i$) ├ (I, i$) ├ (I, $) ├ (F, )
Thus: (S, a2i$) ├* (F, )

28. Some Conventions
Explicit terminator used only for program as a whole, not each token.
An unlabeled arc represents any other valid input symbol.
Recognition of a token ends in a final state.
Recognition of a non-token transitions back to start state.

29. Recognition of end symbol (end of file) ends in a final state.
Automaton must be deterministic.
Drop keywords; handle separately.
Must consider all sequences with a common prefix together.

32. Lexer Code Parser calls lexer whenever it needs a new token.
Lexer must remember where it left off.
Greedy consumption goes 1 character too far
peek function
pushback function
no symbol consumed by start state

33. From Design to Code private char ch = ‘ ‘; public Token next ( ) {
do {
switch (ch) {
... }
} while (true);

34. Remarks Loop only exited when a token is found
Loop exited via a return statement.
Variable ch must be global. Initialized to a space character.
Exact nature of a Token irrelevant to design.

35. Translation Rules Traversing an arc from A to B:
If labeled with x: test ch == x
If unlabeled: else/default part of if/switch. If only arc, no test need be performed.
Get next character if A is not start state

36. A node with an arc to itself is a do-while.
Condition corresponds to whichever arc is labeled.

37. Otherwise the move is translated to a if/switch:
Each arc is a separate case.
Unlabeled arc is default case.
A sequence of transitions becomes a sequence of translated statements.

38. A complex diagram is translated by boxing its components so that each box is one node.
Translate each box using an outside-in strategy.

39. private boolean isLetter(char c) {
return ch >= ‘a’ && ch <= ‘z’ ||
ch >= ‘A’ && ch <= ‘Z’; }

40. private String concat(String set) {
StringBuffer r = new StringBuffer(“”);
do {
r.append(ch);
ch = nextChar( );
} while (set.indexOf(ch) >= 0);
return r.toString( ); }

41. public Token next( ) {
do { if (isLetter(ch) { // ident or keyword
String spelling = concat(letters+digits);
return Token.keyword(spelling);
} else if (isDigit(ch)) { // int or float literal
String number = concat(digits);
if (ch != ‘.’)
return Token.mkIntLiteral(number);
number += concat(digits);
return Token.mkFloatLiteral(number);

42. } else switch (ch) {
case ‘ ‘: case ‘\t’: case ‘\r’: case eolnCh:
ch = nextCh( ); break;
case eofCh: return Token.eofTok;
case ‘+’: ch = nextChar( );
return Token.plusTok; …
case ‘&’: check(‘&’); return Token.andTok;
case ‘=‘: return chkOpt(‘=‘, Token.assignTok,
Token.eqeqTok);

43. Source Tokens int main ( char c; ) int i; { c = 'h'; char i = c + 3;
// a first program
// with 2 comments
int main ( ) {
char c;
int i;
c = 'h';
i = c + 3;
} // main
int
main ( ) {
char
Identifier c ;

44. JLex: A Lexical Analyzer Generator for Java
We will look at an example JLex specification (adopted from the manual).
Consult the manual for details on how to write your own JLex specifications.
Definition of tokens
Regular Expressions
JLex
531 notes
Java File: Scanner Class
Recognizes Tokens

45. The JLex tool Layout of JLex file:
user code (added to start of generated file) %%
options %{
user code (added inside the scanner class declaration) %}
macro definitions
lexical declaration
User code is copied directly into the output class
JLex directives allow you to include code in the lexical analysis class, change names of various components, switch on character counting, line counting, manage EOF, etc.
Macro definitions gives names for useful regexps
Regular expression rules define the tokens to be recognised and actions to be taken

46. Java.io.StreamTokenizer
An alternative to JLex is to use the class StreamTokenizer from java.io
The class recognizes 4 types of lexical elements (tokens):
number (sequence of decimal numbers eventually starting with the –(minus) sign and/or containing the decimal point)
word (sequence of characters and digits starting with a character)
line separator
end of file

47. Parsing Some terminology Different types of parsing strategies
bottom up
top down
Recursive descent parsing
What is it
How to implement one given an EBNF specification
(How to generate one using tools – later)
(Bottom up parsing algorithms)
Slides for 531 ([W] = [Watt & Brown])

48. Parsing: Some Terminology
Recognition
To answer the question “does the input conform to the syntax of the language?”
Parsing
Recognition + determination of phrase structure (for example by generating AST data structures)
(Un)ambiguous grammar:
A grammar is unambiguous if there is only at most one way to parse any input (i.e. for syntactically correct program there is precisely one parse tree)

49. Different kinds of Parsing Algorithms
Two big groups of algorithms can be distinguished:
bottom up strategies
top down strategies
Example parsing of “Micro-English”
Sentence ::= Subject Verb Object .
Subject ::= I | a Noun | the Noun
Object ::= me | a Noun | the Noun
Noun ::= cat | mat | rat
Verb ::= like | is | see | sees
The cat sees the rat.
The rat sees me.
I like a cat
The rat like me.
I see the rat.
I sees a rat.

50. Top-down parsing . The sees a . cat rat The The cat cat sees sees a
The parse tree is constructed starting at the top (root).
Subject
Verb
Object .
Sentence
Sentence
Noun
Subject
The
Verb
sees a
Noun
Object .
Noun
cat
Noun
rat
The
The
cat
cat
sees
sees a
rat
rat . .

51. Bottom up parsing The The cat cat sees sees a a rat rat . .
The parse tree “grows” from the bottom (leaves) up to the top (root).
Sentence
Subject
Object
Noun
Verb
Noun
The
The
cat
cat
sees
sees a a
rat
rat . .

52. Top-Down vs. Bottom-Up parsing
LL-Analyse (Top-Down)
Left-to-Right Left Derivative
Scans string left to right
Builds leftmost derivation
LR-Analyse (Bottom-Up)
Left-to-Right Right Derivative
Scans string left to right
Builds rightmost derivation
Reduction
Derivation
Look-Ahead
Look-Ahead

53. Recursive Descent Parsing
Recursive descent parsing is a straightforward top-down parsing algorithm.
We will now look at how to develop a recursive descent parser from an EBNF specification.
Idea: the parse tree structure corresponds to the “call graph” structure of parsing procedures that call each other recursively.

54. Recursive Descent Parsing
Sentence ::= Subject Verb Object .
Subject ::= I | a Noun | the Noun
Object ::= me | a Noun | the Noun
Noun ::= cat | mat | rat
Verb ::= like | is | see | sees
Define a procedure parseN for each non-terminal N
private void parseSentence() ;
private void parseSubject();
private void parseObject();
private void parseNoun();
private void parseVerb();

55. Recursive Descent Parsing
public class MicroEnglishParser {
private TerminalSymbol currentTerminal;
//Auxiliary methods will go here
...
//Parsing methods will go here }

56. Recursive Descent Parsing: Auxiliary Methods
public class MicroEnglishParser {
private TerminalSymbol currentTerminal
private void accept(TerminalSymbol expected) {
if (currentTerminal matches expected)
currentTerminal = next input terminal ;
else
report a syntax error }
...

57. Recursive Descent Parsing: Parsing Methods
Sentence ::= Subject Verb Object .
private void parseSentence() {
parseSubject();
parseVerb();
parseObject();
accept(‘.’); }

58. Recursive Descent Parsing: Parsing Methods
Subject ::= I | a Noun | the Noun
private void parseSubject() {
if (currentTerminal matches ‘I’)
accept(‘I’);
else if (currentTerminal matches ‘a’) {
accept(‘a’);
parseNoun(); }
else if (currentTerminal matches ‘the’) {
accept(‘the’);
else
report a syntax error

59. Recursive Descent Parsing: Parsing Methods
Noun ::= cat | mat | rat
private void parseNoun() {
if (currentTerminal matches ‘cat’)
accept(‘cat’);
else if (currentTerminal matches ‘mat’)
accept(‘mat’);
else if (currentTerminal matches ‘rat’)
accept(‘rat’);
else
report a syntax error }

60. Algorithm to convert EBNF into a RD parser
The conversion of an EBNF specification into a Java implementation for a recursive descent parser is so “mechanical” that it can easily be automated!
=> JavaCC “Java Compiler Compiler”
We can describe the algorithm by a set of mechanical rewrite rules
private void parseN() {
parse X }
N ::= X

61. Algorithm to convert EBNF into a RD parser
parse t where t is a terminal
accept(t);
parse N where N is a non-terminal
parseN();
// a dummy statement
parse e
parse XY
parse X
parse Y

62. Algorithm to convert EBNF into a RD parser
parse X*
while (currentToken.kind is in starters[X]) {
parse X }
parse X|Y
switch (currentToken.kind) {
cases in starters[X]:
parse X
break;
cases in starters[Y]:
parse Y
default: report syntax error }