Syntax: 10/18/2015IT 3271 Semantics: Describe the structures of programs Describe the meaning of programs Programming Languages (formal languages) -- How to describe them? -- How to use them? (machine and human) Grammars --Ambiguous (sometimes) Textbook, manuals --Confusing (always) solution: denotation semantics (for nuts only) solution: using unambiguous only

English Grammar The man hit the ball. subject verb object 10/18/2015IT 3272 The man saw the girl with a telescope. subject verb object The purpose of grammar: To have a device to generate all valid sentences in the target language (from a root). To tell whether a sentence is valid. Chomsky: (old fashion)

Noam Chomsky /18/2015IT Syntactic Structures (1957) Generative Grammar A valid sentence is generated from a root according to some fixed rules (grammar).

A generative grammar in Syntactic Structures 10/18/2015IT 3274 S NP VP TN Verb the | a man | ball | car hit | take | took | run | ran NP T N + VP Verb + + ….. root terminal symbols non-terminal symbols

Syntactic Structures 10/18/2015IT 3275 S NP VP TN Verb theman theball hit NP TN the man hit the ball

Backus-Naur Form, BNF 10/18/2015IT 3276 ::= ::= the ::= man | ball ::= hit | took ::= ::= loves | hates|eats ::= a | the ::= dog | cat | rat Grammar 1 Grammar 2 ::= | ::= loves | hates|eats |hit | took ::= a | the ::= the ::= dog | cat | rat|man | ball

Deviation: the sequence of processes that generate a sentence 10/18/2015IT 3277 <S><S> the the man the man hit the man hit the the man hit the ball ::= ::= the ::= man | ball := hit | took Grammar 1 the man hit the ball

10/18/2015IT 3278 Parse: v. To break (a sentence) down into its component parts of speech with an explanation of the form, function, and syntactical relationship of each part. (American Heritage Dict.) the dog loves the cat the loves dog the cat loves the dog the cat × ×

A Parse Tree 10/18/2015IT 3279 <S><S> the dog the cat loves Grammar ::= ::= loves | hates|eats ::= a | the ::= dog | cat | rat “the loves dog the cat” doesn’t have a parse tree

A grammar for Arithmetic Expression 10/18/2015IT ::= + | * | ( ) | a | b | c Example: ((a+b)*c) Is this expression valid? ( ) ( * ) (( ) * ) (( + ) * ) (( a + ) * ) (( a + b ) * ) ((a+b)*c) Yes

A Parse Tree for ((a+b)*c) 10/18/2015IT ( ) * ( ) a b c

Parse Trees for a+b*c 10/18/2015IT * ab c * + bc a ? What is the meaning of a+b*c

Restrictions on Grammars 10/18/2015IT Unrestricted Grammars (type-0) Why context sensitive grammars have less restrictions than context free grammars? Right/Left Linear Grammars (type-3) Context Sensitive (type-1) Context Free (type-2) Diagram in terms of the sizes of the set of restrictions

Chomsky Hierarchy 10/18/2015IT Regular Expressions (type-3) Computable (formal) languages (type-0) Context-free languages (type-2) Context-sensitive languages (type-1) Diagram in terms of the sizes of the language families

A BNF grammar consists of four parts: –The finite set of tokens (terminal symbols) –The finite set of non-terminal symbols –The start symbol –The finite set of production rules 10/18/2015IT ::= ::= the ::= man | ball ::= hit | took Grammars in BNF (Backus-Naur Form)

Constructing Grammars Using divide and conquer to simplify the job. Data types, variable names (identifiers) One variable, one type (this is not grammar’s job to make sure) 10/18/2015IT float a; boolean a, b, c; int a, b;

Primitive type names Using divide and conquer 10/18/2015IT ::= ; ::= boolean | byte | short | int | long | char | float | double ::= |, ::= | =

Tokens: How is such a program file (a sequence of characters) divided into a sequence of tokens? 10/18/2015IT e.g. identifiers ( const, x, fact ) keywords ( if, const ) operators ( == ) constants ( ), etc. Programs stored in files are just sequences of characters, but we want to prepare them into tokens before further analysis. Reserved words Tokens are atoms of the program

Lexical Structure And Phrase Structure Grammars so far have defined phrase structure: how a program is built from a sequence of tokens We also need to define lexical structure: how a text file is divided into tokens 10/18/2015IT 32719

Separate Grammars Usually there are two separate grammars –to construct a sequence of tokens from a file of characters ( Lexical Structure) –to construct a parse tree from a sequence of tokens ( Phrase Structure) 10/18/2015IT ::= | ::= | | ::= | | ::= | | | …

Separate Compiler Passes Scanner  tokens string parser  parse tree (more to do afterwards) 10/18/2015IT 32721

Historical Note #1 Early languages sometimes did not separate lexical structure from phrase structure –Early Fortran and Algol dialects allowed spaces anywhere, even in the middle of a keyword –Other languages like PL/I or Early Fortran allow keywords to be used as identifiers This makes them difficult to scan and parse It also reduces readability 10/18/2015IT 32722

Historical Note #2 Some languages have a fixed-format lexical structure -- column positions are significant –One statement per line (i.e. per card) –First few columns for statement label –Etc. Early dialects of Fortran, Cobol, and Basic Almost all modern languages are free-format: column positions are ignored 10/18/2015IT 32723

Other Grammar Forms BNF variations EBNF variations Syntax diagrams 10/18/2015IT 32724

BNF Variations Some use  or = instead of ::= Some leave out the angle brackets and use a distinct typeface for tokens Some allow single quotes around tokens, for example to distinguish ‘ | ’ as a token from | as a meta-symbol 10/18/2015IT Sir, please Step away from the ASR-33 Interesting operator!! Or not!

EBNF Variations Additional syntax to simplify some grammar chores: –{x} to mean zero or more repetitions of x –[x] to mean x is optional (i.e. x | ) –() for grouping –| anywhere to mean a choice among alternatives –Quotes around tokens, if necessary, to distinguish from meta-symbols 10/18/2015IT 32726

EBNF Examples Anything that extends BNF this way is called an Extended BNF: EBNF There are many variations 10/18/2015IT ::= { ;} ::= if then [else ] ::= { ( | ) ;}

Syntax Diagrams Syntax diagrams (“railroad diagrams”) 10/18/2015IT ifthenelse exprstmt if-stmt ::= if then else

Bypasses 10/18/2015IT ifthenelse exprstmt if-stmt ::= if then [else ]

Branching 10/18/2015IT ::= + | * | ( ) | a | b | c

Loops 10/18/2015IT ::= {+ }

Syntax Diagrams, Pro and Con Easier for human to read (follow) Difficult to perceive the phrase structures (syntax tree)? Harder for machine to read (for automatic parser-generators) 10/18/2015IT 32732

Conclusion We use grammars to define programming language syntax, both lexical structure and phrase structure Connection between theory and practice –Two grammars, two compiler passes –Parser-generators can produce code for those two passes automatically from grammars (compiler tools) 10/18/2015IT 32733