1. ©University of Sheffieldcom2010 Com2010 - Functional Programming Demos: LexPrs & PrsRes & Software Engineering Design and Coding Marian Gheorghe Lecture 18 Module homepage Mole & http://www.dcs.shef.ac.uk/~marian

2. Computational models utilised lexical analysis and parsing (regular expressions, finite state machines, EBNF, syntax diagrams etc) Haskell key concepts: –Complex data structures: algebraic types (simple, compound, polymorphic, recursive) –Recursive functions – recursive descent parser –Higher order functions – three basic parsing diagrams –List comprehension Software engineering approach on designing, coding and maintenance What have we learned? ©University of Sheffieldcom2010

3. Compiling AST ©University of Sheffieldcom2010 Lexical analysisParsing Compiler Semantics; code, evaluation… Source fileToken unitsAST "a = a + 1 " ++ “/*comment*/" [(1,"a"),(5,"="),(1,"a"), (3,"+"),(2,"1"),(0,"Eop")] ?? ? AST – Abstract Syntax Tree Key - codeLexical unit - string spaces, comment only in source

4. Note how lex_aut represents various transitions ( Move constructor) It recognizes: identifiers, numbers, comments, delimiters ( +,-,=,; ), spaces Invoked as: lex_an lex_aut inLexN for inLex1 = "ident 23 + - = ; /*comment*/ anotherIdentThenNumber12 124" ~~> OK inLex2 = "something 1 /*wrong_identifier*/ ident_y ok" ~~> ident_y is not identifier inLex3 = "something 2 /*wrongNumber*/ 1.24 ok" ~~> 1.24 is not number inLex4 = "something 3 /*wrongDelimiter*/ < ok" ~~> < is not delimiter LexPrs: Lexical analysis ©University of Sheffieldcom2010

5. Compiling AST ©University of Sheffieldcom2010 Lexical analysisParsing Compiler Semantics; code, evaluation… Source fileToken unitsAST "a = a + 1 " ++ “/*comment*/" [(1,"a"),(5,"="),(1,"a"), (3,"+"),(2,"1"),(0,"Eop")] ?? ? AST – Abstract Syntax Tree Key - codeLexical unit - string spaces, comment only in source

6. Note how basic diagrams are implemented Parser is invoked as fProgram (lex_an lex_aut inPr1) Tests: inPr1 = "a = 0 ; a = a + 1 ; b = a + a - 2“ ~~> Ok inPr2 = "a = 0 a = a + 1 ; b = a“ ~~> missing ; inPr3 = "a = 0 ; a = a + ; b = a" ~~> missing operand inPr4 = "a = 0 ; a = a 1 ; b = a" ~~> missing operator ( + ) LexPrs: parsing ©University of Sheffieldcom2010

7. Compiling AST ©University of Sheffieldcom2010 Lexical analysisParsing Compiler Semantics; code, evaluation… Source fileToken unitsAST "a = a + 1 " ++ “/*comment*/" [(1,"a"),(5,"="),(1,"a"), (3,"+"),(2,"1"),(0,"Eop")] ?? ? AST – Abstract Syntax Tree Key - codeLexical unit - string spaces, comment only in source

8. PrsRes: Postfix notation eval ©University of Sheffieldcom2010 SetOf TokenUnit replaced by ParserUnit Changes in the Parser code (effect terminal rules) Extract postfix notation Invoked as snd(snd(fProgram (lex_an lex_aut inp,([],[])))) Or extractOut inp Tests inp1 = "a = 0 ; a = a + 1 ; b = a + a - 2“ ~~> [(1,"a"),(2,"0"),(5,"="),(6,";"),(1,"a"),(1,"a"),(2,"1"),(3,"+"),(5,"="),(6,";"),(1,"b"), (1,"a"), (1,"a"),(3,"+"),(2,"2"),(4,"-"),(5,"="),(0,"Eop")]

9. PrsRes: Postfix notation eval (2) ©University of Sheffieldcom2010 Extract expressions Invoked as extractExp(extractOut inp,[]) Tests inp2 = "a = 6 + 2 ; b = 8 ; c = 2 - 1 + 3" ~~> ([],[("a",["6","2","+"]),("b",["8"]),("c",["2","1","-","3","+"])]) Expression evaluation Invoked as evalSA(snd(extractExp(extractOut inp2,[]))) Tests inp2 – above [("a",8),("b",8),("c",4)]

10. Software Engineering Haskell Projects ©University of Sheffieldcom2010 Lexical analyser requires changes in the associated model (FSM), lex_aut –see the code Parser consists of a set of recursive functions – it requires a stepwise process with short iterations Implementing all SA functions (15) in one step simply does not work why? – get a complex set of recursive invocations impossible to manage and debug need a strategy... Initial step:

11. 1. Program :: =(S)7. Trm::=(A) StmtList EopIdentifier 2. StmtList ::=(I)Number Assign8. Operator ::=(A) DelimAddOp 3. Assign ::=(S)MinOp LHandS RestAss9. LHandS ::= ident (T) 4. RestAss ::=(S)10. AssSymb ::= assg (T) AssSymb Exp11. Identifier ::= ident (T) 5. Exp ::=(I)12. Number ::= no (T) Trm13. Delim ::= sc (T) Operator14. Addop ::= pls (T) 6. Eop ::= eop(T)15. MinOp ::= mns (T) Instead of the Entire SA ©University of Sheffieldcom2010

12. Create an Initial Step ©University of Sheffieldcom2010 Use only diagrams 1, 6 and change 2 to a simpler form, i.e.: 2. StmtList ::= ident So, you have only 3 simple diagrams: a sequence (1) and two terminals (2, 6) These are simple to be tested. Test program: (See the code...) “a /*just_an_identifier*/”

13. Next Step ©University of Sheffieldcom2010 Revert diagram 2 to its initial form, i.e.: 2. StmtList ::= Assign {Delim Assign} where Assign is simply a terminal and Delim keeps its definition, i.e.: 3. Assign ::= ident 13. Delim ::= sc You have: a sequence (1), an iteration (2) and two terminals (6, 13) These will be tested by: (See the code...) “a ; a1 ; b /*just_identifiers_and_sc*/”

14. Generic Step ©University of Sheffieldcom2010 Based on the previous step, add on one or two new diagrams The associated components are made terminals Alternation and/or iteration sets, if any, might be adjusted Write a test set for this new iteration Looks like an XP approach!

15. Finally, the Back-end ©University of Sheffieldcom2010 To implement the back-end, the following transformations are made into the parser The generic SetOf TokenUnit is replaced Adequate output is implemented by adding certain functions the terminal rules (usually fTerm transformed into seqOf ) Functions to extract the output Functions implementing requested transformations (postfix notation etc.)

16. Where to find information ©University of Sheffieldcom2010 Lecture notes – Mole Lecture slides + LexPrs & PrsRes code on my page http://www.dcs.shef.ac.uk/~marian