1. CS 153: Concepts of Compiler Design October 29 Class Meeting Department of Computer Science San Jose State University Fall 2014 Instructor: Ron Mak www.cs.sjsu.edu/~mak 1

2. Computer Science Dept. Fall 2014: October 29 CS 153: Concepts of Compiler Design © R. Mak 2 JJDoc  JJDoc produces documentation for your grammar.  Right-click in the.jj edit window.  It generates an HTML file from a.jj grammar file.  Read Chapter 5 of the JavaCC book. Ideal for your project documentation!

3. Computer Science Dept. Fall 2014: October 29 CS 153: Concepts of Compiler Design © R. Mak JJDoc on the Command Line  Example bash script for a Mac: 3 java –classpath Eclipse_path /plugins/sf.eclipse.javacc_1.5.24/javacc.jar jjdoc.jj file #!/bin/bash java -classpath /Applications/Eclipse/plugins/sf.eclipse.javacc_1.5.24/javacc.jar jjdoc $1

4. Computer Science Dept. Fall 2014: October 29 CS 153: Concepts of Compiler Design © R. Mak 4 Assignment #6  Generate a working parser with JavaCC This assignment is the start of your compiler project. This assignment due Monday, November 10.  By the end of the semester, you must implement enough of your compiler to be able to write nontrivial programs in your chosen source language, compile them, and run them.  Your final compiler project will be due Friday, December 12.

5. Computer Science Dept. Fall 2014: October 29 CS 153: Concepts of Compiler Design © R. Mak 5 Assignment #6, cont’d  Keep the grammar of your source language simple! Add features incrementally. This assignment represents a snapshot of your early thinking about language design. You can change or add new features later. _

6. Computer Science Dept. Fall 2014: October 29 CS 153: Concepts of Compiler Design © R. Mak 6 Assignment #6: Suggested Implementation Order 1. Expressions with numeric constants and scalar variables no type checking yet no arrays and records yet 2. Assignment statements 3. Control statements 4. Variable declarations no type definitions yet 5. Procedure and function declarations 6. Procedure and function calls 7. Type definitions type checking arrays and records For this assignment, do at least the first 3. (Some control statements; you can add more later.)

7. Computer Science Dept. Fall 2014: October 29 CS 153: Concepts of Compiler Design © R. Mak 7 Assignment #6: Turn in a Zip File  The EBNF production rules for your language. At least 20 rules for this assignment (you can add more later). Generate with JJDoc.  A.jj file based on your EBNF rules.  A sample program (or a set of statements) written in your source language. Show off features of your parser as far as you implemented it for this assignment. Can be error-free (i.e., no syntax error handling yet). Output from compiling your sample program.

8. Computer Science Dept. Fall 2014: October 29 CS 153: Concepts of Compiler Design © R. Mak 8 Assignment #6  Add syntactic actions to print messages when the parser recognizes major source language constructs (such as statements and expressions) in the source program. Doesn’t have to be fancy output. _

9. Computer Science Dept. Fall 2014: October 29 CS 153: Concepts of Compiler Design © R. Mak 9 JJTree  JJTree builds an Abstract Syntax Tree (AST); i.e., a parse tree.  It works as a preprocessor for JavaCC grammars. _

10. Computer Science Dept. Fall 2014: October 29 CS 153: Concepts of Compiler Design © R. Mak 10 JJTree  JJTree augments a JavaCC grammar by inserting tree-building code. When a parser is generated from the augmented grammar, and The generated parser is executed against a source program … An AST is created..jjt JJTree JavaCC Generated Parser source program AST JavaC.jj.java

11. Computer Science Dept. Fall 2014: October 29 CS 153: Concepts of Compiler Design © R. Mak 11 Example: Simple Calculator void Expression() : {} { {System.out.println("EXPRESSION STARTS");} Operator() {System.out.println("EXPRESSION ENDS");} } void Operator() : {} { Operand() "+" {System.out.println("Operator: " + tokenImage[PLUS]);} Operand() } void Operand() : {Token t;} { t= {System.out.println("Operand: " + t.image);} } calculator_simple.jj

12. Computer Science Dept. Fall 2014: October 29 CS 153: Concepts of Compiler Design © R. Mak 12 Example: A Simple AST SimpleNode Expression() : {} { Operator() {return jjtThis;} } void Operator() : {} { Operand() "+" Operand() } void Operand() : {} { } PARSER_BEGIN(Calculator) import java.io.*; public class Calculator { public static void main(String[] args) { Reader sr = new StringReader(args[0]); Calculator calc = new Calculator(sr); try { SimpleNode node = calc.Expression(); node.dump(">"); } catch (ParseException ex) { ex.printStackTrace(); } PARSER_END(Calculator) jjtThis represents the current AST node. dump() is the AST print method. SimpleNode implements the JJTree Node class. calculator_tree.jjt

13. Computer Science Dept. Fall 2014: October 29 CS 153: Concepts of Compiler Design © R. Mak 13 Example: Print AST Node Images SimpleNode Expression() : {} { Operator() {return jjtThis;} } void Operator() : {Token t;} { Operand() t="+" {jjtThis.setImage(t.image);} Operand() } void Operand() : {Token t;} { t= {jjtThis.setImage(t.image);} } calculator_tree_image.jjt Modify the generated SimpleNode class to add the image field and the setImage() method.  You can modify the generated SimpleNode class.

14. Computer Science Dept. Fall 2014: October 29 CS 153: Concepts of Compiler Design © R. Mak 14 Example: Operating Calculator SimpleNode Expression() : {} { Operator() { SimpleNode operator = (SimpleNode) jjtThis.jjtGetChild(0); SimpleNode first = (SimpleNode) operator.jjtGetChild(0); int firstValue = (Integer) first.jjtGetValue(); SimpleNode second = (SimpleNode) operator.jjtGetChild(1); int secondValue = (Integer) second.jjtGetValue(); if (((String) operator.jjtGetValue()).equals("+")) { jjtThis.jjtSetValue(firstValue + secondValue); } else { System.out.println("Unknown operator"); } return jjtThis; } } calculator_calculating.jjt Second child First child

15. Computer Science Dept. Fall 2014: October 29 CS 153: Concepts of Compiler Design © R. Mak 15 The Visitor Design Pattern  Purpose: Visit the different node types of a tree structure. Perform operations on the nodes without needing to modify the nodes.  The visitor interface declares a visit() method for each node type that wants to be visited. visit(NodeTypeA node) visit(NodeTypeB node)  Each tree node has an accept() method to accept a visitor. accept(Visitor v)

16. Computer Science Dept. Fall 2014: October 29 CS 153: Concepts of Compiler Design © R. Mak 16 The Visitor Design Pattern  A tree node’s accept() method calls the visitor’s visit() method that’s appropriate for the node type, passing itself as an argument. Accept a visitor  call the visitor’s visit() method “Here’s the house key. Go visit my house now.”  In order to perform its operation, the visitor’s visit() method uses the node argument (the house key) to access the node’s public data and to call the node’s public methods.

17. Computer Science Dept. Fall 2014: October 29 CS 153: Concepts of Compiler Design © R. Mak 17 Calculator and the Visitor Design Pattern  The visitor interface declares an overloaded visit() method for each node type.  Each tree node has an accept() method to accept a visitor.  A tree node’s accept() method calls the visitor’s visit() method that’s appropriate for the node type, passing itself as an argument.  The visitor’s visit() method uses the node argument to access the node’s public data and to call the node’s public methods in order to perform its operation. JJTree generates the black classes. You write the blue classes.

18. Computer Science Dept. Fall 2014: October 29 CS 153: Concepts of Compiler Design © R. Mak 18 Calculator and the Visitor Design Pattern  First create a visitor object from class SumVisitor. There is only one visitor object!  Tell the root node of the expression parse tree to accept the visitor. Call the root node’s accept() method and pass the visitor object that will visit the root node.  The root node’s accept() method makes the visit happen by calling the visitor’s overloaded visit() method and passing itself (the node = house key) as a parameter. The parameter’s node type determines which visit() method of the visitor is called. Reader sr = new StringReader(args[0]); Calculator calc = new Calculator(sr); SimpleNode root = calc.Expression(); SumVisitor visitor = new SumVisitor(); root.jjtAccept(visitor, null); System.out.println("Sum is " + visitor.sum);

19. Computer Science Dept. Fall 2014: October 29 CS 153: Concepts of Compiler Design © R. Mak 19 Calculator and the Visitor Design Pattern public class CalculatorVisitorAdapter implements CalculatorVisitor { public Object visit(SimpleNode node, Object data) { return node.childrenAccept(this, data); } public Object visit(ASTExpression node, Object data) { return node.childrenAccept(this, data); } public Object visit(ASTOperator node, Object data) { return node.childrenAccept(this, data); } public Object visit(ASTOperand node, Object data) { return node.childrenAccept(this, data); } } The default action when visiting each type of tree node is to tell each of the node’s children to accept the visitor. (In other words, recursively walk down the tree.) This action can be overridden by subclasses. JJTree generates the CalculatorVisitor interface. One visit() method for each tree node type that can accept visitors.

20. Computer Science Dept. Fall 2014: October 29 CS 153: Concepts of Compiler Design © R. Mak 20 Calculator and the Visitor Design Pattern, cont’d  SumVisitor ’s visit() method is called whenever the tree node is an ASTOperand (i.e., an ASTOperand node is being visited). The SumVisitor object simply adds the value stored in the operand node to its running sum. Then it performs the default action for any children.  SumVisitor is a subclass of CalculatorVisitorAdaptor. The adaptor implements visit() methods for the other node types. public class SumVisitor extends CalculatorVisitorAdapter { public int sum = 0; public Object visit(ASTOperand operand, Object data) { sum += (Integer) operand.jjtGetValue(); return super.visit(operand, data); } After summing, do the default action ( super.visit ) for any children. (This is a preorder traversal.) Only override the visit method of the ASTOperand node. calculator_visitor.jjt

21. Computer Science Dept. Fall 2014: October 29 CS 153: Concepts of Compiler Design © R. Mak 21 JJTree: New Node Names  By default, JJTree creates an AST where the nodes are named after the nonterminals in the grammar. This makes the tree design tightly-coupled to the source language and its grammar.  Use node descriptors to change the name of a node. SimpleNode Expression() #Expr : {} { Operator() {return jjtThis;} } void Operator() #AddOp : {} { Operand() "+" Operand() } void Operand() #Opnd : {} { } node_name_change.jjt

22. Computer Science Dept. Fall 2014: October 29 CS 153: Concepts of Compiler Design © R. Mak 22 JJTree: Node Descriptors  You can use node descriptors to combine nodes by reusing node names.  Now both production rules Expression and SimpleExpression will generate Expr nodes. SimpleNode Expression() #Expr : {} { Operator() {return jjtThis;} } SimpleNode SimpleExpression() #Expr : {} { Operator() {return jjtThis;} }

23. Computer Science Dept. Fall 2014: October 29 CS 153: Concepts of Compiler Design © R. Mak 23 JJTree: Node Descriptors, cont’d  Use the node descriptor #void to cause a production rule not to generate an AST node.  There are other things you can do with node descriptors. Read Chapter 4 of the JavaCC book. SimpleNode Expression() #Expr : {} { Operator() {return jjtThis;} } void Operator() #void : {} { Operand() "+" Operand() } void Operand() #Opnd : {} { } node_void.jjt

24. Computer Science Dept. Fall 2014: October 29 CS 153: Concepts of Compiler Design © R. Mak 24 JJTree: Tree Shaping  Will our simple calculator grammar accept 1+2+3+4 ?  Change to  Tree: void Operator() : {Token t;} { Operand() "+" Operand() } void Operator() : {Token t;} { Operand() ( "+" Operand() )* } Expression Operator Operand calculator_tree_shape_1.jjt

25. Computer Science Dept. Fall 2014: October 29 CS 153: Concepts of Compiler Design © R. Mak 25 JJTree: Tree Shaping, cont’d  Solution: Use embedded definite node descriptors:  Tree: void Operator() #void : {Token t;} { Operand() ( "+" Operand() #add(2) )* } Expression add Operand calculator_tree_shape_2.jjt

26. Computer Science Dept. Fall 2014: October 29 CS 153: Concepts of Compiler Design © R. Mak 26 What JJTree, JJDoc, and JavaCC Do  You feed JJTree a.jjt grammar file Token specifications using regular expressions Production rules using EBNF  JJTree produces a.jj grammar file  JavaCC generates a scanner, parser, and tree-building routines Code for the visitor design pattern to walk the parse tree.  JJDoc produces a.html containing the ENBF

27. Computer Science Dept. Fall 2014: October 29 CS 153: Concepts of Compiler Design © R. Mak 27 What JJTree, JJDoc, and JavaCC Do  However, JJTree and JavaCC will not: Generate code for a symbol table Generate any backend code You have to provide this code!

28. Computer Science Dept. Fall 2014: October 29 CS 153: Concepts of Compiler Design © R. Mak 28 Pcl  Pcl is a teeny, tiny subset of Pascal.  Use JavaCC to generate a Pcl parser and integrate with our Pascal interpreter’s symbol table components parse tree components  We’ll be able to parse and print the symbol table and the parse tree in our favorite XML format  Sample program test.pcl : PROGRAM test; VAR i, j, k : integer; x, y, z : real; BEGIN i := 1; j := i + 3; x := i + j; y := 314.15926e-02 + i - j + k; z := x + i*j/k - x/y/z END.

29. Computer Science Dept. Fall 2014: October 29 CS 153: Concepts of Compiler Design © R. Mak 29 Pcl Challenges  Get the JJTree parse trees to build properly with respect to operator precedence. Use embedded definite node descriptors!  Decorate the parse tree with data type information. Can be done as the tree is built, or as a separate pass. You can use the visitor pattern to implement the pass.  Hook up to the symbol table and parse tree printing classes from the Pascal interpreter.

30. Computer Science Dept. Fall 2014: October 29 CS 153: Concepts of Compiler Design © R. Mak 30 Pcl, cont’d options{ JJTREE_OUTPUT_DIRECTORY="src/wci/frontend"; NODE_EXTENDS="wci.intermediate.icodeimpl.ICodeNodeImpl";... } PARSER_BEGIN(PclParser)... public class PclParser { // Create and initialize the symbol table stack. symTabStack = SymTabFactory.createSymTabStack(); Predefined.initialize(symTabStack);... // Parse a Pcl program. Reader reader = new FileReader(sourceFilePath); PclParser parser = new PclParser(reader); SimpleNode rootNode = parser.program();...

31. Computer Science Dept. Fall 2014: October 29 CS 153: Concepts of Compiler Design © R. Mak 31 Pcl, cont’d... // Print the cross-reference table. CrossReferencer crossReferencer = new CrossReferencer(); crossReferencer.print(symTabStack); // Visit the parse tree nodes to decorate them with type information. TypeSetterVisitor typeVisitor = new TypeSetterVisitor(); rootNode.jjtAccept(typeVisitor, null); // Create and initialize the ICode wrapper for the parse tree. ICode iCode = ICodeFactory.createICode(); iCode.setRoot(rootNode); programId.setAttribute(ROUTINE_ICODE, iCode); // Print the parse tree. ParseTreePrinter treePrinter = new ParseTreePrinter(System.out); treePrinter.print(symTabStack); } PARSER_END(PclParser) Demo