Center for Software Sciences Northeastern University A domain specific language for Traversal Specification Johan Ovlinger Mitchell Wand Northeastern University Center for Software Sciences

Talk Outline Introduction –introduce visitor pattern –point out some of its shortcomings –introduce our motivating, and running, example –show how it is unsatisfactorily solved by visitor pattern, despite the obvious applicability. Center for Software Sciences Northeastern University

Center for Software Sciences Northeastern University Introducing the Visitor Pattern Visitor pattern is useful for –removing structural assumptions from programs, by separating navigational and behavioral concerns –adding behavior over classes independently of their definition.

Center for Software Sciences Northeastern University Critiquing Traversals Visitor is guided around object graph by traversal (iterator, strategy) Unfortunately, the traversal serializes the hierarchical structure of the object being traversed into a series of visits. One global traversal is typically hard- coded into classes, restricting its interface to Visitors to LCD.

Center for Software Sciences Northeastern University Motivating Example Taken from real world experience, writing a compiler. We want to verify that all variables have been declared before use. Obvious use of visitor pattern (suggested in “Design Patterns”).

Center for Software Sciences Northeastern University Useless language Our example is a very simple language that only has the ability to bind and reference variables. Lets are non recursive No Shadowing (for simplicity) Epitomizes a common task: Finding undeclared variables. Leave grammar up to imagination.

Center for Software Sciences Northeastern University AST Class Diagram Prog LetRefFun StringBindEmpty Exp BindList

Center for Software Sciences Northeastern University A small AST object LetBind Empty Let Ref foo bar Fun Ref x y foo Bind Empty grokRefbar grok LET foo = FUN (x) {y} bar = foo IN LET grok = bar IN grok Prog

Center for Software Sciences Northeastern University Visitor Considerations Strategy: –Calculate new bindings for one level and also analyze their bodies at same time –Remember which variables are in scope when entering a Let, and reset on exiting. –Since body of binding may also include Lets, must also remember List of new bindings to be added to scope. –Calculate list of bindings to add from let and process their bodies simultaneously.

Center for Software Sciences Northeastern University animation

Center for Software Sciences Northeastern University Visitor’s view of Traversal LetBind Empty Let Ref foo bar Fun Ref x y foo Bind Empty grokRefbar grok Prog before Prog visitin scopeto add administrator: these are missing the Bind middle visit administrator: these are missing the Bind middle visit

Center for Software Sciences Northeastern University Visitor’s view of Traversal LetBind Empty Let Ref foo bar Fun Ref x y foo Bind Empty grokRefbar grok Prog before Prog before Let visitin scopeto add

Center for Software Sciences Northeastern University before Prog before Let before Bind visitin scopeto add Visitor’s view of Traversal LetBind Empty Let Ref foo bar Fun Ref x y foo Bind Empty grokRefbar grok Prog

Center for Software Sciences Northeastern University before Prog before Let before Bind before Fun x visitin scopeto add Visitor’s view of Traversal LetBind Empty Let Ref foo bar Fun Ref x y foo Bind Empty grokRefbar grok Prog

Center for Software Sciences Northeastern University before Prog before Let before Bind before Fun x before Ref x visitin scopeto add Visitor’s view of Traversal LetBind Empty Let Ref foo bar Fun Ref x y foo Bind Empty grokRefbar grok Prog

Center for Software Sciences Northeastern University before Prog before Let before Bind before Fun before Ref x after Ref x visitin scopeto add Visitor’s view of Traversal LetBind Empty Let Ref foo bar Fun Ref x y foo Bind Empty grokRefbar grok Prog

Center for Software Sciences Northeastern University before Prog before Let before Bind before Fun before Ref x after Ref x after Fun visitin scopeto add Visitor’s view of Traversal LetBind Empty Let Ref foo bar Fun Ref x y foo Bind Empty grokRefbar grok Prog

Center for Software Sciences Northeastern University before Prog before Let before Bind before Fun before Ref x after Ref x after Fun before Bind visitin scopeto add Visitor’s view of Traversal LetBind Empty Let Ref foo bar Fun Ref x y foo Bind Empty grokRefbar grok Prog

Center for Software Sciences Northeastern University... before Bind before Fun before Ref x after Ref x after Fun x before Bind before Ref after Ref before Empty after Empty after Bind bar visitin scopeto add Visitor’s view of Traversal LetBind Empty Let Ref foo bar Fun Ref x y foo Bind Empty grokRefbar grok Prog

Center for Software Sciences Northeastern University... before Fun before Ref x after Ref x after Fun x before Bind before Ref after Ref before Empty after Empty after Bind bar after Bind foo,bar visitin scopeto add Visitor’s view of Traversal LetBind Empty Let Ref foo bar Fun Ref x y foo Bind Empty grokRefbar grok Prog

Center for Software Sciences Northeastern University... before Ref x after Ref x after Fun x before Bind before Ref after Ref before Empty after Empty after Bind bar after Bind foo,bar middle Let foo,bar visitin scopeto add Visitor’s view of Traversal LetBind Empty Let Ref foo bar Fun Ref x y foo Bind Empty grokRefbar grok Prog

Center for Software Sciences Northeastern University... after Bind bar after Bind foo,bar middle Let foo,bar before Let foo,bar before Bind foo,bar before Ref foo,bar after Ref foo,bar before Empty foo,bar after Empty foo,bar after Bind Let foo,bargrok middle Let foo,bar,grok visitin scopeto add Visitor’s view of Traversal LetBind Empty Let Ref foo bar Fun Ref x y foo Bind Empty grokRefbar grok Prog

Center for Software Sciences Northeastern University... middle Let foo,bar before Let foo,bar before Bind foo,bar before Ref foo,bar after Ref foo,bar before Empty foo,bar after Empty foo,bar after Bind Let foo,bargrok middle Let foo,bar,grok before Ref foo,bar,grok visitin scopeto add Visitor’s view of Traversal LetBind Empty Let Ref foo bar Fun Ref x y foo Bind Empty grokRefbar grok Prog

Center for Software Sciences Northeastern University... before Let foo,bar before Bind foo,bar before Ref foo,bar after Ref foo,bar before Empty foo,bar after Empty foo,bar after Bind Let foo,bargrok middle Let foo,bar,grok before Ref foo,bar,grok after Let foo,bar visitin scopeto add Visitor’s view of Traversal LetBind Empty Let Ref foo bar Fun Ref x y foo Bind Empty grokRefbar grok Prog

Center for Software Sciences Northeastern University... before Ref foo,bar after Ref foo,bar before Empty foo,bar after Empty foo,bar after Bind Let foo,bargrok middle Let foo,bar,grok before Ref foo,bar,grok after Let foo,bar after Let after Prog visitin scopeto add Visitor’s view of Traversal LetBind Empty Let Ref foo bar Fun Ref x y foo Bind Empty grokRefbar grok Prog

Center for Software Sciences Northeastern University Implementation in Java We assume a “typical” traversal distributed through classes which calls appropriate accept method on Visitor. Visitor is a superclass of all visitors, and has default do-nothing accept methods.

Center for Software Sciences Northeastern University Visitor in Java class UDVisitor extends Visitor { Vector inscope,boundbylet; Stack oldscope, oldbound; void before(Ref host) { if (!inscope.contains(host.str)) complain(host); } void after(Bind host) { boundbylet.addElement(host.str); } void before(Fun host) { inscope.addElement(host.str); } void after(Fun host) { inscope.removeElement(host.str); }

Center for Software Sciences Northeastern University Visitor Cont void before(Let host) { oldscope.push(inscope.copy()); oldbound.push(boundbylet.copy()); boundbylet = new Vector(); } void middle(Let host) { inscope.add_from(boundbylet); } void after(Let host) { inscope = oldscope.pop(); boundbylet = oldbound.pop(); }

Center for Software Sciences Northeastern University Visitor Interface Critique –Since all traversal code is outside our control, visitors have to pass parameters to subsequent visits by side-effecting own variables. Very imperative. –We assume a more intricate interface than normally accepted -- rather than just accept, we have before, middle and after. Otherwise, it would be hard to know when to add new bindings to scope. administrator: We could do with just accept by adding bound at empty and checking on the way down. administrator: We could do with just accept by adding bound at empty and checking on the way down.

Center for Software Sciences Northeastern University Structural Critique –The structure cannot be ignored -- the programmer needs to know when to save/restore the parameter passing variables. –Programmer needs to save and restore state manually –Restoring state depends on our non- standard visitor interface. –The order of traversing parts is vitally important. administrator: the fact that Binding’s body may contain lets forces Let to s/r bound. Also imp is when not to restore vars administrator: the fact that Binding’s body may contain lets forces Let to s/r bound. Also imp is when not to restore vars administrator: if we s/r around Bindings, then all is lost administrator: if we s/r around Bindings, then all is lost administrator: to restore state with only accept would be hard. We know after will close before administrator: to restore state with only accept would be hard. We know after will close before administrator: who ever said that it could? administrator: who ever said that it could? administrator: Bindings before exp administrator: Bindings before exp

Center for Software Sciences Northeastern University Visitor Critique Summary –Visitors want to return results and accept arguments like recursive functions –The visitor needs to control the order in which parts are traversed. –A more elaborate traversal/visitor interface is desirable. accept is least common denominator. administrator: it is natural to expect that complex behavior will need more complicated behavior than before / middle / after administrator: it is natural to expect that complex behavior will need more complicated behavior than before / middle / after

Talk Outline Introduction Traversal Specifications –compare to std traversal Center for Software Sciences Northeastern University

Center for Software Sciences Northeastern University Traversal Specification –Define traversals with domain specific language. –Traversal is external to classes, and can be tailored to just one visitor, or be generic. –Traversal is explicit about traversing all parts. Assume only local knowledge of the object structure. Non-local knowledge cannot be expressed in Traversal Spec. administrator: surprisingly, we don’t have the faults we pointed out with visitor pattern. administrator: surprisingly, we don’t have the faults we pointed out with visitor pattern.

Center for Software Sciences Northeastern University Features of our Language Separate Behavior, Navigation, and Class Definition Specify which parts of an object are to be traversed, and which order Traversals and Visitors return results Traversal may bind intermediate results Behavior may influence Navigation Iterate over collections automatically administrator: There is a lot of talking about this slide administrator: There is a lot of talking about this slide

Center for Software Sciences Northeastern University traversal everywhere() = Program => visit before () traverse exp () visit after () Ref => visit before () visit after() Let => visit before () traverse bindings() visit middle () traverse exp() visit after () Binding => visit before () traverse exp () visit middle () traverse bindings() visit after ()... Go Everywhere Traversal administrator: this makes an interface everywhere_visitor that looks just like the Visitor class. Interface, not class, because we will have many traversals, with different requirements, while the std model has just one traversals, or if several, all w/ subsets of one big interface. administrator: this makes an interface everywhere_visitor that looks just like the Visitor class. Interface, not class, because we will have many traversals, with different requirements, while the std model has just one traversals, or if several, all w/ subsets of one big interface. administrator: introduce concept of subtraversal administrator: introduce concept of subtraversal

Center for Software Sciences Northeastern University The Go Everywhere Traversal Will generate an interface with before, after, middle methods, just like the traversal will impose its p Each traversal imposes different requirements on visitors it is used with By generating Java interfaces instead of abstract classes, a visitor may be used with several traversals.

Talk Outline Introduction Traversal Specifications –compare to std traversal –overview Center for Software Sciences Northeastern University

Center for Software Sciences Northeastern University Syntax of the Language –Variables are declared with C/C++/Java syntax, with optional types –A Traversal is a named set of TraversalEntries, named after the class of objects they traverse. –TraversalEntry describes how to traverse objects of a certain class, using a list of Actions including visit and traverse.

Center for Software Sciences Northeastern University More syntax –On entering an object, the most specific TraversalEntry for that object is executed, by performing the Actions in order and returning the result of the last one. –Visit actions name the visitor method to invoke, and which arguments to pass (current object is always passed). –Traversal actions specify the part of the current object to traverse, along with arguments to the traversal.

Talk Outline Introduction Traversal Specifications –compare to std traversal –overview –running example detailed interface to visitor recursion Center for Software Sciences Northeastern University

Center for Software Sciences Northeastern University Our Strategy Write a traversal that calls specific methods on visitor, at appropriate time. Visits return their results to traversal. Generate a Java Interface from the traversal to be implemented by visitor. Traversal uses recursion to save/restore visitor variables automatically.

Center for Software Sciences Northeastern University Traversal Spec for AST traversal unbound(Vector scope) = Program => traverse exp (scope) Ref => visit checkref(scope) Let => Vector bound = traverse bindings(scope) Vector newscope = visit concat(scope,bound) traverse exp(newscope) Binding => Vector bound = traverse bindings(scope) visit addtobound(bound) traverse exp(scope) bound Empty => visit emptybound Fun => visit addvar(scope) traverse exp(scope) visit remvar(scope)

Center for Software Sciences Northeastern University Interface for AST Traversal interface unbound_visitor { void checkref(Ref host, Vector scope); Vector concat(Let host,Vector a,Vector b); void addtobound(Binding host, Vector bound); Vector emptybound(Empty host, Vector bound); void addvar(Fun host,Vector scope); void remvar(Fun host,Vector scope); }

Center for Software Sciences Northeastern University Visitor Definition class UBVisitor implements unbound_visitor { Vector concat(Let host,Vector a,Vector b) {...} void checkref(Ref host, Vector inscope) { if (!inscope.contains(host)) complain(host); } void addtobound(Binding host, Vector bound) { bound.addElement(host.var); } Vector emptybound(Empty host, Vector bound) { return new Vector(); } void addvar(Fun host,Vector scope) { scope.addElement(host.var); } void remvar(Fun host,Vector scope) { scope.removeElement(host.var); } }

Center for Software Sciences Northeastern University Compare to Visitor pattern Surprisingly, we much prefer the separation of our version. Navigation and Behavior are specified separately, but are tailored

Talk Outline Introduction Traversal Specifications –compare to std traversal –overview –running example –influencing navigation Center for Software Sciences Northeastern University

Center for Software Sciences Northeastern University Behavior influences Navigation In several cases, it is necessary to decide dynamically the navigational details. The only way to do this with the visitor pattern is to hard code another traversal both in the classes, and even worse, in the visitor.

Center for Software Sciences Northeastern University Example: Recursive Lets Recursive Let adds bound variables to scope before checking bodies. Let has boolean variable indicating whether it is recursive. Strategy: –First calculate variables bound by Let –Add these to scope at appropriate time during processing

Center for Software Sciences Northeastern University In Java Separate traversal, launched from before(Let) collects bound vars, with separate visitor. The new traversal doesn't enter bodies of Let or Bindings. administrator: so this example needs two traversals, as if we enter the bodies of bindings, we’re hosed. administrator: so this example needs two traversals, as if we enter the bodies of bindings, we’re hosed.

Center for Software Sciences Northeastern University Recursive Lets in Java class UDVisitor extends Visitor {... void before(Let host) { oldscope.push(inscope.copy()); oldbound.push(boundbylet.copy()); GBVisitor gbv = new GBVisitor(); host.traverse_bindings(gbv); boundbylet = gbv.boundbylet; if (host.rec) inscope.add_from(boundbylet); } void middle(Let host) { if (!host.rec) inscope.add_from(boundbylet); } void after(Let host) { inscope = oldscope.pop(); boundbylet = oldbound.pop(); }

Center for Software Sciences Northeastern University GBVisitor class GBVisitor extends Visitor { Vector boundbylet = new Vector(); void before(Bind host) { boundbylet.addElement(host.var); }

Center for Software Sciences Northeastern University Java Critique The traversal_bindings is hard-coded in visitor, mixing navigation and behavior. We need another visitor, as we do not want to calculate bindings twice, and the before, middle, after interface is to simple to allow two different modes of use for visitor administrator: and although we would get the same result with just two travs (by duplicating binding collection), that would be inefficient. administrator: and although we would get the same result with just two travs (by duplicating binding collection), that would be inefficient.

Talk Outline Introduction Traversal Specifications –compare to std traversal –overview –running example –influencing navigation Done right with Traversal Specifications –Thunks –Calling other traversals Center for Software Sciences Northeastern University

Center for Software Sciences Northeastern University In Traversal Specs Named visits make it easy to use the same (extended) visitor with both traversals. Auxiliary traversal is called from main traversal, not visitor. Thunks make objects out of Traversal Snippets, allowing the visitor to decide which option to use, without knowledge of navigational details.

Center for Software Sciences Northeastern University Traversal Spec traversal unbound(Vector scope) = Program => traverse exp (scope) VarRef => visit checkvarref(scope) Let => Vector bnd = othertrav genbnd bindings() Vector newscope = visit concat(scope,bnd) Thunk_void reccase = Thunk { Vector traverse bindings(newscope) } Thunk_void normcase = Thunk { Vector traverse bindings(scope) } visit choosecase(reccase,normcase) traverse exp(newscope) Binding => Vector bnd = traverse bindings(scope) traverse exp(scope) Empty => visit emptybound Fun => visit addvar(scope) traverse exp(scope) visit remvar(scope)

Center for Software Sciences Northeastern University Traversal Spec, cont and getbnd() = Binding => Vector bnd = traverse bindings() visit addtobound(bnd) bnd Empty => visit emptybound class UBVisitor implements unbound_visitor, getbnd_visitor {... void choosecase(Let Host, Thunk_void reccase, Thunk_void normcase) { if (host.rec) reccase.invoke(); else normacase.invoke(); }

Center for Software Sciences Northeastern University Dynamically Controlling Behavior Thunks encode current object and a list of actions for later invocation. Thunks are first class objects that can be passed as arguments to visitors. The return type of the Thunk's actions determines the type of the Thunk.

Center for Software Sciences Northeastern University The interfaces generated interface getbnd_visitor { Vector emptybound(Empty host, Vector bound); void addtobound(Binding host, Vector bound); } interface unbound_visitor { void checkvarref(VarRef host, Vector scope); Vector concat(Let host,Vector a,Vector b); void addvar(Fun host,Vector scope); void remvar(Fun host,Vector scope); void choosecase(Let Host, Thunk_void reccase, Thunk_void normcase); }

Center for Software Sciences Northeastern University Thunks Close over (for reading only) traversal variables at definition site. Can be nested, and can be returned from traversals and other thunks. Are typed by return value. Are first class objects.

Center for Software Sciences Northeastern University Uses of Thunks Allow complicated behaviors to be programmed –Traversals over cyclic structures –Fixpoint iterations –Searches in Binary Trees

Center for Software Sciences Northeastern University Calling Other traversals Could just invoke a traversal in the standard way from a visitor, but then navigational details enter the visitor. othertrav allows mutually recursive traversals to be written to share the same visitor. Used this to collect the bound variables

Center for Software Sciences Northeastern University Encoding State in Traversals We have mutually recursive traversals This allows us to encode traversal state in traversal name –When we entered Let, we were in “check” state, then entered “collect” state to get bindings, and then exited that to “check” state again. –We don’t have tail recursion of course.

Center for Software Sciences Northeastern University Inlaws use traversal to encode state

Talk Outline Introduction Traversal Specifications –compare to std traversal –overview –running example –influencing navigation –Wrapping up modifying the class graph Miscellania Center for Software Sciences Northeastern University

Center for Software Sciences Northeastern University Class Graph Modifications For all but the most trivial modifications to the class graph, the traversal specification must be modified as well. Traversals can only express local knowledge, so the modifications will be local as well.

Center for Software Sciences Northeastern University Other abilities –Traversing Enumerations Visitor must provide combine method to combine traversal results for each element to one result for whole traversal. –Traversing Results Can invoke traversal on values returned from visits or traversals, not just parts of the host object –Traverse TraversalEntries for superclasses. Extend behavior

Center for Software Sciences Northeastern University Traversing Collections We have a for-each Action which allows us to iterate over collections Impacts on visitors.

Center for Software Sciences Northeastern University Traversing Variables We want to traverse dynamically generated objects So we are able to traverse not only parts but also results from visitors.

Center for Software Sciences Northeastern University Super classes The super directive allows a traversal entry to invoke the next most precise Traversal Entry; the Traversal Entry defined on its super class. The root has a do-nothing entry implicitly, so there is always something to invoke.

Center for Software Sciences Northeastern University Semantic Issues In this section we describe the semantics of the Traversal Specifications

Center for Software Sciences Northeastern University Type Checking Need simple type checking to alleviate explicit type annotation

Center for Software Sciences Northeastern University Most Precise Wrapper Can't use inheritance; not in class graph Dynamically search object

Center for Software Sciences Northeastern University Translation to Java Details

Talk Outline Introduction Traversal Specifications Conclusion –future work –summary Center for Software Sciences Northeastern University

Center for Software Sciences Northeastern University Conclusion Future work –Investigate Multiple Extension Dimensions Visitors can be subclasses TraversalEntries for superclasses Traversal should be extensible Summary –We allow Traversals and Visitors to have more detailed interface to visitors be more recursive -- return results, pass args Behavior needs to influence Navigation