The Evolution of the StringTemplate Engine Presented to the UC Berkeley Harmonia group 15 December 2004 Terence Parr University of San Francisco

Overview Introduction Introduction what is StringTemplate? what is StringTemplate? why did I build it? why did I build it? the nature of text generation the nature of text generation Design goals Design goals The evolution; 4 phases The evolution; 4 phases Experience Experience Lessons Lessons

What is StringTemplate Template engine designed with Tom Burns (CEO, jGuru.com) while building commercial sites over many years in response to JSP Template engine designed with Tom Burns (CEO, jGuru.com) while building commercial sites over many years in response to JSP small: 170k uncompressed binary (w/o test rig) small: 170k uncompressed binary (w/o test rig) Evolved from simple “document with holes” into a functional language capable of generating large class of languages Evolved from simple “document with holes” into a functional language capable of generating large class of languages Well suited to generative programming as well as dynamic page generation; being used in new ANTLR parser generator for code generation Well suited to generative programming as well as dynamic page generation; being used in new ANTLR parser generator for code generation Distinguishing characteristic: strictly enforces separation of model and view Distinguishing characteristic: strictly enforces separation of model and view

Canonical Operations Attribute reference: Attribute reference: Template references (possibly recursive): Template references (possibly recursive): Apply template to multi-valued attribute: or ;}> Apply template to multi-valued attribute: or ;}> Conditional include: extends Name: Guest Conditional include: extends Name: Guest

Example* Translated “Program Manipulation via Interactive Transformations” example to actual StringTemplate template definition: Translated “Program Manipulation via Interactive Transformations” example to actual StringTemplate template definition: Integration with existing code clean, easy Integration with existing code clean, easy generateMemSize(node) ::= << int mem_size() { int result = 0;.mem_size();}> return result; >> *shameless StringTemplate plug

Semantics Side-effect free expressions Side-effect free expressions No “state” No “state” No defined order of execution No defined order of execution Lazy evaluation Lazy evaluation Template inheritance group to subgroup Template inheritance group to subgroup Dynamically scoped; values inherited Dynamically scoped; values inherited Recursive template instantiation Recursive template instantiation

Design Goals Optimized for enforcement of separation not for Turing completeness nor expressive “one-liners” Optimized for enforcement of separation not for Turing completeness nor expressive “one-liners” Conceptual integrity Conceptual integrity single-minded fanaticism against entanglement single-minded fanaticism against entanglement stick to a few concepts: attributes and templates stick to a few concepts: attributes and templates consistency; no special cases; driven by design not implementation details consistency; no special cases; driven by design not implementation details proper language formalisms, semantics, sane syntax proper language formalisms, semantics, sane syntax Simple as possible Simple as possible useful to nonprogrammers useful to nonprogrammers makes it fast, easy to build/maintain makes it fast, easy to build/maintain Powerful as possible Powerful as possible

Phase I, “Document with holes”; only tag refs “Document with holes”; only tag refs Could nest templates by setting a tag value to template; dynamic scoping (“value inheritance”) Could nest templates by setting a tag value to template; dynamic scoping (“value inheritance”) Not intended for full page generation Not intended for full page generation File of template definitions loaded into hash table; code manually created template instances File of template definitions loaded into hash table; code manually created template instances INSERT INTO Topics VALUES (, ' ' );

Phase II, Rewrite of jGuru.com; learned entanglement allowed by JSP is horrible! Rewrite of jGuru.com; learned entanglement allowed by JSP is horrible! Decided to enhance StringTemplate, letting needs dictate feature set; would not be Turing- complete! Decided to enhance StringTemplate, letting needs dictate feature set; would not be Turing- complete! Moved to single template per file format with $$…$$ indicating attribute ref Moved to single template per file format with $$…$$ indicating attribute ref Needed “include” Needed “include” added template reference rather than #include, macro added template reference rather than #include, macro subconciously supported recursion subconciously supported recursion

Phase II, (Cont’d) Needed to walk multi-valued attributes; added “apply” syntax: Needed to walk multi-valued attributes; added “apply” syntax: $names:bold(item=names)$ $names:bold(item=names)$ $names:bold(item=names, separator=“, “)$ $names:bold(item=names, separator=“, “)$ Allowed anonymous templates Allowed anonymous templates $names:” $names$ ”$ $names:” $names$ ”$ Iterated value naming convention evolved Iterated value naming convention evolved $names:bold(item=names[i])$ $names:bold(item=names[i])$ Needed multiple array walk for relational DB Needed multiple array walk for relational DB $a,b:foo(x=a[i], y=b[i])$ $a,b:foo(x=a[i], y=b[i])$ Finally, needed conditional inclusion Finally, needed conditional inclusion $if(userName)$ Name: $userName$ $endif(userName)$

Phase III, 2003-Summer 2004 Wanted to say $names:bold()$ w/o arg Wanted to say $names:bold()$ w/o arg defined default attribute attr defined default attribute attr Cleaned up IF: $if(foo)$…$endif$ Cleaned up IF: $if(foo)$…$endif$ Nested anonymous blocks: $names:{…}$ Nested anonymous blocks: $names:{…}$ Round-robin template application Round-robin template application $users: rowOdd(), rowEven()$ $users: rowOdd(), rowEven()$ Properties: $user.name$ calls user.getName() Properties: $user.name$ calls user.getName() Indirect template ref: $(tname)()$ ; added in context of “immediate evaluation” Indirect template ref: $(tname)()$ ; added in context of “immediate evaluation” Template inheritance via StringTemplateGroup Template inheritance via StringTemplateGroup

Phase IV, 2004 Added group file format: mutually- referential templates with formal arguments Added group file format: mutually- referential templates with formal arguments method(type,name,args,body) ::= << public ( ) { } >> assign(lhs,expr) ::= “ = ;” …

Future Named args for anonymous blocks ala Smalltalk Named args for anonymous blocks ala Smalltalk $list:{x | $x$ }$ $list:{x | $x$ }$.mem_size();}> iterator could also choose single argument from named templates iterator could also choose single argument from named templates Add parallel array walking back in Add parallel array walking back in Syntax for super group? Syntax for super group? group sub : super; group sub : super; Lots of little clean up; e.g., whitespace, … Lots of little clean up; e.g., whitespace, …

Experience (ANTLR v3.0) Tree walker (controller) collects data from AST (model), pushes data into templates (view) Tree walker (controller) collects data from AST (model), pushes data into templates (view) Decouples order of computation from order of output (this is huge) Decouples order of computation from order of output (this is huge) Enforced separation guarantees easy retargeting, no code duplication, … Enforced separation guarantees easy retargeting, no code duplication, … no code in template no code in template no output strings in code generator no output strings in code generator Previous code generator hopelessly entangled Previous code generator hopelessly entangled Group file format (output grammar) is great! “Executable documentation” Group file format (output grammar) is great! “Executable documentation”

Practical Lessons Separate pattern matching from templates! Separate pattern matching from templates! don’t make templates also analyze model don’t make templates also analyze model lazy evaluation essential; decouple order of computation from order of output; build in any order lazy evaluation essential; decouple order of computation from order of output; build in any order Isolates templates (“user code”) from model changes, which will surely evolve Isolates templates (“user code”) from model changes, which will surely evolve Templates may be reusable w/o embedded logic Templates may be reusable w/o embedded logic Recursion required for nested structures Recursion required for nested structures Attributes Attributes dynamic scoping really handy dynamic scoping really handy push don’t pull attributes; attributes are inert push don’t pull attributes; attributes are inert Simple language is fast, easy to learn, easy to use, promotes retargetability Simple language is fast, easy to learn, easy to use, promotes retargetability

Philosophical Lessons Say what you mean! e.g., no FOR-loops Say what you mean! e.g., no FOR-loops Focus on principles, conceptual integrity Focus on principles, conceptual integrity focus on what not how focus on what not how other tools either doc with holes or Turing complete other tools either doc with holes or Turing complete Difficult to stick to your principles and still create usable tool; grey areas appear Difficult to stick to your principles and still create usable tool; grey areas appear Language design is hard; implementation is relatively easy Language design is hard; implementation is relatively easy Selection pressure results in StringTemplate Selection pressure results in StringTemplate output conforms to a language, hence, a grammar output conforms to a language, hence, a grammar strict enforcement of separation leads to grammar strict enforcement of separation leads to grammar

Demo Generating Java / XML with same model Generating Java / XML with same model