[ 1 ] Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009 ETAPS/LDTA, York Banana Algebra: Jacob Andersen [ ] Aarhus University Claus Brabrand [ ] IT University of Copenhagen Syntactic Language Extension via an Algebra of Languages and Transformations

[ 2 ] Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009 ETAPS/LDTA, York Abstract We propose an algebra of languages and transformations as a means for extending languages syntactically. The algebra provides a layer of high-level abstractions built on top of languages (captured by CFGs) and transformations (captured by constructive catamorphisms). The algebra is self-contained in that any term of the algebra specifying a transformation can be reduced to a constant catamorphism, before the transformation is run. Thus, the algebra comes "for free" without sacrificing the strong safety and efficiency properties of constructive catamorphisms. The entire algebra as presented in the paper is implemented as the Banana Algebra Tool which may be used to syntactically extend languages in an incremental and modular fashion via algebraic composition of previously defined languages and transformations. We demonstrate and evaluate the tool via several kinds of extensions.

[ 3 ] Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009 ETAPS/LDTA, York Introduction: "What is a Banana?" Bananas for Language Transformation Language Extension Pattern Banana Algebra Examples Implementation Related Work Conclusion Outline

[ 4 ] Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009 ETAPS/LDTA, York What is a 'Banana' ? Datatype; "list": Banana ("sum-of-list"): Separation of recursion and evaluation Implicit recursion on input structure bottom-up re-combination of intermediate results list = Num N | Cons N * list [Num n] = n [Cons n l] = n + [l] list  N (aka. "Catamorphism" ) (| n.n, (n, l ).n+ l |)

[ 5 ] Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009 ETAPS/LDTA, York Language Transformation Bananas (statically typed): Source language: ' L S ' Target language: ' L T ' Nonterminal-typing: '  ' Reconstructors: ' c ' list = Num N | Cons N * list tree = Nil | Leaf N | Node N * tree * tree [Num n] = Leaf n [Cons n l] = Node n (Nil) [l] [list -> tree] (| L S -> L T [  ] c |) Type-check'able! L S -> L T 

[ 6 ] Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009 ETAPS/LDTA, York Banana properties: Simple (corresponds to: “simple recursion”) Safe (syntactically safe + always terminate) Efficient (linear time in size of input + output) (Expressive) (…enough for interesting extensions) Banana Algebra “for free” (16 banana ops): Modular Incremental Simple Safe Efficient (Expressive) Statically reduce: Banana Algebra (term)  Banana (term)  Banana Properties "The metafront System: Safe and Extensible Parsing and Transformation" [ Claus Brabrand | Michael Schwartzbach ] ( LDTA 2003, SCP J ) "Growing Languages with Metamorphic Syntax Macros" [ Claus Brabrand | Michael Schwartzbach ] ( PEPM 2002 )

[ 7 ] Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009 ETAPS/LDTA, York Introduction: "What is a Banana?" Bananas for Language Transformation Language Extension Pattern Banana Algebra Examples Implementation Related Work Conclusion Outline

[ 8 ] Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009 ETAPS/LDTA, York Language Extension Pattern [var V] = var V [lam V E] = lam V [E ] [app E 1 E 2 ] = app [E 1 ] [E 2 ] [zero] = lam z (var z) [succ E] = lam s [E ] [pred E] = app [E ] (lam z (var z)) Numeral extension:Lambda-Calculus: Nonterminal typing: Reconstructors: [Exp -> Exp] Exp : var Id : lam Id * Exp : app Exp * Exp : zero : succ Exp : pred Exp Exp : var Id : lam Id * Exp : app Exp * Exp 'LT''LT' 'LS''LS' ' '' ' 'c ''c ' (| L S -> L T [  ] c |) Catamorphism: Using very simple numeral encoding

[ 9 ] Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009 ETAPS/LDTA, York Algebraic Solution Exp : var Id : lam Id * Exp : app Exp * Exp Exp : zero : succ Exp : pred Exp (| ln -> l [Exp -> Exp] [zero] = lam z (var z) [succ E] = lam s [E ] [pred E] = app [E ]... |) idx + lnl ln  l ln+l  l l  ll  l

[ 10 ] Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009 ETAPS/LDTA, York Banana Algebra Languages (L): l v L \ LL \ L L + LL + L src ( X ) tgt ( X ) let v = L in L letx w = X in L Transformations (X): x w X \ LX \ L X + XX + X X XX X idx ( L ) let v = L in X letx w = X in X (| L -> L [  ] c |) {  CFG  }

[ 11 ] Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009 ETAPS/LDTA, York Algebraic Laws Idempotency of '+': Commutativity of '+': Associativity of '+': Source-identity: … L  L + L L 1 + L 2  L 2 + L 1 L 1 + (L 2 + L 3 )  (L 1 + L 2 ) + L 3 L  src(idx(L)) L  tgt(idx(L)) Target-identity:

[ 12 ] Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009 ETAPS/LDTA, York Introduction: "What is a Banana?" Bananas for Language Transformation Language Extension Pattern Banana Algebra Examples Implementation Related Work Conclusion Outline

[ 13 ] Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009 ETAPS/LDTA, York Example Revisited { Id = [a-z] [a-z0-9]* ; Exp.var : Id ; Exp.lam : "\\" Id "." Exp ; Exp.app : "(" Exp Exp ")" ; } { Exp.zero : "zero" ; Exp.succ : "succ" "(" Exp ")" ; Exp.pred : "pred" "(" Exp ")" ; } --- "l.l" "ln.l" --- let l = "l.l" in let ln = "ln.l" in idx(l) + (| ln -> l [Exp -> Exp] Exp.zero = '\z.z' ; Exp.succ = '\s.$1' ; Exp.pred = '($1 \z.z)' ; |) --- "ln2l.x" ---

[ 14 ] Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009 ETAPS/LDTA, York Numerals + Booleans l lb  l idx + lb+l  l l  ll  l lb l ln  l idx + ln+l  l ln + l+ln+lb  l …with Nums …with Bools …with Nums & Bools?

[ 15 ] Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009 ETAPS/LDTA, York Java + Repeat let java = "java.l" in let repeat = "repeat.l" in idx(java) + (| repeat -> java [Exp -> Exp, Stm -> Stm] Stm.repeat = 'do $1 while (!($2));' ; |) { Stm.repeat : "repeat" Stm "until" "(" Exp ")" ";" ; } 7 lines ! { Java... "try" Stm "catch"... Name.id : Id ; } 575 lines --- "java.l" "repeat.l" "repeat2java.x" ---

[ 16 ] Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009 ETAPS/LDTA, York Concrete vs. Abstract Syntax Exp.or : Exp1 "||" Exp ;.exp1 : Exp1 ; Exp1.and : Exp2 "&&" Exp1 ;.exp2 : Exp2 ; Exp2.add : Exp3 "+" Exp2 ;.exp3 : Exp3 ;  Exp7.neg : "!" Exp8 ;.exp8 : Exp8 ; Exp8.par : "(" Exp ")" ;.var : Id ;.num : IntConst ; Exp (with explicit assoc./prec.) : Stm.repeat = Stm.do(, Exp.exp1( Exp1.exp2( Exp2.exp3( Exp3.exp4( Exp4.exp5( Exp5.exp6( Exp6.exp7( Exp7.neg( Exp8.par( ) ))))))))) ; Abstract syntax: Stm.repeat = 'do $1 while (!($2));' ; Concrete syntax: NB: Tool supports BOTH ! (unambiguous: concrete  abstract)

[ 17 ] Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009 ETAPS/LDTA, York "FUN" Example Lambda Calculus Nums→  Unsigned arithmetic + booleans + definitions + pairs Bools→  Defs→  Pairs→  +++ Fun Literals Literals→Nums Fun grammar transform The "FUN" Language: used for Teaching Functional Programming Basically The Lambda Calculus with…: numerals, booleans, arithmetic, boolean logic, local definitions, pairs, literals, lists, signs, comparisons, dynamic types, fixed-point combinators, … (at Aarhus University)

[ 18 ] Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009 ETAPS/LDTA, York Literals Literals→Nums "FUN" Example Lambda Calculus Nums→  Unsigned arithmetic + booleans + definitions + pairs Bools→  Defs→  Pairs→  +++ Fun Fun grammar transform Signed arith→Nums Literals→Nums Fun grammar transformFunSigned GT + FunFunSigned + Component re-use

[ 19 ] Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009 ETAPS/LDTA, York "FUN" Example Lambda Calculus Nums→  Unsigned arithmetic + booleans + definitions + pairs Bools→  Defs→  Pairs→  +++ Fun GTFunSigned GT + FunFunSigned + FunCompareFunTypesafe ++ FunCompare GT + FunTypesafe GT + 245x Banana Algebra ops  4 MB Banana !

[ 20 ] Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009 ETAPS/LDTA, York "FUN" Usage Statistics Usage statistics (245x operators) in "FUN": 58x { …cfg… } Constant languages 51x "file.l" Language inclusions 28x L + L Language additions 23x v Language variables 17x (|L  L [  ] c|) Constant transformations 17x X + X Transformation additions 14x "file.x" Transformation inclusions 10x let-in Local definitions 9x idx(L) Identity transformations 8x X X Compositions 4x L \ L Language restriction 4x w Transformation variables 2x src(X) Source extractions

[ 21 ] Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009 ETAPS/LDTA, York Other Examples Self-Application (The tool on itself!): SQL embedding (in ): My-Java (endless variations): [L 1 << L 2 ] = '(L 1 \ L 2 ) + L 2 ' [X 1 << X 2 ] = '(X 1 \ src(X 2 )) + X 2 ' Stm.select = 'factor ( ) { if ( ) return ( # \+ ( ) ); }' java ( + sql) ( \ loops) o syntaxe_francais

[ 22 ] Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009 ETAPS/LDTA, York Implementation [ / ] The 'Banana Algebra' Tool: (3,600 lines of O'Caml) Uses (underlying technologies): 'dk.brics.grammar': for parsing, unparsing, and ambiguity analysis ! 'XSugar': for transformation: "concrete syntax  abstract XML syntax" 'XSLT': for transformation: "XML  XML"

[ 23 ] Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009 ETAPS/LDTA, York Introduction: "What is a Banana?" Bananas for Language Transformation Language Extension Pattern Banana Algebra Examples Implementation Related Work Conclusion Outline

[ 24 ] Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009 ETAPS/LDTA, York Related Work (I/III) Macro Systems: "The metafront System: Safe and Extensible Parsing and Transformation" [ Claus Brabrand | Michael Schwartzbach ] ( LDTA 2003, SCP J ) "Growing Languages with Metamorphic Syntax Macros" [ Claus Brabrand | Michael Schwartzbach ] ( PEPM 2002 )

[ 25 ] Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009 ETAPS/LDTA, York Related Work (II/III) Attribute Grammars: Language transformation (and extension)… …via computation on AST's (using "inherited" or "synthesized" or … attributes) E.g., Eli, JastAdd, Silver, … Rewrite Systems: Language transformation (and extension)… …via syntactic rewriting, using encodings…: gradually rewrite " S -syntax" to " T -syntax" E.g., Elan, TXL, ASF+SDF, Stratego/XT, … S  TS  T S  TS  T Both; compared to bananas: More ambitious (expressivity) No termination guarantees (safety) Transformation "indirect" (simplicity)

[ 26 ] Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009 ETAPS/LDTA, York Related Work (III/III) Functional Programming: Catas mimicked by "disciplined style" of fun. programming …aided by: Traversal functions (auto-synthesized from datatypes) Combinator libraries "Shortcut fusion" (to eliminate ' ' at compile-time) Category Theory: A lot of this work can be viewed as Category Theory: Basically ye olde issue: GPL vs. DSL

[ 27 ] Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009 ETAPS/LDTA, York Conclusion IF bananas are sufficiently: (Expressive) THEN you get…: Banana Algebra “for free” (16 banana ops): Incremental Modular Simple Safe Efficient "Niche" Statically reduce: Banana Algebra (term)  Banana (term) 

[ 28 ] Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009 ETAPS/LDTA, York BONUS SLIDES - Reduction Semantics - " Syntactic Language Extension via an Algebra of Languages and Transformations " [ Jacob Andersen | Claus Brabrand ] ( ITU Technical Report, Dec ) If you want all the details:

[ 29 ] Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009 ETAPS/LDTA, York Reduction Semantics Environments: Reduction relations: Abbreviations:...as a short-hand for: ENV L = VAR L  EXP L ENV L  ENV X  EXP L  EXP L  '  L ' ENV X = VAR X  EXP X ENV L  ENV X  EXP X  EXP X  '  X ' ,  | - L  L l ( , ,L,l)  '  L ' ( , ,X,x)  '  X ' ,  | - X  X x environment of languages environment of transformations

[ 30 ] Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009 ETAPS/LDTA, York Semantics (L) ,  l  L l [ CON L ] ,  v  L  (v) [ VAR L ] ,  L \ L'  L l l' [ RES L ] ,  L'  L l' ,  L  L l ,  L + L'  L l l' [ ADD L ] ,  L'  L l' ,  L  L l l l l ~ l' l l wfl

[ 31 ] Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009 ETAPS/LDTA, York Semantics (L) ,  src (X)  L l S [ SRC L ] ,  X  X (| l S -> l T [  ] c |) ,  tgt (X)  L l T [ TGT L ] ,  X  X (| l S -> l T [  ] c |) ,  let v=L in L'  L l' [ LET L ]  [v=l],  L'  L l' ,  L  L l

[ 32 ] Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009 ETAPS/LDTA, York Semantics (X) ,  X \ L  X x l [ RES X ] ,  L  L l ,  X  X x ,  X + X'  X x x' [ ADD X ] ,  X'  X x' ,  X  X x x x x ~ x' x ,  (| L S -> L T [  ] c |)  X (| l S -> l T [  ] c |) [ CON X ] ,  L S  L l S ,  L T  L l T (| l S -> l T [  ] c |) wfx ,  w  X  (w) [ VAR X ]

[ 33 ] Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009 ETAPS/LDTA, York Semantics (X) ,  X' X  X (| l S -> l T ' [  '  ] c' c |) [ COMP X ] ,  X'  X (| l S ' -> l T ' [  '] c' |) ,  idx (L)  X (| l -> l [id  (l)] id c (l) |) [ IDX X ] ,  letx w=X in X'  X x' [ LET L ] ,  [w=x] X'  X x' ,  X  X x ,  X  X (| l S -> l T [  ] c |) ,  L  L l l T l S ' l

[ 34 ] Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009 ETAPS/LDTA, York BONUS SLIDES - More Examples -

[ 35 ] Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009 ETAPS/LDTA, York Numeral & Boolean Extension Numeral Extension (catamorphism): Boolean Extension (catamorphism): [var V] = var [V ] [lam V E] = lam [V ] [E ] [app E 1 E 2 ] = app [E 1 ] [E 2 ] [zero] = lam z (var z) [succ E] = lam s [E ] [pred E] = app [E ] (lam z (var z)) [var V] = var [V ] [lam V E] = lam [V ] [E ] [app E 1 E 2 ] = app [E 1 ] [E 2 ] [true] = lam a (lam b (var a)) [false] = lam a (lam b (var b)) [if E 1 E 2 E 3 ] = app (app [E 1 ] [E 2 ]) [E 3 ] Exp : var Id : lam Id * Exp : app Exp * Exp : zero : succ Exp : pred Exp Exp : var Id : lam Id * Exp : app Exp * Exp Exp : var Id : lam Id * Exp : app Exp * Exp : true : false : if Exp Exp Exp Exp : var Id : lam Id * Exp : app Exp * Exp

[ 36 ] Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009 ETAPS/LDTA, York Lambda with Booleans Exp : true : false : if Exp Exp Exp (| lb -> l [Exp -> Exp] [true] = '\a.\b. a' [false] = '\a.\b. b' [if E 1 E 2 E 3 ] = '(([E 1 ] [E 2 ]) [E 3 ])' |) idx + lbl lb  l lb+l  l l  ll  l Exp : var Id : lam Id * Exp : app Exp * Exp

[ 37 ] Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009 ETAPS/LDTA, York Incremental Development let l = "l.l" in idx(l) + (| "ln.l" -> l [Exp -> Exp] Exp.zero : '\z.z' ; Exp.succ : '\x.$1' ; Exp.pred : '($1 \z.z)' ; |) let l = "l.l" in idx(l) + (| "li.l" -> l [Exp -> Exp] Exp.id : '\z.z' ; |) { Exp.zero : "zero" ; Exp.succ : "succ" Exp ; Exp.pred : "pred" Exp ; } { Id = [a-z] [a-z0-9]* ; Exp.var : Id ; Exp.lam : "\\" Id "." Exp ; Exp.app : "(" Exp Exp ")" ; } --- "ln.l" "ln2li.x" "l.l" "ln2l.x" --- let l = "l.l" in idx(l) + (| ln -> l+"li.l" [Exp -> Exp] Exp.zero : 'id' ; Exp.succ : '\x.$1' ; Exp.pred : '($1 id)' ; |) { Exp.id : "id" ; } --- "li.l" "li2l.x" --- "li2l.x" o "ln2li.x" --- "ln2l.x" ---

[ 38 ] Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009 ETAPS/LDTA, York Example cont'd Both statically reduce to same catamorphism: (| Exp.app : Exp.app($1, $2) ; Exp.lam : Exp.lam($1, $2) ; Exp.pred : Exp.app($1, Exp.lam(Id("z"), Exp.var(Id("z")))) ; Exp.succ : Exp.lam(Id("x"), $1) ; Exp.var : Exp.var($1) ; Exp.zero : Exp.lam(Id("z"), Exp.var(Id("z"))) ; |) { Id = [a-z] [0-9a-z]* ; Exp.app : "(" Exp Exp ")" ; Exp.lam : "\" Id "." Exp ; Exp.var : Id ; } { Id = [a-z] [0-9a-z]* ; Exp.app : "(" Exp Exp ")" ; Exp.lam : "\" Id "." Exp ; Exp.pred : "pred" Exp ; Exp.succ : "succ" Exp ; Exp.var : Id ; Exp.zero : "zero" ; } -> [Exp -> Exp, Id->Id]

[ 39 ] Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009 ETAPS/LDTA, York Java + Repeat Java : Exp : Exp "+" Exp... Stm : Exp ";" : "if" "(" Exp ")" Stm : "while" "(" Exp ")" Stm... Stm : "repeat" Stm "until" "(" Exp ")" ";" (| repeat -> java [Exp -> Exp, Stm -> Stm] [repeat S until (E);] = … ; |) idx + repeat java repeat  java java+repeat  java java  java 575 lines Entire extension: 7 lines !

[ 40 ] Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009 ETAPS/LDTA, York Usage Scenarios Programmers: May extend existing languages (~ syntax macros) Developers: May embed DSLs into host languages (SQL in Java) Developers (and teachers): May incrementally specify multi-layered languages Compiler writers: May rely on tool and implement only a small core (and then specify the rest externally as extensions)

[ 41 ] Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009 ETAPS/LDTA, York BONUS SLIDES - Parsing & Error Reporting -

[ 42 ] Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009 ETAPS/LDTA, York Parsing Parsing (XSugar): Variant of Earley's algorithm: O ( |  | 3 ) Can parse any context-free grammar Closed under union of languages Support for production priority Tool easily adapts to other parsing algorithms

[ 43 ] Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009 ETAPS/LDTA, York Unparsing: Canonical whitespace Ambiguity: parsing  unparsing AST L / ~ L. L.  Parsing: Grammar ambiguity AST L / ~ L. L. .

[ 44 ] Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009 ETAPS/LDTA, York Ambiguity Analysis Ambiguity Analysis: Using implementation ( ) on: Source language; Target language; and/or …all intermediate languages (somewhat expensive) (Note: Ambiguity analysis comes with XSugar tool) "Analyzing Ambiguity of Context-Free Grammars" [ Claus Brabrand | Robert Giegerich | Anders Møller ] ( CIAA 2007 ) " dk.brics.grammar " [ by Anders Møller ]

[ 45 ] Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009 ETAPS/LDTA, York Error Reporting Error reporting: Static parse-error (O'Caml-lex): Static transformation error (XSugar): (is actually a parse-error in a cata reconstructor) Dynamic parse-error (XSugar): Dynamic transformation error: impossible :-) *** In ln2l.x (4,4)-(4,7): Parse error at "Exp" *** Parse error at character 6 (line 1, column 7) in /tmp/shape84e645.txt *** Parse error at character 23 (line 1, column 24) in /dev/stdin Could be improved Prototype