1. 5/10/2015 1 Domain-Specific Embedded Languages with Boost.Proto Or, "How I Learned to Stop Worrying and Love Expression Templates"

2. copyright 2007 Eric Niebler 5/10/2015 2 Talk Overview Goal: Designing declarative domain-specific libraries using Boost.Proto. 1. Domain Specific Embedded Languages 2. DSEL Design with Boost.Proto 1. Expression construction 2. Expression evaluation 3. Expression introspection 4. Expression transformation 3. Extending Boost.Proto

3. copyright 2007 Eric Niebler 5/10/2015 3 Hello, Boost.Proto! // #includes... using namespace boost::proto; terminal ::type cout_ = { std::cout }; template void evaluate( Expr const & expr ) { default_context ctx; eval(expr, ctx); } int main() { evaluate( cout_ << "hello" << ',' << " world" ); }

4. copyright 2007 Eric Niebler 5/10/2015 4 Hello, Boost.Proto! // #includes... using namespace boost::proto; terminal ::type cout_ = { std::cout }; int main() { display_expr( cout_ << "hello" << ',' << " world\n\n" ); return 0; }

5. 5/10/2015 5 Domain-Specific Embedded Languages Or, "Operator Abuse for Fun and Profit!"

6. copyright 2007 Eric Niebler 5/10/2015 6 DSEL Example: Boost.Lambda DSEL for creating unnamed function objects (and very long compiler errors) Jaakko Järvi std::for_each( a.begin(), a.end(), std::cout << _1 << ' ' );

7. copyright 2007 Eric Niebler 5/10/2015 7 DSEL Example: Boost.Spirit Parser Generator  similar in purpose to lex / YACC DSEL for declaring grammars  grammars can be recursive  Approximates Backus-Naur Form Joel de Guzman

8. copyright 2007 Eric Niebler 5/10/2015 8 Infix Calculator Grammar In Extended Backus-Naur Form group ::= '(' expr ')' fact ::= integer | group term ::= fact (('*' fact) | ('/' fact))* expr ::= term (('+' term) | ('-' term))*

9. copyright 2007 Eric Niebler 5/10/2015 9 template struct Op {... }; template struct BinOp : Op > { L const & left; R const & right;... }; template BinOp const operator >> ( Op const & l, Op const & r ) {... } What is an Expression Template? Tree representation of an expression Built using templates and operator overloading Builds the tree representation of " expr1 >> expr2 ". BinOp<> is a container for two sub-expressions.

10. copyright 2007 Eric Niebler 5/10/2015 10 group ::= '(' expr ')' fact ::= integer | group term ::= fact (('*' fact) | ('/' fact))* expr ::= term (('+' term) | ('-' term))* Infix Calculator Grammar In Extended Backus-Naur Form In Boost.Spirit spirit::rule group, fact, term, expr; group = '(' >> expr >> ')'; fact = spirit::int_p | group; term = fact >> *(('*' >> fact) | ('/' >> fact)); expr = term >> *(('+' >> term) | ('-' >> term));

11. copyright 2007 Eric Niebler 5/10/2015 11 rule group = '(' >> expr >> ')'; What is an Expression Template? DSELs like Spirit interpret expression trees in a domain-specific way. >> ')' '(' expr BinOp< RShift, BinOp< RShift, Term, Term >, Term >

12. copyright 2007 Eric Niebler 5/10/2015 12 rule group = '(' >> expr >> ')' What is an Expression Template? DSELs like Spirit interpret expression trees in a domain-specific way. struct rule { template rule & operator= ( Op const & op ) {... magic happens here... return *this; }... }; Somehow interpret op as a grammar rule. This is hard!

13. copyright 2007 Eric Niebler 5/10/2015 13 DSELs in C++, the Sad Truth Very difficult to design Code is hard to read and maintain Terrible compiler error messages So why not a DSEL for defining DSELs?!  Expression construction  Expression evaluation  Expression introspection  Expression transformation

14. 5/10/2015 14 DSEL Design with Boost.Proto Or, “The Importance of Being Grammatically Correct"

15. copyright 2007 Eric Niebler 5/10/2015 15 A Calculator DSEL template struct placeholder : Int {}; terminal > >::type _1; terminal > >::type _2; int main() { (_1 + _2); return 0; } Expression construction expr > >, 0 > &, expr< tag::terminal, term > >, 0 > & >, 2>

16. copyright 2007 Eric Niebler 5/10/2015 16 Calculator, Reloaded // To be defined... struct calculator_context //... int main() { // Placeholders have values 1. and 2. respectively: calculator_context ctx(1., 2.); // Evaluate a calculator expression with the // calculator_context std::cout << eval(_1 + _2, ctx) << std::endl; } Expression evaluation

17. copyright 2007 Eric Niebler 5/10/2015 17 Calculator, Reloaded // This describes how to evaluate calculator expressions struct calculator_context : callable_context { double d[2]; typedef double result_type; calculator_context(double d1, double d2) { d[0] = d1; d[1] = d2; } template double operator()(tag::terminal, placeholder i) const { return d[i-1]; // handle argument placeholders } }; Expression evaluation, continued

18. copyright 2007 Eric Niebler 5/10/2015 18 The Context Concept All context types must look like this: struct Context { template struct eval { typedef unspecified result_type; result_type operator()(Expr &expr, Context &context) const; }; Helpers for building contexts  default_context / default_eval  callable_context / callable_eval

19. copyright 2007 Eric Niebler 5/10/2015 19 default_context, detail. struct default_context { template struct eval : default_eval {}; }; template //, class T = typename Expr::tag_type struct default_eval { typedef ?????? result_type; result_type operator()(Expr &expr, Ctx &ctx) const { return eval(left(expr),ctx) + eval(right(expr),ctx); } }; struct default_context { template struct eval : default_eval {}; }; template //, class T = typename Expr::tag_type struct default_eval { static Expr &sexpr; static Ctx &sctx; typedef BOOST_TYPEOF(eval(left(sexpr),sctx) + eval(right(sexpr),sctx)) result_type; result_type operator()(Expr &expr, Ctx &ctx) const { return eval(left(expr),ctx) + eval(right(expr),ctx); } };

20. copyright 2007 Eric Niebler 5/10/2015 20 callable_context, detail. template struct callable_context { template struct eval : callable_eval {}; }; template //, int I = Expr::proto_arity struct callable_eval { typedef typename Derived::result_type result_type; result_type operator()(Expr &expr, Derived &ctx) const { return ctx(typename Expr::tag_type(), arg_c (expr), arg_c (expr)); } };

21. copyright 2007 Eric Niebler 5/10/2015 21 Expression Introspection Query the properties of expression trees at compile time Utilities for defining grammars proto::matches<> for grammar- checking your expressions. “Be grammatical!"

22. copyright 2007 Eric Niebler 5/10/2015 22 An Input/Output DSEL // Terminals for lazy input/output expressions: terminal ::type cin_ = { std::cin }; terminal ::type cout_ = { std::cout }; template void do_io( IO const & io ) { eval( io, default_context() ); } int main() { int i = 0; do_io( cin_ >> i ); // reads an int do_io( cout_ << i ); // writes an int } What if do_io() needs to know whether IO represents an input or an output operation?

23. copyright 2007 Eric Niebler 5/10/2015 23 Answer: Input/Output Patterns // A pattern to match Input expressions struct Input : shift_right, _ > {}; // A pattern to match Output expressions struct Output : shift_left, _ > {}; template void do_io(IO const &io) { if( matches ::value ) std::cout << "Input!\n"; if( matches ::value ) std::cout << "Output!\n"; //... } Evaluated at compile time! Matches " cin_ >> i " Matches " cout_ << i "

24. copyright 2007 Eric Niebler 5/10/2015 24 Whoops! A Problem... // A pattern to match Output expressions struct Output : shift_left, _ > {}; Why doesn't this expression match the Output pattern? cout_ << 1 << 2; shift_left< terminal ::type, terminal ::type >::type, terminal ::type >::type ( cout_ << 1 ) << 2;

25. copyright 2007 Eric Niebler 5/10/2015 25 Solution: A Recursive Pattern // A pattern to match Output expressions struct Output : or_< shift_left, _ >, shift_left > {}; or_<> alternates tried in order Short-circuit evaluation Notice recursion on Output pattern! // A pattern to match Output expressions struct Output : or_< shift_left, _ >, shift_left > {};

26. copyright 2007 Eric Niebler 5/10/2015 26 From Patterns to Grammars struct Calculator; struct Double : terminal {}; struct Placeholder : terminal > {}; struct Plus : plus {}; struct Minus : minus {}; struct Multiplies : multiplies {}; struct Divides : divides {}; struct Calculator : or_< Double, Placeholder, Plus, Minus, Multiplies, Divides > {}; A Proto Grammar for our Calculator:

27. copyright 2007 Eric Niebler 5/10/2015 27 Proto vs. EBNF Are Proto grammars as powerful as EBNF? group ::= '(' expr ')' fact ::= double | group term ::= fact (('*' fact) | ('/' fact))* expr ::= term (('+' term) | ('-' term))* struct Expr; struct Dbl : terminal {}; struct Plus : plus {}; struct Minus : minus {}; struct Mult : multiplies {}; struct Div : divides {}; struct Expr : or_ {}; EBNFProto Why don’t Proto grammars need: Sequencing? Repetition? Precedence? Associativity?

28. copyright 2007 Eric Niebler 5/10/2015 28 What are grammars good for? // A function that only accepts valid calculator expressions! template typename enable_if >::type evaluate( Expr const & expr ) { //... } enable_if<> tricks to prune overload sets mpl::if_<> for type selections Compile-time assertions Defining tree transformations Controlling Proto's operator overloading "It's the Grammar, stupid."

29. copyright 2007 Eric Niebler 5/10/2015 29 Expression Transformation Turns an expression tree into some other object Transformations attached to grammar rules Transformations provided by Proto or user defined Challenge! For any calculator expression, find the arity. Eg.: _1 + 42 has arity 1 _2 * 42 has arity 2

30. copyright 2007 Eric Niebler 5/10/2015 30 Arity Calculation ExpressionArity double0 _11 _22 Left op Rightmax(arity(Left), arity(Right)) Calculator expression arity is calculated as follows:

31. copyright 2007 Eric Niebler 5/10/2015 31 From Grammars to Transforms // Ye olde calculator grammar struct Arity; struct Double : terminal {}; struct Placeholder : terminal > {}; struct Plus : plus {}; struct Minus : minus {}; struct Multiplies : multiplies {}; struct Divides : divides {}; struct Arity : or_< Double, Placeholder, Plus, Minus, Multiplies, Divides > {};

32. copyright 2007 Eric Niebler 5/10/2015 32 // Ye olde calculator grammar struct Arity; struct Double : terminal {}; struct Placeholder : terminal > {}; struct Plus : plus {}; struct Minus : minus {}; struct Multiplies : multiplies {}; struct Divides : divides {}; struct Arity : or_< Double, Placeholder, Plus, Minus, Multiplies, Divides > {}; Double Transform struct Double : terminal {}; struct Arity : or_< case_ () >,... > {}; Anything that matches Double is transformed into mpl::int_

33. copyright 2007 Eric Niebler 5/10/2015 33 Placeholder Transform template struct placeholder : Int {}; terminal > >::type _1; struct Placeholder : terminal > {}; struct Arity : or_<..., case_,... > {}; Proto’s _arg transform pulls the placeholder<> argument out of the terminal<>. And a placeholder<> IS-A compile-time integer representing its arity.

34. copyright 2007 Eric Niebler 5/10/2015 34 Binary Operator Transform // A Lambda for evaluating the max arity of a binary // calculator expression. typedef mpl::max MaxArity; struct Arity : or_<..., case_, MaxArity() >,... > {}; Anything that matches plus is transformed according to the MaxArity() transform.

35. copyright 2007 Eric Niebler 5/10/2015 35 Fun with C++ Declaration Syntax What are all the things that this can mean? X( Y ) Function call Constructor call Function type Function-style cast

36. copyright 2007 Eric Niebler 5/10/2015 36 mpl::max () Lambda Transforms Step 1: Recursively evaluate nested lambdas mpl::max () mpl::max, mpl::int_ >() Evaluate left and right sub- expressions according to the Arity transform.

37. copyright 2007 Eric Niebler 5/10/2015 37 mpl::max, mpl::int >() mpl::max, mp::int >::type() mpl::int_ () Lambda Transforms Step 2: Evaluate meta-function ` Look for a nested ::type typedef. `

38. copyright 2007 Eric Niebler 5/10/2015 38 mpl::int_ () Lambda Transforms Step 3: Call object constructor The function type is interpreted as a constructor invocation. Proto returns a default constructed mpl::int_ !

39. copyright 2007 Eric Niebler 5/10/2015 39 The Complete Transform struct Arity; struct Double : terminal {}; struct Placeholder : terminal > {}; struct Plus : plus {}; struct Minus : minus {}; struct Multiplies : multiplies {}; struct Divides : divides {}; typedef mpl::max MaxArity; struct Arity : or_ () >, case_ > {};

40. copyright 2007 Eric Niebler 5/10/2015 40 It works! // Defined as before struct Arity... ; // Display the arity of a calculator expression template void print_arity(Expr const & expr) { BOOST_MPL_ASSERT((matches )); int i = 0; // dummy, not used std::cout << Arity::call(expr, i, i) << std::endl; } int main() { print_arity(_1 + 42); print_arity(_2 * 42); }

41. 5/10/2015 41 Extending Proto Giving Proto expressions extra smarts.

42. copyright 2007 Eric Niebler 5/10/2015 42 Calculator, Reloaded // To be defined... struct calculator_context //... int main() { // Placeholders have values 1. and 2. respectively: calculator_context ctx(1., 2.); // Evaluate a calculator expression with the calculator_context std::cout << eval(_1 + _2, ctx) << std::endl; } Expression evaluation flashback Q: Is eval() always needed to make proto expressions do something?

43. copyright 2007 Eric Niebler 5/10/2015 43 Calculator, Reloaded int main() { double data[] = { 1., 2., 3., 4. }; // Use the calculator DSEL to square each element std::transform( data, data + 4, data, _1 * _1 ); } A Calculator expression as an STL functor? Whoops! " _1 * _1 " doesn't have an operator() that takes and returns a double.

44. copyright 2007 Eric Niebler 5/10/2015 44 Calculator, Revolutions struct calc_domain; template struct calc : extends, calc_domain > { calc( Expr const &expr = Expr() ) : extends, calc_domain >( expr ) {} typedef double result_type; result_type operator()( double d1 = 0.0, double d2 = 0.0 ) const { calculator_context ctx( d1, d2 ); return eval( *this, ctx ); } }; Expression wrappers to add behaviors The "domain" of our wrapped expressions calc extends Expr Define operator() to evaluate expr with calculator_context

45. copyright 2007 Eric Niebler 5/10/2015 45 Calculator, Revolutions // OK, _1 and _2 are calc<> expressions in the calc_domain calc > >::type > _1; calc > >::type > _2; Wrap our terminals... struct calc_domain : domain > {}; Hook Proto's expression generator... In calc_domain, wrap all expression types in our calc<> wrapper.

46. copyright 2007 Eric Niebler 5/10/2015 46 Calculator, Revolutions It works!

47. copyright 2007 Eric Niebler 5/10/2015 47 Summing up... Proto makes C++ DSEL design easy and fun! Good for expression template... ... construction ... evaluation ... introspection ... transformation Powerful extensibility mechanisms Used by xpressive, Spirit-2, Karma, etc... Available in Boost Subversion and the File Vault

48. 5/10/2015 48 Questions?