1. A Theory of Hygienic Macros David Herman, Mitchell Wand Northeastern University

2. 2 A Theory of Hygienic Macros  Introduction to macros and hygiene  The challenge of formalizing hygiene  m and its properties

3. 3 In the beginning MACDEF SUM A,B,C LOAD A ADD B STORE C ENDDEF

4. 4 The essence of macros  Derived (user-defined) syntactic constructs  Defined by rewriting to existing constructs  Translated at compile-time (“macro expansion”) repeat s until e  begin s ; while (not e) s end end

5. 5 The essence of macros  Derived (user-defined) syntactic constructs  Defined by rewriting to existing constructs  Translated at compile-time (“macro expansion”) (repeat s until e)  (begin s (while (not e) s))

6. 6 Example: short-circuit or (or (foo "bar") (baz 42)) (define-macro (or e1 e2) (with (t e1) (if t t e2))) evaluate only if (foo "bar") produces #f evaluate first; evaluate only once pattern; matches subexpressions e1 and e2 template; rewrites expression using e1 and e2

7. 7 The power of macros  Abstractions for control, scope, program structure  Powerful meta-programming idioms  Simplifies language core (“extensible languages”) functions, recursion, iteration, primitive datatypes local bindings, custom loop forms, test harnesses, data conditionals, constructors, regular expressions, language designer library writer little lang- uages, … short-circuit operators,

8. 8 A brief history of hygiene

9. 9 Naïve macro expansion (define-macro (or e1 e2) (with (t e1) (if t t e2))) (with (x 1) (or #f x))  (with (x 1) (with (t #f) (if t t x))) (with (t 1) (or #f t))  (with (t 1) (with (t #f) (if t t t))) 

10. 10 Naïve macros are leaky abstractions (define-macro (or e1 e2) (with (t e1) (if t t e2))) A leaky abstraction is a brittle abstraction. Please don’t refer to any variables named t in expression e2 ! Oh, and don’t use or in a context that shadows the names with or if !

11. 11 For reasons of hygiene it will always be assumed that the bound variables that occur in a certain expression are different from the free ones. This can be fulfilled by renaming bound variables. —H. P. Barendregt Kohlbecker: This calls for hygiene

12. 12 Hygienic macro expansion (define-macro (or e1 e2) (with (t e1) (if t t e2))) (with (x 1) (or #f x))  (with (x 1) (with (t #f) (if t t x))) (with (t 1) (or #f t))  (with (t 1) (with (t ′ #f) (if t ′ t ′ t)))

13. 13 Hygiene and syntactic abstraction functions, recursion, iteration, primitive datatypes libraries functions, recursion, iteration, primitive datatypes local bindings, custom loop forms, test harnesses, data conditionals, constructors, regular expressions, little lang- uages, … short-circuit operators, language

14. 14 Hygiene and syntactic abstraction functions, recursion, iteration, primitive datatypes custom loop forms, derived datatypes, first-class module systems, object- orientation, test harnesses, data constructors, laziness, short-circuit boolean operators, iteration protocols, static analyses, parser generators, partial evaluation, debuggers, coroutines, exceptions, threads, type checkers, introspection, etc… local bindings, software contracts, language libraries

15. 15 What do we know about hygiene?

16. 16 Hygiene by example (define-macro (or e1 e2) (with (t e1) (if t t e2))) (with (x 1) (or #f x))  (with (x 1) (with (t #f) (if t t x))) (with (t 1) (or #f t))  (with (t 1) (with (t #f) (if t t t)))

17. 17 Hygiene by informal rules Kohlbecker et al, ’86: Generated identifiers that become binding instances in the completely expanded program must only bind variables that are generated at the same transcription step. Clinger and Rees, ’91: 1. It is impossible to write a macro that inserts a binding that can capture references other than those inserted by the macro. 2. It is impossible to write a macro that inserts a reference that can be captured by bindings other than those inserted by the macro. SRFI 72: A binding for an identifier can only capture a reference to another if both were present in the source or introduced during a single evaluation of a syntax or quasisyntax form, with the understanding that the evaluation of any nested, unquoted syntax or quasisyntax forms counts as part of the evaluation of an enclosing quasisyntax.

18. 18 Hygiene by implementation

19. 19 But which implementation? ? Kohlbecker et al, ’86Clinger & Rees ’91Dybvig et al, ’93Van Tonder, ’05

20. 20 Insight #1: Lexical scope (define-macro (or e1 e2) (with (t e1) (if t t e2))) (with (x 1) (or #f x))  (with (x 1) (with (t #f) (if t t x))) (with (t 1) (or #f t))  (with (t 1) (with (t ′ #f) (if t ′ t ′ t))) == ==

21. 21 Hygienic macro expansion preserves  -equivalence. The essence of hygienic macros

22. 22 What variables are bound by with ? (define-macro (with (a e1) e2) ) (with (x 1) (or #f x)) (with (t 1) (or #f t)) == …

23. 23 What is the binding structure? (define-macro (murky a e) (begin (set! a e) (lambda (a) e))) (lambda (foo) (murky foo (+ foo 1)))

24. 24 What is the binding structure? (define-macro (turing-machine e) )

25. 25 What is the binding structure? (define-macro (indecisive a e) … (set! a e) … (lambda (a) e)) (lambda (foo) (indecisive foo (+ foo 1)))

26. 26 What variables are bound by with ? (define-macro (with (a e1) e2) …) ) Standard answer: look at results of expansion (with (x 1) (or #f x))  ((lambda (x) ((lambda (t) (if t t x)) #f) 1) (with (t 1) (or #f t))  ((lambda (t) ((lambda (t ′ ) (if t ′ t ′ t)) #f) 1) …

27. 27 Using expansion to define scope  To specify hygienic expansion, we need to know the binding structure (scope) of programs.  To specify the binding structure of programs, we need to know the results of hygienic expansion. Oops.

28. 28 Inside every large language is a small language struggling to get out… —Igarashi, Pierce, and Wadler

29. 29 Insight #2: Specifying lexical scope

30. 30 What variables are bound by with ? (define-macro (with (a e1) e2) …) ) In practice, programmers write binding specifications: with takes the form (with (a e1) e2) and produces an expression, where: a is an identifier e1 and e2 are expressions a is bound in e2 …

31. 31 A shape type for with (with ( expr ) expr a ) → expr binder expression in same environment expression in scope of a

32. 32 Explicitly type-annotated macros let val x = 1 in … x … end (with (x 1) … x …) (define-macro (with (a e1) e2) : expr ((a : var) (e1 : expr) (e2 : expr a)) ) … …

33. 33 Shape-directed  -conversion (with ( expr ) expr a ) → expr (with (x 1) (or #f x)) (with (t 1) (or #f t)) ==

34. 34 Shape-directed  -conversion (with ( expr ) expr a ) → expr (with (z 1) (or #f z)) (with (t 1) (or #f t)) ==

35. 35 Shape-directed  -conversion (with ( expr ) expr a ) → expr (with (z 1) (or #f z)) (with (z 1) (or #f z)) ==

36. 36 Quick recap  Macro shape types  binding structure  Binding structure   -equivalence   -equivalence  correctness of hygienic expansion So, what of hygienic expansion?

37. 37 m : a model of hygienic macros (with (x 1) x) ( x.x ) 1 S-expressions compiler’s core language

38. 38 m : a model of hygienic macros e ::=var | var.e | e e | let syntax var = m in e end | ( op s … ) core language source

39. 39 Hygienic macro expansion let syntax x = m in e end →  e[m/x] if BV(e) ∩ FV(m) =  ( ( macro p => e) s … ) →   (e) if  (p) = s for some substitution  and BV(s) ∩ FV(e) =  and BV(e) # s

40. 40 Properties of m

41. 41 Confluence (with (x 1) (or #f x)) ( x.(with (t #f) (if t t x)) ) 1 ( x.(or #f x) ) 1 (with (x 1) (with (t #f) (if t t x)))    

42. 42 Hygiene For (well-typed) e 0 =  e ′ 0 and fully-expanded e and e ′, if e 0 →  * e and e ′ 0 →  * e ′, then e =  e ′. e0e0 e′0e′0   e e′e′ ** **

43. 43 Hygiene…from confluence   ** ** s s1s1 s′1s′1 s′s′

44. 44 Hygiene…from confluence   ** ** s0s0 s1s1 s′1s′1 s2s2 s′0s′0 s′2s′2  

45. 45 Contributions  Semantics for hygienic expansion, accommodating:  Lexically scoped macros  Binding forms of arbitrary shape  Formal definitions and proofs   -equivalence in presence of macros  Shape-directed  -conversion  Confluence and hygiene

46. 46 What we’ve gained  Formalized programmers’ mental model of the binding structure of macros.  Showed the model has good theoretical properties.  Validated intuition: hygienic macro expansion frees the programmer to  -convert with impunity.  But—more features yet to model…

47. 47 Thank you. Hygienic macro expansion preserves  -equivalence. Moral: