A Theory of Hygienic Macros David Herman Northeastern University

2 The power of macros Derived (user-defined) syntactic constructs Defined by rewriting to existing constructs Translated at compile-time (define-macro (increment! a) (set! a (+ a 1)))

3 The power of macros Abstractions for scope, control, program structure Powerful meta-programming idioms Allow for a minimal language core functions, recursion, iteration, primitive datatypes custom loop forms, derived datatypes, first-class module systems, object- orientation, test harnesses, data constructors, laziness, short-circuit boolean operators, iteration protocols, static analyses, parser generators, partial evaluation, debuggers, coroutines, exceptions, threads, type checkers, introspection, etc… local bindings, software contracts, language designer language user

4 The need for hygiene (define-macro (or e1 e2) (with ([t e1]) (if t t e2))) (with ([x 1]) (or #f x)) => (with ([x 1]) (with ([t #f]) (if t t x))) (with ([t 1]) (or #f t)) => (with ([t 1]) (with ([t #f]) (if t t t)))

5 The need for hygiene (define-macro (or e1 e2) (with ([t e1]) (if t t e2))) (with ([x 1]) (or #f x)) => (with ([x 1]) (with ([t #f]) (if t t x))) (with ([t 1]) (or #f t)) => (with ([t 1]) (with ([t ′ #f]) (if t ′ t ′ t)))

6 What are hygienic macros?

7 (define-macro (or e1 e2) (with ([t e1]) (if t t e2))) (with ([x 1]) (or #f x)) => (with ([x 1]) (with ([t #f]) (if t t x))) (with ([t 1]) (or #f t)) => (with ([t 1]) (with ([t #f]) (if t t t)))

8 What are hygienic macros?

9 KFFD ’86: Generated identifiers that become binding instances in the completely expanded program must only bind variables that are generated at the same transcription step. i.e., not provided as the macro’s input i.e., a single rewriting of a macro application

10 What are hygienic macros? Clinger and Rees ’91: 1. It is impossible to write a macro that inserts a binding that can capture references other than those inserted by the macro. 2. It is impossible to write a macro that inserts a reference that can be captured by bindings other than those inserted by the macro. again: not provided as the macro’s input

11 Why isn’t this enough?

12 Exotic macros in Scheme Impossible with hygienic macros, right? (loop (begin (when (prime? i) (break i)) (increment! i)))

13 Exotic macros in Scheme Contrast with the explicit version: (loop/explicit break (begin (when (prime? i) (break i)) (increment! i))) provided as input

14 Exotic macros in Scheme Petrofsky extraction: (loop (begin (when (prime? i) (break i)) (increment! i)))

15 Exotic macros in Scheme Dumpster diving: (loop (begin (when (prime? i) (break i)) (increment! i)))

16 Exotic macros in Scheme Dumpster diving: (loop/explicit break (begin (when (prime? i) (break i)) (increment! i)))

17 Exotic macros in Scheme (define-macro (murky a e) (begin (set! a e) (lambda (a) e))) (let ([foo 10]) (murky foo (+ foo 1)))

18 Exotic macros in Scheme (define-macro (indecisive ([a e]) body) (if-zero …complicated computation… ((lambda (a) body) e) ((lambda (a) e) body)))

19 What do we know about hygiene?

20 Hygiene and lexical scope (define-macro (or e1 e2) (with ([t e1]) (if t t e2))) (with ([x 1]) (or #f x)) => (with ([x 1]) (with ([t #f]) (if t t x))) (with ([t 1]) (or #f t)) => (with ([t 1]) (with ([t ′ #f]) (if t ′ t ′ t))) ==

21 Hygienic macro expansion is insensitive to  -conversion. The essence of hygienic macros In other words, hygienic = respects  !

22 What is the scope of with ? (define-macro (with ([a e1]) e2) …) First try: look at results of expansion (with ([x 1]) (or #f x)) => ((lambda (x) ((lambda (t) (if t t x)) #f) 1) (with ([t 1]) (or #f t)) => ((lambda (t) ((lambda (t ′ ) (if t ′ t ′ t)) #f) 1)

23 Using expansion to define scope Goal: define hygienic macro expansion in terms of  -equivalence. Strategy: define  -equivalence in terms of hygienic macro expansion. Oops.

24 What is the scope of with ? (define-macro (with ([a e1]) e2) …) Better idea: provide a binding specification with consumes an argument of shape (with ([a e1]) e2) and produces an expression, where: a is an identifier e1 and e2 are expressions a is bound in e2

25 What is the scope of with ? (define-macro (with ([a e1]) e2) …) Better idea: provide a binding specification with : (with ([ expr ]) expr a ) → expr

26 Annotated with macro (define-macro (with ([a e1]) e2) : expr [(a : binder) (e1 : expr) (e2 : expr a)] …) States macro writer’s intention about scope Admits rigorous  -equivalence definition Provides correctness criterion for hygienic macro expansion

27 A Theory of Hygienic Macros m, a model of hygienic macros 1. Macro binding specifications (types) 2. Theory of  -equivalence 3. High-level expansion semantics 4. Definition and proof of hygiene

28 A Theory of Hygienic Macros m, a model of hygienic macros 1. Macro binding specifications (types) 2. Theory of  -equivalence 3. High-level expansion semantics 4. Definition and proof of hygiene

29 m : a model of hygienic macros (with ([x 1]) x) ( x.x ) 1 S-expressions -calculus

30 m : a model of hygienic macros →  (with ([x 1]) x) →  ((lambda (x) x) 1) →  ( ( x.x ) 1) →  ( x.x ) 1

31 m : a model of hygienic macros let syntax fun = macro (( a ) e ) : (( ) expr a ) => a.e with = macro (([ a e ]) body ) : (([ expr ]) expr a ) => (( a.body) e) in (with ([f (fun (x) 42 )]) ( f y. y ) ) end

32 A Theory of Hygienic Macros m, a model of hygienic macros 1. Macro binding specifications (types) 2. Theory of  -equivalence 3. High-level expansion semantics 4. Definition and proof of hygiene

33 1. Macro binding specifications Shape types based on Culpepper and Felleisen ’03: with : (with ([ id expr ]) expr ) → expr Types for reasoning about scope inspired by nominal datatypes and nominal logic of Gabbay and Pitts ’01

34 1. Type checking let syntax m = macro p :  => … in x. e end [ m :  → expr] ⊢ x. e : expr

35 1. Type checking let syntax m = macro p :  => … in x. e end [ m :  → expr, x : expr] ⊢ e : expr

36 1. Meta-level scope let syntax m = macro ( e1 e2 ) : ( expr expr ) => t. (e1 e2) in … end [ t : expr], [e1 : expr, e2 : expr] ⊢ e : expr

37 1. Two dimensions of binding  : program bindings (object-level)  : macro pattern bindings (meta-level) ,  ⊢ s : 

38 1. Syntactic types Base types:  ::= expr |  → expr Binding types:  ::= | expr a,a… Shape types:  ::=  |  | (  … )

39 A Theory of Hygienic Macros m, a model of hygienic macros 1. Macro binding specifications (types) 2. Theory of  -equivalence 3. High-level expansion semantics 4. Definition and proof of hygiene

40 2. Theory of  -equivalence Adapted to macro applications by exploiting shape type annotations Relies on novel shape-directed  -conversion Based on Gabbay/Pitts “swapping” formulation of  -equivalence

41 2.  -equivalence via swapping let val x = x in x (fn x => x) end =  let val y = x in y (fn w => w) end binding occurrence

42 2.  -equivalence via swapping let val z = x in x (fn x => x) end =  let val z = x in y (fn w => w) end free

43 2.  -equivalence via swapping let val z = x in x (fn x => x) end =  let val z = x in y (fn w => w) end

44 2.  -equivalence via swapping let val z = x in z (fn z => z) end =  let val z = x in z (fn w => w) end

45 2.  -equivalence via swapping let val z = x in z (fn z ′ => z ′ ) end =  let val z = x in z (fn z ′ => z ′ ) end

46 2.  -equivalence via swapping For each binder x, pick fresh name z Rename the binding occurrence of x In the body, swap all occurrences of x with z Recur on subterms Compare results syntactically

47 2.  -equivalence for m (with ([x x]) (x (fun (x) x))) =  (with ([y x]) (y (fun (w) w))) ?

48 2. Shape-directed  -conversion (with ([x x]) (x (fun (x) x))) (with ([ expr ]) expr a ) (with ([y x]) (y (fun (w) w))) binding occurrence

49 2. Shape-directed  -conversion (with ([z x]) (x (fun (x) x))) (with ([ expr ]) expr a ) (with ([z x]) (y (fun (w) w))) not in scope of a ; not converted

50 2. Shape-directed  -conversion (with ([z x]) (x (fun (x) x))) (with ([ expr ]) expr a ) (with ([z x]) (y (fun (w) w))) entering scope of a ; swap all occurrences of x / y with z

51 2. Shape-directed  -conversion (with ([z x]) (z (fun (z) z))) (with ([ expr ]) expr a ) (with ([z x]) (z (fun (w) w)))

52 2. Shape-directed  -conversion (with ([z x]) (z (fun (z) z))) (fun ( ) expr b ) (with ([z x]) (z (fun (w) w)))

53 2. Shape-directed  -conversion (with ([z x]) (z (fun (z ′ ) z ′ ))) (fun ( ) expr b ) (with ([z x]) (z (fun (z ′ ) z ′ )))

54 2.  -equivalence for m (with ([z x]) (z (fun (z ′ ) z ′ ))) =  (with ([z x]) (z (fun (z ′ ) z ′ )))

55 But what is hygienic expansion?

56 A Theory of Hygienic Macros m, a model of hygienic macros 1. Macro binding specifications (types) 2. Theory of  -equivalence 3. High-level expansion semantics 4. Definition and proof of hygiene

57 3. Hygienic macro expansion Simple substitution semantics: let syntax x = m in e end →  e[m/x] if BV(e) ∩ FV(m) =  ( m. s ) →  transcribe(e, match(p, s)) if m = ( macro p :  => e) and BV(s) ∩ FV(e) =  and BV(e) # s

58 3. Expansion up to  x.let syntax m = macro ( e ) : ( expr ) => (e x ) in x.(m x) end

59 3. Expansion up to  x.let syntax m = macro ( e ) : ( expr ) => (e x ) in x ′.(m x ′ ) end →→ x. x ′.( ( macro (e) : ( expr ) => (e x )) x ′ ) →→ x. x ′. ( x ′ x )

60 3. Confluence (with ([x 1]) (or #f x)) (( x.(with ([t #f]) (if t t x)) ) 1 ) (( x.(or #f x) ) 1 ) (with ([x 1]) (with ([t #f]) (if t t x)))    

61 3. Confluence Theorem: Let s be a term such that ,  ⊢ s : . If s →  s 1 and s →  s 2 then there exists an s ′ such that s 1 →  * s ′ and s 2 →  * s ′.   ** ** s s1s1 s2s2 s′s′

62 A Theory of Hygienic Macros m, a model of hygienic macros 1. Macro binding specifications (types) 2. Theory of  -equivalence 3. High-level expansion semantics 4. Definition and proof of hygiene

63 4. Hygiene The expansion relation →  is hygienic if for any well- typed expressions e 0 and e ′ 0 such that e 0 =  e ′ 0 and fully-expanded expressions e and e ′ such that e 0 →  * e and e ′ 0 →  * e ′, then e =  e ′. e0e0 e′0e′0   e e′e′ ** ** Hygienic = respects  !

64 4. Hygiene Remember confluence?   ** ** s up to  -equivalence s1s1 s2s2 s′s′

65 4. Hygiene Theorem: the relation →  is hygienic. e0e0 e′0e′0     e′e′ e

66 Related and Future Work

67 Related work Hygienic macro systems Kohlbecker et al ’86: Hygienic macro expansion Kohlbecker and Wand ’87: Macro-by-example Bawden and Rees ’88: Syntactic closures Clinger and Rees ’91: Macros that work Dybvig et al ’93: Syntactic abstraction in Scheme Krishnamurthi ’01: Linguistic reuse Models of macros Griffin ’88: Notational definitions Bove and Arbilla ’92: Confluent calculus of expansion Gasbichler ’06: Modules with hygienic macros

68 Related work Macro types Culpepper and Felleisen ’03: Well-shaped macros Variables and binding Higher-order abstract syntax, de Bruijn, “locally nameless” Gabbay, Pitts: Nominal logic Staged programming Davies and Pfenning: Modal analysis of staged computation Taha et al: Multi-stage programming, MetaML, MacroML Nanevski: Meta-programming with names and necessity. Kim et al ’06: Polymorphic modal type system for Lisp-like multi-staged languages

69 Future work Core features: Recursive macros Case dispatch Advanced features: Macro-defining macros Conveniences: Variable-length sequences Ellipsis patterns (Kohlbecker and Wand ’87)

70 Future work Joint confluence with evaluation: (with ([x (( y.y ) 1 ) ]) x) (with ([x 1]) x) (( x.x ) (( y.y ) 1 )) (( x.x ) 1 )    

71 Hygienic = respects  ! Thank you.