1. A Theory of Hygienic Macros PhD Thesis Proposal David Herman

2. 2 The power of macros Derived (user-defined) syntactic constructs Defined by rewriting to existing constructs Translated at compile-time (define-macro (increment! a) (set! a (+ a 1)))

3. 3 The power of macros Abstractions for scope, control, program structure Powerful meta-programming idioms Allow for a minimal language core functions, recursion, iteration, primitive datatypes custom loop forms, derived datatypes, first-class module systems, object- orientation, test harnesses, data constructors, laziness, short-circuit boolean operators, iteration protocols, static analyses, parser generators, partial evaluation, debuggers, coroutines, exceptions, threads, type checkers, introspection, etc… local bindings, software contracts, language designer language user

4. 4 The need for hygiene (define-macro (or e1 e2) (with ([t e1]) (if t t e2))) (with ([x 1]) (or #f x)) => (with ([x 1]) (with ([t #f]) (if t t x))) (with ([t 1]) (or #f t)) => (with ([t 1]) (with ([t #f]) (if t t t)))

5. 5 The need for hygiene (define-macro (or e1 e2) (with ([t e1]) (if t t e2))) (with ([x 1]) (or #f x)) => (with ([x 1]) (with ([t #f]) (if t t x))) (with ([t 1]) (or #f t)) => (with ([t 1]) (with ([t ′ #f]) (if t ′ t ′ t)))

6. 6 What are hygienic macros?

7. 7 (define-macro (or e1 e2) (with ([t e1]) (if t t e2))) (with ([x 1]) (or #f x)) => (with ([x 1]) (with ([t #f]) (if t t x))) (with ([t 1]) (or #f t)) => (with ([t 1]) (with ([t #f]) (if t t t)))

8. 8 What are hygienic macros?

9. 9 KFFD ’86: Generated identifiers that become binding instances in the completely expanded program must only bind variables that are generated at the same transcription step. i.e., not provided as the macro’s input i.e., a single rewriting of a macro application

10. 10 What are hygienic macros? Clinger and Rees ’91: 1. It is impossible to write a macro that inserts a binding that can capture references other than those inserted by the macro. 2. It is impossible to write a macro that inserts a reference that can be captured by bindings other than those inserted by the macro. again: not provided as the macro’s input

11. 11 Why isn’t this enough?

12. 12 Pathological macros in Scheme Impossible with hygienic macros, right? (loop (begin (when (prime? i) (break i)) (increment! i)))

13. 13 Pathological macros in Scheme Contrast with the explicit version: (loop/explicit break (begin (when (prime? i) (break i)) (increment! i))) provided as input

14. 14 Pathological macros in Scheme Petrofsky extraction: (loop (begin (when (prime? i) (break i)) (increment! i)))

15. 15 Pathological macros in Scheme Dumpster diving: (loop (begin (when (prime? i) (break i)) (increment! i)))

16. 16 Pathological macros in Scheme Dumpster diving: (loop/explicit break (begin (when (prime? i) (break i)) (increment! i)))

17. 17 Pathological macros in Scheme (define-macro (murky a e) (begin (set! a e) (lambda (a) e))) (let ([foo 10]) (murky foo (+ foo 1)))

18. 18 Pathological macros in Scheme (define-macro (indecisive ([a e]) body) (if-zero …complicated computation… ((lambda (a) body) e) ((lambda (a) e) body)))

19. 19 What do we know about hygiene?

20. 20 Hygiene and lexical scope (define-macro (or e1 e2) (with ([t e1]) (if t t e2))) (with ([x 1]) (or #f x)) => (with ([x 1]) (with ([t #f]) (if t t x))) (with ([t 1]) (or #f t)) => (with ([t 1]) (with ([t ′ #f]) (if t ′ t ′ t))) ==

21. 21 Hygienic macro expansion is insensitive to  -conversion. The essence of hygienic macros In other words, hygienic = respects  !

22. 22 What is the scope of with ? (define-macro (with ([a e1]) e2) …) First try: look at results of expansion (with ([x 1]) (or #f x)) => ((lambda (x) ((lambda (t) (if t t x)) #f) 1) (with ([t 1]) (or #f t)) => ((lambda (t) ((lambda (t ′ ) (if t ′ t ′ t)) #f) 1)

23. 23 Using expansion to define scope Goal: define hygienic macro expansion in terms of  -equivalence. Strategy: define  -equivalence in terms of hygienic macro expansion. Oops.

24. 24 What is the scope of with ? (define-macro (with ([a e1]) e2) …) Better idea: provide a binding specification with consumes an argument of shape (with ([a e1]) e2) and produces an expression, where: a is an identifier e1 and e2 are expressions a is bound in e2

25. 25 What is the scope of with ? (define-macro (with ([a e1]) e2) …) Better idea: provide a binding specification with : (with ([ expr ]) expr a ) → expr

26. 26 Annotated with macro (define-macro (with ([a e1]) e2) : expr [(a : binder) (e1 : expr) (e2 : expr a)] …) States macro writer’s intention about scope Admits rigorous  -equivalence definition Provides correctness criterion for hygienic macro expansion

27. 27 Thesis The key to specifying and verifying hygienic macro expansion is specifying the binding structure of macros.

28. 28 A Theory of Hygienic Macros m, a model of hygienic macros 1. Macro binding specifications (types) 2. Theory of  -equivalence 3. High-level expansion semantics 4. Definition and proof of hygiene

29. 29 A Theory of Hygienic Macros m, a model of hygienic macros 1. Macro binding specifications (types) 2. Theory of  -equivalence 3. High-level expansion semantics 4. Definition and proof of hygiene

30. 30 m : a model of hygienic macros (with ([x 1]) x) ( x.x ) 1 S-expressions -calculus

31. 31 m : a model of hygienic macros →  (with ([x 1]) x) →  ((lambda (x) x) 1) →  ( ( x.x ) 1) →  ( x.x ) 1

32. 32 m : a model of hygienic macros let syntax fun = macro (( a ) e ) : (( ) expr a ) => a.e with = macro (([ a e ]) body ) : (([ expr ]) expr a ) => (( a.body) e) in (with ([f (fun (x) 42 )]) ( f y. y ) ) end

33. 33 A Theory of Hygienic Macros m, a model of hygienic macros 1. Macro binding specifications (types) 2. Theory of  -equivalence 3. High-level expansion semantics 4. Definition and proof of hygiene

34. 34 1. Macro binding specifications Shape types based on Culpepper and Felleisen ’03: with : (with ([ id expr ]) expr ) → expr Types for reasoning about scope inspired by nominal datatypes and nominal logic of Gabbay and Pitts ’01

35. 35 1. Type checking let syntax m = macro p :  => … in x. e end [ m :  → expr] ⊢ x. e : expr

36. 36 1. Type checking let syntax m = macro p :  => … in x. e end [ m :  → expr, x : expr] ⊢ e : expr

37. 37 1. Meta-level scope let syntax m = macro ( e1 e2 ) : ( expr expr ) => t. (e1 e2) in … end [ t : expr], [e1 : expr, e2 : expr] ⊢ e : expr

38. 38 1. Two dimensions of binding  : program bindings (object-level)  : macro pattern bindings (meta-level) ,  ⊢ s : 

39. 39 1. Syntactic types Base types:  ::= expr |  → expr Binding types:  ::= | expr a,a… Shape types:  ::=  |  | (  … )

40. 40 1. Two dimensions of binding macro (([ a e ]) body ) : (([ expr ]) expr a ) => (( a.body) e) meta-level scopeobject-level scope

41. 41 A Theory of Hygienic Macros m, a model of hygienic macros 1. Macro binding specifications (types) 2. Theory of  -equivalence 3. High-level expansion semantics 4. Definition and proof of hygiene

42. 42 2. Theory of  -equivalence Adapted to macro applications by exploiting shape type annotations Relies on novel shape-directed  -conversion Based on Gabbay/Pitts “swapping” formulation of  -equivalence

43. 43 2.  -equivalence via swapping let val x = x in x (fn x => x) end =  let val y = x in y (fn w => w) end binding occurrence

44. 44 2.  -equivalence via swapping let val z = x in x (fn x => x) end =  let val z = x in y (fn w => w) end free

45. 45 2.  -equivalence via swapping let val z = x in x (fn x => x) end =  let val z = x in y (fn w => w) end

46. 46 2.  -equivalence via swapping let val z = x in z (fn z => z) end =  let val z = x in z (fn w => w) end

47. 47 2.  -equivalence via swapping let val z = x in z (fn z ′ => z ′ ) end =  let val z = x in z (fn z ′ => z ′ ) end

48. 48 2.  -equivalence via swapping For each binder x, pick fresh name z Rename the binding occurrence of x In the body, swap all occurrences of x with z Recur on subterms Compare results syntactically

49. 49 2.  -equivalence for m (with ([x x]) (x (fun (x) x))) =  (with ([y x]) (y (fun (w) w))) ?

50. 50 2. Shape-directed  -conversion (with ([x x]) (x (fun (x) x))) (with ([ expr ]) expr a ) (with ([y x]) (y (fun (w) w))) binding occurrence

51. 51 2. Shape-directed  -conversion (with ([z x]) (x (fun (x) x))) (with ([ expr ]) expr a ) (with ([z x]) (y (fun (w) w))) not in scope of a ; not converted

52. 52 2. Shape-directed  -conversion (with ([z x]) (x (fun (x) x))) (with ([ expr ]) expr a ) (with ([z x]) (y (fun (w) w))) entering scope of a ; swap all occurrences of x / y with z

53. 53 2. Shape-directed  -conversion (with ([z x]) (z (fun (z) z))) (with ([ expr ]) expr a ) (with ([z x]) (z (fun (w) w)))

54. 54 2. Shape-directed  -conversion (with ([z x]) (z (fun (z) z))) (fun ( ) expr b ) (with ([z x]) (z (fun (w) w)))

55. 55 2. Shape-directed  -conversion (with ([z x]) (z (fun (z ′ ) z ′ ))) (fun ( ) expr b ) (with ([z x]) (z (fun (z ′ ) z ′ )))

56. 56 2.  -equivalence for m (with ([z x]) (z (fun (z ′ ) z ′ ))) =  (with ([z x]) (z (fun (z ′ ) z ′ )))

57. 57 But what is hygienic expansion?

58. 58 A Theory of Hygienic Macros m, a model of hygienic macros 1. Macro binding specifications (types) 2. Theory of  -equivalence 3. High-level expansion semantics 4. Definition and proof of hygiene

59. 59 3. Hygienic macro expansion Simple substitution semantics: let syntax x = m in e end →  e[m/x] if BV(e) ∩ FV(m) =  ( m. s ) →  transcribe(e, match(p, s)) if m = ( macro p :  => e) and BV(s) ∩ FV(e) =  and BV(e) # s

60. 60 3. Expansion up to  x.let syntax m = macro ( e ) : ( expr ) => (e x ) in x.(m x) end

61. 61 3. Expansion up to  x.let syntax m = macro ( e ) : ( expr ) => (e x ) in x ′.(m x ′ ) end →→ x. x ′.( ( macro (e) : ( expr ) => (e x )) x ′ ) →→ x. x ′. ( x ′ x )

62. 62 3. Confluence (with ([x 1]) (or #f x)) (( x.(with ([t #f]) (if t t x)) ) 1 ) (( x.(or #f x) ) 1 ) (with ([x 1]) (with ([t #f]) (if t t x)))    

63. 63 3. Confluence Theorem: Let s be a term such that ,  ⊢ s : . If s →  s 1 and s →  s 2 then there exists an s ′ such that s 1 →  * s ′ and s 2 →  * s ′.   ** ** s s1s1 s2s2 s′s′

64. 64 A Theory of Hygienic Macros m, a model of hygienic macros 1. Macro binding specifications (types) 2. Theory of  -equivalence 3. High-level expansion semantics 4. Definition and proof of hygiene AKA, paydirt

65. 65 4. Hygiene The expansion relation →  is hygienic if for any well- typed expressions e 0 and e ′ 0 such that e 0 =  e ′ 0 and fully-expanded expressions e and e ′ such that e 0 →  * e and e ′ 0 →  * e ′, then e =  e ′. e0e0 e′0e′0   e e′e′ ** ** Hygienic = respects  !

66. 66 4. Remember confluence?   ** ** s up to  -equivalence s1s1 s2s2 s′s′

67. 67 4. Hygiene Theorem: the relation →  is hygienic. e0e0 e′0e′0     e′e′ e

68. 68 Research Plan

69. 69 Additional properties and proofs Joint confluence with evaluation: (with ([x (( y.y ) 1 ) ]) x) (with ([x 1]) x) (( x.x ) (( y.y ) 1 )) (( x.x ) 1 )    

70. 70 Evaluation of classic algorithms KFFD ’86 Macros that Work (Clinger and Rees ’91) Proof of correctness, if possible

71. 71 Elements of syntax-rules Core features: Recursive macros Case dispatch Advanced features: Macro-defining macros Conveniences: Variable-length sequences Ellipsis patterns (Kohlbecker and Wand ’87)

72. 72 Research plan Formal model (Completed.) Correctness proofs (Completed.) Elements of syntax-rules (Expected May 2008.) Evaluation of classic algorithms (Expected June 2008.) Additional properties and proofs (Expected August 2008.) Dissertation and defense (Expected November 2008.)

73. 73 Related Work

74. 74 Related work Hygienic macro systems Kohlbecker et al ’86: Hygienic macro expansion Kohlbecker and Wand ’87: Macro-by-example Bawden and Rees ’88: Syntactic closures Clinger and Rees ’91: Macros that work Dybvig et al ’93: Syntactic abstraction in Scheme Krishnamurthi ’01: Linguistic reuse Models of macros Bove and Arbilla ’92: Confluent calculus of expansion Gasbichler ’06: Modules with hygienic macros

75. 75 Related work Macro types Culpepper and Felleisen ’03: Well-shaped macros Variables and binding Higher-order abstract syntax, de Bruijn, “locally nameless” Gabbay, Pitts: Nominal logic Staged programming Davies and Pfenning: Modal analysis of staged computation Taha et al: Multi-stage programming, MetaML, MacroML Nanevski: Meta-programming with names and necessity. Kim et al ’06: Polymorphic modal type system for Lisp-like multi-staged languages

76. 76 Hygienic = respects  ! Thank you.

77. 77 Thank you.