Macros

1
Macros

• Jonathan Bachrach
• MIT AI Lab

2
Macro

• Syntactic extension to core language
• Defines meaning of one construct in terms of
  other constructs
• Their declarations are called macro definitions
• Their uses are called macro calls

3
unless Example

• Definition
• (ds (unless ,test ,_at_body) (if (not ,test)
  ,_at_body))
• Call
• (unless done? (go))
• Expansion
• (if (not done?) (go))
• Evaluation

4
Macro Motivation

• From now on, a main goal in designing a language
  should be to plan for growth.
• Guy Steele, Growing a Language, OOPSLA-98,
  invited talk
• Power of abstraction where neither functional nor
  object abstraction will suffice, affording
• Clarity
• Concision
• Implementation hiding
• People are lazy
• do the right thing

5
Mathematical Notation

• Continually inventing new abbreviations
  appropriate for expressing new concepts
• Many mathematical concepts become usable only
  after someone invents a suitable notation to
  express them.

6
Proper Macros

• Macros control the evaluation of their arguments
• Four basic types of control
• Transformation pick apart before eval
• Binding vars directly in code
• Conditional Evaluation e.g., unless
• Multiple Evaluation e.g., do

7
Useful Macros

• With macros
• (with-open-file (s dump direction output)
  (princ 99 s))
• Conditional evaluation
• (nif x p z n)
• Iteration
• (forever (yell))
• Anaphoric macros
• (aif (big-long-calculation) (foo it))

8
Improper Macros

• General Rule It is inelegant to use a macro
  where a function would do.
• Example 1 versus while
• Specific Rule dont use macros for inlining.
• Caveat sometimes macros are used when function
  calls would be awkward (e.g., closures)

9
Quasiquote

• WSYWIG specification of lists
• Permits the writing a template of desired list
  with substitutions in place of the variable parts
  of the list
• Evaluation
• yields a list
• turned off by default
• turned on with unquote
• Unquote - ,expr - inserts value of expr
• Splicing - ,_at_expr - splices in value of expr
• Example
• (let ((x 1) (y (2 3))) (,x ,_at_y)) gt (1 2 3)

10
Pattern Matching

• An intuitive and accessible way of expressing a
  parser
• Permits the writing out of the general shape of
  the expected input, but with pattern bindings in
  place of the variable parts of the code
• Proto uses quasiquote notation for patterns
• Default is to match exactly and recursively
• Unquote - ,name - matches one expr bound to name
• Splicing - ,_at_name - matches zero or more exprs
  and binds them to name
• Example
• (mif ((,x ,_at_y) (1 2 3)) (lst x y)) gt (1 (2 3))

11
for Example

• (for (e (1 2 3)) (println out e))
• gt
• (let ((c (1 2 3)))
• (rep loop ((s (ini-state c)))
• (unless (fin-state? S c)
• (let ((e (cur-elt s c)))
• (println out e)
• (rep (nxt-state s c)))))))

• (ds (for (,name ,collection) ,_at_body)
• (let ((c ,collection))
• (rep loop ((s (ini-state c)))
• (unless (fin-state? s c)
• (let ((,name (cur-elt s c)))
• ,_at_body
• (rep (nxt-state s c)))))))

12
TestSuite Macros

• Want test-suite thats
• Concise to write tests
• runs to completion
• records intelligent failure messages
• (test-group whales
• (test-eqv (moby) 37)
• )
• (ds (test-eqv ,input ,output)
• (do-test-eqv group-name
• ,(to-str input)
• (fun () ,input) (fun () ,output)))
• (dm do-test-eqv (group-name doc input-thunk
  output-thunk)
• (unless ( (input-thunk) (output-thunk))
• (sig (cat group-name " " doc))))

13
Declarative Data

• Defining tables can be awkward
• (deftab t (jb gt boy) (kk gt girl))

14
Declarative Data Examples
f(list(x, y, z))
Studies course Math101 title
Mathematics 101 2 points fall term
exclusions Math101 ltgt MathA Math102
ltgt MathB prerequisites (Math101,
Math102) lt (Math201, Math202, Math204)
(CS101,CS102) lt (CS201, CS203)
15
Compilation

• Parse into sexprs
• Recursively expand macros top-down
• Create IR
• Optimize IR
• Emit code

16
Scope of Macros

• Macros usually are limited to known positions
• In lisp,
• Particular forms
• Head named lists
• Symbols
• In particular contexts
• Avoids macro calls in parameter specs, let
  bindings,

17
Evaluation with Macros

• Preparation for Macros
• (run (prepare expression))
• Preparation can be split into many phases
• Lexical e.g., read macros
• Syntactic
• Good design dictates that
• run and prepare are strictly separated
• Their relation is clarified
• And most importantly that experiments can be
  repeated

18
Preparation

• Multiple worlds
• Preparation produces a result often stored in a
  file
• Preparation and running typically execute in
  distinct processes
• Expression being prepared is only means of
  communicating between preparer and evaluator
• Unique world
• All computations cohabit same memory
• Communication can be neither limited nor
  controlled

19
Exogenous Mode

• macroexpand is provided independently of
  expression to prepare, for example, by
  preparation directives
• (dm prepare (expression directives)
• (let ((macroexpand (generate-macroexpand
  directives))
• (really-prepare (macroexpand expression))))
• Macro expansion might either occur as
• A cascade using piping
• A synthesis of a new compiler
• Ensures separation between macro expansion and
  evaluation
• No connection between language of macro expander
  and the language that is being prepared

20
Endogenous Mode

• All information necessary for expansion must be
  found in the expression to prepare
• (dm prepare (expression)
• (really-prepare (macroexpand expression)))
• Macro expansion algorithm preexists
• Finds macro definitions as well as calls
• Use eval rather than invent a special language
• Complicated because requires an evaluator inside
  the macro expander
• Must not confuse language for writing macros with
  the language that is being prepared
• There are actually two different evaluators or at
  least evaluation contexts

21
Protos Endogenous Macros

• Can side-effect macro evaluator with ct special
  form
• (ct (dm map-boot-forms (f) ))
• ct forms can then be used during the expansion of
  subsequent macros
• (ds (define-parents)
• (seq ,(map-boot-forms (fun (x) ))))
• ct forms can be stripped from resulting image if
  in compilation setting
• Experiments can be more faithfully repeated
  because ct world is insulated from rt world

22
Macro Expansion

• Classic Mode
• Result returned by expander can include more
  macro calls
• Result is re-expanded until it no longer contains
  any macro calls
• Expansion Passing Style
• Expander must return an expression without macro
  calls
• Pass expander macroexpand function as argument
• Easy to express local macros

23
Macro-Defining Macros

• Patterns in code signal need for abstraction
• Same applies to macros
• Example
• (ds (mvbind ,_at_args) (multiple-value-bind
  ,_at_args))
• (alias mvbind multiple-value-bind)
• (ds (mvbind ,_at_args)
• (let ((name multiple-value-bind)) (,name
  ,_at_args))
• (ds (,short ,_at_args)
• (let ((name ,long)) (,name ,_at_args))
• (ds (,short ,_at_args) (,,long ,_at_args))
• (ds (alias ,short ,long)
• (ds (,short ,_at_args) (,,long ,_at_args)))

24
Self Generating Code Quote
Fragment f Fragment f ??f ?f
Fragment f ??f ?f
Based on Mike McMahons self generating
quasiquote solution published in Alan Bawdens
quasiquote paper.
(let ((let '(let ((let ',let)) ,let))) (let
((let ',let)) ,let))
25
Unexpected Captures

• (ds (apair ,key ,value ,alist)
• (let ((f pair)) (f (f ,key ,value) ,alist)))
• (let ((pair lst)
• (f f))
• (apair false f ()))
• Captures
• Definition pair
• Call f

26
Simple Hygiene Solutions

• Renaming for call captures
• Use gensym for bindings introduced by macro
• Call Example
• (ds (apair ,key ,value ,alist)
• (let ((f (gensym))
• (let ((,f pair)) (,f (,f ,key ,value)
  ,alist)))
• Package prefixing for definition captures
  (doesnt work though for local macros)
• Definition Example
• (ds (apair ,key ,value ,alist)
• (let ((f protopair)) (f (f ,key ,value)
  ,alist)))

27
Semi-automatic Hygiene

• Explicitly mention the words whose meaning is to
  be frozen
• (ds (apair ,key ,value ,alist)
• (let ((f (gensym)))
• (with-aliases ((cons pair))
• (let ((,f ,cons))
• (,f (,f ,key ,value) ,alist)))

28
Automatic Hygiene

• Variables record definition (e.g., module) origin
• Expansion variables record hygiene context which
• Is unique number
• Is created afresh during each macro expansion
• After expansion
• Names are renamed such that names connect if they
  have both same name and hygiene context
• If free then use module origin to start lookup
• Example
• (ds (apair ,key ,value ,alist)
• let ((f pair)) (f (f ,key ,value) ,alist))

29
Hygiene Escapes

• There are important macros that arent hygienic
  there is a need to escape hygiene
• (ds (loop ,_at_body)
• lab (exit) (rep again () ,_at_body (again)))
• (let ((i 0))
• (loop (if ( i 10) (exit 37)) (set i ( i 1))))

30
R5RS Macros Examples

• (define-syntax or
• (syntax-rules ()
• ((or) f)
• ((or test) test)
• ((or test1 test2 ...)
• (let ((x test1)) (if x x (or test2 ...))))))
  
• (define-syntax let
• (syntax-rules ()
• ((let ((name val) ...) body1 body2 ...)
• ((lambda (name ...) body1 body2 ...) val
  ...))
• ((let tag ((name val) ...) body1 body2 ...)
• ((letrec ((tag (lambda (name ...)
• body1 body2 ...)))
• tag)
• val ...))))

31
R5RS Macros

• syntax-rules is not procedural
• Two environments
• is cute but brittle
• Pattern variables are defaults
• No hygiene escapes
• Local syntax
• Other Scheme macro systems exist

32
Code Walking

• Think of macros as compiler extensions
• Macros are still limited (no with-slots nor
  series)
• Cant access lexical context
• Cant build up form dependent information
• Code walking is a framework that makes this
  possible
• Current state of the art is AST/OO walkers
• See LiSP

33
Conventional Syntax Macros

• Power of macros has been limited to Lisp family
  of languages because Lisp has
• a very regular syntax based on lists
• a large library of list fabs and xforms
• Has been some recent work towards remedying this
  situation

34
Grammar Extensions

• Programmable Syntax Macros
• by Weise and Crew and
• Growing Languages with Metamorphic Syntax
  Macros
• by Brabrand and Schwartzbach
• User writes new rewrite rules
• Example
• Syntax ltstmgt unless (ltexp Egt) ltstm S1gt
• if (!ltEgt) ltSgt

35
Grammar Extensions Critique

• Advantages
• Can ensure that expansions produce admissible
  code fragments

• Disadvantages
• Users need to know grammar (e.g., nonterminals)
• Limited to rewrite rules only and thus awkward to
  write more complicated macros

36
Dylan Macros

• Simple regular conventional syntax
• Converts surface syntax to SST form
• Offers constraint-based pattern matching and code
  quotes all with automatic hygiene
• Rewrite rule only but allows auxiliary rules that
  take arguments
• Simple Example
• define macro unless
• unless ?testexpression ?body
• gt if (?test) ?body
• end macro

37
Sophisticated Dylan Example

• define macro iterate
• iterate ?name (?bindings) ?body end
• gt local method ?name (?_at_vars ?bindings )
  ?body end
• ?name(?_at_inits ?bindings )
• vars
• gt
• ?variable ?expression, ...
• gt ?variable, ...
• inits
• gt
• ?variable ?expression, ...
• gt ?expression, ...
• end macro

38
Dylan Macro Critique

• Advantages
• High level
• Surface syntax
• Hygiene support

• Disadvantages
• Rewrite rule only
• No guarantees about producing admissible code
  fragments

39
Procedural Macro Motivation

• Analysis and rewriting no longer constrained
• Simplified pattern matching and rewrite rule
  engine
• Can package and re-use syntax expansion utilities
• Pattern matching engine is extensible

40
Java Syntactic Extender

• Builds on Dylan macros having constraint-based
  pattern matching and code quotes
• Adds procedural macros
• Macros are classes
• Uses CLASSPATH to find them
• Example
• public syntax unless
• case unless (?testexpression)
• ?statement
• return if (!?test) ?statement

41
JSE Assert Example

• assert(moby() 37)
• public syntax assert
• case assert(?expression)
• return if (?expression) System.out.println(
  "Error")
• case assert(?expression, ?messageexpression
  )
• return if (?expression) System.out.println(
  ?message)
• public class assertSExpander implements
  SyntaxExpander
• public Expansion expand (Fragment fragments)
  throws SyntaxMatchFailure
• syntaxSwitch (fragments)
• case assert(?expression)
• return if (?expression) System.out.println("
  Error")
• case assert(?expression, ?messageexpression)
  
• return if (?expression) System.out.println(?
  message)

42
JSE Parallel forEach Example

• public syntax forEach
• case forEach (?clauses) ?statement
• Fragment inits
• Fragment preds true
• Fragment nexts
• return ?(loop(clauses, statement, inits,
  preds, nexts))
• private Fragment loop
• (Fragment clauses, Fragment statement,
• Fragment inits, Fragment preds, Fragment
  nexts) throws MatchFailure
• syntaxSwitch (clauses)
• case
• return ?inits while (?preds) ?nexts
  ?statement
• case ?type ?name in ?cexpression, ...
  
• Fragment newInits ?inits Iterator i
  ?c.iterator()
• Fragment newPreds ?preds i.hasNext()
  
• Fragment newNexts ?nexts ?name
  (?type)i.next()
• return ?(loop(..., statement, newInits,
  newPreds, newNexts))

43
Open Problems

• Conventional syntax macros
• More flexibility in macro shapes
• Provable admissibility

44
Reading List

• Bawden Quasiquotation in Lisp
• Queinnec Lisp In Small Pieces
• Graham Advanced Techniques for Common Lisp
• Kelsey, Clinger, and Rees R5RS
• Clinger Hygienic Macros through Explicit
  Renaming.
• Shalit The Dylan Reference Manual
• Braband and Schwartzbach Growing Languages with
  Metamorphic Syntax Macros