Growing Languages with Metamorphic Syntax Macros

1
Growing Languages with Metamorphic Syntax Macros

• Claus Brabrand
• Michael Schwartzbach
• BRICS, University of Aarhus, Denmark

2
Outline

• Introduction
• Metamorphisms
• vDSL
• Specificity parsing
• Related and future work
• Conclusion

3
Lexical Macros

• MLEX (TOKENS)n ? TOKENS, n ? 0

4
Lexical Macros

• MLEX (TOKENS)n ? TOKENS, n ? 0

define square(X) XX
5
Lexical Macros

• MLEX (TOKENS)n ? TOKENS, n ? 0

define square(X) XX square(y1) ? y1y1
6
Lexical Macros

• MLEX (TOKENS)n ? TOKENS, n ? 0
• ?

define square(X) XX square(y1) ? y1y1
7
Lexical Macros

• MLEX (TOKENS)n ? TOKENS, n ? 0
• ?
• ?

define square(X) XX square(y1) ? y1y1
define square(X) (X)(X) square(y1) ?
(y1)(y1)
8
Lexical Macros

• MLEX (TOKENS)n ? TOKENS, n ? 0
• ?
• ?
• Problem Independent of syntax!
• Unsafe parse errors discovered at invocation-time

define square(X) XX square(y1) ? y1y1
define square(X) (X)(X) square(y1) ?
(y1)(y1)
9
Invocation Syntax
define swap(X,Y) int tX XY Yt
10
Invocation Syntax
define swap(X,Y) int tX XY Yt
if (agtb) swap(a,b) else b0
parse error!
11
Invocation Syntax
define swap(X,Y) int tX XY Yt
if (agtb) swap(a,b) else b0
?
parse error!
12
Invocation Syntax
define swap(X,Y) int tX XY Yt
if (agtb) swap(a,b) else b0
?
parse error!
define swap(X,Y) do int tX XY Yt while
(0)
if (agtb) swap(a,b) else b0
?
13
Invocation Syntax
define swap(X,Y) int tX XY Yt
if (agtb) swap(a,b) else b0
?
parse error!
define swap(X,Y) do int tX XY Yt while
(0)
if (agtb) swap(a,b) else b0
?

• Problem fixed invocation syntax!
• same for exp / stm /

M(x,y,z)
14
Syntax Macro

• MSYN (AST)n ? AST, n ? 0
• Typed with nonterminals of the host grammar
• Safe no parse errors as a conseq. of expansion

15
Syntax Macro

• MSYN (AST)n ? AST, n ? 0
• Typed with nonterminals of the host grammar
• Safe no parse errors as a conseq. of expansion

stm ? repeat stm until ( exp )
16
Syntax Macro

• MSYN (AST)n ? AST, n ? 0
• Typed with nonterminals of the host grammar
• Safe no parse errors as a conseq. of expansion

stm ? repeat stm until ( exp )
macro ltstmgt ? repeat ltstm Sgt until ( ltexp Egt )
bool first true while (first
!ltEgt) ltSgt first false
17
Syntax Macro

• MSYN (AST)n ? AST, n ? 0
• Typed with nonterminals of the host grammar
• Safe no parse errors as a conseq. of expansion

stm ? repeat stm until ( exp )
macro ltstmgt ? repeat ltstm Sgt until ( ltexp Egt )
bool first true while (first
!ltEgt) ltSgt first false

• Invocation syntax
• grammar extension

18
Syntax Macro

• MSYN (AST)n ? AST, n ? 0
• Typed with nonterminals of the host grammar
• Safe no parse errors as a conseq. of expansion

stm ? repeat stm until ( exp )
macro ltstmgt ? repeat ltstm Sgt until ( ltexp Egt )
bool first true while (first
!ltEgt) ltSgt first false

• Invocation syntax
• grammar extension

• Transformation
• morphing into
• host syntax

19
Syntax Flexibility?
macro ltdeclsgt ? enum ltid Xgt macro
ltdeclsgt ? enum ltid Xgt, ltid Ygt macro
ltdeclsgt ? enum ltid Xgt, ltid Ygt, ltid Zgt
20
Syntax Flexibility?
macro ltdeclsgt ? enum ltid Xgt macro
ltdeclsgt ? enum ltid Xgt, ltid Ygt macro
ltdeclsgt ? enum ltid Xgt, ltid Ygt, ltid Zgt
decls ? enum id ? enum id , id
? enum id , id , id
21
Syntax Flexibility?
macro ltdeclsgt ? enum ltid Xgt macro
ltdeclsgt ? enum ltid Xgt, ltid Ygt macro
ltdeclsgt ? enum ltid Xgt, ltid Ygt, ltid Zgt
decls ? enum id ? enum id , id
? enum id , id , id

• Problems
• Only fixed (finite) arity
• Highly redundant

22
Syntax Flexibility

• Scheme
• special list constructor ...

23
Syntax Flexibility

• Scheme
• special list constructor ...

decls ? ( enum id )
24
Syntax Flexibility

• Scheme
• special list constructor ...
• MS2
• lists /?, options ?, tuples , and
  token-separated lists ?T

decls ? ( enum id )
25
Syntax Flexibility

• Scheme
• special list constructor ...
• MS2
• lists /?, options ?, tuples , and
  token-separated lists ?T

decls ? ( enum id )
decls ? enum id ?,
E-BNF
26
Syntax Flexibility!

• Allow user-defined nonterminals(in invocation
  syntax)

27
Syntax Flexibility!

• Allow user-defined nonterminals(in invocation
  syntax)

decls ? enum id enums enums ? , id
enums ? ?
28
Syntax Flexibility!

• Allow user-defined nonterminals(in invocation
  syntax)
• Transformation?
• without compromising safety

decls ? enum id enums enums ? , id
enums ? ?
29
Metamorphisms

• Our solution Metamorphisms
• Attach host nonterminals to a user-defd
  nonterminal

metamorph ltngt m()
30
Metamorphisms

• Our solution Metamorphisms
• Attach host nonterminals to a user-defd
  nonterminal

metamorph ltngt m() macro ltstmgt ? ltm Agt
ltAgt
31
Metamorphisms

• Our solution Metamorphisms
• Attach host nonterminals to a user-defd
  nonterminal
• Specify morphing (into host syntax) inductively

metamorph ltngt m() macro ltstmgt ? ltm Agt
ltAgt
32
Metamorphisms

• Our solution Metamorphisms
• Attach host nonterminals to a user-defd
  nonterminal
• Specify morphing (into host syntax) inductively

metamorph ltngt m() macro ltstmgt ? ltm Agt
ltAgt morph ltmgt ?

33
Metamorphisms

• Our solution Metamorphisms
• Attach host nonterminals to a user-defd
  nonterminal
• Specify morphing (into host syntax) inductively
• Non-local transformations (multiple results)

metamorph ltngt m() macro ltstmgt ? ltm Agt
ltAgt morph ltmgt ?

34
Metamorphisms

• Our solution Metamorphisms
• Attach host nonterminals to a user-defd
  nonterminal
• Specify morphing (into host syntax) inductively
• Non-local transformations (multiple results)

metamorph ltn,ngt m() macro ltstmgt ? ltm A, Bgt
ltAgt ltBgt morph ltmgt ?

35
Metamorph Example enum
decls ? enum id enums enums ? , id
enums ? ?
36
Metamorph Example enum
decls ? enum id enums enums ? , id
enums ? ?
const int x 0 const int y 1 const int z
2
?
enum x, y, z
37
Metamorph Example enum
decls ? enum id enums enums ? , id
enums ? ?
const int x 0 const int y 1 const int z
2
?
enum x, y, z

• Without compile-time programminglanguage (with
  AST values)

38
Metamorph Example enum
decls ? enum id enums enums ? , id
enums ? ?
int e 0 const int x e const int y
e const int z e
?
enum x, y, z

• Without compile-time programminglanguage (with
  AST values)

39
Metamorph Example enum

• metamorph ltdeclsgt enums()

decls ? enum id enums enums ? , id
enums ? ?
40
Metamorph Example enum

• metamorph ltdeclsgt enums()
• macro ltdeclsgt ? enum ltid Xgt ltenums Dsgt
  
• int e 0
• const int ltXgt e
• ltDsgt

decls ? enum id enums enums ? , id
enums ? ?
41
Metamorph Example enum

• metamorph ltdeclsgt enums()
• macro ltdeclsgt ? enum ltid Xgt ltenums Dsgt
  
• int e 0
• const int ltXgt e
• ltDsgt
• morph ltenumsgt ? , ltid Xgt ltenums Dsgt
• const int ltXgt e
• ltDsgt

decls ? enum id enums enums ? , id
enums ? ?
42
Metamorph Example enum

• metamorph ltdeclsgt enums()
• macro ltdeclsgt ? enum ltid Xgt ltenums Dsgt
  
• int e 0
• const int ltXgt e
• ltDsgt
• morph ltenumsgt ? , ltid Xgt ltenums Dsgt
• const int ltXgt e
• ltDsgt
• morph ltenumsgt ?

decls ? enum id enums enums ? , id
enums ? ?
43
Metamorph Example reserve
stm ? reserve ( res ) stm res ? id res ?
?
acquire(a) acquire(b) acquire(c) ...
release(c) release(b) release(a)
?
reserve ( a b c ) ...
44
Multiple ResultsExample reserve
metamorph ltstms,stmsgt res()
stm ? reserve ( res ) stm res ? id res ?
?
45
Multiple ResultsExample reserve
metamorph ltstms,stmsgt res() macro ltstmgt ?
reserve ( ltres S1,S2gt ) ltstm Sgt ltS1gt
ltSgt ltS2gt
stm ? reserve ( res ) stm res ? id res ?
?
46
Multiple ResultsExample reserve
metamorph ltstms,stmsgt res() macro ltstmgt ?
reserve ( ltres S1,S2gt ) ltstm Sgt ltS1gt
ltSgt ltS2gt morph ltresgt ? ltid Xgt ltres S1,S2gt
acquire(ltXgt) ltS1gt ltS2gt
release(ltXgt)
stm ? reserve ( res ) stm res ? id res ?
?
47
Multiple ResultsExample reserve
metamorph ltstms,stmsgt res() macro ltstmgt ?
reserve ( ltres S1,S2gt ) ltstm Sgt ltS1gt
ltSgt ltS2gt morph ltresgt ? ltid Xgt ltres S1,S2gt
acquire(ltXgt) ltS1gt ltS2gt
release(ltXgt) morph ltresgt ?
stm ? reserve ( res ) stm res ? id res ?
?
48
Metamorph Advantages

• Flexibility
• Safety
• Simplicity

49
Metamorph Advantages

• Flexibility
• Tree structures
• Non-local transformations (multiple results)
• Safety
• Simplicity

50
Metamorph Advantages

• Flexibility
• Tree structures
• Non-local transformations (multiple results)
• Safety
• No parse errors as a conseq. of macro expansion
• Guaranteed termination
• Simplicity

51
Metamorph Advantages

• Flexibility
• Tree structures
• Non-local transformations (multiple results)
• Safety
• No parse errors as a conseq. of macro expansion
• Guaranteed termination
• Simplicity
• Based entirely on declarative conceptsgrammars
  and substitution

52
vDSLvery Domain Specific Language

• studies
• course Math101
• title Mathematics 101
• 2 point fall term
• exclusions
• Math101 ltgt MathA
• Math102 ltgt MathB
• prerequisites
• Math101, Math102 lt Math201, Math202,
  Math203
• Math101, CS101 lt CS202

53
Specificity Parsing
macro ltexpgt ? select ltid Igt from ltexp Egt where
ltexp E2gt macro ltexpgt ? select all from ltexp
Egt where ltexp E2gt
54
Specificity Parsing
macro ltexpgt ? select ltid Igt from ltexp Egt where
ltexp E2gt macro ltexpgt ? select all from ltexp
Egt where ltexp E2gt

• Challenge rounds
• Select most specific productions (wrt. FIRST sets)

55
Specificity Parsing
macro ltexpgt ? select ltid Igt from ltexp Egt where
ltexp E2gt macro ltexpgt ? select all from ltexp
Egt where ltexp E2gt

• Challenge rounds
• Select most specific productions (wrt. FIRST
  sets)
• Resolves many ambiguities
• Independent of definition-order
• Overloading
• Avoids keywordification
• Commit ? no branch explosion, no backtracking

56
Related Work Macro Survey

• 8x Macro languages
• CPP, M4, TEX, Dylan, C Templates, Scheme,
  JTS, MS2

57
Related Work Macro Survey

• 8x Macro languages
• CPP, M4, TEX, Dylan, C Templates, Scheme,
  JTS, MS2
• 31x Macro properties
• Level of operation, Programmability, Definition
  scope, Termination, Argument syntax, Error
  trailing,

58
Related Work Macro Survey

• 8x Macro languages
• CPP, M4, TEX, Dylan, C Templates, Scheme,
  JTS, MS2
• 31x Macro properties
• Level of operation, Programmability, Definition
  scope, Termination, Argument syntax, Error
  trailing,

59
Related Work

• Extensible Syntax with Lexical Scoping
• - by Cardelli, Matthes, and Abadi
• Not a macro language, but a parser generator
• Target language (not host language)
• Extend ? recompile parser
• Localized transformation only
• Disjoint productions
• Keywordification
• Explicit alpha conversion

60
Future Work metafront

• Extensible syntax processor, based on
• Specificity parsing and
• Metamorphic syntax macros

base grammar L macros L ? L program in L
metafront
program in L
61
Future Work metafront

• Extensible syntax processor, based on
• Specificity parsing and
• Metamorphic syntax macros
• Safe transformation L ? L

base grammar L macros L ? L program in L
metafront
program in L
62
Future Work metafront

• Extensible syntax processor, based on
• Specificity parsing and
• Metamorphic syntax macros
• Safe transformation L ? L
• Safe transformation L ? L, (vDSL)

base grammar L macros L ? L program in L
metafront
program in L
63
Conclusion

• Metamorphic Syntax Macros is a
• flexible
• safe
• simple
• way of Growing Languages

64
Conclusion

• Metamorphic Syntax Macros is a
• flexible
• safe
• simple
• way of Growing Languages
• Fully implemented in ltbigwiggt
• (language for developing interactive Web services)

http//www.brics.dk/bigwig/
65
The End

• next bonus slides

66
Pretty Printing Error Reporting

• Pretty Printing
• Terminal printersASCII, LA EX, HTML(/-
  expansion)
• Error Reporting
• stdout, HTML

T
symbol errors test.wig7 Identifier
inf not declared in macro argument S
in macro invocation reader (test.wig7)
defined in std.wigmac44
67
?-Conversion
macro ltstmgt ? repeat ltstm Sgt until ( ltexp Egt )
bool first true while
(first !ltEgt) ltSgt first false

68
?-Conversion

• metamorph ltdeclsgt enums()
• macro ltdeclsgt ? enum ltid Xgt ltenums Dsgt
  
• int e 0
• const int ltXgt e
• ltDsgt
• morph ltenumsgt ? , ltid Xgt ltenums Dsgt
• const int ltXgt e
• ltDsgt
• morph ltenumsgt ?

decls enum id enums enums , id
enums ?
69
Example switch

• metamorph ltstmgt swb()
• macro ltstmgt ? switch ( ltexp Egt ) ltswb Sgt
  
• var x ltEgt
• ltSgt
• morph ltswbgt ? case
• ltexp Egt ltstms Ssgt break ltswb
  Sgt
• if (xltEgt) ltSsgt else ltSgt
• morph ltswbgt ? case ltexp Egt ltstms Ssgt break
  
• if (xltEgt) ltSsgt

stm switch ( exp ) swb swb case exp
stms break swb case exp stms break
70
Metamorph Wellformedness

• Check at definition time
• no left recursion (guarantees termination)
• ? ?
• derivability (metamorphisms must derive
  something finite)
• ? ?

xlist ? xlist X ? ?
xlist ? X xlist ? ?
xlist ? X xlist ? ?
xlist ? X xlist
71
Representation
macro ltidsgt ? MXY ( ltids Isgt ) X, ltIsgt, Y
A, MXY(B, C), D ? A, X, B, C, Y, D
weave

• Weaving yields transparency!

72
Syntax Macros

• MSYN (AST)n ? AST , n ? 0
• square( ) ?

exp

macro ltexpgt ? square ( ltexp Egt ) ltEgt
ltEgt
exp E
exp E
exp

exp
y 1
exp E
exp E
y 1
y 1