Code Generation with Jasper, a Reflective Syntax Processor

1
Code Generation with Jasper, a Reflective Syntax
Processor

• Dmitry Nizhegorodov
• Oracle Corp., USA
• Dmitry.Nizhegorodov_at_oracle.com

2
Jasper
Reflective syntax processor for Java, implemented
in Java

• Compile-Time Meta-Object Protocol
• Very high-quality pretty-printer
• Syntactic Extensions
• Macros
• Templates

3
Compile-Time MOP

• Parsed Code Objects
• Better than ASTs.

Standard meta-classes Identifier Expression
Statement Type SourceField SourceClass
4
Compile-Time MOP

• Abstract Meta-Object Factory
• Syntax-to-Syntax Transforms

Closure implementing interface
TSyntaxTransform T transform(T)
5
Compile-Time MOP

• Pending Transform List
• Visitor Pattern Code Walkers

For each attribute a of type T or T public
T follow (T a) public T follow (T a)
6
Visitor
class t ... ... void traverse(Visitor v)
... a v.follow(a) ...
... class V implements Visitor public t
process(t a) apply pending transforms to a
public t follow(t a) if(a ! null)
a.traverse(this) return process(a) public
t follow(t a) for(int
i0ilta.lengthi) ai follow(ai)
...
7
Syntactic Extensions

• Parser Plugins keyword-handler pairs
• Extensibility maps guarantee syntactic type
  safety
• Handler a closure, a syntax transform

Handler can call back to the parser public T
parse () ... p.parseEx?ression()
p.parseStatement() ...
8
Macros

• Programmable
• MOP-based, Syntactic, Type-Safe
• Easy to use

MACRO defines a plugin a transform MACRO
Statement TRIVIAL (Statement s) return s
9
Macro TRIVIAL
MACRO Statement TRIVIAL (Statement s) return
s --gt public class TRIVIAL_Statement
extends StatementParserExtension public
void init(Parser p) init(p, "TRIVIAL")
public Statement parse (Identifier kwd)
return new TRIVIAL_StatementExpander(p.pa
rseStatement()) class
TRIVIAL_StatementExpander extends
StatementExpander Statement s
TRIVIAL_StatementExpander(Statement s) this.s
s public Statement expand () return s
public void print(SoursePrintStream out) ...

10
MACRO as a Transform

• MOP-based syntactic extension defining other
  extensions
• MACRO itself can be defined as MACRO

MACRO SourceClass MACRO (SourceField spec)
body
11
MACROs body
type of the macro T, its name N, the forms to
consume from the parse-stream Pn pn MACRO T
N (PT pn...) code gt public class N_T
extends TParserExtension public void
init(Parser p) init(p, "T") public T
parse (Identifier k) return new
N_TExpander(PT pn...) ... class
N_TExpander extends TExpander PT pn ...
N_TExpander(PT pn...) this.pn pn ...
public T expand () code ...
12
MACRO is meta-helical
package implemented EXTEND implementing.DefMac
ro public class DefMacro extends
ExtensionPackage ... MACRO SourceClass
MACRO (SourceField spec) body ...

13
MACRO can be meta-circular
package jasper.macro EXTEND
jasper.macro.DefMacro public class DefMacro
extends ExtensionPackage ... MACRO
SourceClass MACRO (SourceField spec) body
...
14
Templates

• Syntax forms defining code patterns
• writing body becomes a as easy as cut-paste-edit
• Interpreting templates pattern is parsed and
  instantiated at runtime
• Compiling templates pattern is compiled into
  executable code
• Type-safe

15
Interpreting Templates

• Closure parsing a template body from an input
  stream
• String x y
• File foo/bar/mytmpl1.txt
• Parameters are plugins!
• Reifying handlers

Lets take a look at how setters and getters can
be generated using an interpretive template
16
SLOT
MACRO SourceField SLOT (Type type,
Identifier name, Statement body) return
(new Template (parser, "private
type slot" "public type setter(type v)
slot v return v " "public type
getter() " " if (slot null) slot
computeSlot()" " return slot "
"private type computeSlot() body "))
.bind("type", type) .bind("slot", name)
.bind("body", body) .bind("getter",
Accessor.makeGetter(name.toString()))
.bind("setter", Accessor.makeSetter(name.toString(
))) .bind("computeSlot", Accessor.makeInitia
lizer(name.toString())) .parseFields()

17
Reifying Parameter Handlers

• Typed
• A handler holds a parameter as a Syntax value

public class StatementReifier extends
StatementParserExtension private Statement
reifiedValue public StatementReifier (String
nm, Statement v) reifiedValue v ...
... public Statement parse (Identifier
keyword) return reifiedValue
18
Interpreting !! (or pros)

• Very flexible
• Change in a template does not necessarily require
  recompilation (file, Java property)

19
Interpreting ((( (or cons)

• Slow
• Error detection is deferred until macro-use time
• A macro-expansion tower materializes at at
  macro-use time

20
Compiling Templates
Inspired by Lisp backquote (setq ,x
5) --gt (list 'setq x 5) x 5 --gt new
AssignmentExpression(x,
new IntExpression(5))
21
Compiling Templates

• Template body is converted to executable code at
  template-definition time
• Expansion builder is verified at template
  compilation time
• Error detection is early
• runtime surprises are rare and small

22
Compiling Templates NEW

• Expression-domain plugin NEW builds newing
  expression constructs
• NEW converts a syntax-form-with-parameters to an
  expression building such form

NEW Syntax --gt Expression
23
NEW Syntax --gt Expression
Specification NEW ltmop-typegt ltparameter-listgt
ltsyntax-formgt Examples Expression my(Statement
body) return NEW Statement(Statement
body) for() body NEW
Expression(Expression arg) arg 1
24
NEW Applies Build Conversion

• Build-conversion is a fundamental property of our
  MOP
• Method constructor()

public class Syntax public Expression
constructor() return buildNewingForm(this)

25
NEW Applies Build Conversion
NEW ltmop-typegt ltparameter-listgt ltsyntax-formgt At
template-definition time, the handler of NEW,
class Compil- ingTemplate, consumes the syntactic
arguments of NEW and re- turns a build-conversion
of ltsyntax-formgt (0) class
CompilingTemplate extends ExpressionParserExtensio
n (1) public Expression parse (Identifier
macro) (2) Type formType p.parseType()
(3) ParameterList parms p.parseParameterList
() (4) for each (t, arg) in parms bind arg
as a handler of type t (5) Syntax
form_with_reifiers parse according to formType
(6) restore old bindings of all parms (7)
return form_with_reifiers.constructor() (8)

26
Reification in NEW
NEW ltmop-typegt ltparameter-listgt
ltsyntax-formgt The parameters of NEW are
reified by-name, not by-value. The
materialization happens at build-conversion time,
line (7), not at template-parsing time, line (5),
and requires no interaction with the parser and
meta-level shifts, unlike Template class
StatementParameterReifier Expression use
StatementParameterReifier(Expression e) use
e StatementParameterReifier(Identifier
name) use new IdentifierExpression(name)
public Statement parse (Identifier name)
return this public Expression constructor ()
return use
27
Example
MACRO Statement UNLESS (Expression test,
Statement body) return NEW Statement
(Expression test, Statement body) if (test
false) body --gt class
UNLESS_Statement extends StatementParserExtension
... class UNLESS_StatementExpander extends
StatementExpander Expression test
Statement body public Statement expand ()
return new IfStatement (0,
new EqualExpression(0, test, new
BooleanExpression(0, false)), body,
null)
28
Type-Safety

• Each materialized name reference is
  type-checked, in the domain of standard
  metaclasses, at expansion-compilation time

foo.jsl --gt foo.java --gt foo.class bar,jsl
EXPORT foo
29
Statement-level NEW

• Conveniency feature for use as macro body
• Macros parameters become template parameters

MACRO Statement UNLESS (Expression test,
Statement body) NEW if (test
false) body
30
Escapes

• Generalization of parameter reifiers
• Analogous to Lisps ,

class StatementEscapeForm extends
StatementParameter ... public Statement
parse (Identifier keyword) unbind the
template parameters Expression e
parser.parseExpression() re-bind the
template parameters return new
StatementParameterReifier(e)
31
Expansion-Passing Style

• Expanders operate just in time
• direct continuation
• Macro AS for expander invocation, substituting
  the direct continuation

MACRO Statement UNLESS_NULL(Expression e,
Statement s) e NEW Expression(Expression
e) e null return AS Statement UNLESS(e,
s)
32
Partial Expansions Problem
Consider MACRO Statement
UNLESS_NULL(Expression e, Statement s)
NEW UNLESS(e null) s Here the expander of
UNLESS transforms parameter-reifiers of NEW, not
real syntactic forms.
33
Partial Expansions Solution
Augment NEW with a transform that avoids
premature expansion of forms that are partly
specified NEW T(..ti ai,..) M form1 .. ai
.. --gt AS T M(..constructor_of_ai..).expand()
34
Partial Expansions Research

• Roots in Partial Evaluation
• Online PE defining a specialized
  build-conversion for N_TExpander closures
• Offline PE Type analysis and Static Binding Time
  Analysis (BTA)
• Conservative approach expansion closure passing
  style
• Avoids premature expansion

35
Summary

• Compile-time MOP
• Syntax extensibility protocol type-safe plugins
• Syntax-to-syntax transforms
• MACRO plugintransform generator
• NEW newing code generator
• Expansion Passing semantics

36
Comparison

• IT, CT and PSM both programmable and type-safe,
  but PSM is not reflective
• IT is much like MOO
• Novelty is in CTs build-convertion

PSM Weise, D., Crew, R. Programmable Syntax
Macros. In Proc. of PLDI'93, ACM SIGPLAN
Notices, 1993. MOO Chiba, S. Macro Processing
in Object-Oriented Lan- guages. TOOLS Pacific'98.
IEEE Press, 1998.
37
Applications at Oracle

• Ahead of time native Java compiler written in
  Java with macros
• C codegen backend based on NEW
• Introspective Java source interpreter
• CLOS multimethods and generic functions for Java
• EJB deployment and publishing descriptor
  generation