The Evolution of the StringTemplate Engine

1
The Evolution of the StringTemplate Engine

• Presented to the UC Berkeley Harmonia group15
  December 2004
• Terence Parr
• University of San Francisco
• parrt_at_cs.usfca.edu

2
Overview

• Introduction
• what is StringTemplate?
• why did I build it?
• the nature of text generation
• Design goals
• The evolution 4 phases
• Experience
• Lessons

3
What is StringTemplate

• Template engine designed with Tom Burns (CEO,
  jGuru.com) while building commercial sites over
  many years in response to JSP
• small 170k uncompressed binary (w/o test rig)
• Evolved from simple document with holes into a
  functional language capable of generating large
  class of languages
• Well suited to generative programming as well as
  dynamic page generation being used in new ANTLR
  parser generator for code generation
• Distinguishing characteristic strictly enforces
  separation of model and view

4
Canonical Operations

• Attribute referencelttypegt
• Template references (possibly recursive)ltstateme
  ntList()gt
• Apply template to multi-valued attributeltdeclsd
  ecl()gt orltdeclsltit.typegt ltit.namegtgtltitemsr
  ed(), blue(), green()gt
• Conditional includeltif(superClass)gtextends
  ltsuperClassgtltendifgtltif(name)gtName
  ltnamegtltelsegtGuestltendifgt

5
Example

• Translated Program Manipulation via Interactive
  Transformations example to actual StringTemplate
  template definition
• Integration with existing code clean, easy

generateMemSize(node) ltlt int mem_size()
int result 0 ltnode.fieldsresult
ltit.namegt.mem_size()gt return result gtgt
shameless StringTemplate plug
6
Semantics

• Side-effect free expressions
• No state
• No defined order of execution
• Lazy evaluation
• Template inheritance group to subgroup
• Dynamically scoped values inherited
• Recursive template instantiation

7
Design Goals

• Optimized for enforcement of separation not for
  Turing completeness nor expressive one-liners
• Conceptual integrity
• single-minded fanaticism against entanglement
• stick to a few concepts attributes and templates
• consistency no special cases driven by design
  not implementation details
• proper language formalisms, semantics, sane
  syntax
• Simple as possible
• useful to nonprogrammers
• makes it fast, easy to build/maintain
• Powerful as possible

8
Phase I, 1999-2001

• Document with holes only tag refs
• Could nest templates by setting a tag value to
  template dynamic scoping (value inheritance)
• Not intended for full page generation
• File of template definitions loaded into hash
  table code manually created template instances

lttemplate name"db.topic.insert"gt INSERT INTO
Topics VALUES ( ltTIDgt,
'lttopicnamegt' ) lt/templategt
9
Phase II, 2001-2003

• Rewrite of jGuru.com learned entanglement
  allowed by JSP is horrible!
• Decided to enhance StringTemplate, letting needs
  dictate feature set would not be
  Turing-complete!
• Moved to single template per file format with
  indicating attribute ref
• Needed include
• added template reference rather than include,
  macro
• subconciously supported recursion

10
Phase II, 2001-2003 (Contd)

• Needed to walk multi-valued attributes added
  apply syntax
• namesbold(itemnames)
• namesbold(itemnames, separator, )
• Allowed anonymous templates
• namesltbgtnamesltbgt
• Iterated value naming convention evolved
• namesbold(itemnamesi)
• Needed multiple array walk for relational DB
• a,bfoo(xai, ybi)
• Finally, needed conditionalinclusion

if(userName) Name userName endif(userName)
11
Phase III, 2003-Summer 2004

• Wanted to say namesbold() w/o arg
• defined default attribute attr
• Cleaned up IF if(foo)endif
• Nested anonymous blocks names
• Round-robin template application
• users rowOdd(), rowEven()
• Properties user.name calls user.getName()
• Indirect template ref (tname)() added in
  context of immediate evaluation
• Template inheritance via StringTemplateGroup

12
Phase IV, 2004

• Added group file format mutually-referential
  templates with formal arguments

method(type,name,args,body) ltlt public lttypegt
ltnamegt( ltargsarg() separator,gt )
ltbodygt gtgt assign(lhs,expr) ltlhsgt
ltexprgt
13
Future

• Named args for anonymous blocks ala Smalltalk
• listx ltbgtxlt/bgt
• ltnode.fieldsf result ltf.namegt.mem_size()gt
  
• iterator could also choose single argument from
  named templates
• Add parallel array walking back in
• Syntax for super group?
• group sub super
• Lots of little clean up e.g., whitespace,

14
Experience (ANTLR v3.0)

• Tree walker (controller) collects data from AST
  (model), pushes data into templates (view)
• Decouples order of computation from order of
  output (this is huge)
• Enforced separation guarantees easy retargeting,
  no code duplication,
• no code in template
• no output strings in code generator
• Previous code generator hopelessly entangled
• Group file format (output grammar) is great!
  Executable documentation

15
Practical Lessons

• Separate pattern matching from templates!
• dont make templates also analyze model
• lazy evaluation essential decouple order of
  computation from order of output build in any
  order
• Isolates templates (user code) from model
  changes, which will surely evolve
• Templates may be reusable w/o embedded logic
• Recursion required for nested structures
• Attributes
• dynamic scoping really handy
• push dont pull attributes attributes are inert
• Simple language is fast, easy to learn, easy to
  use, promotes retargetability

16
Philosophical Lessons

• Say what you mean! e.g., no FOR-loops
• Focus on principles, conceptual integrity
• focus on what not how
• other tools either doc with holes or Turing
  complete
• Difficult to stick to your principles and still
  create usable tool grey areas appear
• Language design is hardimplementation is
  relatively easy
• Selection pressure results in StringTemplate
• output conforms to a language, hence, a grammar
• strict enforcement of separation leads to grammar

17
Demo

• Generating Java / XML with same model