SchemeLisp at N Echo

1
Scheme/Lisp at Nü Echo

• June 16, 2004

2
Agenda

• Introduction
• Potential uses of Scheme
• Which Scheme system?
• Our set of tools
• Implementing Kawa in the production env.
• Demos

3
Your Host

• Dominique Boucher
• Ph.D. in CS with Marc Feeley link-time
  optimization of multimodule Scheme programs
• 1996-1997 Schemers Inc.
• 07/1997-12/2001 Locus Dialogue
• 12/2001-03/2003 SGDL systems
• 03/2003-08/2003 Cogniscience
• 08/2003-... NüÉcho Inc.
• Author of lalr-scm, a free portable LALR(1)
  parser generator for Scheme initially based on
  GNU bison.
• Author of IDyl, a Dylan interpreter written in
  Scheme.

4
Your Host

• Why I like Scheme
• Parentheses (yes, really!)
• Functional subset
• Dynamic (but strong) typing
• Call/cc
• And .. Macros!
• I really like defining new special forms, new
  embedded languages, DSLs

5
Introduction
6
NüÉcho

• Who we are
• 8 Former employees of Locus Dialogue
• A strong team of dedicated individuals
• Complementary competencies
• Speech technologies
• Web development/infrastructure
• CTI
• Programming languages and systems
• Voice User Interface (VUI)
• etc.

7
NüÉcho

• What we do
• Develop and sell packaged speech-enabled,
  VoiceXML-based applications
• Eg. Change of Address
• Develop custom speech-enabled VoiceXML
  applications
• Develop technologies and tools to develop better
  VUIs, faster
• Developing such applications is HARD
• Offer various consulting services in the speech
  industry
• Voice data collection, etc.

8
NüÉcho

• Development
• Mostly Java servlets
• Grammar authoring
• Prompts authoring
• Some command-line tools
• Less VoiceXML than ever!

9
VoiceXML vs HTML
Web Browser
HTML Images Audio
Back-end
VoiceXML Grammars Audio
VoiceXML Platform
10
Why VoiceXML?

• Industry standard largely supported
• Portable (at least in theory)
• Reuses the WEB expertise
• Allows the separation of control and interface
  (eg MVC)
• Uses existing standards (XML, HTTP, etc...)
• Can coexist with other Web-based applications
• Reuses the Web-based infrastructure and
  technologies (JSP, servlets, Velocity, ...)

11
BUT...

• High learning curve lots of languages to learn
• CCXML (Call-control)
• VoiceXML
• ECMAScript
• SRGS (Speech Recognition Grammar Specification)
• Application language (mostly Java)
• Template langage (ex Velocity, JSP)
• Taglibs, etc., etc.

12
NüÉcho

• Showcase
• TalkToSantaClaus (www.talktosantaclaus.com)
• My first project when I joined NüÉcho
• Won two  Flèche dor  (Relationship Marketing
  Association)
• VUI coded in 2 days
• The Speech app. has been entirely coded in Scheme
  (using an early prototype)
• Certainly the most stable part of the whole
  application!
• Received gt70,000 calls in 3 weeks
• DEMO

13
Potential uses of Scheme
14
Three main areas

• Symbolic computations
• DSL for application development
• Development tools

15
Symbolic computation

• After all, S-expression Symbolic expression
• Scheme/Lisp is ideal for symbolic applications
  (compilers, interpreters, etc.)
• Symbolic computation at NüÉcho
• Parsers, lexical analyzers
• Scheme is SO natural here
• lalr-scm
• pattern-matching SXML
• Phrase generators
• from ABNF/XML grammars
• Sentence interpretation
• Conversions
• XML lt-gt ABNF

16
DSL

• Dialog Component Language
• We needed a language for encoding sophisticated
  state-based dialogs ...
• State machines are the usual way of encoding
  dialog logics
• VoiceXML is not good a encoding state machines
• ... providing high reusability, composability,
  configurability
• Features not provided by any existing commercial
  product

17
DSL

• Our approach
• Represented by S-expressions
• Compiled to Java objects for runtime execution
• Advantages
• Simple, regular, extensible syntax
• Easily read (with read!)
• Parsing eased by our pattern-matching library
• Seamless integration of code fragments (scripts,
  embedded actions)
• Everything is compiled to Java bytecode
• even scripts, thanks to Kawa
• No more VoiceXML messing, no JSP, no taglibs,
  etc.!!!
• We now code our applications at the right level
  of abstraction

18
Development tools

• We need authoring tools on top of our technology
• easy access to all our programmatic tools
• provide automated consistency checks,
  validations, etc.
• Our choice XEmacs
• Cross-platform
• Easy to extend
• Scripting language Emacs Lisp (!)
• Best free editor for Lisp-based languages

19
Which Scheme system?
20
Kawa

• Scheme programs must run in a Java environment
• Kawa generates bytecode for the JVM
• Excellent performances
• When properly annotated, compares to Java code
• With a good JIT, can outperform Scheme-gtC
  compilers
• Java/Scheme interface is really natural
• One can declare classes in Scheme, visible from
  Java
• Actively supported by Per Bothner
• Small, but responsive user community

21
Drawbacks

• No debugger!
• must resort to stack traces, (format t ...), ...
• No true call/cc
• no tail-call elimination
• at least by default, but enabling them degrades
  performances a lot for programs in CPS
• I only miss multimethods (à la CLOS)

22
Other Schemes for the JVM

• Bigloo
• Code must run with noverify flag
• Can be a problem when sharing the Servlet engine
  with other applications requiring bytecode
  verification
• I did not like its Java/Scheme interface (YMMV)
• too verbose
• SISC
• Only interpreted
• We have some CPU-intensive algorithms

23
Kawa Performance

• Consider the following (contrived) example
• (define (f)
• (let loop ((i ltintgt 1)
• (sum ltintgt 0))
• (if (lt i 10)
• (loop ( i 1) ( sum i))
• sum)))

24
Kawa Performance

• Method java.lang.Object f()
• 0 iconst_1
• 1 iconst_0
• 2 istore_2
• 3 istore_1
• 4 iload_1
• 5 bipush 10
• 7 if_icmpgt 21
• 10 iload_1
• 11 iconst_1
• 12 iadd
• 13 iload_2
• 14 iload_1
• 15 iadd
• 16 istore_2
• 17 istore_1
• 18 goto 4
• 21 iload_2
• 22 invokestatic 12 ltMethod gnu.math.IntNum
  make(int)gt

Generated code
25
Kawa Modules

• Static modules
• Mainly serve as namespaces
• Importation/exportation capabilities
• Compile to Java classes with only static members
  (well, almost)
• Helps type inference
• Dynamic modules
• can be instantiated like ordinary classes

26
Kawa Modules

• Static module example
• (module-name ltmymodulegt)
• (module-static t)
• (module-export foo bar)
• (require ltanother_modulegt)
• (define (foo) ..)
• (define bar ...)

27
Application development tools
28
Generic MVC-like architecture

• XEmacs
• provides the View and the Controller
• Scheme server
• provides the Model
• Communicate using S-expressions

29
Client-server architecture
Request (S-expression)
stdin
Scheme Server
stdout
Response (S-expressions)
30
Client-server architecture

• Benefits
• The Scheme server can be debugged on the
  command-line!
• DEMO

31
Grammar editor

• For editing ABNF grammars
• DEMO!

32
Grammar editor
33
DCF components editor

• Major mode for editing Speech components
• DEMO

34
DCF components editor
35
Implementing Scheme
36
Biggest challenges

• Integrate Kawa in the build process
• Ant
• Scheme modules can see Java modules, but not the
  contrary (at compile-time)
• Provide good editing capabilities to Java
  programmers
• Most of them use Eclipse, rely heavily on code
  completion, etc.
• They should feel they gain something, not loose
  something

37
The build process

• Ant
• XML-based build tool (à la make)
• Now de facto standard for most Java projects
• integrated in Eclipse, NetBeans, etc.
• Problem
• No built-in task for compiling Kawa modules
• Kawa does not track module dependencies
• like javac does

38
The build process

• Solution
• Build a dedicated Ant task
• Wrap the Kawa compiler
• First compute the module dependency graph (DAG)
• Each module must include a module-name clause
• Like any Java source file, anyway
• Does not allow cycles
• OK in practice
• Selective recompilation
• Recompile a file only if it has changed or one of
  its direct dependencies has changed
• DEMO

39
Development tools

• Kawa Scheme XEmacs mode
• Use the client-server architecture
• The server is a module repository/module compiler
• First scans all Scheme files
• under a given root directory
• Controls completion,  find-definition ,
  tooltips, etc.
• DEMO

40
Conclusion
41
Conclusions

• Kawa Scheme is viable in a Java environment
• Good performance
• Integrates well with Java code
• Initial resistance internally, but now 3
  developers use it daily (at least our DSL) and
  are quite happy with it
• XEmacs is a nice text editor for Lisp languages
• Easy to program (for a Lisp guru -)
• No Lisp editor for Eclipse/NetBeans
• But...
• Not a modern UI
• No widgets other than text editors