Steffen Bleul, Wolfgang Mueller, Robbie Schfer Paderborn UniversityCLAB Paderborn, Germany

1
Steffen Bleul, Wolfgang Mueller, Robbie Schäfer
Paderborn University/C-LABPaderborn, Germany
Multimodal Dialog Description for Mobile Devices
2
Motivating Example
Control and monitoring of electronic devices
Winamp Mediaplayer PC User Interface
Motivation MIRS DISL Implemen- tation Summar
y Conclusion
Problem UI does not scale for other platforms /
devices / modalities
3
MIRS - Multimodal Interaction and Rendering
System
S-DISL Interpreter
Motivation MIRS DISL Implemen- tation Summar
y Conclusion
S-DISL
UI Spec (DISL)
HW UserProfile
abstract to concrete mapping
Apps
XSLT
http tcp
Server
(Mobile) Client

• Dialog and Interface Specification Language -
  DISL
• Structural modelDefinition of generic widgets
  (no specific modality)(inherited from UIML with
  ideas from Paterno, Plomp...)
• Control/Dialog modelVariables, Rules, Actions,
  Events(inherited from Dialog Specification
  Notation DSN)

4
Dialog and Interaction Specification Language -
DISL

• Modification of UIML Interface Section
• structure - widget organization
• Generic widget (modality independent widget)
• New generic widget elements
• style - properties
• defines properties of generic widgets
• behavior - user interaction
• defines behavior for generic widgets
• extension for advanced state-oriented dialog
  specification
• states
• rules
• events
• communication

Motivation MIRS DISL Implemen- tation Summar
y Conclusion
5
DISL Structure Elements
Motivation MIRS DISL Implemen- tation Summar
y Conclusion
1. Container Elements
- variablefield - textfield - genericfield -
widgetlist
2. Interaction Elements
a. Input
b. Misc
- variablebox - textbox - genericbox - choice
- command - confirmation - genericcommand
6
DISL Structure/Style/Behavior
ltinterface ...gt
ltstructuregt ltwidget id"Play"
generic-widget"command"/gt
... lt/structuregt ltstylegt ltpart
generic-widget id"Play" gt ltproperty
id"title"gtTitlelt/propertygt ltproperty
id"description"gt This is
a Remote Control for Audio Playing
Devices lt/propertygt ltproperty idhelpgtPress
Button to Start Playinglt/propertygt ltproperty
idvisible gttruelt/propertygt ltproperty
idselected gtfalselt/propertygt ltproperty
idactivatedgtfalselt/propertygt
lt/partgt lt/stylegt ltbehaviorgt ... lt/behaviorgt
Motivation MIRS DISL Implemen- tation Summar
y Conclusion
lt/interfacegt
7
DISL - Behavior

• ltinterface idMediaPlayer" stateinit"gt
• ltbehaviorgt
• ltvariable ...gt ... lt/variablegt
• ...
• ltrule ...gt
• ltconditiongt...lt/conditiongt
• ltevent...gt ...lt/eventgt
• lt/rulegt
• ...
• lttransition ...gt
• ltif-true .../gt
• ltactiongt
• ltstatement ...gt lt/statementgt
• ...
• lt/actiongt
• lt/transitiongt
• ...
• lt/behaviorgt

Motivation MIRS DISL Implemen- tation Summar
y Conclusion
8
Dialog Specification Notation (DSN)Basic
Concepts
Events TogglePower SetVolume ToggleMute PlayTitl
e StopTitle TogglePause NextTitle ... States Pow
er isOn,isOff Volume 0,1,,100 Mute
MuteOn,MuteOff Playing PlayingOn,PlayingOff
Paused PauseOn,PauseOff ...
Motivation MIRS DISL Implemen- tation Summar
y Conclusion
9
Dialog Specification Notation (DSN)Basic
Concepts (contd)
Motivation MIRS DISL Implemen- tation Summar
y Conclusion
Rules Power 1. isOn TogglePower ?
isOff Power 2. isOff TogglePower ?
isOn Mute 1. isOn MuteOn ToggleMute ?
MuteOff Mute 2. isOn MuteOff ToggleMute ?
MuteOn Playing 1. isOn PlayTitle ?
PlayingOn PauseOff Playing 2. isOn
NextTitle ? PlayingOn PauseOff Playing 3.
isOn PreviousTitle ? PlayingOn
PauseOff Playing 4. isOn PlayingOn
StopTitle ? PlayingOff PauseOff Paused 1.
isOn PausedOn ? PausedOff PlayingOn Paused
2. isOn PausedOff PlayingOn ? PausedOn
PlayingOff
10
DSN to DISL
States Power isOn,isOff Events TogglePower
Motivation MIRS DISL Implemen- tation Summar
y Conclusion
ltvariable id"Power" type"boolean"gtfalselt/vari
ablegt
ltrule id"isOn"gt ltconditiongt ltequalgt
ltvariable idPower"gttruelt/variablegt
lt/equalgt lt/conditiongt lt/rulegt
ltrule id"isOff"gt ltconditiongt ltequalgt
ltvariable idPower"gtfalselt/variablegt
lt/equalgt lt/conditiongt lt/rulegt
ltrule id"TogglePower"gt ltconditiongt
ltequalgt ltproperty generic-widget"ToggleP
ower" id"selected"gttruelt/propertygt
lt/equalgt lt/conditiongt lt/rulegt
11
DSN to DISL
Rules Power 1. isOn TogglePower ? isOff
Motivation MIRS DISL Implemen- tation Summar
y Conclusion

lttransitiongt ltif-true rule-id"TogglePower"
expression"and"gt ltif-true
rule-id"isOn"/gt lt/if-truegt ltactiongt
ltcall source"http//localhost8080/mediaplayer/se
rvlet" id"tryoff"
synchronized"yes" timeout"5s"gt
ltparameter id"request"gt
ltvalue-ofgtturnofflt/value-ofgt
lt/parametergt lt/callgt ltstatementgt
ltvariable-content idisOngtltconstgtfalselt/co
nstgtlt/variablegt lt/statementgt
lt/actiongt lt/transitiongt
2 conditions to be true

• 2 actions
• to execute
• http request with 1 parameter
• 1 variable assignment

12
DISL Additional Features
Periodic Events
lttransitiongt ltevent idcheckstatus
repeatyes time5sgt ltactiongt ltcall
source... synchronized"yes" timeout"5s"gt
lt/actiongt lt/eventgt lt/transitiongt
Motivation MIRS DISL Implemen- tation Summar
y Conclusion
More Complex Expressions
ltconditiongt ltop operatorandgt
ltop-expression expressiongreaterthangt ...
lt/op-expressiongt ltop-expression
expressionlessthangt ... lt/op-expressiongt
lt/opgt lt/conditiongt
Change of Interface at Runtime
lttransitiongt ltif-true ...gt ... lt/if-truegt
ltactiongt ltinterface id"VolumeControl"/gt
lt/actiongt lt/transitiongt
Content of virtually any element can be modified
13
Current Implementation
kXML Parser
Motivation MIRS DISL Implemen- tation Summar
y Conclusion
S-DISL
UI Spec (DISL)
http
S-DISL InterpreterJ2ME/MIDP1.0
XSLT/Saxon Processor
Now with Bluetooth support!!
PC Tomcat http-server
Siemens M55
14
Summary Future Work

• Summary
• XML-based specification language for general UIs
• advanced state-oriented dialog specificationby
  variables, rules, actions, events(basic concepts
  inherited from Dialog Specification Notation
  DSN)
• definition of generic widgets NOT for a specific
  modality, e.g., text, graphics, speech(basic
  concepts inherited from UIML, Paterno, Plomp...)
• Implementation for Java enabled phones
• Future Work
• use shorthand vocabulary
• Better structuring through object orientation
  (step from DSN to ODSN)
• Cooperation with high level modelling techniques
  (bridging the gap between e.g. task models and
  rendered UI)
• Proof of concept with more modalities (Speech and
  Gesture)

Motivation MIRS DISL Implemen- tation Summar
y Conclusion