A Transformational Approach for Developing Multimodal Web User Interfaces

1
A Transformational Approach for Developing
Multimodal Web User Interfaces

• Adrian Stanciulescu

2
Plan

• Context
• Motivations
• Thesis statement
• Requirements for multimodal web UIs
• Proposal
• Reference framework for multimodal web UIs
• Design space for multimodal web UIs
• Transformational method for producing multimodal
  UIs
• Tool support
• Case studies
• Conclusion Future work

3
Context

• Graphical user interfaces classical
  human-computer interface paradigm
• Users with limited literacy and typing skills
• Handheld devices
• Need for hands and eyes free
• Difficult to support mouse and keyboard
  interaction
• Web access anyplace, anywhere, anytime
• Need for newer, better interface paradigm
• Multimodal user interface is the answer
• Adaptation to the context of use (user, platform,
  environment)
• More natural and efficient human-computer
  interaction

4
Motivations

• Modality interaction flexibility
• Faster interaction
• Lower incidence of errors and esier error
  recovery
• True device mobility
• Usability improvement
• Robust systems
• Enhancement of device effectiveness
• Improved experience
• Lack of multimodal applications

5
Thesis statement

• Development of multimodal web user interfaces
• Three types of user interfaces
• Graphical
• Vocal
• Multimodal
• Conceptual and methodological aspects for a
  model-to-model transformational approach
• Development life cycle based on a design space
  composed of design options

6
Requirements

• Support for multimodal input and output
• CARE properties support for input and output
  modalities
• Approach based on design space
• Openness to new modalities
• Separation of modalities

7
Requirements (contd)

• Ontological independence of modalities
• Transformation-based development
• Machine processable
• Human readable
• Ontological homogeneity
• Reuse of specification
• Methodological explicitness
• Methodological extendibility
• Support for tool interoperability

8
Reference framework

• Camelon reference framework
• UsiXML selected language
• Structured according to four levels of
• abstraction
• Progressive transformation from
• Task Concept level to Final UI
• Support for the extension of the transformational
  
• methodology (graph transformation)
• Underlying unique formalism (graph-based syntax)
• Support for reusability of specification and
  transformations thanks to the XML structure
• Support for modality independence
• Allows introduction of new interaction modality
• Cross-toolkit development

9
Vocal Concrete Interaction Objects

• Shortcomings of UsiXML previous version
• No specialized containers for user-system dialog
• No specialized container for multiple selection
• No differentiations between the systems output
  types
• No object dedicated to system interruptions

10
Vocal Concrete Interaction Objects (contd)

• VocalContainers
• VocalGroup
• VocalForm
• VocalMenu
• VocalConfirmation
• VocalIndividualComponents
• VocalOutput
• VocalFeedback
• VocalPrompt
• VocalMenuItem
• VocalInput
• VocalNavigation
• Break
• Exit
• EventTypes error, help, noInput, noMatch

11
Vocal Concrete Interaction Objects (contd)

• Example virtual ice cream shop

12
Multimodal Concrete Interaction Objects

• MultimodalCIO combination of graphical CIO and
  vocalCIO

Multimodal
Graphical
Vocal
outputText inputText vocalInput
vocalFeedback guidance
vocalPrompt vocalInput
outputText inputText
groupBox radioButtons
vocalMenu vocalItems vocalInput
groupBox radioButtons vocalMenu vocalItems
vocalInput guidance
13
Concrete UI Relationship

• GraphicalRelationships
• GraphicalTransition
• GraphicalAdjacency
• GraphicalContainment
• VocalRelationships
• VocalTransition
• VocalAdjacency
• VocalContainment
• Synchronization between vocalCIOs and
  graphicalCIOs

14
Multimodal Concrete Interaction Objects

• Syntax example multimodal text field
• with equivalent input and redundant output

15
Specification of transformation

• Graph-based patterns
• Syntactically typed patterns
• Textual concrete syntax of patterns UsiXML syntax

16
How transformation rules work

• LHS Find an occurrence of LHS into G
• NAC Check that NAC does not match into G. If
  there is a match then skip to another occurrence
  of LHS
• RHS Replace LHS by RHS

17
UsiXML syntax of transformation rules
NAC
NAC
RHS
mappings
18
The four steps of the transformational approach

• Step 1 Construct the TaskDomain Models
• Step 2 From Task Domain to AUI
• Step 3 From AUI to CUI
• Step 4 From CUI to FUI

19
Sub-steps of the transformational approach
20
Design space for web UIs

• Support for the progressive development life
  cycle towards a final system
• Composed of a set of design options
• Each design option has a finite set of design
  values
• Advantages
• Clarifies the development process in terms of
  options
• Facilitates the incorporation in the development
  life cycle of design options as an abstraction
  covered by a software
• Supports the tractability of more complex design
  problems

21
Design options for graphical web UIs
22
Design options for graphical web UIs
23
Design options for multimodal web UIs

• Prompt vocal (A), graphical (A), multimodal (E,
  C, R)
• Grouping/distinction of items
• Input vocal (A), graphical (A), multimodal (E)
• Output vocal (A), graphical (A), multimodal (E,
  C, R)
• Immediate feedback vocal (A), graphical (A),
  multimodal (E, C, R)
• Guidance
• Input textual (A), Iconic (A), Acoustic (A),
  Speech (A), Multimodal (E, C, R)
• Immediate feedback textual (A), Iconic (A),
  Acoustic (A), Speech (A), Multimodal (E, C, R)

24
Step 2 From Task Domain to AUI

• Separated sub-task presentation
• Combined sub-task presentation

NAC
LHS
RHS
NAC
LHS
RHS
25
Step 3 From AUI to CUI

• Multimodal text field
• Prompt multimodal (R)
• Input multimodal (E)
• Immediate feedback multimodal (R)
• Guidance
• Input iconic (A)
• Output iconic (A)

26
Transformation rule
NAC
LHS
RHS
27
Case studies

• 2 case studies Virtual Polling system and Car
  rental system

28
Tool support

• IdealXML
• TransformiXML
• GrafiXML
• CFB Generator
• Communication Flow Builder
• XHTMLVoice Generator tool

29
Conclusions

• Model-drive approach for the development of
  multimodal web user interfaces
• Requirements fulfillment
• Support for multimodal input/output
• CARE properties support for input/output
  modalities
• Approach based on design options
• Methodology covers a subset of the UIs which can
  be developed with XV

30
Conclusions

• Advantages of combining design options into a
  design space

• Descriptive
• Generative
• Comparative

31
Conclusions

• Work positioned at the intersection of SE and HCI
• Focus on the feasibility of code generation (2
  case studies)
• Usability point of view
• Inexistent published studies on multimodal UI
  design in SE and HCI
• Few ongoing works caused by the reduced number of
  multimodal applications
• Internal validation of the Ph.D. (study based on
  qualitative principles derived from the cognitive
  psychology)

32
Future work

• Investigate the implementation of a new
  transformation engine MOF QVT
• Reinforce existing vocal components
• Extend the ontology with new vocal components
• Design space improvement
• Extend transformation catalogs
• Adaptation of multimodal web user interface to
  the context of use