Multimodality and contextaware adaptation

1
Multimodality and context-aware adaptation

• Quentin Limbourg, Jean Vanderdonckt

2
Context-awareness

• What is in the context of use?
• User
• Platform
• Environment
• Organisation
• Socio-psychological factors
• Two examples of adaptation due to
  context-awareness
• Adaptability adaptation by the user
• Adaptivity adaptation by the system

3
Example of adaptability

• Personalisation
• Requires many editing
• Is hard to achieve
• Does not necessarily factor out common parts
• Does not necessarily separate fixed parts from
  variable parts
• Goal
• To have a system for automated generation of UIs
• With support of personalisation
• That can generate multiple variants of the same
  UI
• For this purpose, we use
• A feature model
• Generative Programming

4
Whats a Feature Model?
Exclusive
Optional
C
C
F1
F1
F2
Alternate
Dependencies
1.
C
2.
F1
F2
5
Example of a Feature Model
6
Feature model
Model Specifications
C
C
0
1
1
F1
F2
F3
F1
F2
F3
1
1
0
0
F2a
F2b
F3a
F3b
F2a
F2b
F3a
F3b
ltCgt ltF1gt0lt/F1gt ltF2gt1
ltF2agt1lt/F2agt ltF2bgt0lt/F2bgt lt/F2gt
ltF3gt1 ltF3agt0lt/F3agt ltF3bgt1lt/F3bgt
lt/F3gt lt/Cgt
USIXML Specifications
7
Generative Programming
8
Example of adaptivity

• Computer-aided surgery by mixed reality

9
Multi-platform

• Extending approaches for multiple contexts of use
• Context is now restrictedto one computing
  platformat a time

10
Multi-platform

• Traditional approach last level only

11
Multi-platform

• Traditional approach Atomica

12
Multi-platform
UIML,1999

• Shortcomings of the traditional approach (ctd)
• Repetition of development and maintenance

Application 1
Application 2
Application 3
13
Multi-platform
Translation relationship Transformation at
the same level of abstraction

• Physical level deployment

14
Multi-platform

• Physical level deployment Galaxy

Platform 1 Personal Computer
Platform 2 Macintosh Mac OS
Platform 3 Workstation OSF/Motif
Physical User Interface
Physical User Interface
Physical User Interface
Windows UI code
Macintosh UI code
OSF/Motif UI code
15
Multi-platform

• Physical level deployment Galaxy

16
Multi-platform

• Logical level deployment

17
Multi-platform

• Logical level deployment UIML

18
Multi-platformUIML Skeleton Part 1
Abrams et al., 1999
lt?xml version"1.0" ... ?gt ltuiml version"2.0"gt
lt/uimlgt

• What parts comprise the UI whats their
  relationship?

19
Multi-platform

• Task Concepts level deployment

20
Multi-platform
Thevenin,2001

• Multiple levels of translation/reification

21
Multi-platform

• Translation at Task Concept level

Platform 2
Task Concepts
Logical User Interface
Physical User Interface
Final User Interface
22
Multi-platform

• Translation at Task Concept level
• Task model primitives
• Sub-tasks (nodes)
• Remove sub-task for different reasons
• Add sub-task unlikely to appear
• Modify sub-task switch, reshuffle, relax
  constraints
• Hierarchical temporal operators (edges)
• Modify temporal operator reordering
• Cutting sub-trees
• Concepts model primitives

23
Multi-platform

• Task translation

gtgt

24
Multi-platform
Märtin,1990

• Concepts removal CT-UIMS

25
Multi-platform

• Translations TC logical levels

Platform 2
Task Concepts
Logical User Interface
Physical User Interface
Final User Interface
26
Multi-platform

• Translations TC logical levels

27
Multi-platform

• Translations TC logical levels

28
Multi-platform

• Translations TC logical levels

29
Multi-platform

• Translations TC logical physical levels

Platform 2
Task Concepts
Logical User Interface
Physical User Interface
Final User Interface
30
Multi-platform

• Translations TC logical physical levels
• TC logical remain constant over time or
  coordinated with changes
• Graceful degradation of presentation
• Widgets replacement, reduction, resizing
• Text text summarization, sentence elision,
  automatic summary
• Images color reduction, cropping, resizing,
• Animation reduction to still images,
• Video reduction of frame, refreshing,

31
Multi-platform

• Graceful degradation of presentation
• Widgets

32
Multi-platform

• Graceful degradation of presen-tation
• Images

33
Multi-platform

• Graceful degradation of presentation
• Images

34
Multi-platform

• Translations combined at all levels

Platform 2
Task Concepts
Logical User Interface
Physical User Interface
Final User Interface
35
Multi-platform

• Multiple transformations possible

Platform 2
Final User Interface
36
Multi-platform
Calvary, Coutaz Thevenin, 2001

• Plastic UI UI that can accomodate multiple
  changes of context (here, platforms) while
  preserving some usability properties

Platform 2
Platform 3
Final User Interface
Final User Interface
37
Multi-platform
Grolaux, 2001

• Plastic UI Virtual keyboard

38
Multi-platform
Grolaux, Van Roy, Vanderdonckt, 2002

• Plastic UI FlexClock

39
Multi-platform

• Plastic UI adaptable bounded value

40
Multi-platform
Abstraction relationship Transformation into
amore abstract level ofabstraction

• Reverse engineering

Platform 1
Platform 2
Task Concepts
WebRevenge
ReversiXML
Logical User Interface
Logical User Interface
ReversiXML
Physical User Interface
Physical User Interface
Final User Interface
Final User Interface
41
XIML2WML generator
42
Multi-platform
Crossing relationship Transformation
betweendifferent levels ofabstraction

• Reverse engineering

Bouillon Vanderdonckt,2002
Platform 1
Platform 2
Retargeting
Logical User Interface
Physical User Interface
Final User Interface
Final User Interface
43
Multi-directional User Interfaces

• A UI is represented by a suite of models that are
• Analysable
• Editable
• Transformable
• Transformational approach
• Any UI is subject to any transformation at any
  time
• From one language to another
• From one computing platform to another
• From reverse engineering to forward engineering

44
Multi-directional User Interfaces

• Model engineering
• Forward engineering (top-down)
• Reverse engineering (bottom-up)
• Bidirectional engineering (both ways)
• Model transformations
• Reification, abstraction
• Translation
• Crossing (retargeting, revamping,)

45
Multiple levels of consideration

• Channels plus
• User behaviour
• User expectations
• User experience
• User groups
• Physical constraints
• Environment

• Modalities plus
• Market segments
• Time of usage
• Place of usage
• Network, Wi-fi
• Various tasks

• Platforms plus
• Modalities of interaction
• Mono vs multimodality

• Devices plus
• Software characteristics
• Hardware characteristics
• Languages programming, markup
• Tools renderer, interpreter, compiler

46
USIXML
47
An overview of USIXML tools

• Running prototypes
• GrafiXML CUI Hi-Fi editor Code Generator (Java
  Swing, XHTML)
• FormiXML CUI Hi-Fi editor for Forms (XForms,
  HTML)
• SketchiXML CUI Lo-Fi Sketching
• VisiXML CUI Mid-Fi (MS Visio Plug-in)
• FlashiXML CUI renderer for SVF
• ReversiXML reverse engineering from HTML to CUI,
  AUI
• TransformiXML API transformation tool
• Teresa (F. Paternò, CUI level to multi FUI
  generation)

Demo
Demo
Demo
48
Thank you very much for your attention!
Credits