Towards Virtualization of User Interfaces based on UsiXML

1
Towards Virtualization of User Interfaces based
on UsiXML
Web3D Symposium 2005 March 29 - April 1 2005,
Bangor, UK

• J. P. Molina1,2, J. Vanderdonckt1, F. Montero1,2,
  P. González1
• 1 Université catholique de Louvain
• 2 University of Castilla-La Mancha

www.usixml.org
2
Contents

• Overview
• Motivation
• Related work
• Cameleon reference framework
• The Concrete UI level and UsiXML
• Mapping from CUI to VRML/X3D
• VRML97 X3D toolkits
• A case study
• Continuum of User Interfaces
• Conclusion

3
Contents

• Overview
• Motivation
• Related work
• Cameleon reference framework
• The Concrete UI level and UsiXML
• Mapping from CUI to VRML/X3D
• VRML97 X3D toolkits
• A case study
• Continuum of User Interfaces
• Conclusion

4
Overview

• A model-based approach is presented for
  structuring a development process of UIs based on
  UsiXML, a XML-compliant language.
• UsiXML provides a Concrete UI description that
  remains independent from any platform (HTML,
  Java, VRML).
• To support the rendering of this description in a
  virtual world, two toolkits have been developed
  for VRML97 and X3D worlds.

5
Contents

• Overview
• Motivation
• Related work
• Cameleon reference framework
• The Concrete UI level and UsiXML
• Mapping from CUI to VRML/X3D
• VRML97 X3D toolkits
• A case study
• Continuum of User Interfaces
• Conclusion

6
Motivation

• It is difficult to build a UI that relies solely
  on 3D direct manipulation.
• Some tasks require 1D/2D interaction.
• Support for legacy applications is one
  requirement for new 3D desktops.
• 3D environments must handle 2D very well to break
  into desktop market.
• 3D F/X of UI elements should be rendered in 3D,
  matching much better the mental model of the
  user.
• There is a need to transform 2D GUIs into 3D.

7
Contents

• Overview
• Motivation
• Related work
• Cameleon reference framework
• The Concrete UI level and UsiXML
• Mapping from CUI to VRML/X3D
• VRML97 X3D toolkits
• A case study
• Continuum of User Interfaces
• Conclusion

8
Related work

• Automated generation of a 3D UI from its
  specifications demonstrated in several domains.
• Maquils tool Maquil, 2004.
• Produces 3D dialog boxes describing the UI in
  terms of the StudierStube environment.
• The tool is equipped with an automatic layout
  facility of the widgets.

9
Related work

• VRML Widgets Working Group.
• Aimed to define a standard set of widgets.
• Two different architectures
• Layered approach Isaacs Becker.
• Time control and pointer-surface interaction.
• Interpretation and control of interaction (feel
  of a widget).
• Geometry (look of a widget).
• Component-based approach Seidman.
• Five components sensors, internal glue,
  geometry, external glue and Virtual Reality User
  Interfaces (VRUIs).

10
Related work

• VRML authoring tools.
• ParallelGraphics Internet Scene Assembler.
• Library of prototypes button, toggle button,
  slider, checkbox, text label, etc.
• Basic set for building HUD (Head-Up Display)
  interfaces for PC desktop virtual worlds.

11
Related work

• X3D-related work.
• Project CONTIGRA.
• Defines a XML-based language for the
  specification of widgets, metaphors and
  interaction techniques.
• X3D language is used to describe the final
  appearance of geometry of each 3D widget.
• The set of components has been designed taking
  geometry apart from behaviour.
• Many components have not been developed or are
  not publicly available yet.

12
Related work

• Virtual Reality toolkits.
• The INQUISITIVE toolkit Boyd, 1999.
• Implemented using the MAVERIK toolkit, includes
  window, dial, slider, etc.
• VIEWL Larimer, 2003.
• Based on Qt and DIVERSE, windows are shown as
  polygons that are tangent to a sphere that
  surrounds the user.
• VRIXML Cuppens, 2004.
• A XML-based languge for the creation of 2D/3D
  hybrid interfaces.

13
Contents

• Overview
• Motivation
• Related work
• Cameleon reference framework
• The Concrete UI level and UsiXML
• Mapping from CUI to VRML/X3D
• VRML97 X3D toolkits
• A case study
• Continuum of User Interfaces
• Conclusion

14
Cameleon Reference Framework

• Defines UI development steps for multi-context
  interactive applications Calvary et al., 2003.
• Final UI (FUI).
• UI running on a particular computing platform
  (HTML, Java, VRML).
• Concrete UI (CUI).
• A mock-up that is independent of any computer
  platform.
• Abstract UI (AUI).
• Defines abstract containers and individual
  components, independent of any modality of
  interaction.
• Task Concepts (TC).
• Describe tasks and domain-oriented concepts.

15
Cameleon reference framework

• Multi-path UI development Abstraction,
  Reification and Translation.

16
Contents

• Overview
• Motivation
• Related work
• Cameleon reference framework
• The Concrete UI level and UsiXML
• Mapping from CUI to VRML/X3D
• VRML97 X3D toolkits
• A case study
• Continuum of User Interfaces
• Conclusion

17
The CUI level and UsiXML

• From a same specification of CUI, a GUI can be
  obtained in parallel to a virtualization of this
  GUI.
• A description of a CUI can be obtained by
  successive forward engineering from the TC
  level, the AUI level or directly.
• UsiXML, a XML-compliant User Interface
  Description Language url http//www.usixml.org,
  provides a CUI description that remains
  independent from any toolkit, whether graphical
  or virtual.
• This quality is often referred as platform
  independence.

18
The CUI level and UsiXML

• A CUI model consists of a hierarchical
  decomposition of Concrete Interaction Objects
  (CIOs).
• Each CIO can be sub-typed into sub-CIOs depending
  on the interaction modality chosen graphicalCIO,
  auditoryCIO, etc.
• Every final element of the CUI inherits
  properties depending on the category they belong
  to.
• The properties chosen for UsiXML belong to the
  intersection of property sets of major toolkits
  and window managers Windows GDI, Java AWT and
  Swing, HTML, etc.

19
The CUI level and UsiXML

• Each graphicalCIO has specific properties such as
  isVisible, isEnabled, fgColor, and bgColor.
• Each graphicalCIO is sub-typed into one of the
  two possible categories graphicalContainer, and
  graphicalIndividualComponent.

20
Contents

• Overview
• Motivation
• Related work
• Cameleon reference framework
• The Concrete UI level and UsiXML
• Mapping from CUI to VRML/X3D
• VRML97 X3D toolkits
• A case study
• Continuum of User Interfaces
• Conclusion

21
Mapping from CUI to VRML/X3D

• When mapping a CIO from CUI to a VRML or X3D
  world, different cases are assessed
• Direct mapping.
• It is not possible to set a one-to-one mapping,
  as there is no VRML/X3D primitives that can be
  matched with a CIO. These Web3D languages specify
  basic elements that must be used together to
  create interactive elements.

22
Mapping from CUI to VRML/X3D

• When mapping a CIO from CUI to a VRML or X3D
  world, different cases are assessed
• Direct mapping.
• New mapping.
• There is a need to fill a gap by introducing a
  new widget by appropriate implementation.
• Some attributes defined in the CUI model are not
  used in the 3D world, while new attributes must
  be added in order to describe the interface
  elements in the 3D world, such as position and
  size.

23
Mapping from CUI to VRML/X3D

• When mapping a CIO from CUI to a VRML or X3D
  world, different cases are assessed
• Direct mapping.
• New mapping.
• No possible mapping.
• There are CIOs whose VRML/X3D counterpart have
  not been done yet or are impossible or difficult
  to implement. An example of this case is the CIO
  box.

24
Mapping from CUI to VRML/X3D

• Input/output mapping from desktop-oriented 2D GUI
  to VRML/X3D virtual world.
• Display Screen PROTO.
• Rectangular area where screen units (pixels) are
  converted into world units (meters).
• Serves as a container of UI elements (children
  field).
• Mouse Users pointing device.
• VRML/X3D sensors.
• Keyboard?
• VRML requires non-standard extensions (PGs
  KbdSensor).
• X3D StringSensor.
• Virtual keyboard.

25
Contents

• Overview
• Motivation
• Related work
• Cameleon reference framework
• The Concrete UI level and UsiXML
• Mapping from CUI to VRML/X3D
• VRML97 X3D toolkits
• A case study
• Continuum of User Interfaces
• Conclusion

26
VRML97 X3D toolkits

• The architecture for both toolkits is based on
• The prototyping mechanism available in both
  languages.
• Some ideas borrowed from the VRML Widget WG.
• For each CIO, there is a final widget implemented
  as a PROTO that consists of
• An interface based on a set of fields.
• Shapes for the geometry and appearance of the
  widget.
• Sensors that allow user-widget interaction.
• Internal logic that manages the state of the
  widget.
• Routes that links shapes, sensors and logic.
• Differences between both toolkits are meant to
  take advantage of new features of X3D
  specification.

27
VRML97 X3D toolkits

• GUIToggleButton VRML97 PROTO interface.

PROTO GUIToggleButton USIXML cio ...
USIXML graphicalCio ... USIXML
graphicalIndividualComponent ... USIXML
toggleButton field SFBool defaultState
FALSE VRML97 GUI Toolkit fields field
SFInt32 top 0 field SFInt32 left
0 field SFInt32 width 75 field
SFInt32 height 25 exposedField MFNode label
eventOut SFTime touchTime eventIn
SFBool set_state eventOut SFBool
state_changed eventIn SFInt32
set_top eventIn SFInt32 set_left eventIn
SFInt32 set_width eventIn SFInt32
set_height eventOut SFInt32
top_changed eventOut SFInt32
left_changed eventOut SFInt32
width_changed eventOut SFInt32
height_changed

• From attributes specified for toggleButton in the
  CUI model.
• Some UsiXML attributes have not a meaning in the
  toolkit.
• Other attributes are added, plus set_ and
  _changed fields.
• Type of attributes must be adapted, e.g.
• boolean SFBool
• string? If represents a colour, SFColor seems
  more appropriate.

28
VRML97 X3D toolkits

• GUIToggleButton X3D PROTO interface.

• inputOutput fields were not possible with VRML97
  as this language does not allow us to match them
  with Script fields.

ltProtoDeclare name"GUIToggleButton"gt ltProtoInterf
acegt lt!-- USIXML cio --gt lt!-- . . . --gt lt!--
USIXML graphicalCio --gt lt!-- . . . --gt lt!--
USIXML graphicalIndividualComponent --gt lt!-- . .
. --gt lt!-- USIXML toggleButton --gt ltfield
accessType"initializeOnly" name"defaultState"
type"SFBool" value"false"/gt lt!-- X3D GUI
Toolkit fields --gt ltfield accessType"inputOutpu
t" name"top" type"SFInt32"
value"0"/gt ltfield accessType"inputOutput"
name"left" type"SFInt32" value"0"/gt ltfield
accessType"inputOutput" name"width"
type"SFInt32" value"75"/gt ltfield
accessType"inputOutput" name"height"
type"SFInt32" value"25"/gt ltfield
accessType"inputOutput" name"state"
type"SFBool"/gt ltfield accessType"inputOutput"
name"label" type"MFNode"/gt ltfield
accessType"outputOnly" name"touchTime"
type"SFTime"/gt
29
VRML97 X3D toolkits

• GUIToggleButton VRML97 vs. X3D PROTO.

PROTO GUIToggleButton USIXML cio ...
USIXML graphicalCio ... USIXML
graphicalIndividualComponent ... USIXML
toggleButton field SFBool defaultState
FALSE VRML97 GUI Toolkit fields field
SFInt32 top 0 field SFInt32 left
0 field SFInt32 width 75 field
SFInt32 height 25 exposedField MFNode label
eventOut SFTime touchTime eventIn
SFBool set_state eventOut SFBool
state_changed eventIn SFInt32
set_top eventIn SFInt32 set_left eventIn
SFInt32 set_width eventIn SFInt32
set_height eventOut SFInt32
top_changed eventOut SFInt32
left_changed eventOut SFInt32
width_changed eventOut SFInt32
height_changed
ltProtoDeclare name"GUIToggleButton"gt ltProtoInterf
acegt lt!-- USIXML cio --gt lt!-- . . . --gt lt!--
USIXML graphicalCio --gt lt!-- . . . --gt lt!--
USIXML graphicalIndividualComponent --gt lt!-- . .
. --gt lt!-- USIXML toggleButton --gt ltfield
accessType"initializeOnly" name"defaultState"
type"SFBool" value"false"/gt lt!-- X3D GUI
Toolkit fields --gt ltfield accessType"inputOutpu
t" name"top" type"SFInt32"
value"0"/gt ltfield accessType"inputOutput"
name"left" type"SFInt32" value"0"/gt ltfield
accessType"inputOutput" name"width"
type"SFInt32" value"75"/gt ltfield
accessType"inputOutput" name"height"
type"SFInt32" value"25"/gt ltfield
accessType"inputOutput" name"state"
type"SFBool"/gt ltfield accessType"inputOutput"
name"label" type"MFNode"/gt ltfield
accessType"outputOnly" name"touchTime"
type"SFTime"/gt
30
Contents

• Overview
• Motivation
• Related work
• Cameleon reference framework
• The Concrete UI level and UsiXML
• Mapping from CUI to VRML/X3D
• VRML97 X3D toolkits
• A case study
• Continuum of User Interfaces
• Conclusion

31
A case study

• Internet Radio Player.
• This case study is used as an example running
  from the topmost level of the reference framework
  until the bottom level, here in VRML97 and X3D.
• The following tools are used at different levels,
  all of them based on UsiXML language
• TC and AUI level IdealXML, by F. Montero
• CUI level GrafiXML, by B. Michotte
• FUI level VRML97 X3D toolkits

www.usixml.org
32
A case study

• Task Concept Model.
• Users actions, independently of any
  implementation.

33
A case study

• Abstract User Interface Model (AUI).
• From the task model, a set of transformations can
  be applied to generate an AUI, later manually
  refined.

34
A case study

• Concrete Model (CUI).
• New transformations produce a CUI that can be
  refined with GrafiXML. Again, everything is saved
  in UsiXML.

35
A case study

• Final User Interface (FUI).
• According to given mappings, VRML97 and X3D
  versions of the FUI can be generated from UsiXML
  specifications. The rendered FUIs are genuine 3D
  objects, with real 3D effects.

36
A case study

• Augmenting the GUI with virtual platform.

• A virtual version of the old environment could be
  used to run the old applications Roberts, 2000.

37
Contents

• Overview
• Motivation
• Related work
• Cameleon reference framework
• The Concrete UI level and UsiXML
• Mapping from CUI to VRML/X3D
• VRML97 X3D toolkits
• A case study
• Continuum of User Interfaces
• Conclusion

38
Continuum of User Interfaces

• Milgram and Kishino 1994 continuum distributes
  user interfaces on one dimension ranging from the
  real environment to the virtual environment by
  going through two forms of Mixed Reality (MR)
  Augmented Reality (AR) and Augmented Virtuality
  (AV).
• We have extended this continuum by adding a more
  continuous range of UIs in the virtual part, and
  a second dimension to represent the degree of
  immersion.

39
Continuum of User Interfaces
40
Contents

• Overview
• Motivation
• Related work
• Cameleon reference framework
• The Concrete UI level and UsiXML
• Mapping from CUI to VRML/X3D
• VRML97 X3D toolkits
• A case study
• Continuum of User Interfaces
• Conclusion

41
Conclusion and further work

• UsiXML is a XML-compliant language that allows
  the designer to describe a UI, whether 2D or 3D,
  graphical or virtual.
• It supports a multi-path framework that can be
  followed to produce, from a same specification, a
  GUI and a virtualization of such UI.
• The implemented VRML97 X3D toolkits make it
  possible to render the FUI in a 3D world.
• The look is similar to 2D widgets, but the
  feel is a genuine 3D behaviour.
• The continuum of user interfaces has been
  extended to accommodate Virtualized User
  Interfaces.

42
Conclusion and further work

• With these toolkits, it also possible to simulate
  in the virtual world the look and feel of the
  same UI in virtual versions of different
  platforms, such as PDAs, PCs or even Interactive
  Worlds.
• Current work deals with supporting the migration
  of a UI from one platform to another, by visually
  representing this transition.

43
Thank youMercíGracias

• More info
• www.usixml.org
• Contact
• jpmolina_at_info-ab.uclm.es