Direct Manipulation of User Interfaces for Migration

1
Direct Manipulation of User Interfaces for
Migration
Intelligent User Interfaces 2006 Jan 29 - Feb 1
2005, Sydney, Australia

• J. P. Molina1,2, J. Vanderdonckt1 and P.
  González2
• 1 Université catholique de Louvain
• 2 University of Castilla-La Mancha

www.usixml.org
2
Contents

• Introduction
• MigriXML
• Case study
• Migration process in detail
• Conclusion

3
Contents

• Introduction
• MigriXML
• Case study
• Migration process in detail
• Conclusion

4
Introduction

• End users of modern interactive systems face a
  wide variety of computing platforms to support
  their interactive tasks, from mobile phones to
  wall screens
• The specific needs of the users environment at a
  given moment, or its evolution, can make
  necessary to remove one application from one
  platform and make it run in another one, that is,
  to migrate the application

5
Introduction

• Migration can be
• Total, when the whole application is migrated
• Partial, only the UI is migrated if it is
  decomposed into control and presentation, then
  migration can be
• Control-oriented, migrates only the control
  component
• Presentation-oriented, the control remains
• Mixed, parts of both control and presentation are
  migrated
• A user interface (UI) is said to be migratable if
  it holds the migration ability

6
Introduction

• A special UI is required for controlling the
  run-time migration of the UI of the interactive
  systems
• This UI is referred to as the meta-user
  interface, implemented in very different ways
• Sytem-initiated, the system initiates the
  migration
• User-initiated, the user initiates the migration
• Mixed-initiated, user and system collaborate to
  perform the migration

7
Introduction

• Related work

8
Introduction

• Elements represented in in the meta-UI

9
Introduction

• Tasks are usually distributed among platforms,
  but without any consistent way to distribute
  them, the space between platforms introduces a
  discontinuity that may disrupt the fluency of the
  interactive task
• Direct manipulation is proposed to tackle that
  discontinuity, and thus avoid possible usability
  problems

10
Contents

• Introduction
• MigriXML
• Case study
• Migration process in detail
• Conclusion

11
MigriXML

• In order to make user interface migration as
  natural as direct manipulation, a meta-UI in the
  form of a virtual environment has been developed
• MigriXML enables the user to carry out that
  migration process without the physical presence
  of the involved platforms, in a virtual and
  interactive way, relaying on UsiXML language and
  a set of models

12
MigriXML

• To generate the virtual reality scene
  representing the migration environment, a suite
  of models is used describing relevant aspects of
  the problem in terms of UsiXML
• The Concrete User Interface
• The User Model
• The Platform Model
• The Environment Model
• Describes properties of interest, which can be
  physical, psychological and organizational

13
MigriXML Environment Model

• The physical part consists of
• a scene model based on VRML97/X3D,
• the multi-surface interaction ontology,
• and the Stanford topology model.
• Each scene is composed of physical resources and
  a series of planes, which are decomposed into
  areas
• Each area may be an interaction surface a
  monitor on a table, a wall screen, or any area
  where the UI is projected and/or recognized with
  computer vision
• Each area is described by attributes like
  dimensions, relative position, relationships with
  other objects, etc

14
MigriXML architecture

• Interactive system architecture
• GUI application
• Arch meta-model or Slinky meta-model
• Platform
• Layered structure

15
MigriXML architecture

• General MigriXML architecture
• Users environment
• GUI application
• Arch meta-model fits the UsiXML models
• Platform
• Virtual Hardware (VHW)
• Virtual Machine (VM)
• Virtual Window Manager (VWM)
• Migration Manager (MM)

16
Implementation

• This work focuses on the original part of
  providing a virtual control environment for
  migrating parts or whole of a UI from one
  platform to another
• To address its implementation, the VRML97/X3D
  languages have been used
• User Interfaces and the Arch meta-model
• Interaction toolkit component VUIToolkit
• Domain-specific component Java and Javascript

17
Implementation

• VUIToolkit
• A set of VRML97/X3D PROTOs that transforms the
  standard plain widgets into a truly 3D
  representation (Virtualized User Interface)
• This toolkit was developed starting from the
  object classes described in the Concrete Model of
  UsiXML language

18
Implementation

• Java and Javascript
• Most VRML97/X3D browsers can interpret Javascript
  code and execute Java code
• VRML97/X3D specifications describe two
  programming interfaces to access the scene graph
• SAI (Scene Authoring Interface)
• EAI (External Authoring Interface)
• Allowed us to leave aside, for this first
  implementation of MigriXML, the complex
  integration of emulators for each specific
  platform

19
Implementation

• Interactive Java/Javascript application,
  virtualized user interface rendered in the 3D
  window of VRML97/X3D browser

20
Implementation

• Interactive application, virtualized user
  interface and virtual platform in MigriXML

21
Contents

• Introduction
• MigriXML
• Case study
• Migration process in detail
• Conclusion

22
Case study

• The users environment is a small office, with
  five different platforms

23
Case study

• Display characteristics of each platform

24
Case study

• UsiXML was used to describe the environment
• The virtual world was the result of a structured
  process of creation
• Requirements from UsiXML specifications,
  designers knew the details of the environment
• Preparation compilation of audio-visual
  material sketches and mock-ups were created
• Design, test and optimize model creation, edit
  appearance, enhance world

25
Case study

• User interfaces were specified in UsiXML too
• Different tools have been developed to help the
  UI designer to visually create this models
• IdealXML Montero et al., 05

26
Case study

• User interfaces were specified in UsiXML too
• Different tools have been developed to help the
  UI designer to visually create this models
• IdealXML
• GrafiXML B. Michotte

27
Case study

• Two different user interfaces

28
Case study

• Migrating from one laptop to the other

Aspire 200015, 1610 1280x800
PT-LB10SU43 800x600
Latitude15, 43 1024x768
29
Case study

• Migrating from a laptop to the Pocket PC

Aspire 200015, 1610 1280x800
e7503,8, 34240x320
30
Contents

• Introduction
• MigriXML
• Case study
• Migration process in detail
• Conclusion

31
Migration process in detail

• The migration process is divided in four stages
  (A, B, C and D), each one having a finite set of
  steps, representing a total amount of 14 steps

32
Migration process in detail

• Stage A Select an interactive application

33
Migration process in detail

• Stage A Select an interactive application

The user presses button (M) to grab the window,
which follows the user wherever she points at
34
Migration process in detail

• Stage A Select an interactive application

The action is transmitted to the corresponding
Virtual Window Manager (VWM) by an event
35
Migration process in detail

• Stage A Select an interactive application

The VWM forwards the information to the Migration
Manager (MM)
36
Migration process in detail

• Stage B Select target platform

37
Migration process in detail

• Stage B Select target platform

The MM broadcasts a message to all platforms
38
Migration process in detail

• Stage B Select target platform

Platforms change their state to
wait-for-selection, a TouchSensor is activated
39
Migration process in detail

• Stage B Select target platform

The user selects the target platform by clicking
on its screen
40
Migration process in detail

• Stage B Select target platform

The Virtual Hardware (VHW) casts an event to
inform the VWM
41
Migration process in detail

• Stage B Select target platform

The VWM forwards the message to the MM
42
Migration process in detail

• Stage C Migrate to target platform

43
Migration process in detail

• Stage C Migrate to target platform

The MM commands to detach the application from
source platform
44
Migration process in detail

• Stage C Migrate to target platform

10
The MM broadcast amessage to the rest
ofplatforms, which changeto normal execution
45
Migration process in detail

• Stage C Migrate to target platform

The MM takes the UsiXML specification and
re-generates the app UI
46
Migration process in detail

• Stage D Adapt to target platform

47
Migration process in detail

• Stage D Adapt to target platform

The MM asks the target platform to host the
re-generated application
48
Migration process in detail

• Stage D Adapt to target platform

The application is launched, and its virtualized
UI added to the scene graph of the target VHW
49
Migration process in detail

• Stage D Adapt to target platform

The virtualized UI is adapted to the target
screen and windowing system
50
Contents

• Introduction
• MigriXML
• Case study
• Migration process in detail
• Conclusion

51
Conclusion

• MigriXML andprevious work

52
Conclusion

• MigriXML and previous representations

53
Conclusion

• MigriXML highlights
• As an environment for simulating migration, there
  is no need to have the physical device to carry
  on the process of adapting a previous UI to a new
  platform
• As a meta-UI for migration, it could also render
  remote spaces, moving us forward tele-presence
  applications
• In both cases, all involved elements are
  graphically rendered and the migration is
  continuously depicted during the process,
  avoiding disruption in the users task, and thus
  improving usability

54
Conclusion

• As future work, evaluations are desirable
• On the one hand, expert and user evaluation in
  order to assess the influence of the meta-display
  problem, that is, how the resolution of the
  display that renders the meta-UI can affect
  utility and usability of this approach
• On the other hand, summative evaluation with
  other meta-UI representations, such as the iconic
  2D map of ARIS, in order to assess which approach
  is better for a given scenario

55
Thank youMercíGracias

• More info
• www.usixml.org
• Contact
• jpmolina_at_info-ab.uclm.es