Diapositive 1

1
Multimodal and 3D Graphic Man Machine Interfaces
to improve Operations
M. Ould, P. Palanque, A. Schyn, R. Bastide,
D. Navarre, F. Rubio CNES (Centre National
dEtudes Spatiales), France CS Communication
Systèmes, France LIIHS-IRIT, University
Toulouse 3, France
SpaceOps 2004, May 17-21 2004, Montréal, Canada
2
Reports, Problems, Risks and Objectives (1/3)

• Reports (about Spacecraft CommandControl graphic
  MMIs)
• Report1 (R1) important part (about 30 of
  ground data processing software)
• C2 increasing complexity impossible to avoid
  (related to the market distribution)
• Problems
• Pb1 important and increasing cost (MMI Cost gt
  30 of the global development cost of a ground
  data processing software)
• Pb2 difficult to qualify (cf. C2)
• Risks
• R1 reduction in reliability (cf. Pb2)
• R2 no control over the development cost
• R3 no control over the development time

3
Reports, Problems, Risks and Objectives (2/3)

• Objectives
• Obj1 To Solve the problems (pb1, pb2) and to
  avoid the risks (R1, R2 and R3)
• Obj2 exploit human capabilities while designing
  interaction to improve the MMIs usability and
  efficiency
• The Human interaction is by nature multimodal
• Try to draw a rectangle on a sheet of paper
• Combination of gesture and vocal interaction "put
  that there" (Bolt 80)

4
Evolution of Technology
5
Increase of Bandwidth user-gt system
6
Increase of Bandwidth system -gt user
7
Reports, Problems, Risks and Objectives (3/3)

• Objective to improve the MMIs ergonomics
• Advantages of the multimodality
• Adaptativity helps the adaptation with
  situations and user different profiles and
  different context
• Effectiveness in particular for graphic objects
  handling and spatial selection
• Robustness by the redundancy of the modes
• Users satisfaction many studies shows their
  preference for UI offering Multimodal Interaction

8
Types of Multimodality
Use of Media
Sequential
Parallel
Exclusive
Interpretation
Combined
Alternate
Synergistic
Caelen 91 Nigay 94
9
Solutions and Expected Results (1/2)

• Solutions
• S1 Multimodal MMIs
• "Traditional" modes (screen-keyboard-mouse)
• and of the "New" interaction modes (tactile
  interaction, vocal order / voice synthesis)
• Use of Formal Description Technique to improve
  Software reliability
• S2 3D Visualization
• Web X3D, MPEG4, VRML, Java3D et RM3D,
• Non Web OpenGL/Direct3D.
• Expected results
• ER1 costs reduction
• Development reuse of already qualified
  components
• Operation direct access to a function by vocal
  order.

10
Solutions and Expected Results (2/2)

• ER2 More reliability
• Development improved reliability because of
  tool-supported development process
• Operation improved reliability for critical
  functions thanks to interaction redundancy (e.g.
  combination keyboard-mouse vocal order)
• ER3 increased operability (operation) thanks to
  improved usability and efficiency
• ER4 control and perpetuation of CNES future
  Spacecraft Control Centers
• ER5 anticipation of the future MMIs and checking
  of adequation to our needs thanks to new
  technologies

11
Basic Assumptions

• DO NOT trust "HCI designers" (except innovation)
• They will ask for "unfeasible" things
• They will change their mind faster that one can
  implement
• They will ask for trust (when no argument is
  available)
• They usually do not care about safety issues
• They are human being
• DO NOT trust "developers" (except hacking)
• They don't know what to do
• They "mostly" dont know how to do it
• They have a partial view of the problems
• They usually do not care about usability issues
• They are human being
• Control over the process and people is required
  BUT tools are needed

12
Engineering Multimodal Safety Critical Systems

• Specificities
• Relating input devices to other software
  components
• Concurrency
• Quantitative temporal aspects
• Fusion of several input devices
• Fission mechanisms
• Propositions
• Extending architectures
• Extending formal description techniques
• Going from specification to implementation

13
Proposal Definition and Use of Formal Notation

• The user interface requires the same quality as
  the rest of the software
• Completeness (model the entire UI)
• the complex parts must be dealt with too
• the more complex the UI the more likely the
  notation is not be able to deal with it
• Verification techniques

14
Overview of Interactive Cooperative Objects

• Set of cooperating classes
• For each class
• Behaviour (Petri nets)
• Services (availability)
• State (distribution and value of tokens)
• Presentation
• Activation (how users' actions on the input
  devices trigger systems methods)
• Rendering (how state changes are presented to the
  users

15
(No Transcript)
16
User Centered Design Process for Safety Critical
Interactive Systems
17
Architecture (Bass et al 91)
Application
Abstract rendering
Abstract Events
rendering
events
Concrete rendering
18
Relation to ARCH
19
Bolt's System VoiceGesture (low level) 1

• Type of Places in the Petri net
• Speech lt String gt P2 lt Color gt P3 lt Size gt
• UsrMv lt Point gt P1 lt Shape gt

20
Bolt's SystemVoiceGesture (low level) 2
Event Production Function
Activation Function
Rendering Function
21
Bolt's System VoiceGesture (high level) 1
Type of places

• p1, p5, p7, p10, Idle lt gt
• p2, p6, p8, p12 ltShape, Color, Sizegt
• p3, p9, p11 ltPointgt
• p4, p13, Object ltShape, Color, Size, Pointgt

22
Bolt's SystemVoiceGesture (high level) 2
Activation Function
Rendering Function
23
Formal Analysis

• Use from Petri nets theory
• Conservative and repetitive components
• The net is bounded (no production or consumption
  of resources)
• The net (or part of the net) is live i.e. it is
  always ready to process events
• Properties verification
• Safety and liveness
• Interactive systems properties
• CARE properties (dedicated to MM systems)
• More (but not too much) in the paper

24
Prototyping

• Justification of the interaction modes selected
• Multimodal dialogue management tool
• Multimodal MMI prototype
• Mustification of the 3D visualization techniques
  selected
• MMI prototype with 3D visualization
• Experimentation

25
Experimentation

• Multimodal MMI prototype experimentation on an
  operational Space Control Center
• MMI prototype with 3D visualization
  experimentation on an operational Space Control
  Center
• Integrated prototype experimentation on an
  operational Space Control Center

26
A Short Demonstration

• Two handed interaction
• Selection of an range of values
• From Monomodal to Multimodal Interaction
• Formal specification
• Fusion mechanism
• Iterative prototyping