3D Multimodal Interaction

1
3D Multimodal Interaction

• Why 3D / Multimodal ?
• Presence, Naturalness(Cognitive Load),
  Speed/Efficiency
• Interface Specialized choice of hardware /
  software through which a user communicates with a
  computer system
• Interaction A model by which a user accomplishes
  some tasks via the interface

System
H/W
S/W
HMD, Tracker, Glove...
Metaphors,Widgets, ...
Interaction Model
Tasks
Sequence, Context, Hierarchy, Composite, Stage
Navigation, Select, Manipulate, Command, Data
Input,
2
Goals and Problems of Interaction Design

• Performance (Quantitative Measures)
• efficiency, accuracy and productivity
• helps to meet system goals (e.g. agents ?)
• Usability (Qualitative Experience)
• easy to use and learn
• Applicability to other domain (Reuse)
• Problems
• Not enough Guidelines
• Multimodal (Mixing Different Sensory Information)
• Changing Technologies
• Conflicting Goals (Presence vs. Performance)
• Complexity vs. Frequency

3
Some of the General Guidelines

• Structured Approach to the Interaction /
  Interface Design
• Task Analysis
• Identify and consider users (e.g. domain
  specialty, experience, etc. ?)
• What are important high level tasks /
  functionalities of your VR system ?
• Consider the cognitive process (e.g. AI problem
  solving models)
• Task Decomposition
• Sequential / Concurrent / Composite tasks
• Hierarchical Structure
• What are primitive / generic / atomic subtasks ?
• Devising a Metaphor
• Physical, virtual, or reality metaphor
• Use prior research results (Confirm
  applicability)
• Consider hardware constraints
• Implement Interface
• Consider human factors issues (e.g. menu size,
  color, etc.)
• Run Evaluation Tests

4
Structured Approach
5
Other Considerations

• Scale and Proportion of the Virtual Body
• Closed Loop Model of Interaction
• Consider both Display and Input Method
• Information Content of the Display
• Input / Control Parameters and Display
  Correspondence
• Consistency between Multimodalities
• Avoid conflicts (vs. employing conflicting
  multimodalities)
• Natural Metaphors (Metaphors should be a
  universal one)
• E.g. Using an Trackball for rotation control can
  be regarded a metaphor that most anyone can
  accept
• Hardware Constraints
• Decoupling / Integrating control parameters (e.g.
  for rotation control, adjust roll, pitch, yaw
  separately or simultaneously ?)
• Which part of the cognitive load can be reduced
  ?)
• Mapping work to system functions
• Learning and remembering commands
• Magic number 7 humans short term memory
  remembers about 7 items)
• Understanding the Display (Natural Metaphors)
• Recovering from errors and remembering multiple
  states

6
Human Factors Consideration

• Visual (Eyes)
• Accommodation (Focusing by Lens Control)
• Convergence (Rotating of the Eye Balls to Focus)
• Providing Depth Perception
• Resolution (Lateral and Depth)
• Peripheral Vision (Motion is easily detected at
  the outskirts of the fovea)
• Persistence and Flicker (Refresh Rates of
  Displays)
• Sound (Ears)
• Intensity
• Direction
• Kinesthetic / Haptic / Tactile
• Passive Haptics
• Proprioception
• Vestibular Senses
• Fitts Law Time to reach a target whose width is
  W and who is at A is logarithmically realted to
  A/W (can affect placement of tools, widgets and
  menus with respect to users)

W
Object
A
User
7
Display Systems

• Anaglyphs / Polarized Glasses
• HMD Accommodation / Convergence Conflict
• Time Multiplexed (Crystal Eye) Front Object
  Clipping
• Boom Hand(s) Occupied
• Retinal Display
• Fully / Semi / Non Immersive

8
CAVE
9
ImmersaDesk / Workbench
10
(No Transcript)
11
(No Transcript)
12
Auditory Display

• Types
• Simple beep sound
• Iconic simple wave sound
• Natural sound recorded and synthesized (sound
  rendering)
• Issues
• Localization and sonification
• 3D/stereo sound, HRTF (Head Related Transfer
  Functions)
• Graphics and audio synchronization
• Setup methods headphone, speakers

13
Tactile/Haptic Display

• Main types
• Ground referenced Phantom
• Body referenced CyberGrasp
• Tactile CyberTouch
• Issues
• Producing the mechanical device
• Interface design
• must be useful for manipulation
• make it non intimidating
• mimic real world interaction

14
Discrete Input Devices

• Main features
• generate one event at a time based on the user
• examples
• keyboard, Pinch Glove
• Interface design
• useful for discrete commandinterface

15
Continuous Input Devices

• Trackers
• types magnetic, mechanical, acoustic, inertial,
  vision/camera, hybrids
• latency (time from capture to delivery of data to
  system)
• update rate (how many capture per unit time)
• interface design
• correspondence between physical and virtual
  worlds
• Dataglove
• gesture and posture communication

16
Combination Devices and Speech

• Combination/Hybrid devices
• mouse, joysticks, tablets, space mouse, ring
  mouse, fly mouse, BAT, Wand, Flex and Pinch
• interface design
• consider DOF for users interaction
• Speech input
• requires more stable recognition technology
• word-level user-independent
• sentence-level more training, noise
• interface design
• very convenient input
• ideal for multimodal interaction

17
Guidelines for choosing I/O

• Think about what interaction techniques are
  required
• Think about inter-restriction of input devices
  and output devices
• Practically money is a big factor
• designing restricted interaction techniques with
  given devices Vs. buying more advanced devices
• Or, make new devices for accomplishing
  interaction techniques?
• Also can we assume that
• constraints of VR devices will practically
  disappeared in near future ?
• 3D devices will become basic like the mouse
  ?(Space 2050)

18
Generic Interaction Tasks and Their Interfaces

• Navigation travel (walk) way-finding
• Selection choosing one or more objects
• Manipulation specification of object position,
  orientation, scale, shape, others(Selection ?
  Manipulation ?, Menu Selection ?)
• Composite Tasks
• Others almost atomic (chunked) tasks
• Visual information browsing
• Measurement
• Data input
• Following / Flow control

19
Classification of Travel by Subtasks
20
Classification of Travel by Metaphors

• Discrete Target Specification
• Select object in Environment
• Select from List
• Position 2D/3D Cursor
• Automatic Selection
• Continuous Target Specification
• Gaze Directed
• Pointing
• Route Specification
• Series of Markers
• Extension of Discrete Target Specification

21
Things to Consider (1)

• Navigation can be further classified as
• Naïve Search look for something (dont know
  where it is)
• Primed Search find something (knowing where it
  is)
• Exploration just move (and look) around
• Goal of Navigation Spatial Awareness
• Build a Cognitive Map as fast and as accurate as
  possible
• Three Sources Knowledge to Take Advantage of
• Landmarks
• Environment Design (Egocentric Viewpoint)
• Artificial Cues (Landmarks and Frames of
  References)
• Organization (Grids and Sectors)
• Procedural
• Survey (Global Topology)
• Map Design (Exocentric Viewpoint)
• Orientation Map Alignment (reduce mental
  rotation)
• Position You are here markers (where you are
  in the map)

22
Things to Consider (2)

• Map Good for search efficiency (not visiting the
  same place again)
• Grid (Landmarks) Good for remembering where am
  I ?, Where was it ?
• Propriocetive Cues
• Walk and Rotate in Place
• People prefer interfaces based on, in the order
  of, kinesthetic, vestibular, and visual feedback
• Vestibular and Visual Senses are coupled
• Providing Connection between Map and Landmark
  based Interface
• Coupling of Head/View Direction and Travel
  Direction
• Bad for relative motion
• Instant Tele-portation
• Causes Disorientation

23
Navigation Techniques
Worldlet (Elvins, 1997)
World-in-Miniature (Stoakley, 1995)
24
Classification of Selection
25
Classification of Manipulation by Subtasks
26
Classification of Manipulation by Metaphors
27
Other Things to Consider (1)

• Exocentric vs. Egocentric
• World in Minature / Scaled World Grab
  (Excocentric)
• Virtual Hand / Ray / Flashlight / Aperture
  (Egocentric)
• Bring object to me, Bring me to Object,
  Tele-operation
• World in Minature (BMtO)
• Scaled World Grab (BOtM)
• Go-Go / Ray (Teleoperation)
• Use of Constraints (e.g. When translating an
  object constrain translation in x direction, then
  in y and then in z in sequence).
• When using ray/pointer, think about how to define
  the ray
• Virtual Pointer Origin (Not necessarily the
  extension of eye and finger)
• Collision detection / response may be important
  for natural manipulation

28
Things to Consider (2)

• Using two hands
• Dominating Hand does the tasks
• Non Dominating Hand provides local frame of
  reference
• E.g. Rotation (Left hand provides axis of
  rotation and right hand carries out the task)
• Using Props
• Passive Haptic / Tactile
• Two hands
• Physical Metaphors
• Using Proprioception
• Using my own work volume (Scaled World Grab)
• Making use of the body as much as possible (e.g.
  make use of head motion to do something like menu
  item selection instead of using voice)
• Interfaces located with respect to my body (Can
  find them blind folded)

29
Selection / Manipulation Techniques (1)
Simple Virtual Hand
Ray-casting (Bolt, 1980)
30
Selection / Manipulation Techniques (2)
Spotlight (Liang, 1994)
Aperture (Forsberg, 1996)
31
Selection / Manipulation Techniques (3)
Fishing reel (Bowman, 1997)
Sticky Finger (Pierce, 1994)
Scaled-world grab (Mine , 1997)
32
Selection / Manipulation Techniques (4)
World-in-Miniature (Stoakley, 1995)
Go-Go (Poupyrev , 1996) Homer (GoGo Ray Casting)
33
Multimodality Issues

• General Consistency Requirement
• Integration of Different Modalities
• Competition among different sensory input to
  affect processing in the brain,
• Selective Attention yet possible
• Sound and Graphics Synchronization (within 150
  msec or accept as two different events)
• Vestibular and Visual Sense Conflict causes
  sickness
• Gesture and Voice mixture is effective and
  preferable (Put that There (Bolt 80)
• Preference vs. Performance
• Sound Feedback enhances 3D Task Performance
  (Miner 94)
• Not too many things we know about Multimodality
  yet