Display, Interaction and Navigation

1
Display, Interaction and Navigation

• Julian Looser
• HIT Lab NZ
• University of Canterbury

29th January 2009
2
The User Interface

• Human ?? Computer
• Communication mediated by displays and input
  devices
• Human ?? Computer ?? Visualisation
• For visualisation
• How does the user see the visualisation?
• How does the user control the visualisation?

3
2D Desktop Visualisation
4
3D Desktop Visualisation
5
3D Immersive Visualisation
6
What distinguishes these interfaces?

• Display
• Software
• The virtual way the information is presented
• Hardware
• The physical way the information is presented
• Interaction
• Software
• The virtual way the interface is controlled
• Hardware
• The physical way the interface is controlled

7
Display
8
Physical Display Hardware

• Physical Size and Layout
• Monitor, Wall, Head-mounted
• Technology
• CRT, LCD, Projection, Volumetric
• Resolution
• Pixel size and density
• Stereoscopic
• Method (Passive, Active, Auto-Stereoscopic)
• Layout (Head-Mounted, Wall, Caves)

9
Desktop Displays

• Standard PC configuration
• 19 LCD display 1440x900 pixels
• 24 LCD display 1920x1200 pixels
• Multiple monitors
• Cheap upgrade
• Small change to workflow
• Parallel productivity
• Lots of options
• e.g. MatroxTripleHead2Go

Single monitor workstation (2.5M pixels)
Dual monitor workstation (5M pixels)
10
Projection

• Cater to large audiences
• E.g. meetings, control rooms, lectures
• Cheap
• General purpose
• Low spatial resolution (large pixels)
• Focus and Context Screen
• Resolution / price tradeoff

www.patrickbaudisch.com
11
Tiled Displays

• Array of consumer displays
• LCD monitors
• Projectors
• Extremely high resolution
• Multiple PCs required
• Synchonisation issues
• Example applications
• Astronomical visualisation
• Geospatial visualisation
• Visual analytics (e.g. intelligence)

VisBox VisWall-LCD (72M pixels)
NASA Hyperwall-2 (250M pixels)
12
Stereoscopic Displays

• Stereopsis Judging depth by comparing images on
  our two retinas.
• Stereoscopic displays provide two images
• Motivation
• Enhanced realism
• Increase immersive feeling
• Support for spatial perception
• Caveat About 5 of the population lack
  stereopsis (stereoblindness) ?

13
Passive Stereo

• Present two images simultaneously and filter for
  each eye
• Cheap glasses (cardboard or plastic)
• (Mostly) standard hardware
• Common uses
• Entertainment (e.g. Movies, theme parks)
• Education (e.g. Planetariums)
• Visualisation (e.g. GeoWall)
• Methods
• Anaglyph
• Polarisation

14
Anaglyph Method

• Famous Red/Blue glasses
• Red/Green, Red/Cyan also popular
• Very cheap (cardboard acetate)
• Superimpose colour-filtered left/right pairs
• Easy to generate in software
• e.g. OpenGL
• glColorMask(GL_TRUE, GL_FALSE, GL_FALSE,
  GL_TRUE)
• No extra hardware needed
• Works on monitors and projection
• Colour reproduction skewed
• Works best for greyscale images

Paul Bourke, http//ozviz.wasp.uwa.edu.au
15
(No Transcript)
16
(No Transcript)
17
(No Transcript)
18
Polarisation Method

• Similar to anaglyph, but filtered by polarisation
• Requires
• Polarizing filters and glasses
• Two projectors
• Still relatively cheap
• Easy to generate and output graphics
• 1. Horizontal Split desktop
• 2. Dual-output video card
• 3. Two projectors
• Image alignment issues

Paul Bourke, http//ozviz.wasp.uwa.edu.au
19
Passive Stereo Examples

• VisionSpace
• Three walls
• Six projectors

• StereoMirror
• Two LCDs
• Beamsplitter

www.planar3d.com
20
Active Stereo

• Frame sequential Interleave left/right frames
• Requires
• intelligent shutter-glasses
• trigger for synchronisation (IR or wired)
• High refresh rate (e.g. 100Hz 50Hz per eye)
• Video card with stereo support
• Easy desktop solution add tracking for
  fishtank VR
• No projection alignment issues (single image
  source)

21
Active Stereo

• On the desktop
• CRT monitor CrystalEyes
• (New) LCD monitor Nvidia 3D Vision
• Projection
• DepthQ 120Hz projector
• Walls and tables

http//www.nvidia.com
http//www.depthq.com/projector.html
22
CAVE

• Cave Automatic Virtual Environment
• Multiple stereo-projection walls
• Passive or active stereo
• Up to six sides (complete enclosure)
• Single user
• immersion with head tracking
• Multiple users
• Can place real objects in VR
• e.g. car simulators

23
CAVE Video
24
Autostereoscopic Displays

• No glasses required
• Trade resolution for depth
• Half the columns for left eye, half for right
• Methods
• Parallax Barrier
• Lenticular Lenses
• Viewing position crucial for effect
• Sweet spot
• Sony WOWvx

25
Volumetric Displays

• Illuminates locations in physical 3D space
• Voxels rather than pixels
• Correct viewing for multiple users
• No accommodation-convergence conflict
• Cant reach into data
• Small display volume

Perspecta Display 1.9 Acutality Systems
Rush University Medical Center
26
Head Mounted Displays

• Support stereovision
• Immersive
• Block out reality
• Full 360 degrees (with tracking)
• 3D sound support
• Typically low FOV
• Lack of peripheral vision
• Can by bulky and uncomfortable
• Single user solution

5DT HMD
eMagin z800
27
See-Through HMDs

• Video See-Through
• Real world seen through camera(s)
• Optical See-Through
• Real world seen directly
• Virtual Retinal Display
• Developed by HIT Lab, 1991
• Image displayed directly on retina

28
Augmented Reality

• Overlay of computer graphics on the real world
• 3D
• Realtime
• Registered with real world
• Applications
• Medical
• Architecture
• Education
• Entertainment

BMW Repair Concept Video
29
Augmented Reality Examples
Brain Structure Visualization
Collaborative Urban Planning
Aerodynamics Visualization
30
Display Summary

• Lots of options and products
• Consider
• Who is the audience? Single or multiple users?
• What visual resolution do you need?
• What is the cost?
• What level of immersion is required?
• Do you perceive stereopsis?
• How will you interact with the display?...

31
Interaction
32
Interaction Techniques

• Methods (hardware or software) allowing a user to
  accomplish tasks in a user interface
• A successful interaction technique
• Requires appropriate mappings between the user
  input and interface actions
• Must suit both the input device and the display
  type
• Different techniques for different interfaces
• Task Select a particular atom in a molecule
• Desktop Point with mouse, click to select
• CAVE Aim with wand, say select

33
Standard Input Devices

• Keyboard
• Fast text entry
• Plenty of buttons
• Mouse
• Define 1D and 2D positions, regions, paths
• Selection from a menu/list
• Quantify control GUI widgets that affect
  variables
• Scroll wheel for zooming

34
Desktop Interaction Techniques

• Well supported in 2D via WIMP metaphor
• Windows, Icons, Menus, Pointers
• Standard set of input styles
• E.g. Point-and-click, drag-and-drop, pop-up-menu
• Screen as a window into information space
• Pan and Zoom
• ArcBall for 3D rotation

35
Desktop Interaction Techniques
Pan and Zoom
ArcBall Rotation
36
Specialised Desktop Devices

• 3D Position and Orientation without tracking
• Pan, zoom, rotate simultaneously
• Push, pull, twist
• 3D Connexion
• SpaceMouse
• Space Navigator

37
Touchscreens

• Increased directness with visualisation
• Multi points of contact
• Multiple fingers (complex gestures)
• Two hand interaction (kinematic chain)
• Multiple users (collaboration)
• All of the above?
• Multitouch examples
• Microsoft Surface
• Perspective Pixel (Jeff Han)

38
Interaction challenges with Large Displays

• Standard devices become less useful (useless?)
• Where is the mouse cursor?
• How do you move the mouse 10,000 pixels but still
  select small objects?
• Options
• Touch screen input
• Use a proxy, e.g. tablet PC
• Adopt strategies from 3D UIs

Tablet Pointing
39
Interaction in 3D User Interfaces

• No concrete standard
• Different 3D interfaces provide different
  interaction mechanisms
• Key tasks in a 3D user interface
• Navigation
• Selection
• Manipulation
• System Control
• 2D input devices can be used, but typically
  specialised 3D input devices are preferred
• E.g. gloves, tracking

40
3D Input Devices

• Data Gloves
• Natural interaction
• Hand motion tracking
• Finger flexure
• Haptic feedback option
• Accuracy? hygiene? fatigue?
• Wands
• Handheld controllers
• Buttons, knobs, dials, joysticks
• Typically 6DOF tracked for pointing, navigating,
  etc

Measurand ShapeHand
PowerGlove, 1989
Intersense IS-900 Wand
ART Flystick
41
3D Tracking Systems

• Provide input by tracking position and/or
  orientation
• An enabling technology of VR
• Head tracking enhances immersion in VR
• Hand tracking for pointing and manipulation
• Tracked tools
• E.g. paint brush for virtual painting
• Motion capture
• Animation

42
3D Tracking Technologies

• Tracking Technologies
• Optical (IR) E.g. ART Tracker
• Magnetic E.g. Flock of Birds
• Inertial E.g. InertiaCube
• Mechanical E.g. Phantom
• Hybrid
• Some characteristics
• Degrees of Freedom (DOF)
• Update frequency
• Tracking range and accuracy
• Absolute vs Relative tracking
• Tethered / Untethered
• Resilience to external factors (e.g. EMI,
  lighting)

43
3D Navigation

• Travel
• Motor component
• Getting from A to B
• Wayfinding
• Cognitive component
• Where am I? Where do I want to go?

44
3D Travel Techniques

• Travel approaches
• Flying metaphor
• Use head or hand tracking to specify direction
• Grabbing the air
• Pulling-rope metaphor (pinch gloves)
• Smooth transitions between points of interest
• Locomotion devices
• Treadmills, bikes
• Wii Fit
• Display dependence
• Physically turning with an HMD works well
• Physically turning in front of a projection does
  not

45
3D Travel Techniques
Flying
VirtualSphere
46
3D Selection

• Specify one or more objects from a set
• Key task as it leads to most others
• Common approaches
• Virtual hand
• Direct mapping, but limited reach
• Arm extension
• Further reach through non-linear mapping
• Ray casting (laser pointer)
• Remote selection but small, distant targets
  difficult
• Hand jitter
• Image Plane techniques
• Reduce 3D task to 2D
• Occlusion problems

47
3D Selection Techniques
Go-Go Arm Extension Technique
Virtual Hand
Raycasting
Image Plane Technique
48
3D Manipulation

• Modify the position/orientation of a 3D object
• Example techniques
• Direct Mapping
• 11 mapping between object and hand
• World-in-Miniature
• Directly interact with a scale model

49
3D System Control

• The user may need to
• Request the system perform a particular function
• Change interaction mode
• Change system state (e.g. modify visualisation
  parameters)
• Taken for granted in 2D
• Approaches
• Graphical menus (2D and 3D)
• Voice commands
• Gestural commands
• Tools (virtual or physical controller)

50
3D System Control Examples
Menu with Raycasting
Forms in 3D with Raycasting
Head up display in AR
51
Custom Devices

• Increase affordance device suits task
• Narrow scope, takes time and effort
• Examples
• Cubic Mouse
• Approaches
• Repurpose conventional devices (e.g. game
  devices)
• IO modules sensors
• E.g. Arduino

52
Feedback Input Devices

• Utilise additional information channel
• Vibrotactile feedback
• Rumble pack
• Haptic (force) feedback
• SensAble Phantom
• Track 6DOF input, provide resistance

SensAble Phantom
53
Physical Monitoring

• Speech recognition
• Discrete vs continuous
• Gesture recognition
• Physiological signals
• E.g. Heart rate, skin conductance
• Eye tracking
• Brain-computer interfaces
• E.g. Emotiv EPOCH uses Electroencephalography
  (EEG)

Emotiv EPOCH
54
Tangible User Interfaces

• Physical form to digital information
• Direct manipulation of bits
• Benefits
• Direct physical interaction
• Exploit natural dexterity
• Tools suited to the application
• Can merge input/output space
• Disadvantages
• Typically not general purpose

Reactable Collaborative tangible music instrument
55
Tangible User Interfaces
URP, Underkoffler and Ishii, 1999
Ishii, 2008
56
TUI Examples
Placing a cutting plane through CT data
(Hinckley et al.)
Illuminating Clay and SandScape MIT Media Lab
57
Tangible Augmented Reality

• TUI concept applied in Augmented Reality
• Tangible, natural input coupled with overlaid 3D
  display
• Examples
• MagicBook
• Physical Book ? Virtual scenes
• Tangible Molecules
• Physical molecules ? Virtual properties

MagicBook (Billinghurst et al.)
Tangible Molecules (Scripps Institute)
58
Focus and Context
59
Focus and Context

• Focus and Context Problem
• The difficulty the user faces in resolving where
  their current region of interest lies within the
  larger information space.
• Large research effort in HCI to develop
  techniques to address this problem
• Problem exists (even compounded) in 3D

60
2D Focus and Context Techniques

• Explicit focus view and context view
• E.g. Thumbnails, minimap, scrollbars

Thumbnails
61
2D Focus and Context Techniques

• Distortion to expand details and diminish context
• E.g. Fisheye view

62
2D Focus and Context Techniques

• Embedded lenses
• Partition the view into areas of focus and
  context
• Magic Lenses
• Generalise magnification to any type of filtering
• Many visualisation applications!
• Reduce clutter, highlight relevant data

63
Focus and Context Techniques in 3D

• Approaches
• Rendering effects
• Depth of field
• 3D Magic Lenses
• Illustrative visualisation
• Exploded views

Semantic Depth of Field
64
3D Magic Lenses

• 3D volume of interest
• Cutaway views
• Data filtering
• Multiple data layers

3D Lenses for Cutaway Views
Geographic Visualisation with Spherical Lens
3D Lens for managing complexity
65
Illustrative Visualisation

• Incorporate principles from technical
  illustration
• Adapt volume rendering to volume illustration
• Degree of Interest (DOI) function
• E.g. Transparency determined by distance to
  user-defined clipping plane
• E.g. Transparency determined by user viewpoint,
  object of focus, semantic rules

66
http//www.cg.tuwien.ac.at/research/publications/2
005/bruckner-2005-VIS/
67
Exploded Views

• DOI becomes degree of explosion
• Extract parts to illustrate and expose internal
  structure
• Mitigates occlusion problems
• Exposes area of interest while retaining overall
  context

68
Exploded Views Video
69
AR Magic Lens Visualisation

• Visualisation aid for AR
• Tangible Input device
• Augmented Reality
• FocusContext visualisation
• Magnifying glass metaphor

70
AR Magic Lens
71
Summary

• Many types and variations of display and input
  device
• Interaction techniques map user input to
  interface actions
• Must consider input device and display
• Appropriate mappings are vital to usability
• Consider specialised devices and tangibile input
• Challenges
• Finding a combination of tools that suits your
  visualisation needs (and budget!)
• Achieving a high level of usablility
• Wide FOV stereo head tracking can be powerful
• Can also achieve a lot on the desktop!

72
Questions

• Thanks for listening!
• Come see some of this tech at the HIT Lab
• Contact julian.looser_at_hitlabnz.org