Visual Analytics in Vancouver The Western Tide

1
Visual Analytics in VancouverThe Western Tide

• Brian Fisher
• John Dill
• Ron Rensink

2
Initial Canadian involvement in VA

• Aug 2004 - RD Agenda Panel
• John Dill
• William Wright (Oculus)
• Early 2005 - Contribution to Illuminating the
  Path
• John Dill
• William Wright
• Brian Fisher
• Apr 2005 - John Dill named first NVAC Scholar

3
Vancouver VA

• UBC Media Graphics Interdisciplinary Centre
  (MAGIC)
• Use-inspired basic research
• Co-development w. industry
• SFU School of Interactive Arts Technology
• Design school ICT Cogsci
• Interdisciplinary Ph.D, MSc. BSc.
• Joint progs w Business, Cogsci, CompSci,
  Communications
• New SFU Business School partnership

4
Visual Analytics Disciplines
Cognitive Science
Information Systems
Visual Analytics
Mathematical Statistical Methods
5
Cogsci research examples

• Spatial cognition action w. large screens
• Wayfinding in VR (Boeing)
• Space constancy in display environs (AFOSR)
• Multimodal interaction (AFOSR)
• Gesture voice control (NSERC)
• Item tracking in projected 3D displays (HRL)
• Depth judgments w.stereo displays (GMR)
• Psychophysical methods (Fechner)

Focus on cognition in complex displays, how
changes in the rules affect human performance
6
Isnt This Just Usability Testing?

• Powerpoint example
• Usability metric Reduce time/effort/errors
• Cognitive goal Support authoring process
• GMR Visual Eyes VR CAD Cognitive Goal
• Differences in ability of managers to judge
  design
• Test indiv. diffs in perceptual cognition
  (psychometrics)
• Boeing Engineerings Cognitive Goals
• Effective communication in collaborative CAD
• Communicative pragmatics in technology-rich
  environments (grounded theory Clark)

7
Need for a Scientific Approach

• Technical, functional and operational drivers
• Enhanced technical capabilities through
  Perceptually rich interactive displays
• Function spread Cognitive (social, hedonic)
  depth of cognitive performance is goal
• Diversity of users Novices, experts,
  multidisciplinary teams
• Result Complex design space with weak
  performance metrics, user testing gap
• Design stalls for lack of knowledge

8
Steam Power Science Tech.
30
Carnot theory and thermodynamics developed
20
Watt (1864).Separate condenser based on
measurements of latent heat.
thermal efficiency ()
10
1800
1900
1700
9
Small screen interface
Large screen interface
?
?
Accommodate users ranging from novice ? expert
collaborating on tasks that require analysis,
judgment, and coordination
Bill Buxton/Dave Kasik
10
Research approach

• Decompose task based on cognitive architecture
• Identify interaction challenges/opportunities for
  cognitive processing modules
• Dotoy world test of that modules response
• Use-inspired basic research
• Research interaction, not the mind
• Model data for quantitative prediction

11
Scientific Questions from Industry

• Diversity of users (D.Kasik, Boeing)
• We want to derive more value for 3D model data
  outside engineering... Examples 3D for
  assembly instructions, animation for maintenance
  procedures, bids from suppliers.
• Individual differences (R.Smith, GM)
• For a smaller number of individuals.. some
  features are seen to be inappropriately large or
  small, they may appear at the wrong distance, the
  three-dimensional space inside the vehicle may
  appear distorted
• Problems with cognitive processing display
  information by more diverse user population

11
12
VR in GM Design Centre
13
Experiment 1- real vs. virtual
Brian Fisher bfisher_at_sfu.ca
www.interactionscience.org
13
14
Results - Experiment 1
Brian Fisher bfisher_at_sfu.ca
www.interactionscience.org
14
15
Experiment 2 - 2AFC
Brian Fisher bfisher_at_sfu.ca
www.interactionscience.org
16
Results - Experiment 2
Brian Fisher bfisher_at_sfu.ca
www.interactionscience.org
17
Exp 3 Method of Adjustment
18
Results
Brian Fisher bfisher_at_sfu.ca
www.interactionscience.org
18
19
Error - all subjects
Brian Fisher bfisher_at_sfu.ca
www.interactionscience.org
20
Air traffic control w. HRL

• Free Flight ATC fishtank projection
• Change camera position for better view
• How will global motion affect tracking?

21
FINST theory of spatial indexing
22
Multiple object tracking (Pylyshyn)
Brian Fisher bfisher_at_sfu.ca
www.interactionscience.org
23
Multiple Object Tracking (Pylyshyn)
24
Fit human tracking function (Lui)
Brian Fisher bfisher_at_sfu.ca
www.interactionscience.org
25
... Then add display motion
26
Tracking vs object speed
27
Tracking in warped space
Brian Fisher bfisher_at_sfu.ca
www.interactionscience.org
28
Tracking in warped space
Brian Fisher bfisher_at_sfu.ca
www.interactionscience.org
29
Conclusion We track in allocentric space

• Retinal speed of targets does not determine
  performance
• Motion of targets relative to each other does
• But only if motion preserves good metric
  characteristics of space
• Explanation is at the level of a human - display
  cognitive system

30
Applying 2 visual systems to VR

• Regularities of graphical content vs. world
• Object and scene onsets (cuts) common
• Point of view pans, zooms etc.
• Conflict between multimodal cues common
• Conflict between acceleration and vision
• Conflict between synthetic cues (e.g. depth)
• Does immersion magnify these effects?
• Do they differ between the 2 visual systems?

31
Some 2 visual system illusions
32
Apparatus
33
2VS interaction in the large

• Task Localize target with voice (far left, near
  left centre etc.) or pointing
• Induced Roeloffs Effect displaced frame should
  cause ventral system errors (ASC), but not dorsal
  (FSC)

5 target positions 3 frame offsets
34
Experimental protocol

• Display appears, then disappears
• Verbal report pointing tasks
• 1 block of verbal report
• 1 block pointing with no visual feedback
• 1 block pointing with visible pointer
• 1 block pointing with 1/2 second lagged pointer
• Within subject psychophysical analysis
• Meta-analysis of proportion of subjects
  exhibiting effects

35
Findings

• Can you tell if a target is on the left or right?
• 3 out of 7 males, 7 out of 7 females made errors
• Can you point to it without seeing your hand?
• 6 out of 10 who failed 1 were correct
• Are you better with a (simulated) laser pointer?
• Out of 6 who point accurately in 2, all fail
• Will pointing accuracy be affected if visible
  pointer lags pointing?
• 3 of the 6 who failed 3 succeed

36
Subject data for pointing study
37
Results

• Displaced frame leads to verbal errors (ASC)
• Most subjects who made verbal errors did not make
  pointing errors (FSC) w/o cursor.
• Visual feedback hurts (FSC -gt ASC)
• Time lagged feedback (gtFSC) helps
• Fits predictions of 2 visual systems theory
• Pointing (dorsal) more robust against illusion
• Feedback shifts to ventral, increases illusion

Less information better performance
38
Tasks and theories

• Large-screen immersive display dynamics
• FINSTs and indexical cognition
• Multimodal events (sight, sound, touch)
• Sensory integration (FLMP, calibration by
  pairing)
• Understanding and acting in space
• Functional Apparent Space Constancy

39
Isnt This Just Psychology?

• Start with Psych, e.g. Psychophysics Lit.
• Basic perceptual bottlenecks
• Colour theory
• Attention models
• Space constancy
• But Psych does not focus on novel perceptual
  experience cognitive tasks encountered in
  display environments
• Statistical regularities of multiple depth cues
  in VR
• Problem of camera movement in air traffic control
• Large displays and human space constancy

39
40
Science to Support Engineering

• Use-inspired...
• Questions from users infosystem designers
• Focus on understanding cognitive systems fluent
  interaction w. visually rich interfaces for
  timely accurate cognition communication by
  particular set of users
• ... basic research
• Scientific methods, field studies experiments
• Goal is knowledge predict results given initial
  conditions
• Method is (so far) falsification disprove a
  hypothesis, reject an approach

40
41
Expanding Canadian RD Capability
2) Network for comprehensive Canadian VA capacity
1) Build VA science design
3) Bridge disciplines stakeholders
Visual Analytics
42
Build Design Cognitive Science of VA

• NSERC SPP VA for Safety Security
• Application-aware basic research
• Perceptual spatial cognition stream
• Enactive cognition and expert use stream
• Social cognition, metacognition stream
• Boeing, MDA, CAE, Oculus, NVAC, PuRVAC, PARVAC
• Outreach to scientific community
• International Cogsci society VA Symposium
• Outreach to design community
• VA theme in Canadian Design Research NCE

42
43
New Industry Support

• First IRB cheque from Seattle 275K/yr x 5 years
• Canadian government will match 11 or better
• University takes 18 overhead
• Goal improve business intelligence
• Increased reliability, maintainability, safety,
  etc.
• This project will take two essential steps
• Jointly explore VA in industry databases
• Produce core VA curriculum and skilled graduates
• Area experts will apply results more readily
  because we will start with real-world data

43
44
Comprehensive Canadian VA

• Mathematics and Statistics
• SFU Inst. for Research in Mathematics and
  Computing Science (IRMACS)
• Mathematics of Information Technology and Complex
  systems (MITACS) Network of Centres of Excellence
  (NCE) for the Mathematical Sciences.
• Information Communication Technology
• UBC SFU Computer Science Engineering
• National Research Council of Canada
  Communications Research Centre of Canada
• Colleagues in Alberta, Quebec, Maritimes, Ontario

45
Goal Full VA Capability in Canada

• New PSTP support, CNVAC will support CanVAC for
  collaboration with NVAC/RVACs
• Vancouver VA Science workshop Feb 2-4
• Coast-to-coast Canadian gov, academic
  registrants, industry from BC mostly
• JK, JT, all RVAC heads speak
• Launch Can Net of VA Centres
• www.CNVAC.ca portal
• Advanced features Virtual lab, user-controlled
  lightpath shared viz, PNNL Active Products

45
46
Training aspects

• Visual (business, engineering) Analyst
• Understands cognitive constraints
• Perceptually rich human-information discourse
• Tech-mediated social cognition collaboration
• Learning model, personal equation
• Analytics Designer
• Understands cognitive constraints
• Field research skills
• Customize design for specific users tasks
• Interaction scientist
• Builds and applies new research methods that
  focus on knowledge for engineering

47
Western Visual Analytics Cluster
Brian Fisher bfisher_at_sfu.ca
www.interactionscience.org
47
48
New Directions, Challenges
49
Challenge Grounding science in practice

• Field methods to ground research in customer
  tasks, situations, and practices
• Must address core cognitive, perceptual and
  enactive abilities and bottlenecks
• Must specify laboratory studies that can extend
  that knowledge
• Must integrate knowledge in design of new
  technologies
• Need access to stakeholders, situations, users

50
Topic Expert cognition-in-action

• Focus on human-information discourse
• Perceptual motor learning and expertise
• Tight-loop perception/action coordination
• Cognitive processes as supervisory control
• Temporal coordination Rhythms of events flow
  of cognitive processing (individual and group)
• Interaction methods to support underlying
  processes
• Customization for individuals personal
  equation
• Tracking training development of interactive
  expertise
• Experiments on toyworlds, need new DVs and
  analyses

50
51
TopicMetacognition in joint activity

• Grounded theory Psycholinguistic pragmatics
  extended to interactive environments
• Support for familiar F2F metachannels
• Embodied communication (gesture, facial
  expression, body language, prosody
• Interaction methods to support underlying
  metacognitive processes
• Acknowledgement and repair mechanisms
• Advancement
• Layering
• Experiments on toyworlds, need new DVs and
  analyses

52
Tactical challenges

• Plus Success at integrating gov/industry/research
  for IRB funding
• Minus Many toes stepped on, some casualties on
  our side
• Plus Rapid buy in from SFU Business School, UBC
  Cognitive Systems
• Minus What are they telling people?!
• Plus Well-integrated cross-university
  collaboration can happen
• Minus Lots of work, mostly political

53
Levels of analysis of interaction

• Usability Optimizing use of system functions
• Apply knowledge from science ergonomics,
  perception, cognition
• Cognitive task analysis (e.g. info foraging)
• System functions compatible with human cognitive
  processes
• Interaction science
• Study perceptual, cognitive, and control actions
  in technological environments
• Native science of interaction

54

• "Il n'existe pas une catégorie de sciences
  auxquelles on puisse donner le nom de sciences
  appliquées. Il y a la science et les applications
  de la science, liées entre elles comme le fruit à
  l'arbre qui l'a porté"
• Louis Pasteur