CNS

1
Virtual Experiences for Rehabilitation, Training,
and Evaluation

• CNS
• June 3rd, 2004
• Benjamin Lok

2
Overview

• Virtual Experiences Group
• Virtual Limbs for Stroke Rehabilitation
• Virtual Reality for Engineering Design
• Virtual Characters for Interpersonal
  Communications Training

3

• Applications Driven
• Combines
• Interactive Computer Graphics
• Computer Vision
• Human Computer Interaction

4
Virtual Experiences Group

• 2 PhD students
• 1 MS
• 4 Undergraduates
• 15x15x10 wide area tracker
• Virtual Research V8 HMD
• 42 Plasma TV
• 4 data projectors
• 120 passive stereo display
• Tablet PC
• Intersense InertiaCube
• Cyberscan 3D Rapid Color Scanner

• Collaboration with expertise in
• Virtual Reality
• Digital Arts
• Image Processing
• Computational Geometry
• Human Computer Interaction
• Image Based Rendering

5
Virtual Experiences Group

• PhD Students (3)
• Kyle Johnsen (B.S. UF)
• Cyrus Harrison (B.S., M.S. (expected) UF)
• Andrew Raij (B.S., M.S.)
• MS Students (1)
• George Mora (B.S. UF) DAS
• Undergraduates (4)
• Sayed Hashimi (S)
• Robert Dickerson (S)
• Patrick Dugan (S)
• Art Homs (S)

6
Research Interests

• Computer Graphics computer scientists are
  toolsmiths
• Applying graphics hardware to
• 3D reconstruction
• simulation
• Visualization
• Interactive Graphics
• Virtual Reality
• What makes a virtual environment effective?
• Applying to assembly verification clinical
  psychology
• Human Computer Interaction
• 3D Interaction
• Virtual Humans
• Assistive Technology
• Computer Vision and Mobile Technology to help
  disabled

7
Collaborators

• Undergraduate Students
• Patrick Dugan
• Brain Rehabilitation Research Center
• Treven Pickett, Keith White, Charles Levy

8
Virtual Limbs for Stroke Rehabilitation
Before Stroke

• Problem Proper signals to muscles are no longer
  sent after stroke
• Difficulty No positive feedback for progress
  towards goal (all or nothing)
• Chuck Levy, Keith White

Extensenor
Contractor
I want to bend my hand back
9
Virtual Limbs for Stroke Rehabilitation
After Stroke

• Problem Proper signals to muscles are no longer
  sent after stroke
• Difficulty No positive feedback for progress
  towards goal (all or nothing)

Extensenor
Contractor
I want to bend my hand back
10
Virtual Limbs for Stroke Rehabilitation
After Stroke Session 1

• Solution Use VR to provide positive feedback
• With VR, we can provide any visual feedback the
  doctor wishes.

Extensenor
Contractor
I want to bend my hand back
11
Virtual Limbs for Stroke Rehabilitation
After Stroke Session 2

• Solution Use VR to provide positive feedback
• With VR, we can provide any visual feedback the
  doctor wishes.

Extensenor
Contractor
I want to bend my hand back
12
Virtual Limbs for Stroke Rehabilitation
After Stroke Session 3

• Solution Use VR to provide positive feedback
• With VR, we can provide any visual feedback the
  doctor wishes.

Extensenor
Contractor
I want to bend my hand back
13
System Design

• Uses commodity components

14
Virtual Limbs for Stroke Rehabilitation
After Stroke

• Treven Pickett
• Use EEGs to control visual feedback

I want to bend my hand back
15
System Design

• Connect to EEG machine

16
Other possibilities

• Render virtual limbs where no limbs exist
• Render mirrored limbs

17
Collaborators

• Benjamin Lok
• University of Florida
• Danette Allen
• NASA Langley Research Center

18
Overview

• Computer generated characters and environments
• Amazing visuals and audio
• Interaction is limited!
• Reduce applicability?
• Goals
• Create new methods to interact
• Evaluate the effectiveness of these interaction
  methods

Aki from Final Fantasy The Spirits Within
Walking Experiment PIT - UNC
19
Objects in Immersive VEs

• Most are virtual (tools, parts, users limbs)
• Not registered with a corresponding real object
  (limited shape and motion information)
• Example Unscrewing a virtual oil filter from a
  car engine model
• Ideally handle real objects
• improved look, feel, affordance, interaction
• Solution tracking and modeling dynamic objects

20
Merging real and virtual spaces
21
Case Study NASA Langley Research Center
(LaRC)Payload Assembly Task

• Given payload models, designers and engineers
  want to evaluate
• Assembly feasibility
• Assembly training
• Repairability
• Current Approaches
• Measurements
• Design drawings
• Step-by-step assembly
• instruction list
• Low fidelity mock-ups

22
Task

• Wanted a plausible task given common assembly
  jobs.
• Abstracted a payload layout task
• Screw in tube
• Attach power cable
• Determine how much space should be allocated
  between the TOP of the PMT and the BOTTOM of
  Payload A

23
Videos of Task
Having a hybrid environment provides substantial
benefits in prototype design.
24
Results
The tube was 14 cm long, 4cm in diameter.
25
Results

• Late discovery of similar problems is not
  uncommon.

26
Case Study Conclusions

• Object reconstruction VEs benefits
• Specialized tools and parts require no modeling
• Short development time to try multiple designs
• Shows promise for early testing of subassembly
  integration from multiple suppliers
• Possible to identify assembly, design, and
  integration issues early that results in
  considerable savings in time and money.

27
Projects underway

• Getting real objects into VR to aid engineering
  design
• Collaboration w/ Mars Airplane (Langley Research
  Center)
• Get tools, parts, and other (possibly
  distributed) collaborators in a shared space

28
Multimodal Interaction with Life-Sized Virtual
Characters for Training and Education

• JC Cendan, A Stevens, DS Lind
• Department of Surgery (College of Medicine)
• M Duerson
• Department of Community Health and Family
  Medicine (College of Medicine)
• B Lok, C Harrison, K Johnsen,R Dickerson
• Department of Computer and Information Science
  and Engineering (College of Engineering)
• The University of Florida, Gainesville, FL
• M ShinDepartment of Computer Science
• The University of North Carolina at Charlotte,
  Charlotte, NC
• Florida Technology Transfer Conference
• May 17, 2004
• Orlando, FL

29
What is a Virtual Character?

• Virtual character - a character who represents
  the state of a system
• In TRON (1982), humans and humans that represents
  software interacted within a world that
  represented the hardware.

30
What is a Virtual Character

• We look to to have humans and human the represent
  software interact in the real world.
• http//movies.yahoo.com/shop?dhvid1807432839cf
  trailer

31
Motivation

• Doctor, I have a pain in my side!
• What you do next depends on
• Age
• Gender
• Ethnicity
• Visual Cues
• Audio Cues

32
Motivation

• Medical Diagnosis Components
• Patient Interview
• Physical Examination
• Current Teaching Methodologies
• Books
• Standardized Patients (actors)
• Results in
• Minimal training frequency
• Minimal scenario variety
• Lack of immediate feedback
• Medical students are not adequately prepared for
  many diagnosis scenarios

33
Solution

• Interactive Virtual Characters
• Life-Sized
• Computer Generated
• Natural Interaction
• Responds to User
• Responses based on accepted medical procedure

• COTS Equipment
• Projectors
• Laptops
• Web cameras
• Tablet PC

34
Solution

• Interactive Virtual Characters
• Computer Generated
• Natural Interaction
• Responds to User
• Responses based on accepted medical procedure

• Results in
• More Scenarios
• More Training
• Standardized Experiences

35
Why do we want digital characters

• Propose Digital Characters as a new (meta-)
  medium to interact with information
• Why would we want a digital character?
• Effective Interaction
• Better than keyboard and mouse for certain tasks
• Dynamic (output easily augmentable)
• 3D
• Natural interaction
• Low Bandwidth
• Effective Collaboration
• Controlled conveyed visual information
• Non-verbal communication (60)

36
Lets look at interaction

• Each participant in a communication has three
  stages perception, cognition, and response
• Define interaction as both the input and output

Digital Character
Thinking
Perceiving
Responding
Interaction
Responding
Perceiving
Thinking
Participant
37
Combines

• Speech
• Computer Vision
• Eye Gaze
• Gesture Recognition
• Repeat your gestures
• High Quality 3D models
• Animation
• Rendering
• Visualization

38
Integrates

• Computer Science
• Computer Graphics
• Image Processing
• Natural Language Processing
• Medicine
• Training
• Education
• Standardization
• Education
• Multimedia Learning
• Technology based Learning
• Training

39
Current and Future Work

• Current Status
• Initial scenario created with gesture, speech,
  and visualization components integrated
• Evaluating with a group of Medical Students Year
  2
• Future Work
• Formal evaluation studies
• Increase and improve scenarios
• Enhance interactivity

40
Thanks