Immersive Virtual Humans for Interpersonal Skills Education

1
Immersive Virtual Humans for Interpersonal Skills
Education

• Benjamin Lok
• Computer and Information Science and Engineering
  Department
• University of Florida
• University of Central Florida
• November 30th, 2007

2
Virtual Humans

• A virtual character
• Human form
• Application domains
• Games
• Movies
• Simulation

3
Virtual Human Experiences

• Not as a substitute interface
• But the experience is the interaction with the
  virtual human itself
• Fundamentally different than other VEs
• Research goal Through interacting with virtual
  humans, can we
• Teach
• Affect
• Change

4
Immersive Virtual Humans for Educating Medical
and Pharmacy Communication Skills

• K. Johnsen, A. Raij, B. Rossen, A. Kotranza, X.
  Wang, B. Lok
• Computer and Information Science and Engineering
• J. Cendan
• Community Health and Family Medicine D. Beck, C.
  Kimberlin
• Pharmacy
• R. Ferdig
• Education
• A. Deladisma, D. S. Lind
• Surgical Oncology
• S. Chapman, L. Bracegirdle
• Pharmacy

5
Can Virtual Humans Enable
6
VOSCE Project Overview

• Started Spring 04
• n gt 327 students
• Medical
• Nursing
• Physician Assistant
• Pharmacy
• One of the most popular VH/VR experiences
• Three institutes
• University of Florida
• Medical College of Georgia
• Keele University (U.K.)
• Team
• VR/HCI/CS 1 PhD, 6 grad students, 2 undergrads
• Medicine 6 MDs, 2 medical students
• Education 2 PhDs
• Research focus on interfaces

7
T Interpersonal Simulator (IPS)

• Platform for exploring interpersonal interactions
  with VHs
• Natural interaction with a high level of
  immersion
• Real-world application Medical interview
  training
• Simulate a standardized patient encounter
• Virtual patient, DIANA
• Virtual instructor, VIC
• Communication skills

8

• Play Video XXX

9
Why Virtual Humans?

• Students
• Repetition
• Feedback
• Longitudinal learning
• Educators
• Standardization
• Dynamic
• Abnormal findings
• Cultural competency
• Aggregate performance
• Researchers
• Study the extent of impact of VHs
• Easy to run studies (Twiddle one thing)

10
IPS System

• Inputs
• Natural speech
• Tracking data (head, hands, chair, tools)
• Video
• Physiological measures
• Outputs
• Speech and animation
• Life-size projection (or HMD)
• Perspective correct rendering
• Reactive virtual human
• COTS components
• 3 PCs
• 2-4 video cameras
• Data projector, large-screen TV, or HMD
• Wireless microphone
• Bodymedia Sensewear
• lt 10,000(USD)
• Potential
• Every Hospital

11
Natural Interaction

• No Keyboard, No Mouse
• Speech Recognition
• Dragon Naturally Speaking 9 Pro
• Accuracy 90 with 10 minutes training
• 50-70 match to database
• Track Communication Cues
• Non-Verbal (posture)
• Verbal
• Physiological
• Gestures (pointing, handshake)
• DIANA and VIC look at user
• Why this works
• Does not rely on complete sentences
• Constrained scenario
• Students trained on specific questions
• Johnsen VR2005, Presence

12
Studies

• (- July 2007) 14 studies
• Focus
• Validity
• Similarity/differences
• Course integration

13
VP ? SP

• How is experiencing an interpersonal scenario
  with a virtual human similar to and different
  from experiencing an interpersonal scenario
  with a real human?
• Clearly different, but in what important ways?

14
VP ? SP

• n 57
• Two studies
• (between-subjects) Students interviewed VP or SP
• (within-subjects) Students interviewed both VP
  and SP
• Measures
• Content of interaction
• Expert video coding (student behavior, empathy,
  rapport)
• Post-experience questionnaires
• Goal is to identify
• Extent to which interpersonal scenarios can be
  simulated with VH
• Component technologies which need to improve to
  enable effective VH systems
• Raij IEEE VR 2006,TVCG 2006
• Stevens, Southern Group on Educational Affairs
  2006

15
VP cSP (c0..1)
Group V n 49
Group R n 33

• Overall content similar
• Questions asked
• Pain sharp and stabbing (R 100, V 80)
• Sexually active (R 54, V 45)
• Global measures
• Education goals met
• Students rated educational merits similarly
• Students rated difficulty similarly
• Tool to augment SPs

10 minute Interview VH
10 minute Interview SP
Experts assess participants
16
Social Conventions

• Conversation flow is rapid-fire
• Confirmatory phrases (SP 20, VP 3.5)
• Rapport-building different
• Empathetic moments
• Im scared, can you help me?
• Could this be cancer?
• Sneezing
• Responses
• Exists but reduced quantity quality
• Expert ratings higher of SP experiences
• Spontaneous empathy (R 84, V 24)
• Nonverbal behavior different
• Body lean (R 2.9 0.9 , V 2.1 0.6, p lt
  0.001)
• Deladisma, American Journal of Surgery 2007
• Cohen, Southern Group on Educational Affairs
  2006, Association for Surgical Education 2006

17
Validity

• n33
• Each student interviewed VP and SP with similar
  complaint
• Interaction was rated by expert
• Information (r0.34, r20.12)
• Process (r0.19, r20.03)
• Quality (r0.38, r20.14)
• Overall (r0.50, r20.25, plt0.005)
• Predictor
• Good w/ VP Good w/ SP
• Bad w/ VP Bad w/ SP
• Valid experience
• Despite technology, believability
• Proven valid VH experience!
• But measures can be misleading
• Global vs. local
• Highlights current limitations
• Johnsen, CHI 2007
• Bernard, Society of Medical Simulation Meeting
  2006

18
Characterizing a Human-Virtual Human Interaction

• Self-reporting measures
• Easy
• Minimal power
• Expert measures
• Video coding and evaluation
• Powerful
• Logistical issues
• Need a tool/metric that is
• Quantitative
• Encompassing
• Why?
• Evaluate What is a good
• How good is it?
• What makes it good?
• Limits How good can it be?
• For each interaction
• Sensors log
• Analyze log file for data

19
Sample the Interaction
Interaction Log File
Tracking Data User (posture, gaze) Tools Physiolog
ical Data
20
Interaction Log File
Tracking Data User (posture, gaze) Tools Physiolog
ical Data
21
Interaction S(Interaction Signals)

• Challenges
• What signals matter?
• How to sample signal?
• Data can be
• Missing
• Noisy
• Discritized
• How to visualize signals for users?
• Evaluate?

• Interpret interaction as a signal
• Gain insight from signals via
• Signal analysis
• Pattern recognition
• Visualization
• Data mining

22
Interaction S(Interaction Signals)

• User and virtual human talking
• What topics does the user focus on?
• Does the user interrupt the virtual human?
• Empathy
• How well does the user respond empathically?
• User behavior
• How much does posture vary?
• Where does the user gaze?
• Evaluation
• How does an interaction compare to previous
  interactions (student portfolio)?
• How does an interaction compare to others?
• Can we identify poor performing students?
• Goal Automatic evaluation and feedback

23
Interactive Visualization

• Explore Signals
• Display
• Playback
• Multiple views

• Since we log the conversation
• We can recreate any moment of the conversation
• Novel views

24
IPSViz

• n 27
• After an interaction, students receive email and
  dload interaction
• Analogous to flight simulators
• Users
• Students
• Self-reflection
• Feedback
• Educators
• Quantitative analysis
• Identify trends and outliers
• Researchers
• Capturing H-VH interaction
• Analysis
• Process, Filter, Visualize

25
After Action Review of H-VH Interactions

• Users do not remember how they behave in H-VH
  interactions.
• Participants self-reported that (7-point Likert)
• they did not appear as friendly (pre 5.5, post
  4.6, plt0.001)
• or act naturally to the VH
• (pre 3.8, post 2.7, plt0.001)
• Students reevaluated their empathy with a VH
  (scale 1-4)
• (pre 2.5, post 2.2, plt0.001)
• Students reported specific ways they would act
  differently in future real-world interactions.
• It was good to watch myself. See how I act, and
  to hear my voice, how I ask questions the
  intonation in my voice. And after doing it I
  can think about it more, you know, things I could
  have done. And then when I go back and view, I
  can see ways I could have changed.

Raij VR 2008
26
See What It Was Like Talking To Yourself!

• Viz Vid

27
IPSVizn Log files for a student

• Student
• Portfolio of VH interactions
• Accreditation, track learning
• Educator
• Evolution of performance
• Researcher
• Longitudinal learning

28
IPSVizn Multiple students log files

• Educator
• Aggregate performance
• Identify trends/outliers
• Researcher
• Examine logistically complicated topics
• E.g. Uncanny Valley

29
Physiological Measures

• We need new measures to characterize H-VH
  interactions
• Physiological measures
• Blood pressure
• Heart rate
• Heat Flux
• Skin Temperature
• Galvanic Skin Response
• Bodymedia Sensewear
• We can then look at
• Are VH affecting people?
• Are VH changing people?
• Real-time input to simulation
• Scenario
• Anxiety (sex history)
• Empathy challenges
• Study n27
• Experience correlated to anxiety measures w/ a VH!

VHs challenge Can you help me? Im really
scared.
VHs challenge Could this be cancer?
30
Sexual History

• n9, SP only
• n10, VPSP
• Measures Blood pressure, heart rate
• ? SBP 9.31 (SP only), -0.47 (VPSP)
• ? HR 7.76 (SP only), -0.47 (VPSP)
• Taking a sexual history with DIANA makes a
  student less nervous with taking a sexual history
  with a SP
• Self-reported anxiety was not different
• DIANA is changing people, but they dont know it
• Mack, Society for Simulation in Healthcare 2007

31
Bias IPSViz
Interact with VP 1
Interact with VP 2
32
Bias

• Multiple log files per student
• Visualize bias trends differences
• Study n 27
• Interact with either a Caucasian or
  African-American VH
• Explicit and Implicit Bias
• Psychology measures (IAT, Friendliness scales)
• Correlation between users opinion of VH w/
  measures
• VHs can elicit racial biases
• Study (n9), people reported that the
  African-American VH had less education and money
• Identical animations, words spoken
• Differences skin-tone, voice
• Other biases to study
• Race/Ethnicity
• Gender
• Age
• Weight
• Social hot topics

33
Signal Analysis of Bias

• Audio signal analysis
• A students similar audio with two VHs
• Identify differences

34
Component Evaluation

• Study how system components impact interaction
  with a VH
• Recorded Speech vs. Text-to-Speech
• Similar content
• Less engagement with TTS
• Surprisingly, speech understanding does not need
  to be that good
• Dickerson, SIMCHI 2006, MMVR14
• HMD vs. Projector Displays
• HMD more engagement and immersion
• Increased encumbrance
• Perceptions of DIANA affected
• Self-evaluation of empathy
• Johnsen VR 2008

35
Mixed Reality Humans

• Virtual humans have limitations
• Open research problems (e.g. AI, speech)
• No tactile feedback
• Merge the real and virtual spaces
• Real tools
• Real simulators
• Complete patient interaction
• Physical Exams (e.g. eye exam)
• Point at eye chart
• Ophthalmoscope
• Follow my finger
• 1 or 2 fingers?

36
Integrate VP with Simulators

• Medical simulators (e.g. mannequins) are becoming
  accepted educational tools
• However, they
• Focus on a small set of tasks
• Interaction is different than clinical practice
• Integrate virtual patient with simulators
• Interview virtual patient
• Interact with simulator
• VP responds to simulator

37
Breast Simulator Integration

• Breast simulator integrated
• Dr. Carla Pugh, Northwestern
• Student does a patient history
• Asks to remove gown (physiological measures)
• Performs a breast exam
• VP winces at too much pressure
• Future work
• Pelvic simulator
• Central line simulator
• Kotranza VR2008

38
Abnormal Findings

• Conditions difficult to represent with existing
  education methods
• Conditions
• Psychomotor
• Neurological
• Age/race/ethnicity dependent conditions
• Blurry vision scenario
• Cranial Nerve III (due to brain tumor)
• Corneal ulcer
• Retinal detachment
• Benefits
• Curricular planning of medical student exposure
• Supplement SP experiences
• Leverage the dynamic nature of VPs
• Play Video

39
Current Work

• Classroom incorporation
• Communication course
• 2nd year MS
• Surgical rotation 3rd year MS
• Pharmacology 1st year
• ngt120 per year
• Potentially 30,000 interactions a year _at_ existing
  institutions
• Virtual Instructor
• Real-time response to tracked cues
• Physiological measures
• Posture cues
• Verbal cues

40
Museum of Science and Industry

• Science museum in Tampa, FL
• 5th largest in US (75,000 sq ft.)
• 1 million guests
• Integrate a VH interaction
• The Amazing You exhibit
• Lifelong wellness information
• Fall 08
• Inspire, Inform
• Explore race, VH bias

41
Scenarios and VPs

• Acute abdominal pain
• Breast mass
• Dyspepsia
• Sexual history
• Eye exam
• Patients
• DIgital ANimated Avatar (DIANA)
• Elderly Diana (Edna)
• Manniquin Diana (Mandi)
• Building the following VPs
• Male
• Personality (e.g. irate)
• Intelligence (e.g. mentally retarded)
• Appearance (e.g. disfigurement, limbs, burns)

42
Pain Studies

• Prof. Mike Robinson, Clinical and Health
  Psychology
• Perception of pain
• 1 Independent variable

43
Augmented Reality Medical Devices

• Combine the abstract (Virtual Anesthesia Machine)
• With the concrete (Anesthesia Machine)
• Bridge learning

44
Milestones

• August 2004 First participant
• October 2005 First angry student
• May 2007 First participant who expressed a
  behavior change
• October 2007 First crier

45
Virtual Experiences Research Group

• PhD Students
• Kyle Johnsen, Aaron Kotranza, John Quarels,
  Andrew Raij, Xiyong Wang, Brent Rossen
• Undergraduates
• Harold Rodriguez, Anna Vittone
• Funding
• National Science Foundation (CAREER, REU),
  University of Florida Colleges of Engineering and
  Medicine, Medical College of Georgia, Keele
  University, School of Pharmacy

46
Graduating Students (looking for post-docs)

• Kyle Johnsen
• Validation
• Tracking
• Study Design
• http//www.cise.ufl.edu/kjohnsen
• Andrew Raij
• Virtual Human Human?
• Visualization
• http//www.cise.ufl.edu/raij
• Harold Rodriguez
• Graduate school

47
Join Us!

• We need your expertise!
• Looking for grad students
• Additional locations to install and test system
• Thank you!
• http//www.cise.ufl.edu/research/vegroup
• Questions?

48
Studies

• 2004 (n20)
• April Project initiated
• August Prototype (n7) UF
• October Experts (n3) UF
• December Pilot Test (n10) UF
• 2005 (n81)
• June Two Institutions (n16) UF/MCG
• July VP vs SP (n16, n8) UF/MCG
• October Cultural Bias (n16) MCG
• October Class Integration (n33) UF
• 2006 (n27)
• November (n12) MCG, (n15) Keele
• 2007 (n199)
• February (n16) MCG
• Spring Keele (n47)
• May (n27) MCG, 23 med, 4 pa
• July (n10) MCG, pa, testing aarons stuff
• July-October (n60) MCG, basic system (Adeline
  running them)
• October (n39) UF Pharmacy, WoZ