Research Triangle Park, North Carolina

1
Turning Knowledge into Practice

• Research Triangle Park, North Carolina

2
Accessibility and Acceptance of a Virtual
Respondent-Based Interviewer Training Application

• Polly P. Armsby, Michael W. Link, Robert Hubal,
  Curry I. Guinn, Laura Flicker, and Rachel A.
  Caspar

3
Responsive Virtual Human Technology (RVHT) and
Interviewer Training

• Combining voice response and behavioral engines
  to develop virtual practice environments for
  interviewers
• Increases realism of role plays - enables
  repetitive, motivational practice for diverse
  learners
• Natural, interactive dialog between the user and
  emotive, responsive Avatars or virtual humans
• Reduces on-the-job learning of key interaction
  skills
• Effective refusal avoidance training needs to
  focus on
• introduction and tone of voice
• differentiating and dealing with reluctant and
  refusal behavior
• repetitive self-paced practice focusing on
  adaptive responses quick thinking

4
Mechanics of the Training Application

• RVHT-based application simulated the interactions
  during the first 30 to 60 seconds of a telephone
  interview
• Allows interviewers to practice skills in gaining
  cooperation in a self-paced, realistic
  environment
• Software is designed so interviewers begin with
  an introduction and then respond to objections or
  questions raised by the virtual respondent
• Interviewers responses are captured
  electronically and processed by a natural
  language speech processor
• Based on the content of the interviewers speech,
  the software launches another objection/question
  or ends the conversation by either granting the
  interview or hanging-up the telephone

5
Dialogue Map
6
Controlled Environment Evaluation

• Enhance understanding of how people interact with
  responsive virtual humans
• Research questions included
• Does application behave consistently across
  computer platforms?
• Do users perceive differences in performance or
  behavior of the application across platforms?
• Does the behavior of the application effect
  users perceptions of the realism of the training
  tool?
• 12 test subjects - varying levels of expertise
  training telephone interviewers
• 4 computer platforms - 2 PCs, 2 laptops / 2
  high-end, 2 production

7
Controlled Environment Evaluation Protocol

• Each subject completed three conversations with
  the Avatar per machine, then rated the experience
• Rated realism on 7 pt scale in terms of
• Response time
• Overall Conversation
• Objections raised by Avatar
• Rating scale 1 Not at all realistic, 7
  Extremely realistic
• All four machines were evaluated twice (in two
  trials)
• Conversations were taped and transcribed (then
  verified) for coding

8
Application Behavior Measures

• Transcripts were coded in terms of (1) number of
  Avatar-subject exchanges, and (2) exchange
  semantic (meaning)
• Measures of application behavior
• Conversation Exchanges number of times the
  Avatar launched an objection and the subject
  responded
• Conversation Semantics meaning or content of
  each exchange was coded into one of 35
  categories these were collapsed into 6
  categories for analysis.
• Conversation Complexity number of unique
  semantics observed during a conversation with the
  Avatar

9
Findings Controlled Environment Evaluation

• Subject Rating of Application Realism by Platform
• Ideally while performance of an application might
  vary across platforms, behavior of the
  application should not.
• Statistically significant variation was found
  across platforms
• Subject Rating of Realism Based on Application
  Behavior
• Conversation exchanges no significant
  differences noted in perceptions of realism
• Conversation semantics only one significant
  difference -- realism of objections raised
  higher when at least one exchange involved
  setting a callback
• Conversation complexity not significantly
  related to the three realism measures

10
Summary Controlled Environment Evaluation

• Need to determine reason for significant
  variation in application behavior across
  platforms -- to ensure consistency and uniformity
  of training environment
• Developers continuing to look at possible
  explanations for behavioral variation of
  application
• Few differences noted in users perceptions of
  realism -- further analysis to be conducted at
  conversation and exchange level using more
  sophisticated techniques for modeling nested data

11
Live Environment Assessment

• Evaluation Purpose Enhance understanding of how
  people interact with responsive virtual humans
• Approximately fifty telephone interviewers of
  varying backgrounds
• Instructor observations - directly assess the
  interaction between the user and the application
• Structured questionnaire
• perceptions of realism of the interactions with
  the "virtual human"
• ease of use of the software
• the perceived effectiveness of the training
  sessions
• basic background characteristics of the user

12
Findings Live Environment Assessment

• Ease of Use
• Users found the RVHT software very accessible to
  use
• Perception of realism - Mixed results
• Slowness of the response and limited number of
  objections/questions posed limited the sense of
  realism
• Objections/questions seemed realistic and
  trainees could detect and respond to changes in
  tone and emotion
• Impact on skill development
• Most trainees felt the tool increased their
  ability to respond to questions and concerns,
  better gain cooperation, adapt to differences
  tone/voice/moods, and avoid refusals

13
Interviewers Evaluation ofRVHT Training
Application
14
Interviewers Perceptions ofEffectiveness of
RVHT Application
15
Interviewer Recommendations for Future Use of
RVHT Application

• Approximately two-thirds said they found using
  the RVHT software to be fun and enjoyable.
• Nearly three-quarters said they would like to use
  the software again
• 83 said they would recommend the program as a
  training tool for other interviewers

Assessment Yes No Questions Was using RVHT fun
and enjoyable? 65 35 (31) (17) Would you
like to use the RVHT program again as a
training tool? 73 27 (35) (13) Would
you recommend the RVHT program as a training
tool for other interviewers? 83 17
(40) (8)
16
Summary Live Environment Assessment

• Interviewers liked some features
• Repeated practice of frequently asked questions
• Variations in emotional state based on tone of
  voice
• Interviewers disliked other features
• Application response time
• Perceived lack of variety in scenarios presented

17
Possibilities for Future Development

• RVHT holds potential for improving interviewer
  training
• The technology still needs considerable basic
  research to make it a viable, mature technology
• Future development will focus on
• Formal analysis of the speech recognition to
  refine and improve the underlying technology
• Enhancement of dialogues and scenarios to provide
  richer training content