A Conversational Approach to Textbased ComputerAdministered Questionnaires

1
A Conversational Approach to Text-based
Computer-Administered Questionnaires
ASC September 23, 1999 Edinburgh
2
Acknowledgments

• Susan Brennan, Cathy Dippo, Scott Fricker, Susan
  Schnipke and Clyde Tucker
• US Bureau of Labor Statistics
• National Science Foundation
• SBR-97-30140

3
The CASI Era

• Computer Assisted Self-administered Interviews
  (CASI) could revolutionize survey data collection
• WebCASI will (eventually) be able to reach
  anyone in sample
• Because there are no interviewers, CASI is
  cheaper and may improve data quality
• gives respondents a sense of privacy
• eliminates interviewer-related error

4
CASI Instruments as Standardized Interviews

• Embody principles of strict standardization
  (e.g., Fowler Mangione, 1990)
• most major surveys in US subscribe to some
  version of standardization
• Computer presents identical question to all
  respondents (Rs)
• Interpretation of question is left entirely to R
• No interviewer to bias or mislead Rs

5
Standardization can compromise response accuracy

• When human interviews are strictly standardized,
  response accuracy can be poor
• if Rs circumstances do not obviously map onto
  question, responses may not fit official
  definitions
• Examples of such complicated mappings
• Does buying a lamp count as a furniture purchase?
• Does baby-sitting for multiple employers count as
  more than one job?
• Does a person away at college live at home?

6
Conversational Interviewing

• Can lead to far superior accuracy for complicated
  mappings
• (e.g.Conrad Schober, 2000 Schober
  Conrad,1997)
• Interviewers collaborate with Rs to make sure
  they understand questions as intended by
• clarifying concepts at Rs request
• clarifying concepts when they think R doesnt
  understand them
• asking exploratory questions
• Enables Rs to ground their understanding

7
Can conversational techniques improve CASI
response accuracy?

• System would need to support clarification of
  question meaning in self-administration
• enable Rs to ground their understanding
• Possible ways to implement clarification depend
  on which agent (R or CASI system) initiates
  clarification

8
Possibilities for initiative in CASI

• User-initiated
• if user (R) requests clarification, system
  provides it
• requires users to
• recognize they need clarification
• be willing to ask for it

9
Initiative (contd)

• System-initiated
• system provides (or offers to provide)
  clarification when it diagnoses misunderstanding
• based on user behavior
• e.g. delayed response (text, speech interface)
• e.g. disfluent responding (speech interface)
• makes most sense to implement along with
  user-initiated

10
CASI Media and Modes

• Text
• Users read questions from screen and answer with
  mouse and/or keyboard
• Speech
• Users hear questions and speak answers
• Mixed
• Users hear questions and answer with mouse and/or
  keyboard (e.g. ACASI)

11
CASI and HCI

• Survey applications differ from most in HCI
• In typical HCI application, user retrieves
  information from system
• In survey application, system collects
  information from users

12
Laboratory Experiment

• Computer asks questions from ongoing government
  surveys
• Rs answer from fictional scenarios, so true
  values are known
• Types of clarification (1) User-initiated and
  (2) User- and System-initiated
• clarification at Rs request (clicks unclear
  text)
• clarification when R takes too long to answer and
  at Rs request
• no clarification

13
Will Rs use available clarification?

• People rarely ask for help in testing situations
• e.g. Graesser et al. (1996)
• In conversation, people may not seek
  clarification if stakes are low
• i.e. when grounding criterion is low
• e.g. cocktail party banter versus air traffic
  control tower communication

14
Grounding Criterion

• Varied Rs instructions
• definitions essential
• Rs encouraged to obtain definitions from computer
  because everyday definitions may differ from
  surveys
• definitions available
• definitions available if Rs want them

15
Experimental Conditions
16
Questions

• All Rs were asked the same questions
• 12 questions from three surveys
• CPS (Employment)
• CPI-Housing
• CPOPS (Purchases)
• Official definitions existed for key concepts in
  all questions

17
CASI Interface

• Questions displayed on screen
• Rs use keyboard and mouse to enter responses and
  navigate questionnaire
• Rs request definitions by clicking mouse on
  highlighted text (conditions 2-5)
• Rs are also offered clarification when they take
  too long to respond (conditions 4-5)
• too long longer than median response time for
  that question for No Clarification group

18
No clarification available
19
Definition for live in this house can be
obtained by clicking mouse on blue text
20
Computer offers to provide definition for live
in this house because R took too long to answer.
21
Definition presented for live in this house
22
Mappings

• Rs responded on basis of fictional scenarios
• textual vignettes
• floor plans
• receipts from purchases
• For each R, half the mappings were
  straightforward and half were complicated
• Definitions always clarified mappings

23
Example Scenario
Has Kelly purchased or had expenses for household
furniture?
Straightforward mapping
Complicated mapping
KATZS Furniture Mart Brooks End Table
149.99 713000000075 Tax..
11.99 TOTL 161.98 B112 882000002 4330
749 PM
KATZS Furniture Mart Lumin Floor Lamp
149.99 713000000075 Tax..
11.99 TOTL 161.98 B112 882000002 4330
749 PM
24
Example Scenarios (Contd)
How many people live in this house?
Straightforward mapping
Complicated mapping
The Gutierrez family owns the 3-bedroom house at
4694 Marwood Drive. The family has four members
Maria and Pablo Gutierrez, and their two children
Linda and Marta. There is one bedroom for Maria
and Pablo, one for Marta, and one for Linda.
The Gutierrez family owns the 3-bedroom house at
4694 Marwood Drive. The family has four members
Maria and Pablo Gutierrez, and their two children
Linda and Marta. There is one bedroom for Maria
and Pablo, one for Marta, and one for Linda.
Linda is a college student. Although her legal
address is still 4694 Marwood Drive, she stays at
the college dorms all year, except for holidays
and vacations.
25
Participants

• 54 paid Rs recruited from subject pool
• 10 or 11 in each group
• Characteristics
• 22 F, 32 M
• 13 Black, 38 White, 3 Asian
• range of educational backgrounds
• 24 high school only
• 21 college degrees
• 9 with postgraduate education
• fairly experienced computer users
• 44 every day, 5 once/week, 2 once/month, 3
  once/year

26
Overall Response Accuracy

• For straightforward mappings, accuracy nearly
  perfect
• For complicated mappings, depends on type of
  help and instructions to R (definitions
  essential or available)

27
When did Rs obtain clarification?

• If Rs were told definitions essential, they
  frequently asked for clarification
• far more often than in telephone interviews
  (Schober and Conrad, 1997)
• asking for help with CASI is low cost (see Clark
  Brennan, 1991 Schwarz, et al. 1991)
• If Rs were told definitions merely available,
  they rarely asked for clarification
• slightly less than in previous telephone
  interviews
• Complexity of mappings did not matter

28
User-Initiated
System-Initiated
29
All help increases accuracy

• For complicated mappings, help increases accuracy
  dramatically
• does not matter if Rs request or computer offers
  help

30
Unsolicited Help

• Presented infrequently when grounding criterion
  low (definitions available)
• users probably did not recognize that their
  concepts differed from system
• responded confidently, quickly and inaccurately

31
Clarification takes time

• In Schober and Conrad (97) conversational
  interviews gt 3 times as long as standardized
• Increased time increased interviewer cost
• Current study

• Does not increase interviewer costs but may
  increase costs for R

32
User Satisfaction

• 94 of all users responded favorably to idea of
  future actual surveys like this
• no effect of clarification type or grounding
  criterion
• 85 preferred CASI over interviewer-administered
  or paper instrument for same questions
• no effect of clarification type or grounding
  criterion

33
User Satisfaction (Contd)

• Most Rs recognized value of clarification
• No Clarification group 8 of 11 would have used
  if available
• remainder felt definitions not important
• Definitions Available group 8 of 11
  (User-Initiated) and 5 of 10 (User- and
  System-Initiateed) reported using clarification
• remainder did not because felt not important

34
User Satisfaction (Contd)

• Reaction to unsolicited help depended on
  grounding criterion
• Defs. essential rated as useful (6.0 on 7 pt.
  scale) and not annoying (1.0 on 7 pt. Scale)
• Defs available rated as less useful (3.9) and
  more annoying (4.35)

35
Speech interface

• Bloom and Schober (2000) tested simulated speech
  interface
• CASI system presents spoken questions and
  clarification
• Users speak answers and requests for help
• Wizard-of-Oz technique
• users believe they are interacting with computer
• hidden experimenter actually presents questions
  and scripted clarification

36
Speech study Design

• When system provides clarification
• Never
• User-initiated
• users ask for help explicitly (rather than
  clicking)
• User- and System-initiated
• system also automatically provides full
  definition when users display specific
  uncertainty markers
• ums, uhs, pauses, repairs, talk other than an
  answer
• Always
• full definition presented every time

37
Comprehension accuracy
38
Collaboration

• Users spoke more conversationally when system
  was sensitive to evidence of uncertainty
  (system-initiated)
• asked more questions
• were less fluent

39
Requests for clarification

• Speech users requested clarification far less
  often than text users
• harder to request (not just a mouse click)
• definition unfolded over time, rather than
  appearing all at once
• impossible to shut off (not just a mouse click)
• couldnt reject offers of clarification (text
  users could)

40
Interview duration
41
User satisfaction

• When the system was
• not responsive (no clarification or clarification
  always)
• users wanted more control and felt that
  interacting with the system was unnatural
• responsive (user- or user- and system-initiated)
• users were happier

42
Summary

• Standardized CASI may lead to poor response
  accuracy
• Conversational techniques can be implemented in
  CASI instruments
• respondents can request clarification and
  computer can offer to clarify concepts
• improves response accuracy for complicated
  mappings

43
Summary (Contd)

• Rs often obtain help if told it is essential, but
  rarely obtain it if only told it is available
• Rs request help more often with CASI than with
  human interviewers
• Benefits of obtaining help independent of whether
  R requests it or computer offers it

44
Implications

• Conversational CASI may provide more accurate
  data
• allows Rs to obtain the information they need in
  a systematic way
• at far less cost than human conversational
  interviewing
• But users must recognize own uncertainty and be
  willing to obtain clarification
• and users must display reliable signs of
  uncertainty

45
Implications (Contd)

• May be tempting to directly implement
  standardized techniques
• but this may lead to same problems as
  standardized interviews
• Usable CASI systems enable users to clarify
  question meaning