Cognitive Testing: Washington Group Questions

1
Cognitive Testing Washington Group Questions

• Kristen Miller, Ph.D.
• Question Design Research Lab
• National Center for Health Statistics

2
Cognitive Testing Objectives

• Identify potential response errors related to
  question design
• Understand why and how errors might occur
• Differing interpretations
• Memory difficulty
• Assessment errors
• Insufficient response categories

3
Cognitive Testing Objectives

• Identify socio-cultural factors that might impact
  the question response
• Level of education
• Socio-economic status
• Cultural beliefs
• Racial, ethnic, gender identities
• Access to knowledge and resources

4
Traditional Cognitive Interviews are
Semi-structured

• Core Question- interview is organized by the
  questions that are being tested
• Probe Questions- open-ended, spontaneous, not
  pre-scripted, based on the information that the
  respondent provides

5
Traditional Cognitive Interviewing

• Data collected are qualitative
• Qualitative methodologies are used to analyze
  data
• Small samples approx. 10 30 participants

6
Washington Group Testing Obstacles

• Little or no training
• No understanding of cognitive methods
• Few resources (funding, technology)
• Multiple languages
• Multiple cultures
• Multiple levels of development

7
Protocol Requirements

• Structured interview guide
• Very simple instructions
• Few skip patterns
• Quantitative data and analysis

8
Cognitive Interview Protocol

• Designed to illustrate
• Administration ease
• Interpretations
• Factors considered
• Degree of consistency with physical abilities

9
Cognitive Interview Protocol

• Demographic Section Country, gender, age, SES
• Question Testing Sections (Self-report and
  Proxy-report)
• Core Question
• Interviewer Coding
• Open-ended Follow-up Probe
• Cognitive Follow-up Probe
• Functioning Follow-up Probe
• Health Questions
• Subjective health
• Chronic condition list

10
Interviewer Code

• Did the respondent
• need you to repeat any part of the question?
  Yes/No
• have any difficulty using the response options?
  Yes/No
• ask for clarification or qualify their answer?
  Yes/No

11
Open-ended Follow-up Probe

• Why did you answer that way?

12
Cognitive Follow-up Probes

• How often they had difficulty
• The amount of effort they needed to put into the
  activity
• The degree of concern or worry they had regarding
  the difficulty

13
Functioning Follow-up Probes

• Presence of a condition
• Use of an assistive device
• Degree of difficulty doing related activities

14
Health Status

• General health question
• Why did you answer that way?
• Health conditions

15
Interviewer Debriefing

• Details about the interview and interview setting
• Insight into the quality of the information

16
Analysis Plans

• Patterns in the response process
• Interpretation
• Judgment
• Patterns of response error
• Across countries and demographic variables

17
Design Testing of Cognitive Protocol

• Extensive testing in the United States
• Input from QUEST
• Pilot test in Mexico

18
Pilot Findings

• Strengths
• Relevant findings
• Few resources needed
• Concerns
• Interviewer training (more than expected)
• Protocol translations (critical and will differ
  by country)