Questionnaire Design, Pretesting and Quality Control in Clinical Research

1
Questionnaire Design, Pretesting and Quality
Control in Clinical Research

• Celia P. Kaplan
• Division of General Internal Medicine
• August 27th 2008

2
Good Clinical Practice (GCP)

• A standard for the design, conduct,
  performance, monitoring, auditing, recording,
  analysis, and reporting of clinical trials that
  provides assurances that the data and reported
  results are credible and accurate, and that the
  rights, integrity, and confidentiality of trial
  subjects are protected
• WHO good clinical practice standards

3
Quality Control
4
Sources of Data Errors

• Data acquisition
• instrument problems
• lost data
• falsified data (made up data or changed)
• Data recording
• Data entry
• Data management

5
Quality Control Stages

• Pre-implementation/Developmental Stage
• Design high-quality questionnaires and forms
• Pretest all aspects of the project
• Develop an Operations Manual
• Design a secure data-management system
• Conduct staff training and certification
• Implementation
• Make periodic study- and data-monitoring reports
• Conduct periodic project meetings
• Complete interim staff training and performance
  reviews

6
Quality Control Stages

• Pre-implementation/Developmental Stage
• Design high-quality questionnaires and forms
• Pretest all aspects of the project
• Develop an Operations Manual
• Design a secure data-management system
• Conduct staff training and certification
• Implementation
• Make periodic study- and data-monitoring reports
• Conduct periodic project meetings
• Complete interim staff training and performance
  reviews

7
Questionnaires

• Much of the data in clinical and epidemiological
  research is gathered using questionnaires and
  interviews

8
Questionnaire Measures

• Desired qualities
• Reliability the extent to which the measures
  give consistent results
• Validity the extent to which the measures
  reflect the truth
• Accommodate participants age, literacy,
  cultural, linguistic, and SES characteristics
• Yield good variability in answers
• Result in low percent of missing data

9
Questionnaire Development Pitfalls

• Double-barreled questions Avoid and/or
  statements in questions
• e.g., Does your department have a special
  recruitment policy for ethnic minorities or
  women?
• Long questions and/or questionnaires
• Use of other as a response category

10
Questionnaire Development Order

• First question should be a fact
• Beginning questions should stimulate interest
• Questions should be in logical order and
  groupings
• If ask attitude questions first and then behavior
  questions, may inflate reports of behavior
• Sensitive questions and open-ended questions
  should be at the end of the survey
• e.g., sexual behavior, religion, income

11
Questionnaire Development Language

• Use simple, clear, common, and natural language
• Avoid jargon, complex terms, clichés,
  colloquialisms, e.g., down in the dumps
• Check number of syllables in words
• Check reading level aim for lt 8th grade
• Define ambiguous terms, e.g., family or
  neighborhood
• Keep sentences short

12
Questionnaire Development Recall

• Solutions
• Narrow the reference period
• Average the response ask about a typical day or
  in general (e.g., hours of sleep)
• Use landmark events or milestones to anchor time
  frame Since you moved to the US, or use
  calendar with major holidays to establish major
  life events
• Use cues to generate associations to jog memory
  describe procedure, setting

13
Questionnaire Design
7
Finalize the Questionnaire
Draft survey
Interview methodology
Review existing measures
Conduct exploratory work
Finalized instrument
Translation/ back-translation
Pre-test the survey
14
Questionnaire design

• Selection of interview methodology
• Face-to-face or in-person interview
• Telephone
• Self-administered, paper
• Self-administered, computer/web/e-mail

15
Questionnaire design

• Interview methodology Face-to-Face
• Advantages
• Enables the interviewer to establish rapport with
  respdenont
• Permits more complex questions
• Permits the use of visual aids
• Longer interviews are possible
• Disadvantages
• Cost per interview
• Quality of answers too personal (sensitive
  questions)

16
Questionnaire design

• Interview methodology Telephone
• Advantages
• Faster contact of participants
• Better to elicit personal/sensitive information
• Results are available right away if using
  computer-assisted interviewing (CAI)
• Can do random selection of telephone numbers
• Disadvantages
• Certain sectors of the population are switching
  to cell phones only
• More expensive than mail surveys
• Difficulties in reaching participants during the
  day

17
Questionnaire design

• Interview methodology Self-administered and mail
  surveys (Paper)
• Advantages
• One of the least expensive methodologies
• Needs addresses of potential participants
• Allows participants to answer at their leisure
• Less intrusive
• Elimination of interviewer bias
• Disadvantages
• Response rate is low
• Difficulties to develop clear skip patterns
• It may be answered by a family member
• Very difficult to implement in low literacy
  populations

18
Questionnaire design
Example Self-administered Mail Surveys
19
Questionnaire design

• Interview methodology Self-administered,
  web-based surveys (Computer)
• Advantages
• Elimination of data entry and editing costs
• Get better answers to sensitive questions
• Elimination of interviewer bias
• Ensure that skip patterns are accurately followed
• Allows participants to answer at their leisure
• Disadvantages
• Interviewees must have access to a computer
• Must possess (or purchase) a list of email
  addresses
• Lower response rate
• People may quit in the middle of a questionnaire

20
Questionnaire design
Example Web-based survey
21
Questionnaire design

• Review Existing Questionnaires Measures
• Reference databases
• Medline, Pubmed, Psychinfo, others
• Compendia of measures
• Books that compile various measures and review
  their characteristics
• Request questionnaires from other investigators

22
Questionnaire design

• Review Existing Questionnaires Measures

• Is there evidence the instrument works in your
  target population?
• Examine validity and reliability of measures when
  used in populations similar to your study
  population
• When gold standard does not exist, for construct
  validity, look at relationship of measure to
  other measures to see if in hypothesized direction

23
Questionnaire design

• Conduct Qualitative Exploratory Work
• Focus groups or semi-structured interviews
• Explores a topic before constructing formal
  questions
• Rationale
• Determine what is important
• Discover how respondents think about the topic
• Assess better language/words to use
• Identify response categories

24
Questionnaire design

• Create a Draft Survey Instrument
• Write initial draft
• Process of revision and refinement through cycles
  of translation/back-translation, reviews and
  pre-tests (Steps 5 and 6)

25
Questionnaire design

• Translation/Back-translation Procedures
• Both versions should have the same meaning
• Translation and back translation
• Review by a group of native speakers
• Challenging when more than one language involved
  in the study

26
Quality Control Stages

• Pre-implementation/Developmental Stage
• Design high-quality questionnaires and forms
• Pretest all aspects of the project
• Develop an Operations Manual
• Design a secure data-management system
• Conduct staff training and certification
• Implementation
• Make periodic study- and data-monitoring reports
• Conduct periodic project meetings
• Complete interim staff training and performance
  reviews

27
Quality Control

• Pre-testing the Questionnaire Overview
• Refines
• Instruments and questionnaires
• Evaluates
• Recruitment methods
• Interventions
• Data entry and management system
• Protocols
• Assists with training of personnel

28
Quality Control

• Pre-testing the Questionnaire
• Good to test
• Sensitive questions
• Complex or poorly defined topics
• Questions with terms respondents may not
  understand
• Translations
• Layout and instructions

29
Quality Control

• Pre-testing the Questionnaire
• Field Testing Techniques
• Expert reviews
• Cognitive interviews
• Full pretest

30
Quality Control

• Pre-testing the Questionnaire
• Expert Reviews
• Formal and systematic examination of a
  questionnaire by experts in the field
• Can cover question wording, layout, and skip
  patterns
• Fast method prior to formal testing
• May not be a complete review

31
Quality Control

• Pre-testing the Questionnaire
• Cognitive Interview Overview
• Diagnostic tool for pretesting survey questions
  derived from social and cognitive psychology
• Interview about an interview
• Explores the processes by which respondents reach
  answers
• Based on structured questionnaire and protocol

32
Quality Control

• Pre-testing the Questionnaire
• Cognitive Interview Techniques
• Concurrent proving
• Ask probes immediately after respondent has given
  answer to survey item
• Advantage information is fresh in respondents
  mind
• Disadvantage Disrupts interview flow and
  relationship between questions
• Retrospective probing
• Ask probes after entire interview or block of
  questions
• Advantage able to assess standard administration
  of items
• Disadvantage Participant may not remember
  thought process

33
Quality Control

• Pre-testing the Questionnaire
• Cognitive Interview Techniques
• Probing questions
• Think aloud interviews
• Interviewer asks respondent to think aloud as
  they answer question
• Paraphrasing
• Respondents rephrase the question in their own
  words

34
Quality Control

• Pre-testing the Questionnaire
• Cognitive Interview Key Features
• Comprehension of the question
• Do participants understand words and phrases as
  intended by the researchers (meaning)
• How did you arrive at your estimate of risk?
• What did you understand by the word risk?
• Retrieve of information
• Identify process respondents use to answer
  questions
• Examine strategies used to access memory
• Edit responses Decide what to report
• Are items unacceptable? Is answer embarrassing,
  socially undesirable?

35
Quality Control

• Pre-Test
• Pretest with a sample similar to the population
  to be studied
• Simultaneously pretest recruitment procedures and
  questionnaire implementation

36
Quality Control

• Pre-Test Behavior Coding
• Code interaction between interviewer and
  respondent
• Examples
• R answers before hearing whole question
• R looks puzzled
• R asks for clarification
• I repeats question

37
Quality Control

• Pre-Test Respondent Debriefing
• Structured follow-up questions at the end of the
  interview to assess
• Whether questions were clear
• Ease of completing the questionnaires
• If two versions are tested, which is preferable
• Whether an important question was missed

38
Quality Control

• Pre-Test Interviewer Debriefing
• Interviewers implement pretest with participants
• Interviewers detect problems with questions,
  response categories, and skip patterns
• Interviewers report issues to investigators

39
Quality Control

• Pre-Test Questionnaire Summary
• Testing takes time and resources
• Some procedures take place late in the
  developmental cycle, limiting the amount of
  change possible
• Danger of making changes that are not further
  tested
• Not testing leads to questionnaires that are
  potentially unreliable or will give you invalid
  results

40
Quality Control Stages

• Pre-implementation/Developmental Stage
• Design high-quality questionnaires and forms
• Pretest all aspects of the project
• Develop an Operations Manual
• Design a secure data-management system
• Conduct staff training and certification
• Implementation
• Make periodic study- and data-monitoring reports
• Conduct periodic project meetings
• Complete interim staff training and performance
  reviews

41
Quality Control

• Operations Manual
• Aim is to ensure high quality data
• Clearly written, detailed written instructions to
  obtain uniformity across sites, staff, and
  procedures
• Standardizes procedures for all aspects research
  project. Defines how to
• Recruit participants
• Consent participants
• Measure variables
• Transport clinical samples
• Enter data
• Monitor progress
• Conduct data analysis
• Requires updates, clarifications, and recording
  of decisions made

42
Quality Control

• Data Management System
• Desirable features
• Range and field type checks
• Ease of screen set up and use
• Double data entry (if possible)
• Security features
• protection of human subjects rights (privacy)
• For Web-based systems
• Virus protection to monitor and eliminate
  security threats
• Database server behind firewall
• All data backed up regularly

43
Quality Control

• Staff Training
• Training and certification of the research team
  members
• Rules to conduct interviews
• Rules for recruiting participants, including how
  to obtain consent
• Methods
• Train the trainer model
• Audio-visual techniques
• Certification/recertification to maintain skills

44
Quality Control Stages

• Pre-implementation/Developmental Stage
• Design high-quality questionnaires and forms
• Pretest all aspects of the project
• Develop an Operations Manual
• Design a secure data-management system
• Conduct staff training and certification
• Implementation
• Make periodic study- and data-monitoring reports
• Conduct periodic project meetings
• Complete interim staff training and performance
  reviews

45
Quality Control

• Study Reports
• Recruitment reports (e.g., actual vs. expected
  accrual)
• Follow-up reports (e.g., overdue visits)
• Timeliness of data collection activities
• Data entry error rates

46
Quality Control

• Data Monitoring Reports
• Standard procedures for
• Identifying missing, incomplete data
• Identifying incorrect skip patterns
• Identifying inconsistent and erroneous data
• Verifying correction

47
Quality Control

• Interim Staff Training and Performance Reviews
• Provide continuous supervision
• Hold frequent meetings of the study team
• Review all aspects related to implementation of
  the study, including
• Interviewers activities
• Listen to interviews
• Compare measures between interviewers
• Compare refusal rates between interviewers

48
Good Clinical Practice

• Good Instruments
• Complete and clear protocol manuals
• Extensive pretesting
• Ongoing monitoring procedures