Data Collection Methods: Overview

1
Data Collection Methods Overview

• Modes of Administering Instruments to obtain data
  from people
• (mail, phone, face-to-face interview,
  computer-assisted)
• Introduction to principles of chart/record
  abstraction
• Pre-testing and Pilot testing

2
Modes of Data Collection

• Mailed
• Mass-Administered
• Telephone
• Face-to-Face
• Drop off
• Computer Assisted (CASI, CATI)
• Online surveys
• Other?

3
Decision of DC Mode Factors

• Characteristics of Study variables
• Amount
• Complexity (e.g., skip patterns, use of scales
  with gt3 responses)
• Does good instrument exist?
• Sample degree of dispersion
• Access to source of data
• Resources personnel, budget, deadline

4
Principles Data Collection tools and methods

• Reaction to Uniform Stimulus
• Minimize burden to R (1st) and I (2nd)
• Minimize error steps from Rs brain to Data
  base
• Skills/experience of team
• Triangulation of methods
• Assess/Maximize reliability/validity

5
Comparison of Methods

• Mailed Face to Face Phone

6
Mailing Strategy Example from Dillman

• Series of 4 mailings
• 1st step Advance notice personalized letter
• 2nd step Personalized cover letter,
  Questionnaire, stamped return envelop (1 week
  after 1st step)
• 3rd step Postcard thanking completers and
  urging non-completers to respond (4-8 days after
  2nd step)
• 4th step Personalized cover letter, replacement
  Q, stamped return envelop (3 weeks after 2nd
  step)

7
Mailing Advantages

• Advantages (potential)
• ? Cost
• ? Measurement/Interviewer bias
• ? Anonymity of R
• ? Veracity of R (?)
• ? pressure on R

8
Mailing Disadvantages

• ? ? response rate
• RR chief indicator of quality
• Exceptions special groups professionals,
  alumni, ?house staff
• ? rates of omissions and errors in skip patterns
• ? influence by others, Others may complete
• Limitations of length
• Poor mode for open-ended ?s
• Unable to clarify, probe, control sequence or
  assess comprehension
• Non-Respondents differ from Rs
• Ill-suited to persons with limited reading
  ability

9
Mass Administered

• Suited for special circumstances
• captive population (e.g., students employees)
• Low cost
• High response rates
• Ability to address misunderstandings

10
Telephone Advantages

• Cost
• Ease of supervision
• Quality control (? Interviewer bias appearance)
• Response rate (between mail and face-to-face)
• Relatively fast
• Quality ? Veracity (?)
• Skip patterns controlled by I (or computer CATI)
• Opportunity to clarify ambiguities, answer R
  questions
• Control sequence of ?s

11
Telephone Disadvantages

• Sampling challenges Listed v RDD, homes with gt1
  phone line, availability of Rs (best time
  Sun-Thursday 5-9pm)
• R identity uncertain
• Difficult for complex questions Unable to use
  visual aids
• Limits on Interview length
• I bias (verbal)
• Local application Chicago homes without phone
  30
• High costs of survey lab
• Impact of new technology unknown Caller ID,
  telemarketers

12
Face-to-face Interviews Advantages

• Best response rates
• Control of R identity and item sequence
• Able to handle skip patterns
• Most amenable to ? rapport and R motivation
• Able to use visual aids (cross-cultural research)
• Language spoken interpretation better than
  written
• Compatible with computers CASI

13
Face-to-Face Interviews Disadvantages

• Cost (? training, ? personnel time per
  interview
• ? ? transportation costs)
• Standardization challenging
• Lack of anonymity
• Prone to I bias
• Prone to social desirability bias
• Challenging for dispersed sample
• I fraud potential (more difficult to supervise)

14
Drop-Off

• Instrument is self-administered
• Personally delivered to R (informed of its
  importance)
• Return via mail or leave at collection site
• Evidence of effectiveness ?

15
Computer Assisted Technologies

• Advantages
• Saves timedirect data entry
• ? Meas error
• Suited for sensitive items
• ? accuracy (range checks)
• ? veracity of R
• Handles skip patterns
• Can force R to respond

• Disadvantages
• apprehension of older Rs
• Cost computers and programming
• Settings restricted

16
Sources of Measurement Bias

• Interviewer
• Age, gender, race, voice, body language
• Data gathering procedures
• Social Desirability
• Acquiescence
• R tendency to agree
• Language non-E speaking, translation,
  vernacular terms
• Sampling R and non-R differences (assess)

17
Strategies to Reduce Measurement Bias

• ? social desirability bias
• I training, I non-judgmental, accept full range
  of response
• Explicit/reassurance of Confidentiality
• Items written neutral
• Can assess with SD scale
• ? acquiescence bias
• Balance agree with disagree items
• Avoid extremes, include in-between categories
• Language (pre-test, rigorous translation-back
  translation)

18
Strategies to Increase Response Rates

• Personalized cover letters
• Additional mailings, stamped return envelops
• Minimize refusals (well trained/experience Is)
• Improve Recruitment convenient times for Rs
• Rigorously and explicitly maintain anonymity or
  confidentiality
• Incentives monetary, non-monetary

19
Effective strategies for recruiting minorities

• Advance notice
• Cover letter, media, community leaders/orgs
  endorsement or stationary, health care provider
  endorsement
• Initial contact crucial
• Culturally appropriate, rationale explicit
• Trained and skilled project staff
• Bilingual as needed
• Patience and enthusiasm
• Promote interaction with family

20
Chart/Record Abstraction

• Medical charts amenable to retrospective
  abstraction (can yield prospective data)
• Charts contain indicators about physical
  condition
• Convenience Access, Coding at end of study (or
  in
• parallel)
• (Potentially) cheaper
• Documentation variability
• High variability in diagnoses
• Less variability in laboratory testing results

21
Chart Abstraction Considerations

• High variability in chart documentation
• Differing degrees of completeness
• May not be orderly..Data scattered throughout
• May not record all data needed
• Legibility
• Reliability among coders obtain adequate levels
• Medical laboratory testing/assays may not be
  standard across different provider settings

22
Chart Abstractions Considerations 2

• Standards need to be explicit
• Standards need to encompass typical charting
  styles in all sites
• Rigorous training of abstractors
• Inter-rater reliability
• Medical Professionals need to do the abstraction
• Multi-site options Individual coders at each
  site v. centralized coders
• Time estimates 60 mins/2-3 years data duration

23
Chart Abstraction Effectiveness

• RCT comparing 20 standardized pts to chart
  abstraction ((Luck, J 2000 Am J Med)
• Standardized scoring check list for History, PE,
  Tests, Diagnosis and Rx. Gold Std Observing NP
• Chart abstraction 54, Standarized Pts 68
• Chart abstraction scores lowest for diagnosis
  (40)
• (v Standardized pts 52)

24
Pre-Testing/Pilot Testing

• Terms ambiguous
• Frequently haphazard, rushed, insufficient time
  or not done
• Objectives
• identify unforeseen problems
• Minimize measurement bias (I and instrument)
• Assess quality of Is and instrument
• N20-50, till satisfied with quality
• Sample skilled colleagues, Rs similar to study
  Rs
• Debriefing of R and I

25
Pre-testing Instruments

• Comprehension
• Interpretation
• Flow and sequence
• Response categories
• Task difficulty
• Cultural sensitivity
• Interviewer skill
• Modify Instrument before pilot if major ?,
  pre-test again
• Use qualitative methods interview Rs

26
Pilot Testing

• Dress Rehearsal of Final instrument and
  procedures
• Feasibility
• Identify problems, Develop responses, test
  adequacy of response
• Determine time estimates for study tasks (e.g.,
  recruitment, interviewing)
• Determine adequacy of numbers of eligible
  subjects, participation rates
• Collect Data Assess Data Base
• Analysis sometimes