Session 20: Data Collection in the Field

1
Session 20 Data Collection in the Field

• Ina F. Wallace
• RTI International

IES/NCER Summer Research Training Institute, June
2007
2
Data Collection in the Field Topics Covered

• Methods of Data Collection
• Use of State-of-the-Art Data Collection
  Procedures
• Ensuring Data Quality
• Conducting Small-Scale Validation Studies
• Ethical Issues in Data Collection

3
Methods of Data Collection

• Interviewer Administered
• In Person
• Surveys
• Assessments
• Telephone
• Self-Administered
• Paper and Pencil
• Mail-in
• In-person
• Computer Assisted Self Interview
• Web procedures

4
Data Collection Mode Characteristics
High Data Collector Involvement High Data Collector Involvement Low Data Collector Involvement Low Data Collector Involvement
Paper Computer Paper Computer
Direct Contact with Respondent Face-to-Face Paper and Pencil Interviewing/ Assessment (PAPI) Face-to-Face Computer Assisted Personal Interview/ Assessment (CAPI) (Audio) Computer Assisted Self-Interview (CASI, A-CASI)
Indirect Contact with Respondent Telephone PAPI Telephone Computer Assisted Telephone Interview (CATI) Mail, Fax, E-mail questionnare Touch-tone Data Entry (TDE), Web, Electronic Mail Survey (EMS)
No Contact with Respondent Direct Observation Computer Assisted Data Entry (CADE) Administrative Records Electronic Data Interchange (EDI)
Adapted From Biemer Lyberg (2003).
Introduction to Survey Quality
5
Face-to-Face Interviewing

• Advantages
• Maximum degree of communication and interaction
• Can utilize PAPI or CAI
• Allows for long, complex interviews
• Variety of tactics can be used to gain
  cooperation
• Response rates usually high

• Disadvantages
• High cost due to travel
• Can generate social desirability for sensitive
  questions
• Interviewers can affect respondents
• Can generate interviewer variance
• Presence of others can affect responses
• Falsification of interview

6
Telephone Interviewing

• Disadvantages
• Less flexible
• No ability to use visual aids
• Must be shorter
• Questions must be less complicated
• Less capacity for motivating the respondent
• No coverage of non-telephone units
• Response rates lower

• Advantages
• Cost lower than face to face
• More easy to provide training, supervision, and
  feedback
• Social desirability bias less than with face to
  face
• Interviewer variance less

7
Mail Surveys

• Disadvantages
• Little control over who completes the survey
• Item non-response high
• Response rates lower
• Require a long field period to obtain an
  acceptable response rate
• Respondents must have good literacy skills
• Limit to closed-format questions

• Advantages
• Lowest cost
• Low social desirability
• Can utilize visual aids
• Allows respondent to take time and look up
  information (if necessary)
• No interviewer variance

8
Computerized Procedures

• Computer Assisted Personal Interview (CAPI)
• Computer Assisted Telephone Interview (CATI)
• Computer Assisted Self Interview (CASI)
• Audio Assisted Self Interview (ACASI)
• Telephone Audio Assisted Self Interview (T-ACASI)
• Computer Assisted Data Entry (CADE)
• Web Surveys

9
Computer Assisted Procedures

• Advantages
• Complex skip patterns and branching are easier to
  follow
• Previously acquired information can be programmed
• Ranges, inconsistencies, and illegitimate values
  can be checked
• Help screens can be included
• Data are available for analysis more quickly

• Disadvantages
• Time consuming to develop and program computer
• Must be adequately tested
• Cant change things easily once fielded
• Risk of computer crashing
• Costly

10
Direct Assessment

• Used to obtain psychological and educational
  information about an individual
• Individually administered
• Research driven NOT for clinical or diagnostic
  purposes
• Follows a standardized protocol

11
Observational Techniques

• Involve a researcher making observations within a
  natural setting
• Direct Observation - individuals know they are
  being observed
• Continuous Monitoring recording as much behavior
  as possible
• Time Allocation randomly visiting the sample to
  observe at different times
• Unobtrusive Observation individuals do not know
  they are being observed

12
Observational Techniques

• Advantages
• No or limited participation needed
• May be good for generating hypotheses
• Sensitive social issues can be obtained more
  accurately
• High in validity because of the depth of
  information collected

• Disadvantages
• Variance associated with observers is similar to
  that found with interviewers
• Can be very costly as they are often labor
  intensive
• Time consuming
• Reliability can be low
• Generalizability can be low

13
Summary Issues in Choosing a Strategy

• Type of Population
• Nature of Research Issue
• Question Form
• Question Content
• Response Rates
• Costs
• Length of Data Collection

14
Methods of Administration

• Standardized Interviewing Approach
• A protocol in which interviewers interact with
  the respondent in a manner which is consistent
  across all interviewers
• Conversational Interviewing
• A protocol in which interviewers modify and adapt
  questions to the respondents situation
• Hybrid Interviewing
• Combination of the two approaches

15
Factors that Can Explain Interviewer Variance

• Respondent
• Characteristics
• Knowledge
• Interest/motivation
• Confidence
• Strength of convictions
• Expectations

• Interviewer
• Characteristics
• Appearance
• Motives
• Beliefs/attitudes
• Perceptions
• Expectations
• Skills/Knowledge

Interviewer Effects

• Survey Conditions and Setting
• Mode of interview
• Standardization
• Interviewer training
• Interviewer supervision
• Monitoring/observation

• Questionnaire
• Definition clarity
• Terminology/jargon
• Question form
• Instructions
• Question wording
• Question topic

From Biemer Lyberg (2003). Introduction to
Survey Quality
16
Factors that Can Explain Interviewer Variance

• Respondent
• Characteristics
• Knowledge
• Interest/motivation
• Confidence
• Strength of convictions
• Expectations

• Interviewer
• Characteristics
• Appearance
• Motives
• Beliefs/attitudes
• Perceptions
• Expectations
• Skills/Knowledge

Interviewer Effects

• Survey Conditions and Setting
• Mode of interview
• Standardization
• Interviewer training
• Interviewer supervision
• Monitoring/observation

• Questionnaire
• Definition clarity
• Terminology/jargon
• Question form
• Instructions
• Question wording
• Question topic

From Biemer Lyberg (2003). Introduction to
Survey Quality
17
Role of the Interviewer/Assessor

• Gaining Cooperation
• Converting reluctant participants
• Recognizing and dealing with refusals
• Training and Motivating Respondents
• Interviewers implicitly and explicitly teach
  respondents how to behave
• Obtaining Quality Data
• Ensuring Confidentiality

18
Assessing Children

• Assessor Characteristics
• Flexibility
• Vigilance
• Self-awareness
• Eye contact
• Establishing Rapport
• Allaying apprehensions
• Helping children feel at ease
• Maintaining limits on permissible behavior
• Arranging the Physical Environment

19
Ensuring Quality Data

• Recruiting interviewers/assessors
• Characteristics of all data collectors
• Characteristics of interviewers
• Characteristics of assessors
• Training on standardized procedures
• Certification as data collectors
• Monitoring performance

20
Training

• Content of Training
• Presenting the study
• Contacting participants
• Administering the questionnaire/ assessment
• Probing
• Recording responses
• Confidentiality
• Interpersonal relations
• Administrative procedures

• Procedures for Training
• Written materials
• Lecture and presentation
• Computer-based tutorials
• Planned exercises
• Role Playing
• Observation
• Certification

21
Interviewer Training Agenda

• Introduction to the Study
• Role of the Interviewer
• Confidentiality Procedures
• Review of Questionnaire and Protocols
• Standardized Interviewing Techniques
• Round Robin Interview
• Gaining Cooperation
• Presentation of Scenarios
• Round Robin Demonstrations
• Paired Mocks
• Certification

22
Assessor Training Agenda
Day 1 Large group Introduction to Project Role as an Assessor Best practices with children Small groups Pre-training quiz Introduction to all materials Learn each assessment Observation Round Robins Day 2 Small groups Individual practice with assessments Paired practice with assessments Large group Building rapport and gaining cooperation Administrative responsibilities Data security Informed Consent Day 3 Small groups Review of assessments Paired practice Post-training quiz Large group Concluding remarks Certification
23
Monitoring Performance

• What to Monitor
• Detection and prevention of falsified information
• Compliance with rules and guidelines about
  administering the questionnaire or assessment
• Performance on non-interview/assessment tasks

• How to Monitor
• Telephone verification with a sample of
  respondents
• Direct observation in-person or by telephone
• Audio-recording CARI
• Review of materials
• Performance and production measures

24
Human Subjects Issues

• Institutional Review Boards
• Informed Consent
• Who is carrying out research under whose
  sponsorship
• Description of purposes and procedures
• Duration of involvement
• Risks and Benefits
• Confidentiality
• Voluntary nature of research including stopping
  early and skipping questions
• Compensation
• Contact information
• Confidentiality Issues

25
Human Subjects Issues

• Institutional Review Boards
• Informed Consent
• Who is carrying out research under whose
  sponsorship
• Description of purposes and procedures
• Duration of involvement
• Risks and Benefits
• Confidentiality
• Voluntary nature of research including stopping
  early and skipping questions
• Compensation
• Contact information
• Confidentiality Issues

26
Data Collection Evaluation
Stage of Data Collection Evaluation Method Purpose
Design Expert review of instrument Identify problems with instrument
Pretest Behavior coding Cognitive interviewing Evaluate one or more stages of the response process
Pretest/Post-Data Collection Respondent focus groups Interviewer debriefings Evaluate instrument and data collection procedures
Pretest/Data Collection Supervisor observation Telephone monitoring CARI Evaluate performance of data collector
Post-Data Collection Experimentation Internal Consistency External validation Re-interview/retest Compare alternative methods of data collection Evaluate components of MSE Validate estimates
Adapted From Biemer Lyberg (2003).
Introduction to Survey Quality
27
Post-Data Collection Evaluations

• Experiments
• Observational Studies
• Internal Consistency Studies
• External Validation Studies
• Administrative Record Check Studies
• Re-interview/Re-Test Studies

28
Post-Data Collection Evaluations

• Experiments
• Observational Studies
• Internal Consistency Studies
• External Validation Studies
• Administrative Record Check Studies
• Re-interview/Re-Test Studies