NONSAMPLING ERROR RESEARCH IN PRACTICE

1
NONSAMPLING ERROR RESEARCH IN PRACTICE

• J. Michael Brick and Graham Kalton
• Westat

2
OUTLINE

• Review sources of nonsampling error
• Discuss examples of nonsampling error research
• NHES YATS
• NALS RCGS
• NIPRCS
• Discuss how we choose which methodological
  studies to be conducted

3
TOTAL SURVEY ERROR

• Sampling error
• Nonsampling error
• Missing data
• Coverage error
• Nonresponse error
• Measurement error
• Response error
• Processing (coding, data entry) error

4
COVERAGE ERROR

• Undercoverage to avoid missing persons within
  households
• Undercoverage due to missing households
• Studies of estimation methods to reduce bias
• Studies of efficient designs with lower coverage
  rates
• Studies of the level of coverage bias for
  specific topics.

5
NATIONAL HOUSEHOLD EDUCATION SURVEY (NHES)

• Repeating RDD survey on education topics.
• Two topics of interest in 1989 were high school
  dropouts and preschool enrollment of 3 to 5 year
  olds.
• Concerns about undercoverage lead to an
  evaluation using data from CPS supplement that
  covered these topics and could be classified by
  telephone status.

6
NHES COVERAGE BIAS ESTIMATES

• Coverage rates-14-21 yr olds 92 3-5 yr olds
  88

7
NONRESPONSE ERROR

• Nonresponse bias studies to evaluate the level of
  nonresponse bias in estimates based on
• Frame data,
• Nonresponse follow-ups,
• Simulations.
• Studies evaluating estimation methods (e.g., use
  of different auxiliary variables) to reduce bias.
• Studies evaluating methods of increasing response
  rates.

8
NHES2003 INCENTIVE EXPERIMENT
9
NATIONAL ADULT LITERACY SURVEY (NALS) 1992

• Adults interviewed and given literacy tests.
  Concern that nonresponse was related to literacy.
• An incentive experiment offered 0, 20, and 35.
• Response rates for 20 35 were about 9 pct.
  pts. higher than 0 for minorities 20 pts.
  higher.
• Scores substantially higher for 0 vs. 20 35.
  
• Data collection cost lowest for 20.

10
RESPONSE ERROR

• Studies evaluating the level of errors due to
• Recall
• Questionnaire design
• Sensitive items
• Interviewers

11
NATIONAL IMMUNIZATION PROVIDER RECORD CHECK STUDY

• Parents reported childrens immunizations in a
  supplement to the NHIS.
• Concerns about the accuracy of the parent reports
  (especially if reported by recall rather than
  from shot cards) lead to checks with medical
  providers.
• Provider and parent reports reconciled to create
  best values which are treated as true values.

12
GROSS AND NET DIFFERENCE RATES

• Gross difference rate gdr (B C)/N
• Net difference rate (bias) ndr (B C)/N

13
NDR AND GDR FOR DTP, BY USE OF SHOT CARD,
1994-1996

• Parents substantially underreported DTP.
• Greater underreporting when shot cards used.
• Greater accuracy when shot cards used.

14
NHES 1995 REINTERVIEW STUDY

• The 1995 Adult Education Survey had a response
  variance reinterview (n 1,109 out of 19,722)
• 21 reported work-related (WR) activities
• gdr 12.5 ndr -5.7
• 22 reported personal development (PD)
• gdr 14.3 ndr -1.2

15
NHES INTENSIVE BIAS STUDY

• Used an intensive, cognitive-type reinterview to
  determine true values
• Small sample (n 206) chosen to explore
  reporting AE participation in WR and PD

16
YOUTH ATTITUDE TRACKING STUDY

• Annual cross-sectional RDD survey of 16-24 year
  olds conducted for the DoD to track attitudes
  towards military service.
• Design shifted to include a panel component.
• Annual enlistment propensities declined because
  panel members had lower propensities to enlist.
  

17
YATS ADVISORY GROUP

• Panel attrition and conditioning were the main
  sources considered.
• Few variables consistently related to panel
  attrition and enlistment propensity.
• Revised weighting adjustments did not narrow the
  gap between RDD and panel estimates.
• DoD reverted to a fully cross-sectional design.

18
THE 1991 RECENT COLLEGE GRADUATE SURVEY

• The RCGS included
• A nonresponse study,
• A reinterview study,
• An interviewer variance study,
• A record check study, and
• Other evaluation studies
• Made strong assumptions of additive errors to
  model mean square error of estimates.
• Major contribution is understanding general
  magnitude of errors by source.

19
FACTORS INFLUENCING RESEARCH CHOICES

• Study the major error sources for the specific
  survey design
• Include substantively important variables
• Conduct studies with the potential for assessing
  current estimates and/or designing future surveys
• Take advantage of opportunities for research
• Small studies can be valuable
• Inexpensive studies on low priority issues or
  using less rigorous methods can be worthwhile

20
References

• Brick, J.M., Burke, J., and West, W. (1992).
  Telephone undercoverage bias of 14- to
  21-year-olds and 3- to 5-year-olds (Technical
  report No. 2, NCES 92-101). Washington, DC U.S.
  Department of Education.
• Brick, J.M., Cahalan, M., Gray, L., and
  Severynse, J. (1994). A study of selected
  nonsampling errors in the 1991 Survey of Recent
  College Graduates. U.S. Department of Education,
  Office of Educational Research and Improvement,
  NCES 95-640.
• Brick, J.M., Hagedorn, M.C., Montaquila, J.,
  Roth, S.B., and Chapman, C. (2004). Using an
  experiment to design an RDD survey. Proceedings
  of the Survey Methods Section of the American
  Statistical Association CD-ROM, 4923-4928.
• Brick, J.M., Kalton, G., Nixon, M., Givens, J.,
  and Ezzati-Rice, T. (2000). Statistical issues in
  a record check study of childhood immunization.
  Proceedings of the 1999 Federal Committee on
  Statistical Methodology Research Conference
  (Statistical policy working paper 30, 625-634).
• Brick, J.M., and Morganstein, D. (1996).
  Estimating response bias in an adult education
  survey. Proceedings of the Survey Research
  Methods Section of the American Statistical
  Association, 728-733.
• Brick, J.M., Wernimont, J., and Montes, M.
  (1996). The 1995 National Household Education
  Survey Reinterview results for the adult
  education component (NCES 96-14). Washington, DC
  Office of Educational Research and Improvement,
  U.S. Department of Education.
• Mohadjer, L., Berlin, M., Rieger, S., Waksberg,
  J., Rock, D., Yamamoto, K., Kirsch, I., Kolstad,
  A. (1997). The role of incentives in literacy
  survey research, Chapter 10 pp 209-244 in Adult
  Basic Skills Innovations in Measurement and
  Policy Analysis, eds. Tuijnman, Kirsch, and
  Wagner, Hampton Press, 1997.