Quasiexperimental studies and surveys

1
Quasi-experimental studies and surveys
Susan Purdon, Survey Methods Unit, NatCen
2
Scope

• Trends in policy impact evaluation
• Associated trends in the use of data
• Some challenges
• Some possible options for future development

3
Trends in policy impact evaluation in GB

• No clear trend in RCTs
• Sustained, and possibly increasing, use of
  quasi-experiments
• Typically outcomes for programme participants or
  eligibles are compared to outcomes for a
  comparison group. Difference between the groups
  after controlling for confounders is taken as the
  impact of the programme
• Feature of most quasi-experiments is that they
  are data-hungry (large sample sizes many
  variables)

4
Using admin data for quasi-experiments

• Advantages of using admin data (rather than
  bespoke survey data) are well-recognised
• Large sample sizes inexpensive minimal
  non-response bias (except where consent is
  needed)
• Examples include the longer-term impacts of NDLP
• But not always possible
• Wrong outcome measures
• Confounders missing
• Issues around identifying the eligible pop

5
Using survey data as an alternative

• Existing survey datasets are sometimes used (e.g
  LFS MCS).
• But often a combined problem of smaller sample
  sizes and lack of all necessary variables rules
  it out
• More usual model is bespoke survey data
  collection
• Advantages
• Focus on eligible population
• Identification of a suitable comparison group
• Appropriate primary and secondary outcomes
• Full data on confounders

6
Examples

• Early education pilots for Two Year Old Children.
  
• Before-after interviews with c.1400 users of
  childcare and a similar number of comparison
  group interviews. Tested whether child cognitive
  development in the pilots exceeded that for the
  comparison group
• Evaluation of the Free School Meals pilots
• Involves tracking diets of school children in
  pilot areas v. diets in comparison areas

7
Challenges raised

• Impacts tend to be modest, so large sample sizes
  needed
• But surveys are v.expensive, esp for face-to-face
  interviews
• Budgets limited, and appear to be decreasing
• So the future not clear. Options include
• Cheaper survey methods (e.g postal)
• More focussed evaluations just on those where
  impact is likely to be greatest

8
Possible options for research community

• Increase the pressure for RCTs (which often need
  less data)
• Press for larger (or at least maintained) budgets
• Press for policies that can sensibly be evaluated
• Improve knowledge base so that evaluation designs
  are optimal

9
Improving knowledge base

• Better understanding of how to use surveys to
  control for self-selection bias
• Better understanding of how to minimise mean
  square error (standard error bias)
• Better understanding of how to combine survey and
  admin data approaches

10
(No Transcript)