Lessons from the PACES Voucher Program in Colombia

1
Lessons from the PACES Voucher Program in
Colombia

• Eric Bettinger, Case Western Reserve
• November 17, 2005
• Federal Reserve Bank of Cleveland

2
BACKGROUND WHY VOUCHERS? WHY COLOMBIA?

• As in many LDCs, Colombian enrollment rates drop
  sharply between primary and secondary grades,
  especially for the poor.
• As is in other LDCs, private enrollment in
  Colombia is substantial, even for low-income
  children.
• Public school quality is low, with poorly trained
  teachers, and a shortage of secondary school
  spots.
• This generated calls for demand-side financing
  initiatives such as vouchers.

3
Enrollment Rates in Colombia, 1993

Source Sanchez and Mendes, 1995
4
THE PACES VOUCHER PROGRAM

• Objectives
• Increase enrollment and attainment
• Part of a general reform/decentralization
• Rules
• Applicants entering 6th grade under 16 attended
  a public primary school admitted to a
  participating private secondary school
• Renewable through graduation conditional on
  academic progress
• Awarded by lottery when over-subscribed
• Gold Standard in Research
• Simplicity

5
PROGRAM DETAILS

• Ran from 1992-97 125,00 vouchers awarded
• Elite private schools opted out participating
  schools about like public
• Voucher and tuition amounts
• Private school fees ran about US350/year in 1998
  (for PACES applicants in our data), as did public
  expenditure/pupil
• The voucher was worth about US190

6
SOURCES OF FOLLOW-UP INFORMATION

• Complete and accurate follow-up is the holy grail
  in research on education interventions
• Samples of adults rarely capture information on
  respondents childhood school environment
• Children are difficult to track families are
  highly mobile
• We collected data using a survey, achievement
  tests, and administrative records from college
  admissions testing (ICFES)

7
TIMELINE

• Bogotá 1995 applicants (Calendar A school year)
• Date Activity Grade
• (w/no repetition)
• CY 1994 Apply
• Feb. 1995 Start school 6th
• Feb. 1996 Start school 7th
• Feb. 1997 Start school 8th
• Feb. 1998 Start school 9th
• Feb. 1999 Start school 10th
• Take ICFES exam
• Feb. 2000 Start school 11th
• Feb. 2001 College

8
PREVIOUS RESULTS (SURVEY)

• Private School Attendance
• Voucher Winners More Likely to Attend Private
  School
• 94 Percent of Unsuccessful Voucher Applicants
  Attended Private School in Year After Lottery
• By three years, 54 percent of Control Group and
  70 percent of Treatment Group in Private
  Schools.
• Educational Attainment
• Voucher winners had completed 0.2 years more of
  schooling
• No differences in dropout rates
• Large differences in grade repetition
• 25 percent less likely to have repeated 6th grade
• Voucher Winners Score One Grade Level Higher on
  Standardized Exams
• Results are Robust to a Variety of Statistical
  Specifications

9
LONG RUN CONCERNS

• Effects Observed after 3 Years
• 1/2 of Voucher Winners No Longer Using Voucher
  after 3 Years
• No Difference in Attendance Rates
• Ambiguity of repetition result.
• Reliance on Survey Data and Response Bias
• Response Rates around 55

10
WHY ADMINISTRATIVE RECORDS?

• All college entrants and most high school grads
  take the ICFES college entrance exams. We use
  ICFES registration status and test scores as an
  outcome.
• Advantages
• Long-term outcome of major significance
• No need to survey no loss to follow-up
• Disadvantages
• Score outcomes may be hard to interpret
• Differential record-keeping by win/loss status
  may generate a spurious treatment effect

11
EVALUATION STRATEGIES

• ICFES registration to proxy high-school
  graduation
• Matching Strategies
• Importance of Accurate Record Keeping and
  Randomization
• Key Results
• 30 Percent of Voucher Applicants Take ICFES Exam
• Voucher winners 7 Percentage Points More Likely
  to Take the ICFES Exam

12
EVALUATION STRATEGIES (cont.)

• Conditional-on-registration score effects
• We look at test scores among those who were
  tested.
• These are clearly contaminated by selection bias
  since vouchers affect testing probability.
• The resulting bias probably masks positive
  effects we consider a number of selection
  corrections

13
(No Transcript)
14
(No Transcript)
15
(No Transcript)
16
EVIDENCE FOR NEGATIVE BIAS

• In our 2002 study, the likelihood of taking our
  test is the same by win/loss status (since we
  tested everyone)
• Voucher Winners Scored One Grade Level Higher
• Limiting this sample to the 44 of tested
  students who also took the ICFES does not
  generate any voucher effect
• Students who would not have taken the college
  entrance exam but do so because of the voucher
  have lower ability than other students who took
  the exam

17
Correcting the Bias?

• Difficult Statistically
• Make Assumptions about Non-Takers
• Create Upper and Lower Bounds
• Possible to Estimate the Effect at Different
  Points in the Distribution
• Effect of Voucher on Test Scores on Average
• Effect of Voucher on Students above the 90th
  percentile
• Above the 75th percentile
• Key Results
• Voucher Winners Score Higher on Average and at
  Each Point in the Voucher Distribution

18
(No Transcript)
19
Voucher Mechanisms

• Private Schooling
• Change in Peer Groups
• Income Shocks
• Student Incentives
• Other Mechanisms?

20
Vocational School Voucher Applicants

• Voucher Losers switch to Academic Schools
• Voucher Winners 18 percentage points more likely
  to be in Vocational School
• Vocational Schools had inferior quality
• Lower ICFES scores
• Lower Proportion of Students Eventually Taking
  ICFES
• Voucher winners have better ICFES outcomes

21
CONCLUSIONS

• Angrist, et al (2002) Voucher winners completed
  more schooling much of this is due to reduced
  grade repetition.
• Winners more likely to take college entrance
  exam (5-7 pts) and hence to graduate high school.
  This suggests additional learning and lasting
  benefits.
• Naive score contrasts are compromised by
  selection bias. Corrected estimates show a
  strong treatment effect (.2s) non-parametric
  corrections generate similar results.
• Voucher may not affect students through school
  quality. Voucher winners who had applied to
  vocational schools perform better on ICFES
  despite lower quality schools.

22
Postlude

• Power of Randomization
• Difference is Randomization and Careful Record
  Keeping
• Administrative Data and Follow-up
• Colombia as a Pioneer in Implementing Social
  Policy Allowing for Randomization
• World Banks Implementation Strategies