Scaling up and Evaluation

1
Scaling up and Evaluation

• Tara Vishwanath
• TN Workshop, Oct 5, 2004

2
What Is Evaluation ?

• Serves two types of objectives
• (1) Process evaluation
• Audit and monitoring
• Did the intended policy actually happen ?
• (2) Impact evaluation
• What effect (if any) did the policy have?
• How would individuals who did benefit from the
  program have fared in the absence of the program
• How would those who did not benefit have fared if
  they had been exposed to the program
• Generally both are important .. however.

3
Scaling Up and Impact Evaluation

• Impact evaluation a necessary basis for
  successful scaling up
• Provides the knowledge basis to choose which
  projects should be expanded
• Helps build long term support for development
• The role of international institutions
• Knowledge is a global public good
• Setting up a new standard for evidence

4
Basic Problem

• How well does a particular program work?
• Should we expand or contract a program?
• All evaluations establish a counterfactual.
• What would have happened without the program?

5
Need for Prospective Evaluations

• In general, program beneficiaries are specially
  selected (poor, motivated, etc) and they are
  thus not comparable to non-beneficiaries
• Comparison between beneficiaries before and after
  receiving the program is not informative many
  other things happened over time
• Need to plan the evaluation ex-ante, to ensure a
  reliable comparison group
• Randomized phase-in of a program allows the
  constitution of strictly comparable treatment and
  comparison groups

6
Simple counterfactual

• Randomize people/firms
• From an eligible set
• Some get, some dont
• Chosen explicitly at random
• Compare those who get to those who dont
• Great counterfactual
• Know for sure that only difference is program

7
Opportunities for Evaluation

• Not all projects and programs can be evaluated
  through randomized evaluation, but there are many
  opportunities
• Prospective randomized evaluation
• Pilot projects
• Government sponsored evaluations
• Replication and evaluation of existing projects

8
Pilot Projects

• Small, NGO-sponsored, pilot project
• Seva Mandir was considering adding a second
  teacher in all its NFE centers in Udaipur
  district
• Program was randomly phased-in in half the
  schools, and performance was compared between
  schools who received and did not receive the
  program
• Positive effect on school opening, childrens
  attendance, but not test score
• Seva Mandir decided not to implement the project,
  since it did not have the expected impact

9
Government Sponsored Evaluations

• PROGRESA in Mexico
• Mexican government wanted an evaluation order of
  community phase-in was random
• Results child illness down 23 height increased
  1-4cm 3.4 increase in enrollment
• After evaluation PROGRESA expanded within
  Mexico, similar programs adopted throughout other
  Latin American countries

10
Increasing school participation

• Work in Kenya
• School meals
• Reducing costs of education
• School-based health programs
• Can compare cost effectiveness

11
Evaluation of Existing Programs

• Can pilot project be successfully extended?
• Evaluate program that have already gone to scale
  remedial education in Vadodara and Mumbai
  (Pratham)
• Pratham reaches 121,000 children in 20 cities,
  and employs about 10,000 individuals, and is
  easily replicable

12
Prathams balsakhi Program

• Balsakhi childs friend. A young woman from the
  childrens community is hired to provide remedial
  education to children reaching grade 3 or 4
  without mastering the competencies in
• Program is present in all municipal primary
  school in Vadodara and in parts Mumbai. Schools
  randomly divided into two groups one group
  receives balsakhi in standard 3, one group
  receives her in standard 4
• Competencies evaluated with pre and post test
• Large and significant increases in both years and
  both cities (0.4 standard deviation). Balsakhi
  are 5 times more cost effective than teachers

13
Current Practice

• Evaluation is often planned (e.g., 1 of budget
  of World Bank program)
• Emphasis on process evaluation
• Evaluation not built in ex-ante
• Evaluation is not taken into account in decision
  making
• It is often subcontracted to consultants who are
  not given the training or the resources they need

14
Example of Lost Opportunity DPEP in India

• District Primary Education Program the largest
  world bank sponsored education program
• Grants for teacher training, textbooks, teacher
  aids
• Placement criterion low female literacy and
  greater potential for success
• Large effort of data collection in DPEP districts
• Large number of evaluation studies
  commissioned by DPEP

15
Evaluation Is Jeopardized

• Within low literacy rates districts, those with
  the greatest potential for improvement were
  chosen
• Before/After Indias growth rate at least 5
• DPEP/Non DPEP DPEP in places with low female
  literacy rate, other unobservable differences
• Improvement in DPEP/Non DPEP Placement on the
  basis of potential for improvement
• NO DATA was collected on non DPEP districts

16
Ongoing effortsPartnership with Government of
Punjab Pakistan

• Government of Punjab contemplating initiatives
  regarding new programs for education
• Collaborating with government to evaluate and
  experiment to inform a forward looking education
  policy
• Three districts study in villages that have one
  private school ( and one public school)

17
Evaluation Strategy ( Pak contd)

• Randomized control- treatment design
• Some villages/schools are given intervention
  others not
• Villages that are given the intervention are
  chosen randomly
• Two interventions (a) Report card, (b) School
  councils
• Rationale for (a) addresses missing info problem
  and is financially feasible , can be replicated
  easily
• Rationale for (b) addresses lack of control by
  ultimate clients, financially feasible, but
  replicable?

18
Discussions in AP, India

• Prospective randomized evaluation of schooling
  inputs and teachers incentives through a pilot
  in Andhra Pradesh
• The main objectives of this work improve learning
  achievements
• Provision of child specific inputs ( notebooks,
  slates, pencils),
• Classroom specific inputs ( desks and benches ,
  blackboards)
• Teacher materials (as defined in SSA guidelines)
• Teacher incentives
• Negotiating the plausibility of randomization
  (ongoing)

19
Lessons

• Randomized evaluations are often feasible, and
  have been conducted successfully
• Randomized evaluations are labor intensive and
  costly, but no more so than other data collection
  activities
• Especially cheap when you consider enormous
  ability to learn from them

20
Lessons

• Results from randomized evaluations can be quite
  different from those drawn from retrospective
  ones
• Costs can be reduced and comparability enhanced
  by conducting a series of evaluations in the same
  area
• Most problems of randomized evaluations are
  problems with all evaluations
• Attrition bias
• Spillover effects
• Generalizability beyond one program

21
Lessons

• Publication bias likely huge
• Bigger problem if identification strategy is
  decided ex post
• Randomized evaluations help
• Institutions help even more

22
When to randomize?

• Cost-Benefit calculus
• Good evaluations have huge spillovers
• Multiple retrospective, non-randomized
  evaluations
• OR
• One randomized evaluation
• Certain evidence allows
• Diffusion of knowledge
• Rescaling of programs
• Further testing

23
The role international agencies can play

• Promoting and financing rigorous evaluations
• Defining priorities for randomized evaluation
• Increasing share of randomized evaluations from
  close to zero to 5?
• Need new incentives/structures
• Working with partners.

24
An opportunity for Government, Development
Agencies and Partners

• Randomized trials transformed medicine in the
  20th Century - perhaps randomized evaluations can
  transform development assistance in the 21st!!