Scaling-up education reforms in Kenya: An evaluation of the nationwide Teacher Internship Programme

1
Scaling-up education reforms in Kenya
Anevaluation of the nationwide Teacher
Internship Programme

• Tessa Bold, IIES Stockholm and Goethe University
  Frankfurt
• Mwangi Kimenyi, Brookings Institution
• Germano Mwabu, University of Nairobi
• Justin Sandefur, Center for Global Development
• Alice Nganga, Strathmore University
• December 2011

2
Outline

• Introduction
• Randomization
• Results
• Conclusions

3
Context Kenya's primary education system

• Free Primary Education enacted in 2003
• Abolished fees in all government schools, ending
  fund-raising role of PTA
• FPE increased enrollment in government schools,
  but at the same time unexpected increase in
  private schools enrollment
• Private school market share trebled in last
  decade massive performance deferential.

4
Context Kenya's primary education system

• Focus of the study is government schools
• Education in Kenya is highly centralized under
  MOE limited local gov't role
• Unionized teachers employed directly by Teacher
  Service Commission (TSC)
• Other school expenses funded by a 10/pupil
  annual gov't grant to school bank account
• Corruption in use of FPE funds leading to Aid
  freeze.

5
Context Teacher hiring

• TSC teachers
• Average earnings approximately 250/month
• National pupil-teacher ratio approximately 401,
  but acute shortages in some regions
• Long queue for TSC employment. Excess supply of
  trained teachers
• Teachers in queue often teach on contract or in
  private sector.

6
Context Teacher hiring

• Contract and PTA teachers
• Employed informally out of (unofficial) fee
  revenue
• Average earnings roughly 1/5 of TSC teachers
• Majority have P1 qualifications (requirement for
  intervention described here)

7
Context Teacher hiring
TSC Teachers PTA Teachers
Terms of Employment Permanent Informal Contract
Recruitment District School
Employment Nairobi School
Salary 257/mo 55/mo.
Share 82.5 17.5
8
Genesis of this project

• Ministry of Education
• Desire to formalize the system of contract
  teachers, fill vacancies at lower cost
• Research pilot as means to avoid legal battles
  with Union
• Project researchers
• Goal Work with MOE to integrate rigorous
  evaluation in policy formulation
• Enhance capacity of MOE to evaluate their
  policies.

9
Starting Point Recent RCT Literature

• Growing body of evidence on importance of teacher
  contracts incentives
• Banerjee, Cole, Duo, Linden (2007) Balsakhi
  tutors in Mumbai targeting lagging students
  raised overall performance by 0.28 std. dev.
• Muralidharan Sundararaman (2008) Contract
  teachers in Andhra Pradesh raised performance by
  0.15 std. dev., comparable to (non-experimental
  measure of) benefits of extra civil service
  teachers.

10
Recent RCT Literature

• Kenyan Literature
• MOE Report on an RCT designed to raise pupil test
  performance in Western Kenya
• Class size Massive reduction, from 82 to 43
  pupils ) no effect on scores
• Contract teachers Produced scores 0.21 std.
  dev. higher than TSC teachers.
• SMC training Substitute for contract teachers
  some positive effect, but not for pupils exposed
  to contract teacher.

11
Motivation Challenges of Scaling Up

• RCT evidence from Western Kenya suggests contract
  teachers have a significant, positive effect on
  performance at low cost.
• We focus here on an experiment designed to
  identify the obstacles to scaling-up this
  success.
• 1. Geographic heterogeneity
• Policymakers tend to have less faith in external
  validity than experimentalists!
• Can success in Busia be replicated in poorer
  areas with a thin labor market for teachers? Or
  in Nairobi with higher costs of living?

12
Motivation Challenges of Scaling Up

• 2. Institutional capacity
• RCTs typically run with international NGOs,
  subject to intense researcher supervision.
• Going to scale means working with government.

13
The intervention Contract teachers

• The core intervention is the provision of 1 extra
  teacher in 128 treatment schools
• Randomly assigned to grade 2 or 3 in June 2010
  (all in grade 3 in 2011)
• Instructions not to reassign incumbent teacher
  (split class)
• Treatment reduced class size change in
  contract structure
• Experimental variation in contract used to unpack
  effect of extra teacher
• Central vs. local hiring
• High vs. low salary

14
The intervention SMC training

• School Management Committees
• Comprised of head teacher, teachers, parent
  community reps
• Exercise oversight of central gov't grant monies
• Unclear authority vis-a-vis centrally hired
  teachers
• Training treatment
• Implemented in 1/2 of teacher treatment sample
  Head teacher 1 parent representative invited to
  1-day training
• Curriculum focused on SMC's authorities to
  monitor teachers

15
Cross-cuts
Treatment Control
No of schools 128 64
16
Cross-cuts SMC training

• Within treatment schools

SMC training No SMC training
No. of schools 64 64
17
Cross-cuts NGO vs. Government

• Within treatment schools

SMC training No SMC training
NGO schools 32 32
Government schools 32 32
18
Randomization

• Randomization set up
• Use an optimal multivariate matching algorithm
  to achieve a balanced randomization (see Greevy
  et al. (2004) and Bruhn and McKenzie (2008))
• Treatment and control schools were matched along
  the following dimensions
• Results in nationwide end-of-primary leaving
  exams, results in Grade 1 baseline test
  enrolment no. of classrooms no. of TSC
  teachers no. of contract teachers and average
  pay of contract teachers.

19
Checking balance Specification

• We test the outcome of interest for pupil i in
  school j in period t,
• Let be the treatment in school j
• To examine whether the treatment and control
  schools are comparable prior to the intervention,
  we estimate
• The treatment of interest is whether a school
  received a teacher or not.
• There are two types of outcome variables
• Variables that were used in matching schools
  prior to randomization (all collapsed at the
  school level)
• Additional test score information collected at
  baseline that was not used in conducting the
  randomization.

20
Checking balance School characteristics

• Table Difference in School Characteristics
  between Treatment and Control Schools

Control Treatment Difference
Enrolment 43.33 53.26 9.935 (7.418)
No. of classrooms 11.76 12.48 .715 (1.046)
No. of civil service teachers 10.02 10.21 .195 (1.002)
No. of contract teachers 1.90 2.27 .369 (.347)
Average pay for contract teacher 2843 3393 550.103 (531.535)
Variables in the table were used in the matching
algorithm. Regressions based on 161 schools.
Standard errors in brackets.
21
Checking balance Learning outcomes

• Table Difference in Test Scores between
  Treatment and Control Schools

Control Treatment Difference
KCPE 239.48 235.083 -4.396 (6.783)
Grade 1 English .028 .074 .046 (.166)
Grade 1 Maths .060 .063 .003 (.156)
Variables in the table were used in the matching
algorithm. Regressions based on 161 schools.
Standard errors in brackets.
22
Checking balance additional checks

• Table Difference in Test Scores between
  Treatment and Control Schools

Total English Math
(1) (2) (3)
Grades 2 and 3 .132 (.489) .015 (.887) .117 (.230)
Regressions based on 4187 students in 155
schools. Standard errors clustered at
school. P-values reported in brackets.
23
Results

• The Treatment Effect Specification
• Denote by the outcome of interest for pupil i
  in school j in period t
• Let be the treatment in school j
• To examine whether the effect of the treatment
  following the intervention, we estimate
• The treatment of interest is whether a school
  received a teacher or not.
• The outcome variable of interest is measured as
  performance on English and Mathematics tests in
  Grade 3 and 4.

24
Results Adding a teacher

• Table Effect of an additional teacher on
  learning outcomes

Total English Math
(1) (2) (3)
All grades .297 (.078) .172 (.065) .125 (.128)
Grade 3 .271 (.130) .138 (.157) .132 (.141)
Grade 4 .338 (.086) .223 (.044) .115 (.241)
Regressions based on 7613 pupils in 162 schools.
Standard errors clustered at school
level. P-values reported in brackets. The effect
is equivalent to a 0.16 std increase in test
score
25
Results Comparing Government and NGO

• Table Comparing the effect of an additional
  teacher on learning outcomes in government and
  NGO schools

Total English Math
(1) (2) (3)
Teacher x Government .235 (.198) .150 (.110) .084 (.096)
Teacher x NGO .362 (.208) .195 (.113) .167 (.102)
Regressions based on 7613 pupils in 162 schools.
Standard errors clustered at school level
reported in brackets. Both government and NGO
show positive effect, but only significant for
NGO. For NGO implementation, adding an additional
teacher raised test scores by 0.2 standard
deviations.
26
Results SMC training

• Table Examining the effect of SMC training on
  teacher effectiveness

Total English Math
(1) (2) (3)
Teacher x no SMC training .208 (.185) .152 (.102) .055 (.091)
Teacher x SMC training .398 (.223) .194 (.122) .203 (.107)
Regressions based on 7613 pupils in 162 schools.
Standard errors clustered at school level
reported in brackets. Combining SMC training and
an additional teacher raised test scores by 0.22
standard deviations.
27
Results Salary variation

• Table Examining the effect of salary level on
  teacher effectiveness

Total English Math
(1) (2) (3)
High Salaried Teacher .376 (.301) .212 (.166) .164 (.142)
Low Salaried Teacher .272 (.174) .159 (.096) .113 (.086)
Regressions based on 7613 pupils in 162 schools.
Standard errors clustered at school level
reported in brackets. High salaried teachers
increase test scores more, but difference is not
significant.
28
Results Type of employment contract

• Table Examining the effect of employment
  contract on teacher effectiveness

Total English Math
(1) (2) (3)
Centrally employed teacher .262 (.208) .146 (.114) .117 (.100)
Locally employed teacher .338 (.196) .203 (.107) .135 (.098)
Regressions based on 7613 pupils in 162 schools.
Standard errors clustered at school level
reported in brackets. Locally employed teachers
increase test scores by 0.19 standard deviation.
29
Results TSC Teachers versus Contract Teachers

• Table Comparing civil service teachers and
  community teachers

Total English Math
(1) (2) (3)
TSC teacher -.349 (.381) -.191 (.221) -.158 (.173)
Community contract teacher -.304 (.236) -.178 (.127) -.126 (.117)
Teacher .458 (.205) .265 (.110) .193 (.103)
Regressions based on 7613 pupils in 162 schools.
Standard errors clustered at school level
reported in brackets. No significant difference
between TSC teachers and PTA teachers.
30
Conclusions

• Summary of findings
• Positive impact of reducing class size and of
  similar quantity as existing literature.
• Positive interaction between additional teacher
  and SMC training.
• Both government and NGO administered schools
  show positive effect, but significant only for
  NGO.
• No evidence that TSC teachers do better.
  Contract teachers achieve same outcomes at a
  fraction of the cost.

31

• THANK YOU