Improving School Quality in East Africa: Randomized Evaluation of Policies to Create Local Accountability under Free Primary Education in Kenya and Uganda

1
Improving School Quality in East
AfricaRandomized Evaluation of Policies to
Create Local Accountability under Free Primary
Education in Kenya and Uganda

• Germano Mwabu
• University of Nairobi, Kenya
• Frederick Mugisha
• Economic Policy Research Centre, Uganda
• RESEARCH PROPOSALPresented to PEP-AusAid
  Policy Impact Evaluation Research Initiative

2
Overview

• Randomized controlled trial of institutional
  interventions in the Kenyan and Ugandan primary
  education sector
• Project will take place in close collaboration
  with the Ministries of Education (MoE) in both
  countries
• Both countries characterized by large increases
  in enrollment following introduction of free
  primary education and (perceived) decline in
  educational quality
• Research will focus on two key interventions
• Hiring local contract teachers to address
  shortage of government teachers in Kenya, and
  funding locally determined projects in Uganda.
• Community-Based Monitoring System (CBMS)
• Interventions correspond to the two challenges in
  FPE input shortages and lack of parental
  ownership.

3
Research Question

• Expansion of the primary education sector under
  FPE has
• led to influx of new students creating shortage
  of teachers
• undermined the ability of schools to raise
  funding locally due to the abolition of fees
• weakened community based monitoring of schools.
  (FACTS)
• Main question does local accountability require
  local fundraising or is strengthening CBMS
  sufficient? (question what is local
  accountability)

4
Hypotheses in Detail

• Hiring additional contract teachers in Kenya will
  have a significant effect on student learning
  achievements. (Either by lowering teacher-pupil
  ratios or changing the nature of the contract.)
• Providing school lunches in Uganda will have a
  significant effect on student attendance and
  learning.
• Giving parents a financial stake in the school
  (soliciting co-funding to hire contract teachers)
  will increase local accountability and community
  monitoring relative to pure top-down funding.
• Informing school management committees of
  parents rights and teachers responsibilities
  and providing them with tools to monitor school
  performance will improve school management and
  student performance.
• Community-based monitoring systems are
  particularly effective when parents have a
  financial stake in the school i.e., there are
  complementarities between the interventions.

5
Description of Interventions

• School Management Committee Score Cards
• completed by SMC on a termly basis, gathering
  information on
• teacher performance teacher attendance,
  preparation of schemes and lesson plans, class
  room activities measured via direct class room
  observation and spot checks
• financial administration correspondence between
  budgeted and actual expenditures, appropriate
  input purchases, perceived wastage or
  inefficiency in budget use
• School facilities and maintenance hygiene and
  sanitation, classroom conditions and repairs
• School meals (Uganda only)
• matching grants for locally funded school lunches
  
• Create cooperative decision-making across
  geographically proximate schools

6
Description of Interventions

• Contract Teachers (Kenya only)
• Provide co-funding to SMCs to hire contract
  teachers
• Addresses two issues
• Acute teacher shortage due to constraints placed
  on local fundraising by Free Primary Education
• providing for contract teachers will test the
  importance of local control over teaching staff
  in creating accountability and quality service
  provision
• Hypothesis local contract teachers -- hired on a
  fixed term, renewable contract in which the SMC
  has control over hiring and firing -- will be
  more responsive to local demands

7
Research design pop outcomes

• Population to be studied
• 196 Kenyan schools with a total of 3299 students,
  nationally representative, based on Southern and
  Eastern African Consortium for Monitoring
  Education Quality Survey (SACMEQ)
• Ugandan sample is representative of rural primary
  school students in four districts selected to
  represent poor-performing districts of the 4
  regions
• Outcome of interest
• Student achievement 9 months 1 year after start
  of the intervention measured by national exams
  as well as custom-made literacy and numeracy
  exams (in conjunction with Kenya Institute of
  Education in Kenya project, Uganda National
  Examinations Board)
• Changes in enrolment and transition to secondary
  school 9 months after start of the intervention.

8
Research design randomization

• Control schools chosen randomly and
  simultaneously with the treatment.
• In Kenya, use SACMEQ sample stratification so
  treatment and control schools have equal
  representation from urban and rural areas.
  Successful randomization will be tested using
  pre-intervention information.
• In Uganda, sample from primary data collected at
  district level. Schools representative of rural
  sub-counties, with blocking of the sample based
  on pre-intervention test results to ensure
  representation of low-performing schools.
• Use cross-cutting design treatment group for
  each intervention assigned orthogonally
• Difference-in-differences model exploiting the
  randomized design of the intervention, comparing
  mean changes in academic performance between
  treatment and control schools.

9
Research design analytical framework

• Cross-cutting interventions to test hypotheses
  about interactions
• For instance, is a CBMS program enough to
  increase accountability, or is it primarily
  effective when parents have more control over
  teachers (through contract /or fundraising)
• If Y is individual exam performance, i indexes
  individuals and j indexes schools, TEACH and
  CBMS are treatment dummies then
• The Bx parameter will test the interaction
  between local ownership and community monitoring.
• We plan to go further and test two levels of
  funding as follows

10
(No Transcript)
11
(No Transcript)
12
Similarly for Uganda
No meals intervention Meals intervention Meals intervention
No meals intervention No co-funding (full) Co-funding (partial)
No SMC intervention No SMC intervention 17(no Meals No SMC), Pure Control 8 8
SMC score-cards Bottom-up accountability 17 8 X X 8 X X
SMC score-cards Top-down accountability 17 8 X X 8 X X
13
Minimum detectable effects for alternative
outcome measures and designs, Uganda
(1) (2) (3)
Design parameters j100, n30, P.5 j400, n30, P.125 j100, n50, P.5
Controls district, gender District, gender district, gender
Outcome English 0.69 (0.36) 0.52 (0.27) 0.68 (0.35)
Math 0.53 (0.34) 0.40 (0.25) 0.52 (0.33)
Avg score 2.46 (0.37) 1.84 (0.28) 2.43 (0.37)
Pass indicator 0.12 (0.30) 0.09 (0.22) 0.12 (0.29)
Note Minimum detectable effect sizes in
parentheses.
14
Minimum Detectable Effect Sizes for alternative
outcome measures and test parameters, Kenya
SAQMEQ SAQMEQ SAQMEQ
Parameter estimated (assumptions) Estimated intra-class correlation MDES (P.5, j196, n28) MDES (P.25, j196, n28)
Outcome Reading .38 .28 .32
Math .46 .25 .29
Average .45 .27 .32
15
Data Collection/Sources

• Kenya
• Admin datause of Education Management
  information System, Kenya Certificate of Primary
  Education exam data, Southern and Eastern African
  Consortium to Monitor Education Quality (SACMEQ)
  II and III the Kenya Integrated Household Budget
  Survey (KIHBS).
• Follow-up to SACMEQ II to be conducted with MoE
• Uganda
• Admin data use of Primary Leaving Examination
  results and administrative data on school funding
  and district-level inputs.
• Survey of school management and pupils to be
  collected by EPRC researchers in four districts.
  Tests prepared by UNEB (National Assessment for
  Planning in Education) to be administered to a
  sample of students

16
Example of MoE Database nationwide coverage over
time
17
Consultations

• What have we done to include stakeholders in the
  planning process?
• Kenya
• Meetings held with Teacher Service Commission,
  Kenya Institute of Education and MoE (Quality
  Assurance and Basic Ed Directorates)
• Focus groups discussions with teachers and head
  teachers plus church sponsor
• Training workshop with MOE staff to bring them
  into the project as active participants
• Uganda
• Preliminary meetings with District Education
  officials in each of the 4 districts, plus MOE
  staff in Kampala.
• School visits to meet with teachers head
  teachers in each district
• 3-day intervention design and training workshop
  in Kampala with MoE staff, UBOS, DEOs., and civil
  society organizations.

18
Dissemination Plan

• Standard research model get data, do analysis,
  try to convince policymakers youre right
• Our approach in this project
• Train government officials in methods of analysis
• Rely as much as possible on administrative data
  sources
• Involve gov officials in the implementation and
  analysis
• ? Final results are fully owned by policymakers
  as their own collaborative product

19
Project staff

• Lead Institutions
• a. University of Nairobi, School of Economics,
  Kenya
• Germano Mwabu, Faculty member
• Ms Mumia Phillis Machio, Graduate student
• Ms Racheal Nakhumicha Musitia, Graduate student
• Ms Alice Muthoni Nganga, Graduate student
• b. Economic Policy Research Centre (EPRC), Uganda
• Fredrick Mugisha, Senior Research Fellow
• Lawrence Bategeka, Research Fellow
• Ms Madina Guloba, Assistant Research Fellow
• c. Ministry of Education, Kenya
• Ministry officials
• Ministry of Education and Sports
• Ministry officials

20
Project staff

• Collaborating Institutions
• University of Connecticut, Economics Department
  Samson Mwangi Kimenyi
• University of Oxford, Centre for the Study of
  African EconomiesTessa Bold, Justin Sandefur,
  Roxana Gutierez, Andrew Zeitlin, Stefan Dercon
• ---
• All the above researchers are involved in a
  larger research project funded by DFID to improve
  educational outcomes in Africa and Asia

21
Budget

• Two thirds of the requested resources will fund
  project interventions directly

22
Thank you