Impact Evaluations:

1
Impact Evaluations

• Experiences from Sierra Leone and Ghana

2
Country Summary
Sierra Leone Ghana
Political Post Conflict Relative stability
Governance MII 37th (48 SSA) MII 7th (48 SSA in 2007)
Economic GDP/C US286 PPP US806 GDP GR 4.5 (2005) GDP/C US650 (2007) PPP US2,480 (2005) GDP GR 6 (2006)
Poverty 70 28
Human Devt HDI rank 177th (177) 135 (177 in 2005)
Population 6 million 23 million (2008)
Education 5,000 pri. Schools 1.3 m enrollment 15,200 pri. Schools 3.1 m enrollment (2007)
IE Question Textbooks and TT SMCs
3
Key Evaluation Questions

• Sierra Leone
• Does textbook distribution improve basic pupils
  learning outcomes?
• Does the training of teachers in the use of
  textbooks improve pupils learning outcomes?
• Ghana
• What is the Impact of School Management
  Committees (SMCs) on learning outcomes at basic
  level?

4
Background to Key Evaluation Questions

• Sierra Leone
• Significant expenditures on textbooks
• --Target of 11 is yet to be reached even in
  intervention areas after several years of
  projects
• Teacher in-service training is very important
• --An estimated 40 of Sierra Leones 35,000 to
  40,000 teachers are untrained and unqualified

5
Background to Key Evaluation Questions

• Ghana
• Significant improvements in Primary Completion
  Rate has led to questions of how to improve
  quality of basic education
• Community and school based interventions have
  become very critical with the roll out of
  decentralization in the management and governance
  of basic schools.

6
Evaluation Design

• Sierra Leone
• Four Local Councils out of 19 LCs were selected
  each representing one of the four regions in
  Sierra LeoneLocal Councillors were invited to a
  meeting to randomly select by lottery LCs who
  will participate in the Impact Evaluation.
• Schools were randomly selected into three groups
  from the EMIS data as follows
• -- control (30 primary schools),
• -- textbook treatment (30 primary primary
  schools),
• -- textbooks and teacher training treatment (30
  primary schools)

7
Evaluation Design

• Sierra Leone
• Pupils in Primary 4 and 5 were selected for the
  IE because they could read. Primary 6 was left
  out because they are preparing for the NPSE and
  it was agreed that the IE should not be
  intrusive.
• --Baseline concentrated on achievement test
  scores for P3 and P4 because these cohorts would
  move to P4 and P5 by the time of the actual
  evaluation.

8
Evaluation Design

• Ghana
• 53 deprived districts are already receiving
  additional resources from governmentthese
  districts were selected based on a number of
  indicators spanning inputs, access, quality of
  learning.
• 212 public primary schools were selected randomly
  from these 53 distircts.

9
Evaluation Design

• Ghana
• Three groups were randomly obtained from these
  212 schoolsbased on the EMIS as follows
• --control group (70 basic schools)
• --information about SMC roles and
  responsibilities group (71 basic schools)
• --information capacity building of SMCs group
  (71 basic schools)

10
Components of a Successful IE

• Start-up workshop was/is criticalAbuja, Dakar
• Field Coordinatorwhose key focus on IE
• Government ownership and commitment, Ministry of
  Education involvement at concept stagebut
  remember to diversify involvement
• WB involvement TTLs role in trying to
  coordinate all stakeholders

11
Elements of a Successful IE

• Timely availability of funds (EPDF is great!)
• Follow-up vigorously on the intervention budget
• Understand the issues and take risks
• Keep impact evaluation on the radar at all
  timestake every opportunity to communicate
  effectively on IE

12
Issues and Risks

• Watch out for possible derailment of the process
  and be prepared for this e.g. changes in the
  socio-political environment problems with
  funding the intervention
• Political changes, changes in government
• Ethical concerns, who gets textbooks first?
• Manage potential bad press
• Manage the expectations, because we may find
  nothing
• Create the stage for the next IE

13
Political Change-Sierra Leone

• Change in government in Sierra Leone
• --New councilors were elected last August, thank
  goodness they thought to involve the more
  permanent Chief Administrators in the district
  and school selection workshop
• Commission of Inquiry in Sierra Leone
• --All activities have sort of come to a
  stand-still because of appearances before the
  commission. Textbooks distribution in Pujehun by
  the MCSL and RADA have halted because of
  appearances before the commission.

14
Political Change-Ghana

• Elections in Ghana
• --All sector resources are understandably
  concentrated on the elections
• --First round was on Dec. 7, run-off on Dec. 28.
• Critical implications for interventions budget
• --A new government means a new budget, new
  priorities etc.

15
Status of Impact Evaluation

• Sierra Leone
• --EPDF application and funds obtained
• --Field Coordinator is on board
• --Recruited Statistics Sierra Leone for data
  collection, data entry
• --Textbooks are still being distributed to
  treatment schools
• --Teacher Training Manual is being revised

16
Status of Impact Evaluation

• Sierra Leonenext 12 months
• --Complete textbooks distribution
• --Complete manual revision
• --Roll out teacher training Roll out textbooks
• --Plans to have another small survey, probably
  testing teachers as well
• --Obtain the intervention budgetEFA FTI GoSL
• --Data collection, analysis for follow-up
  survey
• --IE Report

17
Status of Impact Evaluation

• Ghana
• --EPDF application and funds obtained
• --Field Coordinator is on board
• --Questionnaires have been finalized
• --Recruitment of firm for baseline is far
  advancedtechnical and financial evaluation
  reports being finalized

18
Status of Impact Evaluation

• Ghananext 18 months
• --Baseline survey planned for Jan. 2009
• --Data entry and analysis Jun. 2009
• --Training of trainers for SMCs
• --Training of SMCs, capacity building

19
Status of Impact Evaluation

• Ghananext 18 months
• --Mid-Term Review classroom observations and
  surprise visits to measure pupil and teacher
  attendance
• --SMC survey
• --Final Data Collection, data entry and analysis
• --IE Report