Measurement Matters: The Use of PETS and QSDS

1
Measurement MattersThe Use of PETS and QSDS

• Public Expenditure Analysis and Management
  CourseRitva Reinikka
• Development Research Group (DEC)
• Public Services Research Team
• January 13, 2004

2
Increasing public spending is not enough to reach
MDGs
Percent deviation from rate predicted by GDP
per capita Source Spending and GDP from World
Development Indicators database. School
completion from Bruns, Mingat and Rakatomalala
2003
3
Similar changes in public spending can be
associated with vastly different changes in
outcomes
Sources Spending data from World Development
Indicators database. School completion from
Bruns, Mingat and Rakatomalala 2003
4
and vastly different changes in spending can be
associated with similar changes in outcomes.
Sources Spending data for 1990s from World
Development Indicators database. Child mortality
data from Unicef 2002. Other data from World
Bank staff
5
Unit cost and performance in primary education
Mauritania
6
Expenditure incidence tends to favor the
better-off even in health and education
Health
Education
Source Filmer 2003b.
7
Short and long routes of accountabilityin
service delivery
8
The relationship of accountabilityhas five
features
9
Why do we need new tools?

• Limited impact of public spending on growth and
  human development to answer why?
• New demands for evidence on efficiency of
  spending and performance in service delivery
• Lack of reliable data on finance and performance
  obtain them from sample survey ? PETS and QSDS
• New approaches in aid delivery
• Move towards budget support (e.g., PRSC)
• Related fiduciary and accountability concerns

10
Public expenditure tracking surveys PETS

• Diagnostic and monitoring tool to understand
  problems in budget execution
• delays / predictability
• leakage / capture
• discretion in allocation of resources
• Data collected from different levels of
  government, including service delivery units
• Data from record reviews and interviews
• Variation in design depending on perceived
  problems, country, and sector

11
Quantitative service delivery surveys QSDS

• Focus on frontline service providing unit, e.g.
  health facilities and schools
• Inspired by multi-purpose micro-level household
  and firm surveys
• Resource flows (financial and in-kind)
• Availability/adequacy of inputs
• Service outputs and efficiency
• Quality of service
• Focus on costs, dimensions of performance in
  service delivery, ownership categories

12
Hybrid approaches

• Link facility surveys with surveys of
  administrative levels upstream (public
  officials PETS)
• Why different performance in the same system?
• Link facility surveys with household surveys
• Effect of school/facility characteristics on
  household behavior and outcomes?
• Mix quantitative and perception-based approaches
  (e.g., exit polls, staff interviews, focus group
  discussions)
• Relationship between perceptions and observable
  characteristics of schools or facilities?

13
Nonwage funds not reaching schools evidence from
PETS
Country Mean percentage
Ghana 2000 49
Madagascar 2002 55
Peru 2001 (utilities) 30
Tanzania 1998 57
Uganda 1995 78
Zambia 2001 (discretion/rule) 76/10
Source Ye and Canagarajah (2002) for Ghana
Francken (2003) for Madagascar Instituto Apoyo
and World Bank (2002) for Peru Price Waterhouse
Coopers (1998) for Tanzania Reinikka and
Svensson 2002 for Uganda Das et al. (2002) for
Zambia.
14
Capture of public funds (Uganda PETS)

• Large variations in receipts across schools
• Bargaining between local officials and schools
  over nonwage spending
• Election finance and elite capture
• When using actual spending data from PETS,
  neutral benefit incidence became highly
  regressive
• Leakage endogenous to school characteristics
• Parents income most important determinant
• Size of school, teacher qualifications
  significant, too
• Sparked an information campaign which increased
  client power and reduced capture

15
Schools in Uganda received more of what they were
due
Source Reinikka and Svensson (2001), Reinikka
and Svensson (2003a)
16
Impact evaluation of information campaign

• Repeat PETS shows huge reduction in capture of
  capitation grants
• From 80 to 20
• Schools that have access to a newspaper received
  14 percentage points more of their entitlement
• Information campaign was an effective and cheap
  way of reducing capture of funds

17
Ghost workers on payroll (percent)
Country Education Health
Honduras 2000 5 8.3
Uganda 1993 20 -
Source World Bank 2001 Reinikka, 2001.
18
Nigeria QSDS Problems with local government
accountability

• Pervasive non-payment of salaries of primary
  health workers in some states

Percent of staff respondents
20
KOGI (total240)
80
LAGOS (total495)
15
10
5
0
0
1
2
3
4
5
6
7
8
9
10
11
12
Months salary not paid
19
Nigeria QSDS in health care

• Non-payment of staff salaries cannot be explained
  by lack of resources available to local
  governments
• Even when local government spending on staff
  salaries is sufficient to cover actual staff
  costs, survey of staff revealed extensive
  non-payment
• General problem of local accountability in the
  use of public resources transferred from higher
  tiers of government, about which local citizens
  may not be well informed, as they are not the tax
  payers

20
Frontline provider surveys 2002Absence rates
(percent) among teachersand health-care workers
Country Primary schools Health facilities
Ecuador 16 -
Honduras 2000 14 27
Peru 13 26
India (19 states) 25 43
Indonesia 18 42
Uganda 26 35
Zambia 17 -
21
Percent of staff absent in primary schools and
health facilities
50
Primary schools
Primary health facilities
40
30
20
10
0
Bangladesh
Ecuador
India
Indonesia
Papua New
Peru
Zambia
Uganda
Guinea
22
Good reasons for doing PETS/QSDS

• Diagnosing problems shaping the reform agenda
• Analysis guiding reform
• Monitoring over time/benchmarking
• Understanding systems useful for donors and
  governments
• Research collaboration between practitioners
  and researchers
• A good basis for information campaigns to
  increase client power

23
Survey Design Survey what? Why?

• What are the problems? Research question and
  hypothesis? Are there important gaps in
  understanding of the nature, extent, and sources
  of problems?
• Is a quantitative survey the appropriate tool?
  Stand-alone or as a complement? Worth the cost
  (50-150K)?
• Is it feasible? How is the budget structured and
  implemented so that relevant data can be
  collected?
• Who is the audience? Is there a political demand
  for new information (often bad news)?
• Will the information be used? By whom? How to
  ensure impact?

24
Implementation issues Who? How?

• Requires skills similar to other micro surveys
• Steps in implementation
• Concept document
• Buy-in across the board Ministry of Finance,
  sector ministry, local governments, frontline,
  donors, etc.
• Rapid data assessment
• Questionnaire design
• Identifying and contracting implementing agency
• Pilot questionnaires
• Enumerator training
• Field work (quality control and data management)
• Data analysis
• Dissemination ? impact on policies