Intervention Logic and Complex Policy Interventions

1
Intervention Logic and Complex Policy
Interventions

• Karen Baehler
• School of Government
• Victoria University of Wellington, NZ
• Evaluation Summer School
• May 2009

2
(No Transcript)
3
Modeling the system behind a policy problem
involves

• Identifying key steps involved in generating the
  problem (variables)
• Looking for relationships among them
• Cause and effect
• Stimulus and response
• Emergence and chaos
• Identifying contributing / background factors

These may suggest opportunities for interventions.
4
The drug problem system
Other vulnerabilities
Personal problems
s
Underlying mental illness
s
Harmful drug use
s
Note protective factors risk aversion, future
goal orientation, positive role models, support
networks, high opportunity costs and awareness of
them
s
Social problems
Poor life prospects, (low opportunity costs)
s
s
Public costs
Initiation
Impulsiveness, taste for experimentation
Risks (probability of arrest, addiction, overdose
consequences)
Availability, exposure
Price
Peer pressure
5
The education solution
Personal problems
Social problems
s
s
s
s
s
Public costs
Harmful drug use
Public awareness campaigns to deter
regular/harmful use
o
s
Public awareness campaigns to deter
experimentation
Initiation
o
6
The treatment solution
Social problems
s
Personal problems
o
Public costs
s
Needle exchange
s
o
Harmful drug use
Counselling, methadone, other drug treatment
o
s
Initiation
7
The prohibition solution
Social problems
Personal problems
s
s
s
Public costs
s
s
Harmful drug use
?
s
o
Enforcement
Market price
s
Initiation
Law
Social signals
o
s
8
Prohibition The fly in the ointment
s
Social problems
s
Personal problems
s
Public costs
s
s
Harmful drug use
?
Enforcement
o
s
s
Market price
Initiation
s
Social signals
o
Law
9
Assumptions

• Government action is based on some hypothesis
  about cause and effect.
• Policy is itself a hypothesis about how
• outputs (causes)
• will produce
• outcomes (effects).
• Most policy hypotheses contain more than one
  logical step, and some are quite complex in form
  or effects.

10
IVL in policy work

• Articulating and scrutinising these hypotheses,
  including risks and side effects associated with
  them, will lead to better policy choices and
  improved outcomes.
• Policy analysis is evaluation ex ante.

11
END OUTCOME
Social problems/costs will be reduced
then
INTERMEDIATE OUTCOME
if
Fewer people will use X less X will be consumed
then
if
IMMEDIATE IMPACT
Price of X will increase
then
if
OUTPUT
Prohibit drug X
12
Ultimate outcomes

• Does everyone agree on what the programme is
  trying to accomplish?
• If not, then what?

13
SCENARIOS
Social problems/costs will decrease
then
if
Less consumption
Use driven under- ground, unregulated black
markets thrive
then
and
if
Users are impoverished effect on property crime?
Price of X will increase
then
if
Demand shifts to drugs Y and Z?
then
Prohibit drug X
14
END OUTCOME
Climate change will slow
then
INTERMEDIATE OUTCOME
if
Lower emissions
then
if
IMMEDIATE IMPACT
Individuals businesses will change behaviour
then
if
OUTPUT
Tax carbon
15
SCENARIOS
Climate change slows
then
if
Lower GHG emissions
New business opportunities are created?
then
and
Individuals businesses change behaviour
if
Other environmental benefits accrue
then
if
Economic growth slows?
then
Tax carbon
16
Complex interventions

• BMJ

• Systemic policies

• Multiple components
• High variation from site to site
• Active components hard to specify
• Replication difficult

• May have only one mechanism
• But that mechanism changes the whole context of
  behaviours
• Complex web of responses is expected
• Comparisons across jurisdictions are possible,
  but not replication or randomisation

17
(No Transcript)
18
End outcome Increased educational
achievement/smarter kids
Some bad schools lift their game
Good schools get even better
Some bad schools fold
New good schools come on line
Good schools gain pupils
Bad schools lose pupils
Parents choose good schools for their children
Parents seek obtain information about schools
STOP! Where are the weak links in this logic?
Parents aware of understand program
Output School voucher system
19
A device for testing policy options

• Start optimistically
• Outline the logic of how the option is meant to
  achieve the desired objectives
• If it is successful, how will it work?
• Then shift to constructive, creative scepticism
• Identify assumptions behind the logic
• Question the assumptions Are they plausible?
  Are they supported by evidence or experience or
  common sense?
• Identify the risks in the logic
• These are what happens if the assumptions prove
  to be false
• The biggest risks are associated with the weakest
  links in the logic
• Finally, think about altering the policy design
  to improve performance
• Are there features that could be added to remedy
  the risks?

20
Pro-War Logic
End Outcome Reduced terrorism activity
greater global security
Intermediate Outcome New Iraq sets example of
governance for region Other states follow
Intermediate Outcome States that sponsor
terrorism reduce support for terrorist networks
Intermediate Outcome New Iraqi regime proves
stable, Iraqi people see benefits
Intermediate Outcome States that sponsor
terrorism see Anglo-American resolve,
recalculate costs of sponsorship
Intermediate Outcome New democratic, possibly
federalist regime established in Iraq
Intermediate Outcome Saddam Hussein toppled
Output Military attack on Iraq
21
Anti-War Logic
End Outcome Increased terrorism
U.S. allies suffer intl scorn reduced
global power
Flouting of international law weakens U.N.
Antipathy toward U.S. and its allies grows
throughout Middle East
Social disarray breakdown of order sects attack
each other
Land devastated by war conditions dire, people
bitter
International community condemns the war as
a violation of U.N. charter
Opposition is tenacious To win, U.S. allies
must cause more civilian military casualties
physical destruction than planned
Output Military attack on Iraq
22
Assumptions
Pro-War Logic
End Outcome Reduced terrorism activity
greater global security
Assumes The allure of terrorist networks will
fade as demands for liberty, democracy, and
development are met
Risks
Demand for religious fundamentalism
and/or vengeance dominates
New Iraq sets governance example for region
Other states follow
Assumes Pent-up demand for liberty, democracy,
and development among citizens of repressive
Arab regimes
Such states reduce support for terrorist networks
International aid is promised but not delivered
in full other stability factors fail
New Iraqi regime proves stable, Iraqi people
see benefits
Assumes Host of factors required for stability
visible benefits, including large-scale
international aid
State sponsorship of terrorism does not recede,
but becomes more covert threat of force
enflames public backlash against US allies in
Middle East regional instability worsens
States that sponsor terrorism see
Anglo- American resolve, recalculate costs of
sponsorship
Assumes State sponsors of terrorism can be
identified leaders of such states are
influenced by threat of force
New democratic, possibly federalist regime
established
Assumes Availability of new Iraqi leaders
unified public support for change
Cannot form unified government country plunged
into civil war
Saddam Hussein toppled
Assumes Military superiority of U.S. Britain
allies military defeat is sufficient to topple
regime
War prolonged vestiges of Husseins regime
persist after Iraqs military defeat
Output Military attack on Iraq
23
Ultimate Outcome End to poverty
BASIC LOGIC
Community leadership and development
Borrowers empowered, help others, become role
models
Stay out of poverty
More loans made
Further income growth as business develops
Escape from poverty
Microfinance expands
Increased partly used for childrens education,
housing, etc.
Increased partly reinvested in business
Able to pay back loan with interest
Businesses succeed, borrowers income grows
Borrowers start, continue, or expand their
businesses
Grameen Bank
Output Microfinance provided
24
(No Transcript)
25
ASSUMPTIONS
RISKS
REMEDIES
Collective good borrowers stay in community
Successful business owners leave villages, dont
spread knowledge
Build support for 16 decisions, nurture circles
Economic conditions favourable
Natural disaster, economic collapse, death/injury
of borrower
Loan insurance, savings programs, exchange
rate protections
Interest rates reasonable
Interest rates too high, borrowers default
Interest rate formulas needed
Good business plan, good mgt.
Borrowers dont have business knowledge, businesse
s fail
Technical assistance provided
Loan used as intended
All loan spent on consumables, no business
investment
Loan to women in solidarity circles
Grameen Bank continued
26
Group Exercise

• Choose a proposed policy/intervention relevant to
  your work.
• List as many possible outcomes as you can think
  of that might be furthered by that policy.
• Sort the outcomes into a logic chain (backbone).
• Identify associated
• assumptions,
• risks, and
• remedies.
• Discuss implications for evaluation.

27
Alternative forms of evaluation in the face of
complex, systemic interventions?

• Discussion

28
The policy advisers (evaluators?) problem

• Ministers want better advice.
• The most common problems with policy advice
  which can be discerned in recent experience are
• overstatement of what will be achieved
• under-explanation of how policy actions will
  achieve the claimed outcomes.
• From Improving Policy Advice (1993) by G. R.
  Hawke, p. 27

29
Appendix Is IVL a device for discovering
options, too?

• Leave your immediate concerns behind step back
  from the usual ways of viewing your policy area
• Start with a desired goal, a high-level end
  outcome
• Ask How is this goal achieved in reality? What
  are the main conditions that contribute to
  success?
• How are these conditions best nurtured?
• Keep moving down until you spot openings for
  possible government action to support the chain
  of causation.
• Also note risks along the way.

30
End outcome Healthy body weight
Dietary intake
Metabolism
Exercise
Background health factors
Access
Choice
Availability
Income
Price
Genetics, age, environment, etc
Self Control
Conven- ience
Infor- mation
Culture Habit
Tastes
Interventions ???????