Managing for Outcomes Impacts and Intervention Logic

1
Managing for OutcomesImpacts and Intervention
Logic

• Workshops for Public Service Departments
• 22 23 October 2002

2
The Program
8.40 Managing for Outcomes Tony
Hartevelt 9.00 Intervention Logic (IVL) Greg
Claridge 9.50 A Conversation Building an
IVL Greg Claridge Roger Waite 10.10 Introduct
ion to Group Session 1 Roger Waite 10.30 Morning
tea 10.50 Group Session 1 Getting to
Outputs 11.30 Group Session 2 Getting to Your
Outputs 12.30 Feedback from Group Sessions Mary
Slater 12.55 Wrap Up 13.00 Lunch
3
Managing for Outcomes
Iona Holsted 22 October
Peter Bushnell 23 October
4
This workshop is about
The impact of what you do in your department

• Look for a story that explains
• What you are planning to do
• Why you are planning to do it
• How you will know if it is working

5
Why are you doing?

• Providing answers is tough but necessary
• It explains
• the causal link between action and impact
• the basis upon which you give an account

6
Knowing what might work

• Some evidence shows strong causal links
• Look for clear empirical evidence
• Qualitative evidence good logic OK
• Make your assumptions clear
• If experimental approach, plan to verify
• Identify possible unintended outcomes good and
  bad

7
Knowing why might change what

• Understanding why you have chosen a certain
  output should lead you to question
• Are there better outputs for the same result?
• Could these outputs be produced better?
• Is the outcome worthwhile?
• Should you still produce this output?

8
Engaging the Minister

• Essential that
• debates are led from the top and
• the Minister is involved because
• debates about interventions and impacts will
  affect the future shape of the department and
  what it does and stops doing.

9
Departments and central agencies

• Will work with you to challenge the strength of
  your process and story
• Provide support through this and other workshops
• Pathfinder is a key partner in developing this
  support

10
Where does accountability sit?

• Accountability for outputs unchanged
• Accountable for managing for outcomes
• Better outcome definition
• Improved impact measurement
• Explicit rationale - outputs and outcomes
• Able to give an account for what you have done to
  manage for outcomes

11
Managing for OutcomesImpacts and Intervention
Logic

• Workshops for Public Service Departments
• 22 23 October 2002

12
Intervention Logic Guidance
Greg Claridge
13
Intervention Logic

• A reminder the 10 step cycle
• What is it?
• Why is it needed? Benefits ongoing and one-off
• Success factors
• How do you do it? Forms
• Narrative
• Cause and effect diagrams
• Results Chain
• Frameworks
• Logic and Assumption Testing

14
Improving PerformanceA generic 10 step cycle
Strategy Formulation
Optimising Performance
Ex Ante
15
What is it?

• A testable theory of causation K Baehler
• an evidence-based, systematic and reasoned
  description of the links between outcomes and
  outputs. It helps management relate their
  organisational activities to agency outcomes, and
  confirm their business approach is sound.
• Part of a process that enables management to
  systematically prioritise outputs to maximise
  departmental outcomes,
• is a powerful tool that can inform many areas
  of management decision-making and planning.
• BB2, paras 6 and 13.

16
Why is it needed? Ongoing benefits

• Identifying interventions that maximise agency
  outcomes
• Prioritising agency outputs and interventions
• Monitoring agency performance towards outcome
  goals
• Formulation of evaluation approach
• Testing of intervention logic assumptions and
  hypotheses
• Communicating departmental effectiveness

17
Why is it needed? One-off benefits

• Interagency collaboration identifying needs and
  colloboration with other agencies
• Reviewing departmental activities and structure
• Strategic and business planning
• Identifying risks to outcome goals and targets
• Identifying external influences upon agency and
  end outcome(s)
• Communicating and justifying agency activities
  and proposed changes

18

• Success factors
• Clearly defined outcome(s)
• The vital few
• Impact measures
• Research and evidence based
• Simplicity
• Outcomes, not processes
• Logical
• Top down

19
How do you do it? Forms

• Narrative - written description.
• Cause and effect models
• Results chains box and arrow flow diagrams
• Frameworks matrices or tables
• Disciplined Logic and Assumption Testing
  inclusion of IF/THEN to reveal underlying
  assumptions
• Blended Intervention Logic combination of one
  or more of above

20
1. Narrative

• A written description.
• Most useful when it supports other IL forms.

21
2a. Cause and effect diagrams (also Ishikawa or
Fishbone diagrams)
An analysis tool to display possible causes of a
specific problem or condition
http//www.sytsma.com/tqmtools/cause.html
22
2b. Cause and effect diagrams

• Focus attention on one specific issue or problem.
  
• Organize and display graphically the various
  theories about what the root causes of a
  problem may be.
• Show the relationship of various factors
  influencing a problem.
• Cause-and -effect diagrams do not have a
  statistical basis, but are excellent aids for
  problem solving.
• Reveal important relationships among various
  variables and possible causes.
• Provide additional insight into process
  behaviours.
• Focus the team on the causes, not the symptoms.

23
2c. Cause and effect diagrams

• Some points to keep in mind
• Be succinct
• Make sure everyone agrees on the problem
  statement or outcome.
• Resist the temptation to state solutions rather
  than causes.
• Try not to solve problems outside of the group's
  experience or control.
• Take special note of causes that appear
  repeatedly.
• Reach a team consensus if you have identified
  more causes thann you can work with comfortably,
  have team members vote for the most likely
  causes, and concentrate on items that receive the
  most votes.
• Test the most likely causes and verify with data.
  

http//web.mit.edu/tqm/cause_effect.html
24
2d. Cause and effect diagrams

http//www.goalqpc.com/RESEARCH/MJII/CE.html
25
3a. Results Chain

• Box and arrow flow diagrams that link
  departmental outputs to outcomes
• Good for linking many to many
• Simplicity
• Logical
• Identify points for impact evaluation.

26
Results Chains (After OAG Canada)
End/Final Outcomes
Intermediate Outcomes
Immediate Results
Outputs
Activities
Inputs
27
Ultimate outcome Increased educational
achievement/smarter kids
Some bad schools lift their game
Good schools get even better
Some bad schools fold
New good schools come on line
Good schools gain pupils
Bad schools lose pupils
Parents choose best schools for their children
Parents possess appropriate information about
schools
Parents aware of understand program
Karen Baehler, Victoria University of Wellington
Output School vouchers
28
1. Keep it simple!!!!!
2. It should be useful as a powerful
communication tool
3..that helps managers assess performance
against outcome goals and targets.
4..whilst highlighting risks and assumptions.
http//www.acdi-cida.gc.ca/index.htm
29
4a. Frameworks
30
4a. Frameworks
31
4a. Frameworks
32
4b. Frameworks
Uses

• Conventional uses
• Testing existing policy hypotheses
• Testing performance
• Measuring and improving impacts through design,
  management of risk
• Making better use of existing data

• Unconventional uses
• Comparing policy options
• Identifying generic intervention templates for a
  department
• Discovering/
• inventing new interventions

Funnell, S. (1997). Program logic An adaptable
tool for designing and evaluating programs.
Evaluation News and Comment ,6,(1) 5 12.
33
5a. Logic and Assumption Testing

• Powerful enhancement to flow diagrams
• Linked if-then statements
• Action/reaction pairs
• Must use outcome nomenclature not processes
• A chain of conditions to be achieved
• Ultimate/end outcome
• policy goal
• Intermediate outcomes and immediate impacts
• Lead to the end outcome
• But are not ends themselves
• A basis for confirming performance

34
Intervention Logic An Example in NZ Government
Greg Claridge
35
Intervention Logic - Examples

• Cause and effect diagram LTSA
• Logical Results Chain LTSA
• Logical Results Chain Corrections
• Going forward

36

• A Reminder Success factors for Intervention
  Logic
• Clearly defined outcome(s)
• The vital few
• Research and evidence based
• Simplicity
• Outcomes, not processes
• Logical
• Top down

37
LTSA an influencer in the road safety sector
38
1. Cause and effect model LTSA
promoting safety in land transport at reasonable
cost
Targets
Implementation
Interventions
Design and operation of the system
Conditions of entry and exit
Assets/ equipment
User/ Participant
Standards and Rules
Standards and Rules
Compliance
Standards and Rules
Compliance
Compliance
Education
Enforcement
Performance assessment
39
2a. Logical Results Chain LTSA
RCAs
LTSA
Police
Source LTSA, 2010 Strategy
40
2a. Logical Results Chain LTSA
RCAs
LTSA
Police
Source LTSA, 2010 Strategy
41
2a. Logical Results Chain LTSA
RCAs
LTSA
Police
Source LTSA, 2010 Strategy
42
2a. Logical Results Chain LTSA
RCAs
LTSA
Police
Source LTSA, 2010 Strategy
43
2a. Logical Results Chain LTSA
5. IMPACT MEASURES AND RESEARCH
RCAs
LTSA
Police
Source LTSA, 2010 Strategy
44
2a. Logical Results Chain LTSA
RCAs
LTSA
Police
Source LTSA, 2010 Strategy
45
2a. Logical Results Chain LTSA
RCAs
LTSA
Police
Source LTSA, 2010 Strategy
46
2b. Logical Results Chain LTSA

• LTSA uses its intervention logic to drive its
  research agenda.
• Current Examples
• Alcohol case control study
• Drink driving in adolescence and perception of
  risk
• Analysis of speed-related injury accidents on
  State Highway One in Porirua and Kapiti Coast
• Child car restraints infant and child car seat
  use
• Monitoring the effectiveness of road safety
  countermeasures

http//www.ltsa.govt.nz/publications/nzrss-2001/in
dex.html
47
2c. Logical Results Chain LTSA
Interagency collaboration/Sector OCs LTSA uses
its outcome targets and intervention logic to
focus its outputs and activities with Police and
Councils.
http//www.ltsa.govt.nz/publications/nz_rsp/nzrsp0
1_02.html
48
A Conversation Building an Intervention Logic
Greg Claridge Roger Waite
49
RESULTS CHAIN CORRECTIONS
General Outcome

• It is not a linear process
• It is an iterative process that jumps between 4
  phases
• Confirming Outcomes
• Identifying Output options
• Evidence-based Prioritisation of Outputs
• Testing of Intervention Approaches

State Indicator
Interventions
Source Department of Corrections
50
1. CONFIRM OUTCOMES

• Vital few
• Do not lose sight of secondary outcomes eg safe
  and secure
• Buy-in
• Ministers
• Managers
• Stakeholders
• Check for alignment with departmental mission

51
2. IDENTIFY INTERVENTION OPTIONS

• Brainstorm
• What are the barriers?
• What are the drivers?
• Research, literature, and evidence
• Work of other agencies and similar agencies in
  other jurisdictions
• Test using different cultural or intervention
  paradigms
• Departmental vs Sector Intervention Logic

Advocate/Influence

• Case Management Activities
• Assessment of Offenders
• Sentence Management
• Rehab Programmes
• Reintegrative Support

52
3. EVIDENCE-BASED PRIORITISATION OF OPTIONS

• Gather evidence
• Reality check remove the dumb! (feasibility,
  logistic, and ethical tests)
• Eliminate the TESTED and FAILED options
• Cost benefit checks (Quantatative and
  qualatative)

PUNITIVE SERVICES
Activities
Inputs
53
4. TESTING OF INTERVENTION APPROACHES

• Ex ante tests
• Does it work as an Output mix package?
• Coverage Are all target groups covered?
• Compatability Is the delivery style of the
  agency compatible with target groups?
• Are underlying asumptions sound?
• Are risks acceptable
• Ex post tests
• Are assumptions valid
• Are pefromance targets being met?
• Evaluation - including impact on outcomes

• Case Management Activities
• Assessment of Offenders
• Sentence Management
• Rehab Programmes
• Reintegrative Support

54
Group Exercise Identifying alternative and
complementary outputs to improve outcomes
Roger Waite
55
Results Chains (After OAG Canada)
Simple tools for reviewing budget bids?
56
Group Exercises

• September the Thirteenth
• Goodbye Pork Pie
• Trampers Paradise
• 1984 The Orwellian Fantasy

57
(No Transcript)
58
(No Transcript)
59
(No Transcript)
60
(No Transcript)
61
From Results Chains to Management Systems

• Which Outcome?
• Which intermediate outcomes?
• Good output / intervention choices?
• Reiterate are your outcomes right?
• What are your outputs?
• How do outcomes and outputs change in operational
  vs policy roles?