Feeding back Evaluation Results into Policy: Policy Interventions as Complex Dynamic Processes

1
Feeding back Evaluation Results into Policy
Policy Interventions as Complex Dynamic Processes

• Sanjeev Sridharan
• Evaluation Programme
• RUHBC, University of Edinburgh

2
The Background

• In March 2006, the Smoking, Health and Social
  Care (Scotland) Act was implemented in Scotland.
• Seven evaluations being conducted
• The problem
• Dynamic learning across the seven evaluations
• Ongoing project with Sally Haw, Health Scotland

3
Focus of seven studies

• Eight key outcome areas
• Knowledge and attitudes
• ETS exposure
• Compliance
• Increasing support
• Smoking prevalence and tobacco consumption
• Tobacco-related morbidity and mortality
• Economic impacts on the hospitality sector
• Health inequalities.

4
(No Transcript)
5
Questioning the implied causal chain

• Implementation of smoke-free legislation ?
  Increased awareness of health risks ? Reduction
  in exposure to ETS ? Sustained compliance with
  smoke-free legislation ? Reduction in smoking
  prevalence and tobacco consumption ? Reduction in
  tobacco-related morbidity and mortality.
• Plausibility and likelihood of the causal chain
  Are other mediating mechanisms needed to
  enhance the likelihood of the causal chain
  firing?

6
Statement of problem

• The logic model in Figure 1 while complex is
  still incomplete
• A policy such as the smoking ban might lead to
  proximal impacts through perhaps an uncomplicated
  pathway however the logic of the causal chain
  for intermediate and long-term outcomes might be
  incomplete
• Additional programmatic inputs might be needed.
  I
• Information on the nature of such additional
  interventions can perhaps come from a synthesis
  of literature plus additional dialogue with
  stakeholders with an intimate knowledge of the
  processes by which programmes works.

7
The map is not the territory

• In fact, divergence between logic model and
  reality is inevitable because for it to be
  otherwise, an abstract, simplified model would
  have to fully specify a system that is, the
  model would have to describe a deterministic
  system. That is not how real programs in the real
  world operate. A more useful perspective is that
  of strategic planners, who see plans as useful
  for a particular time but in need of constant
  revision as circumstances change (Morell, 2006,
  p. 457).

8
A three-step process

• Step 1
• Identification of outcomes
• Identify the range of outcomes that a complex
  policy can impact.
• Step 2
• Understand likelihood and timeline of impact
• Work with the stakeholders, who understand the
  processes by which components of the complex
  policies work, to identify the anticipated
  timeline of impact and the likelihood of the
  policy to impact the range of outcomes.
• Step 3
• Realist synthesis stakeholder dialogues
• Use both the results of step 2 and a synthesis of
  the literature to frame discussions with
  stakeholders to identify what else needs to be
  done for a complex policy to impact intermediate
  and long-term outcomes

9
Interventions as complex dynamic processes

• Interventions are theories incarnate
• If then types of statements.
• Interventions are active
• Interventions consist of long and thickly
  populated chains
• Intervention theories have a long journey The
  critical upshot of this feature is that
  interventions carry not one, but several implicit
  mechanisms of action. The success of an
  intervention thus depends on the cumulative
  success of the entire sequence of these
  mechanisms as the programme unfolds. The key
  point is that the different theories underlying
  this series of events are all fallible. The
  intended sequence above may misfire at any
  point.. (Pawson and others, 2004, p. 5).

10

• Interventions are always based on a hypothesis
  that postulates If we deliver a programme in
  this way or we manage services like so, then this
  will bring about some improved outcome.
• Interventions generally have their effects by
  the active input of individuals. Active
  programmesonly work through the stakeholders
  reasoning, and knowledge of that reasoning is
  integral to understanding its outcomes

11
Conceptualising the smoking ban as a cause of
long term impacts

• Ban needing other inputs to have long-term causal
  impacts is consistent with definitions of causes
  as INUS conditions
• INUS conditions an Insufficient but Necessary
  part of a condition that is itself Unnecessary
  but Sufficient to cause the result

12
(No Transcript)
13
Smoking ban as an INUS condition

• It is a necessary part of the overall causal
  package (Smoking Ban some other inputs) needed
  to bring about impacts on long-term outcomes it
  is insufficient by itself because other inputs to
  the causal package are also needed.
• The causal package (smoking ban
• other inputs) might be sufficient to cause
  changes in the long-term outcomeshowever, the
  causal package might not be necessary because the
  changes in the long-term outcomes can also happen
  through other causal packages.

14
Participants in the study

• All members of Clean-air Legislation Evaluation
  (CLEAN) collaboration (n57) were invited by
  e-mail to participate in the project.
• Participation was through the web, using a
  private website specially created by Concept
  Systems Incorporated for this project.
• A total of 20 participants completed the concept
  mapping exercise.

15
Data Collection

• The process for collecting data to develop the
  concept map included three main stages
  generation of outcomes, sorting, and rating of
  outcomes.
• 115 outcomes generated
• Sort the 115 outcomes identified in the scoping
  exercise into stacks or clusters based on their
  perceived similarity.
• Respondents were also asked to rate each outcome
  for its importance, likelihood, and timeline of
  measurable impact.

16
Results Concept map
17
Importance ratings

• The concept mapping exercise generates 8 concept
  clusters, but it is striking that the outcomes
  rated as most important belong mainly to the
  immediate health effects and intermediate health
  outcomes clusters, while those that were rated as
  least important fell into the economic impacts
  and social adaptation clusters.
• Outcomes rated as least important were outcomes
  relating to impacts on the drinks and hospitality
  industry - Reduction in total number of staff
  employed by pubs/hotels and Reduction in drinks
  industry profits and Increased conflict between
  managers and employers in workplaces.

18
Likelihood ratings

• Six of the 10 outcomes most likely to happen
  relate to reductions in exposure to environmental
  smoke (ETS), three to changing social attitudes
  and norms about smoking and exposing others to
  ETS, and one to increasing smoking prevalence in
  unenclosed public places.
• Of the ten outcomes rated as least likely to
  occur, five were negative economic impacts, four
  fell into social adaptation cluster and one -
  Reduction in inequalities in smoking prevalence
  was an intermediate health outcome.

19
Pattern Match Importance and Likelihood
20
Anticipated timeline of impact
21
So what?

• Better understanding of the standpoint of the
  group
• Reinforcing initial gains with additional
  program/policy inputs

22
Next Steps

• Realist Synthesis (Pawson, 2006)
• Focus on mechanisms not policy or programmes
• Key questions
• Under what general conditions are public health
  bans likely to work?
• What are the multilevel mechanisms by which
  public health bans enacted at the work place
  impact population health outcomes? Mechanisms
  underlying inequalities?
• Are their certain contexts (say community
  characteristics), under which such legislation
  likely to work?
• Does the review help identify other programmatic
  inputs associated with successful examples of
  legislative interventions such as public health
  bans?
• Stakeholder dialogues

23
Additional stakeholder dialogues

• Range of stakeholders
• Informed by review Understanding mechanisms,
  focus on contingencies, contexts, and conditions
• Multilevel mechanisms
• By highlighting the contingencies on which
  program effectiveness depends, the INUS approach
  reminds evaluators and program planners that a
  policy may need other inputs for it to be
  effective in the long-run.

24
Thinking dynamically about evaluation and
research roles