Conducting realist evaluations

1
Conducting realist evaluations

• Nick Tilley

2
Key principles

• Programs are theories incarnate
• Evaluations are tests of theories
• Theories need to comprise context-mechanism-outcom
  e conjectures
• Mechanisms refer to the ways in which effects are
  brought about
• Contexts refer to the conditions for the
  operation of mechanisms. They are seldom closed.
• Outcomes refer to the effects of mechanisms
  activated in context
• Mechanisms generally (though not always) involve
  reasoning and resources
• Programs work differently amongst different
  subgroups

3
(No Transcript)
4
1. Elicit and formulate theory on what is about
the programme that works for whom in what
circumstances

• The theory takes the form of context, mechanisms,
  outcome patterns.
• The theory focuses on subgroups.
• Theory can be elicited from documents, programme
  architects, practitioners, previous evaluation
  studies and social science literature.
• Theory formulation involves thinking through how
  the programme might work.
• It will seldom be possible to focus on all
  potential theory in a programme?

5
2. Collect data on appropriate mechanisms
contexts and outcomes

• Data of various kinds may be collected.
• Qualitative and quantitative methods may be used.
• Documentary evidence, official records, surveys,
  interviews, observation, focus groups, tests and
  so on may all be used.
• Scavenging takes place for the best data
  available in relation to the realist hypotheses

6
3. Analyse outcome patterns to see which can and
which cannot be explained by initial theory

• The task is to make multiple comparisons
  identifying winners and losers.
• The purpose is to see if the CMOC model will
  explain/predict complex program footprints
• The footprint describes the patterns of success
  and failure, for example by demographic, personal
  attribute, temporal, spatial, and biological
  subgroups.
• There is no single analytic method

7
4. Revise understanding of CMOC configurations as
a prelude to a further round of theory refinement

• Which CMOC theories are corroborated?
• Which CMOC theories are refuted?
• Have unanticipated patterns emerged?
• What revised and refined CMOC theories are
  suggested?
• What scope is there for restarting the cycle for
  further theory improvement?

8
Forms of realist evaluation

• Realist formative evaluation
• Realist program evaluation
• Realist synthesis

9
1. Realist formative evaluation

• Gauge contexts and mechanisms generating the
  presenting problem
• Identify pinch-points that will alter balance of
  mechanisms
• Work out how to put pressure on the pinch-points

10
Realist formative evaluation example

• Ronald Clarke and Herman Goldstein (2002)
• Focus on theft from construction sites in
  Charlotte Mecklenburg
• Review of past responses that had failed to
  reduce problem
• Highlight high number of theft of plug-in white
  goods prior to occupation of new houses
• Suggest non-installation of plug-in white goods
  prior to occupation
• Work out levers for implementation
• Observe drop in specific thefts

11
2. Realist program evaluation

• Elicit/formulate realist theories
• Collect sub-group related data relevant to CMOC
  theories
• Analyse data in the light of the CMOC theories
• Look for ways in which patterns relevant to the
  theories may be generated in unintended ways as
  the program unfolds

12
Realist program evaluation example

• Gloria Laycock (1986, 1992)
• Property marking in South Wales
• High profile, high dosage (70) implementation in
  isolated villages
• Persuasive publicity in confined area hypothesis
• Recovery and detection hypothesis
• Crime drops in houses with stickers
• Crime drops for unmarked as well as marked
  property, gradually rising later
• Renewed crime drop at publication of initial
  findings with no further intervention
• No increase in property recovery or detection
  associated with property marking
• Support for publicity hypothesis

13
3. Realist synthesis

• Look across programs
• Look across program domains
• Look across methods used in studies
• Focus on problems
• Focus on mechanisms
• Focus on contexts
• Develop and test middle range theory

14
Realist synthesisexample

• Pawson and Tilley (2001)
• Altruism and blood transfusion
• Looks across studies on blood transfusion, 1970
  to 1999
• Looks at studies of survey non-response and
  methods of reducing it
• Looks at studies of the meaning of money and
  reward, especially earmarking
• Looks at changes in pathogens, and in the
  technology of blood checking and blood use
• Looks at review of studies of rates bad blood of
  varying kinds
• Looks at the economics and culture of blood
  provision amongst subgroups by sexuality, wealth,
  geography and income
• Looks at varying economics of blood payment in
  different conditions
• Finds variation by time, place and sub-group

15
The problems of conducting realist evaluations

• It departs from experimental orthodoxies
• It is easy to misunderstand
• It involves hard thinking
• It can be technically demanding
• It refuses to provide (meaningless) simple
  (minded) answers that are sometimes wanted
• It can be expensive

16
Conclusion

• Despite practical problems, realist evaluation
  is, we think, based on sound principles
• I would like to hear of Canadian realist
  evaluations
• It may be necessary to educate policy-makers and
  commissioners about realist evaluation
• In Britain there are signs that realism is being
  taken seriously in local authority, health,
  crime, education and skills, and social care
  settings.