Is it What Works that Matters Evaluation and EvidenceBased Policy Making

1

• Is it What Works that Matters? Evaluation and
  Evidence-Based Policy Making
• Professor Ian Sanderson
• Policy Research Institute, Leeds Metropolitan
  University

2
Outline of presentation

• What is evidence-based policy making
• Why should policy-making be evidence-based?
• To what extent is policy informed by evidence?
• How much does evidence of what works matter?
• Conclusion What kind of rationality?

3
What is evidence-based policy making?

• Four key elements
• policy development founded on evidence of
• problems and needs to be addressed
• what has worked in previous policies
• policy option appraisal based on evidence of
  likely effectiveness
• policy evaluation informs decisions on
  continuation/change
• policy articulation based upon overview of
  relevant research (cf. enlightenment)
• Traditional instrumental view
• what about normative aspects?
• Evidence-based practice
• increasing role of evidence in professional
  judgement

4
Why should policy making be evidence-based?

• A self-evidently good idea?
• effectiveness / VFM
• ethical case avoiding harm
• Deep-rooted sense of rationality in Western
  society
• 1960s euphoric sense that social science
  really could change the world (Bulmer, 1987)
• New Labour
• modernising government and policy making
• EBP what counts is what works
• Blunketts appeal to rationality

5
Blunketts appeal to rationality

• It should be self-evident that decisions on
  Government policy ought to be informed by sound
  evidence.
• The Government has given a clear commitment that
  we will be guided not by dogma but by an
  open-minded approach to understanding what works
  and why.
• I believe passionately that having ready access
  to the lessons from high quality research can and
  must vastly improve the quality and sensitivity
  of the complex and often constrained decisions
  we, as politicians, have to make.
• Good government is thinking government rational
  thought is impossible without good
  evidencesocial science research is central to
  the development and evaluation of policy.
• Source DfEE (2000)

6
To what extent is policy informed by evidence?

• But the reality is different!
• powerful political forces of inertia,
  expediency, ideology and finance (Walker, 2000)
• lack of robust evidence!!
• However, there is evidence of increasing
  influence of research and evaluation
• Infrastructure
• Cochrane and Campbell Collaborations Evidence
  Network
• Increased govnt spending on research and
  evaluation
• Examples Welfare-to-Work / New Deals
• BUT much of literature is normative it
  remains an act of faith (Davies, et. al. 2000,
  p.352)

7
How much does evidence of what works matter?

• Instrumental rationality
• evidence improves effectiveness of means to given
  policy ends an exercise in social technology
  (Schwandt, 1997)
• abstracts from political, normative and
  organisational context
• Political context
• overt irrationality of politics
• covert shaping of research agenda
  policy-based evidence?
• Normative context
• policy making as communicative/deliberative
  process requiring judgement on ethical-moral
  concerns (Majone, 1989)
• practical wisdom the key to appropriate
  decisions/action

8
How much does evidence of what works matter?

• Organisational context
• new institutionalism normative order and
  logic of appropriateness
• professional practice tacit knowledge, practice
  wisdom and ethical concerns
• Accountability frameworks
• managerialist accountability performance
  management replacing professionalism external
  authority replacing trust
• performance measurement inept science?
• target culture demonstration v improvement

9
Conclusion What kind of rationality?

• The periodic irrelevance of social science may
  be far easier to explain than we have thought.
  For in the face of ambiguous claims, little time,
  and unsatisfactory data, most practical people,
  amongst them public policy analysts and planners
  have to learn not only about feasible outcomes
  and stable relationships of cause and effect but
  about value in our possible worlds, about
  potential significance, import, consequentiality.
  Practical people in our lives help us to learn
  what to want, what to care about, and what we
  should care for, too. Yet as long as social
  scientists treat value as essentially irrational,
  an epiphenomenal dependent variable, or merely
  the expression of preferences, they will ignore,
  if not fail to understand entirely, the demands
  and opportunities of practical judgement and
  deliberative rationality, the heart of practical
  enquiry in the applied professions. (Forester,
  1995, p.64)

10
Conclusion What kind of rationality?

• Practical rationality
• relevance to assumptive/normative worlds
• reconstructing cognitive/normative frames
• Institutional basis
• Longer-term, dialogical relationships
• Mutual understanding of normative contexts
• Development of high trust relationships
• Responsiveness to timescales of policy makers
• Active forms of dissemination
• Researchers / evaluators as advocates?

11
Conclusion What kind of rationality?

• Applying social science to address social
  problems does matter
• But need for modest expectations for evidence of
  what works
• And practice in dealing with social problems
  cannot be reduced to rational analysis
• Practical wisdom ...rational assessments
  stepping stones to reasonable decisions
• Evaluation needs to guide us in what is
  appropriate

12
Conclusion What kind of rationality?

• the necessity of a deliberate control of
  policies by the method of intelligence, an
  intelligence which is not the faculty of
  intellect honored in text-books and neglected
  elsewhere, but which is the sum-total of
  impulses, habits, emotions, records, and
  discoveries which forecast what is desirable and
  undesirable in future possibilities, and which
  contrive ingeniously in behalf of an imagined
  good. (Dewey, 1993, p.9)