EvidenceBased Policy

1
Evidence-Based Policy
Australian Research Alliance for Children Youth
Brian Head Professor of Governance Griffith
University Public Policy Network
conference Curtin University, Perth, 2-3
February 2006
2
Overview

• -- 1. Background history of linkages between
  Public Policy and Social Science research
• -- 2. How can social research be made more
  relevant?
• (and for whom?)
• -- 3. Can better use of evidence help bridge
  this gap?
• (and is it evidence-based or
  evidence-informed?)
• -- 4. Types of knowledge political/scientific/pr
  actical etc.
• -- 5. Should evidence-based approach focus on
  filling gaps in knowledge?
• OR framing issues and questions (the role of
  values, public communication, professional
  judgement)
• -- 6. What do networks and collaboration
  contribute?
• -- 7. Dealing with complexity and conflict
• -- 8. Conclusion limits on evidence-based
  analysis

3
Public policy and social science where does
evidence-based policy come from?

• General history of mutual distrust between
  government sector social science researchers
• In earlier centuries there was greater scope for
  governmental reliance on appeals to (a)
  authority (b) precedent/tradition and (c)
  fundamental values.
• But a growing governmental usage of social
  information technical knowledge since 19th
  century.
• Tied to rational secular notions of progress
  and development.
• This in turn fuelled expansion of social sciences
  and their increased prestige.
• Question are the Improvement agenda and
  technocratic social control two sides of same
  coin? a la Foucault.

4
Towards relevance

• Modern governments now seek to anchor much of
  their policy/program stance in relevant
  knowledge (usable knowledge)
• Evidence-based policy analysis and research is
  now seen, by govts and by social researchers, as
  especially relevant and useful.
• This is linked to modern emphasis on
• -- problem-solving
• -- risk analysis response
• -- focus on what works? (perhaps the most
  frequent question in modern applied sciences)
• Are some disciplinary frameworks more
  valued/useful? e.g. public health cost/benefit
  economics financial auditing analysis (vis-à-vis
  sociology and cultural studies and history?)

5
The governmental context

• Govts do not simply receive and use research
  they have direct indirect means for influencing
  social sciences.
• Direct methods of influence include
• -- investing in govt-funded research units on
  specific themes/problems
• -- managing policy-research functions inside govt
  agencies
• -- commissioning specific research under
  contract.
• Indirect methods include
• -- setting national priority areas (e.g. for
  competitive research funds)
• -- providing higher rewards for commercial
  technical forms of scientific excellence
• -- diversifying sources of advice, including
  think-tanks consultants.

6
Relevance for government

• The rise and promotion of evidence-based
  orientation in modern governments is not
  accidental
• Evidence-based approaches are supported because
  of their link to governments managerial or
  technocratic interest in efficiency and
  effectiveness
• -- what options will deliver the goods?
• -- how to get greater value for money?
• -- how to promote innovation and competition to
  drive productivity?
• -- how to better achieve outcomes for clients
  stakeholders? (rather than just managing
  programs)
• -- in short, what works?

7
Producing, banking managing information

• Evidence-based approaches claim to fill important
  gaps in data-information-knowledge.
• Knowledge management strategies have been
  introduced to address responsibilities for
  collection, analysis and dissemination.
• Management information systems (decision-support)
  have become widespread to monitor key
  indicators/trends
• Large data sets are systematically collected
• Who is mining these data for useful information
  to construct usable knowledge? govt research
  units? consultants? universities?
• How do we identify gaps that need filling (i.e.
  know that we dont know important things)? And
  are we really sure about what we already claim to
  know?

8
Last frontier? tackling wicked problems

• Longstanding desire to gain greater understanding
  and control over fuzzy and messy realities, not
  just the simpler routine areas of incremental
  adjustment.
• Linked to frustration over persistence of endemic
  problems, on which massive public funds have been
  spent.
• Hence massive investment in data gathering and
  benchmarking to measure future performance.
• Examples of complex challenges
• -- what works in regard to poverty?
• -- crime control recidivism?
• -- preventable diseases?
• -- drug alcohol abuse?
• -- indigenous services?

9
Failure in knowledge? or something else?

• Are endemic social problems a sign of gaps in
  data and basic knowledge?
• And if we had more social data, would this
  provide the superhighway to good policy
  solutions?
• Much as we researchers might want to encourage
  investment in information gathering (research
  reports jobs), there are other possibilities
  beside gaps in data
• poor understanding of the connections between
  various trends/processes?
• poor understanding of program implementation?
• poor understanding of stakeholders motives and
  perceptions?
• poor understanding of how to align and manage
  relations with divergent stakeholders?

10
Types of knowledge in policy analysis

• Major complication not all knowledge takes the
  form of scientific evidence validated by the
  standards of scientific methodology.
• Policy rests on several different evidentiary
  bases.
• Leaving aside the broad milieu of popular culture
  and public opinion, three kinds of knowledge are
  especially important for policy
• -- political know-how
• -- scientific and technical information
• -- practical and professional field experience
• 1. Political knowledge of strategies,
  agenda-setting, priorities, tactics, ideological
  spin, values, communications, accountability,
  support, trade-offs, etc.
• This knowledge inheres primarily in
  politicians, parties, organised groups, public
  affairs media, etc but is also widely diffused
  in popular forms among the public.

11
Types of knowledge

• 2. Scientific (research-based) knowledge of past
  and current conditions and trends, and the causal
  inter-relationships that explain conditions and
  trends.
• Systematic reviews of what works in a
  particular policy field have been championed, esp
  in healthcare and social programs.
• There is a preference for behavioural and
  quantitative data, though qualitative materials
  are increasingly included to help explain the
  conditions for change.
• 3. Practical knowledge of practitioners, field
  managers, program managers and implementers (at
  diverse levels, and in diverse communities of
  practice).
• These people wrestle with everyday problems of
  effectiveness and implementation, whether in the
  office, the clinic, the prison, the school, the
  farm, the factory etc.

12
Types of knowledge evidence

• Each of these types has its distinctive protocols
  of knowledge, expertise, strategy, and what
  counts as evidence
• albeit there is ongoing debate on these matters.
• How do these 3 forms of knowledge fit together?
  is there a hierarchy (e.g., does politics trump
  science)? is there direct competition? is there
  mutual awareness and recognition? can there be a
  coordination mechanism?
• How does the evidence-based research component
  add most value? and where does it best fit in the
  policy cycle? (e.g. problem identification,
  options analysis, evaluation and feedback?)
• How can the practical knowledges be properly
  recognised by policy decision-makers? Arent
  these very important for understanding effective
  implementation?

13
Framing issues and agendas

• Issues and agendas do not arise from bare facts.
• Problem-framing and agenda-setting.
• What is an issue/problem worthy of investigation?
  In practice this is always linked to
• -- a perception of crisis or urgency
• -- the role of political mandates and priorities
• -- the role of professional/managerial judgement
  (public servants, management experts, etc)
• -- organisational and issue histories
• -- the shaping of social values public
  opinion.
• Problem of splits between expert and citizen
  viewpoints on acceptable solutions
• e.g. experts may be 90 in favour of a
  particular measure, but an intense and vocal
  lobby may create political uncertainty and thus
  stall adoption

14
Problems, issues and agendas

• Problem-definition is crucial but often
  contentious (e.g. greenhouse responses,
  industrial relations reform).
• Problem-definition has implications for
  privileging some evidence as relevant and ruling
  out other evidence as irrelevant/ideological etc.
• This means that some policy positions are
  evidence-proof in the sense that their evidence
  base is narrow and buttressed by political
  commitment
• Key concepts have social impacts (e.g. thresholds
  for program eligibility poverty, disability,
  refugee, citizen, etc)
• Problem-definition and focus are very important
  in the commissioning of research and
  consultancies
• -- relatively few projects are open-ended most
  are commissioned to uphold a certain viewpoint.

15
The challenge of consultative approaches

• The 1990s saw the rise of policy processes that
  were less technocratic and more open to
  consultative and network approaches.
• Standard managerialist (NPM) approaches have
  often been supplemented by layers of
• -- community engagement
• -- stakeholder consultation
• -- partnering across sectors (govt - NGOs)
• This has been accompanied by the rhetoric of
• -- collaboration
• -- joined-up services
• -- networks linking stakeholders
• -- devolution and empowerment.

16
The why what how of consultative policy
networks

• Other papers on the rhetoric reality of these
  trends
• not repeated here.
• Decision-makers and stakeholders have widely
  variable understandings of these trends and
  opportunities.
• These divergent frames range across
• -- participatory democrats who advocate broader
  and deeper channels for community involvement
• -- governmental opportunists attracted by
  prospect of government becoming more popular by
  listening to citizens stakeholders
• -- NGOs concerned that consultation/collaboration
  is time-consuming and exhausting, sometimes for
  little gain.

17
Evidence evaluation in networks

• Much of the policy collaboration and network
  literature is not very useful for considering
  evidence and evaluation frameworks e.g.
• -- 1. values-based and aspirational or
• -- 2. practical advice on how to lobby, link,
  connect, or undertake community development
  through local networks/partnerships.
• What would be required for an evidence-based
  framework to evaluate complex networked policy
  programs?
• Traditional evaluation frameworks usually focus
  on
• -- program logic (mapping causal links)
• -- performance information on deliverables
• -- auditing financial probity
• -- cost-effectiveness of options.

18
Implications for evidence

• Do we need more? What other kinds of evidence
  about what works are important in programs
  involving networked governance for tackling
  complex issues?
• Three areas seem especially important to clarify
• 1. Are purposes and directions clearly understood
  and supported by stakeholders? Is divergence well
  handled and adjustments agreed? Perspectives are
  crucial evidence.
• 2. How effective are the multi-stakeholder
  processes of governance (accountabilities, roles,
  decision-making, resourcing, consulting etc)? Is
  there confidence and trust for long term
  sustainability of arrangements?
• 3. Are the short/medium/long term outcomes (e.g.
  stages in achieving desired better services)
  specified in a developmental context? Are program
  outcomes sustainable in long-term?

19
Evidence about emergent relations

• Network approaches aspire to encourage
  innovation new thinking and new solutions
• Dialogue and debate may help deal with blockages
  arising from differing perspectives.
• Enhancing capacities, skills, social capital may
  be very important enablers
• Encouraging long-term thinking about sustainable
  improvements is valuable
• We need to learn more about the features in
  network relationships that are conducive to
  effective performance (success)
• Substantial devolution is unlikely.

20
Policy risks of network devolution

• For government, retaining substantive control
  over policy and programs is always important.
• Two main reasons for this from a govt
  perspective
• 1. political needs to optimise political
  benefits of any success, to focus on their own
  political priorities promises, and to minimise
  collateral damage by actions of 3rd parties
• 2. audit requirements including probity,
  transparency, and accountability for results
  hence the imposition of standards and guidelines
  on NGOs.
• Conclusion there is a major perceived risk for
  govt in devolving control to NGOs and stakeholder
  networks.

21
Conclusions evidence-based analysis

• Analysis of data (evidence) in a suitable
  framework helps to answer the question what
  works or the question what happens if we change
  these settings?
• Policy judgements and decisions are not deduced
  from empirical analysis, but from politics,
  judgement and debate.
• Interplay of facts, norms desired actions.
• Evidence and relevant knowledge are diverse and
  contestable. Arguments are mounted to prove a
  position.
• Some policy positions are data-proof. Only some
  kinds of evidence are noticed in some policy
  contexts.

22
contd

• Real-life policy problems typically involve
  important differences in perspective among
  decision-makers, stakeholders, etc.
• It is difficult/impossible to determine who is
  correct (essentially a political communications
  issue). Not just about facts.
• But we can analyse the likely consequences of the
  preferred policy and other options.
• Qualitative information on perspectives is
  crucial evidence for policy analysis (how will
  they respond? how much cooperation can be
  expected? have their views been taken into
  account? can better communication assist in
  conflict-resolution?)

23
contd

• Arguments about evidence are part of the
  skill-set needed by analysts in drafting a case
  for preferring one approach over another.
• Evidence-based policy analysis and advocacy
  skills can be taught
• Cost/benefit technical skills need to be embedded
  in a broader understanding of good policymaking
  -- political context and competing social values.
• Participatory forums may be a necessary adjunct
  to NPM technocratic approaches.
• Evaluation of policy advice and program success
  is very complex.

24
contd

• We have all learned a lot about community
  engagement, partnering and network approaches in
  recent years.
• Considerable experimentation and support.
• There is also much covert resistance to genuine
  engagement and empowerment, owing to a
  combination of ideology, territorial silos,
  and fear of losing control/influence.
• Networks bring to the table a diversity of
  evidence relevant information,
  interpretations, priorities.
• Network governance for community engagement
  requires flexibility, management of diversity,
  new skills in diplomacy, and capacity-building.
• Capacity building is necessary for all the
  sectors governmental, business and community
  if network approaches are to be more successful.