The potential of longitudinal research in evaluating national government policy: results of a scopin

1
The potential of longitudinal research in
evaluating national government policy results of
a scoping study
Paul Bradshaw
2
Presentation outline

• Overview of Scotlands national performance
  framework
• Objectives of the scoping study
• Evidence gathering
• Analytical process and findings
• Recommendations from the study

3
The Scottish Governments National Performance
Framework (1)
4
National Performance Framework (2)
Purpose
Strategic Objectives
National outcomes
5
The Concordat with Local Authorities

• Local Authorities have more flexibility in how
  they spend their allocated funds (less
  ring-fencing)
• Each LA will have a Single Outcome Agreement
  based on the national outcomes and national
  indicators but with agreed local outcomes
• Performance across LAs will be measured and
  monitored in a more standardised way but LAs will
  have to provide their own data (or at least their
  own reports)

6
Research objectives

• Provide a contextualisation of the policy
  environment in which the evaluation of research
  resources will add value
• Review different longitudinal research resources
  in terms of their relevance to the Scottish
  Governments performance framework
• Review design features and coverage by matching
  resources with particular policy outcomes in such
  a way that their value be understood
• Demonstrate the ways in which different
  longitudinal resources may be used singly or in
  combination to evaluate the strategic outcomes
  weighing their relative value
• Identify gaps in the longitudinal resources
  available that need to be filled by enhancing
  particular studies or supporting new ones
• Identify cross-cutting issues which affect the
  development of the longitudinal portfolio and its
  effective use

7
Sources of evidence

• Detailed review of the content and supporting
  documentation of 29 longitudinal surveys and
  administrative datasets, ranging from those that
  were exclusively designed for Scottish purposes
  and based in Scotland to other UK wide projects
  with much relevance to Scotland.
• Interviews with Government officials and external
  experts
• Expert group

8
Studies reviewed
9
Analysis Vignettes

• Research vignettes where studies were linked
  together around the variables they contained and
  examples given of how they could be used to
  evaluate the outcomes
• Several components to each vignette
• Identification and exploration of the implicit
  proposition that lay behind each of them
• Identification of the longitudinal data that
  would best support explanatory models linking
  circumstances, experiences and personal
  attributes to the given outcome at the
  appropriate level.
• Most outcomes grouped under broad policy domains
  economy, education, health, protection,
  community and public services, environment
• Two standalone inequality and identity
• Identified research exemplars relevant to
  particular outcomes

10
We live longer, healthier lives
Vignette Example 1

• Defining the concepts
• Medicine, public health, physical health, mental
  health, general health, illness, disease,
  disability, lifestyle
• Require data on
• Health-related characteristics of the general
  population
• Specific health information on key population
  sub-groups
• Advantage of longitudinal data
• Key studies
• Scottish Longitudinal Study, West of Scotland
  Twenty-07, British birth cohort studies
• Opportunities and gaps
• The importance of data linkage
• Longitudinal data on older people

11
We have strong, resilient and supportive
communities where people take responsibilities
for their own actions and how they affect others
Vignette Example 2

• Defining the concepts
• Challenge of identifying a community,
  understanding what consitutes strong,
  resilient and supportive
• Require data on communities and individuals
• Characteristics of the community e.g. housing,
  crime levels, reputation (also acts as outcome
  data)
• Mechanisms which operate within e.g informal
  social control, harmony, belonging, honesty and
  trust
• Advantage of longitudinal data
• Key studies
• Go Well Glasgow, West of Scotland Twenty-07,
  ESYTC
• Opportunities and gaps
• Focus on the West
• Generalisability?

12
Policy domain summaries
13
Analysis Cross-cutting issues

• Identification of cross-cutting issues emerging
  from the vignettes and study reviews. Included
• Design features
• Awareness of longitudinal research
• Data linkage
• National data vs. local data
• Capacity

14
Recommendations

• Raise awareness of the value of longitudinal
  research
• Increased cross-government agreement on
  development of longitudinal resources and costs
  shared alongside sustained investment in
  longitudinal resources
• Further operationalisation of the outcomes
• High priority to data linkage and access to
  administrative data sources
• Develop a comprehensive evaluation strategy that
  embraces performance at all levels
• Build as much complementarity between national
  and local measurement as possible through
  development of multi-level design frameworks
• Develop the capacity for local research and
  evaluation
• Develop the capacity for natural experiments
• Ensure that baselines are in place nationally and
  locally
• Invest in capacity building

15
Findings as a resource for analysts
National Outcome or Outcomes of interest tracked
to the vignettes which set out possible research
scenarios
or
Studies relevant to the Outcome or Outcomes
identified via the Outcome by Study matrix
Further technical details of the relevant
resources can be obtained via the project
summaries in the appendix
Visit the studys website and examine
documentation to identify and explore relevant
variables, or obtain dataset
Specify analytical programme
16
Further information

• Bynner, J. and Bradshaw, P., with Sharp, C. and
  Marryat, L. (2008) Use of longitudinal research
  in the evaluation of the Scottish Governments
  National Outcomes Edinburgh Scottish Government
• Available online at
• http//www.scotland.gov.uk/Publications/2008/11/25
  160140/0

17
(No Transcript)