Title: The potential of longitudinal research in evaluating national government policy: results of a scopin
1The potential of longitudinal research in
evaluating national government policy results of
a scoping study
Paul Bradshaw
2Presentation outline
- Overview of Scotlands national performance
framework - Objectives of the scoping study
- Evidence gathering
- Analytical process and findings
- Recommendations from the study
3The Scottish Governments National Performance
Framework (1)
4National Performance Framework (2)
Purpose
Strategic Objectives
National outcomes
5The Concordat with Local Authorities
- Local Authorities have more flexibility in how
they spend their allocated funds (less
ring-fencing) - Each LA will have a Single Outcome Agreement
based on the national outcomes and national
indicators but with agreed local outcomes - Performance across LAs will be measured and
monitored in a more standardised way but LAs will
have to provide their own data (or at least their
own reports)
6Research objectives
- Provide a contextualisation of the policy
environment in which the evaluation of research
resources will add value - Review different longitudinal research resources
in terms of their relevance to the Scottish
Governments performance framework - Review design features and coverage by matching
resources with particular policy outcomes in such
a way that their value be understood - Demonstrate the ways in which different
longitudinal resources may be used singly or in
combination to evaluate the strategic outcomes
weighing their relative value - Identify gaps in the longitudinal resources
available that need to be filled by enhancing
particular studies or supporting new ones - Identify cross-cutting issues which affect the
development of the longitudinal portfolio and its
effective use
7Sources of evidence
- Detailed review of the content and supporting
documentation of 29 longitudinal surveys and
administrative datasets, ranging from those that
were exclusively designed for Scottish purposes
and based in Scotland to other UK wide projects
with much relevance to Scotland. - Interviews with Government officials and external
experts - Expert group
8Studies reviewed
9Analysis Vignettes
- Research vignettes where studies were linked
together around the variables they contained and
examples given of how they could be used to
evaluate the outcomes - Several components to each vignette
- Identification and exploration of the implicit
proposition that lay behind each of them - Identification of the longitudinal data that
would best support explanatory models linking
circumstances, experiences and personal
attributes to the given outcome at the
appropriate level. - Most outcomes grouped under broad policy domains
economy, education, health, protection,
community and public services, environment - Two standalone inequality and identity
- Identified research exemplars relevant to
particular outcomes
10We live longer, healthier lives
Vignette Example 1
- Defining the concepts
- Medicine, public health, physical health, mental
health, general health, illness, disease,
disability, lifestyle - Require data on
- Health-related characteristics of the general
population - Specific health information on key population
sub-groups - Advantage of longitudinal data
- Key studies
- Scottish Longitudinal Study, West of Scotland
Twenty-07, British birth cohort studies - Opportunities and gaps
- The importance of data linkage
- Longitudinal data on older people
11We have strong, resilient and supportive
communities where people take responsibilities
for their own actions and how they affect others
Vignette Example 2
- Defining the concepts
- Challenge of identifying a community,
understanding what consitutes strong,
resilient and supportive - Require data on communities and individuals
- Characteristics of the community e.g. housing,
crime levels, reputation (also acts as outcome
data) - Mechanisms which operate within e.g informal
social control, harmony, belonging, honesty and
trust - Advantage of longitudinal data
- Key studies
- Go Well Glasgow, West of Scotland Twenty-07,
ESYTC - Opportunities and gaps
- Focus on the West
- Generalisability?
12Policy domain summaries
13Analysis Cross-cutting issues
- Identification of cross-cutting issues emerging
from the vignettes and study reviews. Included - Design features
- Awareness of longitudinal research
- Data linkage
- National data vs. local data
- Capacity
14Recommendations
- Raise awareness of the value of longitudinal
research - Increased cross-government agreement on
development of longitudinal resources and costs
shared alongside sustained investment in
longitudinal resources - Further operationalisation of the outcomes
- High priority to data linkage and access to
administrative data sources - Develop a comprehensive evaluation strategy that
embraces performance at all levels - Build as much complementarity between national
and local measurement as possible through
development of multi-level design frameworks - Develop the capacity for local research and
evaluation - Develop the capacity for natural experiments
- Ensure that baselines are in place nationally and
locally - Invest in capacity building
15Findings as a resource for analysts
National Outcome or Outcomes of interest tracked
to the vignettes which set out possible research
scenarios
or
Studies relevant to the Outcome or Outcomes
identified via the Outcome by Study matrix
Further technical details of the relevant
resources can be obtained via the project
summaries in the appendix
Visit the studys website and examine
documentation to identify and explore relevant
variables, or obtain dataset
Specify analytical programme
16Further information
- Bynner, J. and Bradshaw, P., with Sharp, C. and
Marryat, L. (2008) Use of longitudinal research
in the evaluation of the Scottish Governments
National Outcomes Edinburgh Scottish Government - Available online at
- http//www.scotland.gov.uk/Publications/2008/11/25
160140/0
17(No Transcript)