craigforth

1
Evaluation of Supporting People Outcomes Pilot
Interim FeedbackJanuary 2008
2
Scope of the Pilot

• 7 local authorities - Aberdeenshire, Moray,
  Highland, Perth and Kinross, Argyll and Bute,
  North Lanarkshire, City of Edinburgh
• 30 providers involved
• Wide range of services covered older people,
  physical disabilities, learning difficulties,
  Womens Aid, homelessness, young vulnerable
  people, substance misuse, mental health
• Outcomes framework piloted July November 2007
  by providers, results collated by local
  authorities December 2007 and sent to DTZ
• Evaluation at all key stages of pilot

3
(No Transcript)
4
Evaluation Aim

• to assess whether the Outcomes model being
  piloted could be introduced consistently at local
  authority level to capture useful information,
  which in turn could be used as a means of public
  and ministerial reporting at national level

5
Evaluation Framework Developed

• Objective 1 Measures outcomes for individual
  clients
• Objective 2 Measures the impact of the
  programme across each local authority
• Objective 3 Does not place excessive data
  collection burdens
• Objective 4 Provides useful information for
  national and ministerial reporting

6
Approach Taken

• Initial stakeholder interviews
• Baseline survey of all providers
• Telephone interviews with all providers
• Site visits to 14 providers (included observation
  of assessments and talking to service users)
• Interviews with SP Lead Officers in the 7 pilot
  local authorities

7
Next and Final Stages

• Second (follow up) survey of providers
• Review of data (following analysis by DTZ)
• Second set of key stakeholder interviews
• Draft and final report

8
Evaluation Framework Objective 1Measures
Outcomes for Individual Clients
9
Accuracy and Consistency

• No evidence of an overall consistent approach to
  completing the matrix (process or interpretation)
• Service providers each took their own approach
• Staff discussion and training led to greater
  consistency within services (from site visits)
• May be a link between local authority
  input/support and quality of data (to be
  investigated further when data reviewed)

10
Relevance for individual

• May depend on client group and individual
• Distance travelled more useful to some than
  others (e.g. some young people liked goals)
• Over-assessment can be discouraging (issue about
  frequency of assessment for different client
  groups)
• Tailoring matrix to specific client groups was
  issue for service providers and some service
  users
• Generic approach sometimes seen as diluting
  relevance vs. need to be able to compare across
  services

11
Appropriateness

• Matrix being used across all client groups and
  circumstances
• Most providers completed matrix in conjunction
  with service users
• Producing data on distance travelled (even where
  service users could not actively participate in
  matrix)
• Some service users found counterfactual (current
  situation without support) difficult and/or
  upsetting concept

12
Evaluation Framework Objective 2Measures
Outcomes across each local authority
13
Consistency of Completion

• Process of completion varied (on PC, paper
  format)
• Approach taken varied according to client group
  (e.g. more discussion with younger people
  compared to older people)
• Time taken to complete varied depending on degree
  of service user involvement and provider
  understanding
• Most services keen to ensure that the services
  users agreed with information being recorded
  (high degree of self-assessment by service user)
• Interpretation of matrix varied
• Consistency of interpretation varied across local
  authority area for clients with similar needs

14
Accuracy

• Service providers not always confident they were
  completing accurately
• Variation across providers (will be examined in
  more detail in review of dataset)
• Local authorities would use other evidence
  sources to verify accuracy of assessment, e.g.
  own site visits and service reviews

15
Alignment/integration

• No evidence of information sharing across service
  providers and between services (too early in
  process)
• Some (very few) providers had looked at how the
  model could be integrated with other datasets
• No evidence of use of unique individual
  identifier across any local authority
• Some local authorities had considered the
  potential for integrating the matrix with other
  databases e.g. Care First, SWIS (but resource
  issues)

16
Evaluation Framework Objective 3Does not place
excessive data collection burdens
17
Data issues

• May replace some of data required by local
  authorities from providers
• Providers and local authorities saw information
  from matrix as useful addition
• Many support workers have minimal IT skills
  admin resource implications
• Additional resource burden for some providers,
  but not where saw fit with support plans, client
  review process and internal management processes

18
Issues for Discussion

• Value of overall approach to measuring outcomes?
• Where should ownership lie? (With local
  authorities? Scottish Government? Providers?)
• How will the loss of ring-fencing impact on
  implementation?
• How useful is assessment of current situation
  without support (counterfactual)?
• Other issues?