Considering performance measurement in the NHS since devolution: the active role of documents

1
Considering performance measurement in the NHS
since devolution the active role of documents

• Alison Cumming
• Hornet, University of St Andrews
• 24 November 2009

2
Agenda

• Research context performance measurement in
  the post-devolution NHS
• Conceptual framework The Audit Society
• ANT approach to analysing documents
• Research design
• Preliminary findings policy, guidance and
  audit documents
• Next steps

3
Research context performance measurement in
post-devolution NHS

• Devolution as a point of divergence in the
  history of the NHS
• previously, change was only possible at the
  margins (Woods, 2004)
• now we see diverging policy and operational
  models (Greer, 2004), including diverging
  performance measurement systems

4
Conceptual framework

• Is there a systematic trend towards the extreme
  case... a society engaged in constant checking
  and verification, an AUDIT SOCIETY in which a
  particular style of formalized accountability is
  the ruling principle? (Power, 1997)
• Making management auditable
• audit actively constructs the contexts in
  which it operates (Power, 1994)
• install auditable systems to demonstrate
  legitimacy
• evidence of transformation of auditees in
  pursuit of auditable self (Skaerbaek
  Thorbjornsen, 2007 Vikkelso, 2007 Arnaboldi
  Lapsley, 2008)
• national audit bodies shape performance
  measurement systems and gain recognition as
  experts (Power, 2000 Gendron et al., 2007)

5
ANT approach to analysing documents

• Actor-network theory (Callon, Latour, Law)
  assembles the social world by tracing
  interactions between actors, which may be humans
  or objects.
• An object becomes an actor when it makes a
  detectable difference to the action taken by
  another agent.
• Documents represent and make things visible
  They construct and stabilize objects and things,
  identities and processes. (Prior, 2003)
• The ANT researcher must find a way to make
  objects talk (Latour, 2005).

6
Research design

• Audit Scotland
• Overview reports of the NHS for financial years
  1999-2000 to 2007-08, inclusive
• Evolution of titles, for example,
• Overview of the National Health Service in
  Scotland 1999-2000
• Overview of the performance of the NHS in
  Scotland 2004-05
• Overview of Scotlands Health and NHS
  performance in 2006-07
• Overview of the NHS in Scotland 2007-08

• Scottish Government policy and guidance
• Key policy documents
• Our National Health (2000)
• Partnership for Care (2003)
• Delivering for Health (2005)
• Better Health, Better Care (2007)
• Key guidance documents
• Rebuilding Our NHS (2001)
• Performance Assessment Framework 2003/04
• Ministers Key HEAT Targets and Measures
  Guidance 2008/09

7
Preliminary findings policy and guidance
documents

• Traces of networks between Scottish Executive /
  Government, clinicians, managers and especially
  patients
• Recurrence of rhetoric from previous policy
  documents
• Absence of audit mechanism for primary
  performance assessment systems
• But Audit Society evident, for example, in
  arrangements for clinical governance, including
  establishing NHS Quality Improvement Scotland
• Performance Assessment Framework (2001-06)
• Attempt to create holistic view of performance
  shift away from finance and efficiency focus of
  the internal market
• Self-assessment by Boards shared view
• Confusion over purpose of PAF, including
  incentives
• HEAT targets (2006 to present)
• Strengthen central scrutiny of performance more
  assertive role
• Greater focus on national priorities alignment
  with the Scottish Government National Performance
  Framework
• Contribution made by NHS Scotland action oriented

8
Preliminary findings audit reports

• Evolution from consolidated summary of
  findings of financial statement audits and
  national performance audits to integrated
  overview of performance of the NHS in Scotland
• First person narration to third person narration
• From retrospective view to forward focus
• Filling a gap in national performance
  information published by NHS Scotland
• This is the first report of its kind. It draws
  together different sources of published
  information which have never been brought
  together before (An overview of the performance
  of the NHS in Scotland, 2004)
• NHSScotland Chief Executives Annual Report
  2007/08
• Criticism of performance measurement
  arrangements
• There are problems with the targets as they
  stand. (2003/04)
• Call for more systematic target setting process
  and for targets to be made SMART (auditable?)
• Cant say whether additional funding has been
  spent on specific priorities or whether achieving
  outcomes
• Interaction with other actors
• Financial performance recurring vs non-recurring
  distinction becomes embedded in routines of
  Scottish Executive / Government

9
Next steps

• Comparison
• Apply this conceptual framework to analysis of
  NHS England
• Broadening scope of analysis
• Analyse documents at organisational level how
  policy, guidance and audit documents impact upon
  practice
• Strengthening evidence base
• Qualitative interviews to understand
  interaction between documents and human actors
• Analyse documents associated with dissemination
  of policies, guidance and audit reports, e.g.
  press releases, Ministerial statements