Template

1
Tier 2 Vital Signs Benchmarking Analysis
Office of the Chief Analyst Emmi Poteliakhoff,
Jonathan White, Barry McCormick, Alistair Rose
2
The issue

• PCT performance is highly variable on many Vital
  Signs
• e.g. teenage conceptions in Lambeth are over 3
  times as high as those in Kingston upon Thames.
• PCT performance will be driven by (1) their
  effort, (2) prioritisation of resources and (3)
  external factors (e.g. poor educational
  attainment) that they cannot control.

In thinking about PCT performance, we are mainly
interested in PCTs effort and prioritisation, as
the PCT can control these
EFFORT
OUTCOME
RESOURCES
EXTERNAL FACTORS
3
How we are tackling it
1
Gather data at PCT level on different external
factors like education and ethnicity
4 Step Process
2
Develop a model which characterises the
relationship between Vital Signs outcomes and the
external factors
3
Use the model to produce predicted rates that are
based on each PCTs external factors
4
Compare predicted with actual outcomes
Is the PCT doing worse than predicted? Better
than predicted? These questions can help us
isolate PCT effort and prioritisation from the
external factors
4
Potential benefits of our approach

• DH
• Better understanding of which regions are doing
  well or badly given their circumstances
• SHAs
• Better informed and able to engage with PCTs on
  priorities, annual plans and performance
• PCTs
• Enables more nuanced self assessment
• Useful information when allocating resources
• May identify areas where closer working with LA
  will bring benefits

• Imagine a PCT which is trying hard in the face
  of local difficulties.
• Their raw outcome might only be average.
• But our method will highlight their effort
  their good practice could then be applied
  elsewhere.

• A PCT may be doing well given local
  circumstances on a vital sign such as teenage
  conception.
• This shows that its health based actions are
  helping but local conditions mean rates are still
  fairly high.
• To improve outcomes further it could focus on
  working in partnership with the LA.

5
Limitations and challenges to this approach

• There is no perfect model
• There is a degree of subjectivity over which
  external factors should be included, and the form
  of the equation used to create the predicted
  values. This feeds through into the final actual
  versus predicted results.
• However, the high degree of statistical
  significance and explanatory power helps to
  validate our model.
• We have also compared the results for different
  sets of external factors and mathematical forms,
  and the outcomes do not vary a great deal.

So this analysis should be a useful supplement to
existing methods but should not replace them

• So our challenge is to
• develop something useful given that no perfect
  model exists
• trade off transparency and comprehensiveness

6
Which Vital Signs do we cover?

• Those that we have identified as having a
  substantial external component. The latest data
  is used.
• Rate of under-18 conceptions per 1000 girls aged
  15-17
• takeup of DTPP vaccines (incl. booster) by age
  5
• takeup of MMR vaccine (both doses) by age 5
• prevalence of childhood obesity at reception
  age
• prevalence of childhood obesity at year 6 age
• Directly age-standardised mortality per 100,000
  population
• Mortality rates covered (separately) suicide,
  cancer, CVD, all-age-all-cause (both sexes, male
  only, female only)

7
How do we work out predicted values?

• We start by adding data on the following external
  factors into our dataset of PCT Vital Signs
  outcomes.
• These external factors are thought to affect
  Vital Sign performance but are totally outside of
  PCT control.

(NS-SEC National Statistics Socio-Economic
Classification its inclusion helps adjust for
social class)
8
How do we work out predicted values?

• To use the dataset to work out predicted values,
  we apply a technique called multiple regression
  analysis.
• How does this work?
• Imagine we wish to work out predicted values for
  childhood obesity, using the external factors of
  median income and population density.
• The analysis starts with the following form

PCTs predicted obesity value a plus (b times
PCTsMedianIncome) plus (c times
PCTsPopulationDensity)

• It finds values of a, b and c such that the
  predicted values most closely match the actual
  values.
• The equations fit well often 60-70 of
  variation explained.

9
How do we present the results?

• Problem
• With 152 PCTs, 11 Vital Signs and predicted
  actual values, there is a lot of data. Hard to
  present clearly.
• Our solution
• An interactive Microsoft Excel tool. Easy to use
  works on any PC with Microsoft Office (no
  macros needed).
• The user can choose their SHA and the tool can
  then display custom graphs and tables for that
  SHAs constituent PCTs.

Ultimately, the tool makes it easier to see who
is doing better or worse than predicted.