VALUE ADDED MODELS AND METHODS Differences between KS2KS4 CVA and FFT SX DAVE THOMSON AND TREVOR KNI

1
VALUE ADDED MODELS AND METHODSDifferences
between KS2-KS4 CVA and FFT SX DAVE THOMSON
AND TREVOR KNIGHT
2
Objectives of this session

• Statistical philosophy and model building
• The importance of analysis of variance
• The relative importance of explanatory factors
• How DfES and FFT models differs
• Statistical inference
• Use and abuse of statistics from models
• Removing the statistical night-terrors

3
CVA versus FFT SX

• Differences at school level
• 21 of schools have a different significance
  state for KS2-KS4 CVA compared to equivalent SX
  model
• But just 4 (122 schools) have a significantly
  different result
• Differences in variables used
• Differences in statistical methodology
• Multi-level modelling (CVA) and modified OLS
  (SX)
• Differences in purpose
• CVA a measure of whole school performance
• FFT designed for school improvement slicing and
  dicing by pupil groups
• Bias in CVA to certain pupil groups
• Developing criteria for establishing how good a
  model is
• Statistical robustness
• Fairness to all pupil groups and institution
  types
• Interpretability

4
CVA versus FFT SX

• Differences at school level (KS2 to KS4)

5
CVA versus FFT SX

• Differences at school level (KS2 to KS4)

6
The importance of variance

• Pupils KS4 capped points scores vary
• Range from 0 to 464
• Mean in 2004 was 285.8
• Variance
• Measure of how much pupils KS4 scores differ
  from the mean
• Variance in 2004 was 11291
• Value added is about explaining variance in terms
  of prior attainment, pupil contexts and school
  contexts
• If 100 of variance could be explained, every
  pupil and school in the country would have a
  value added score of 0
• In reality, we can account for about 60 based on
  KS2, pupil contexts and school contexts
• The remaining variance is attributed to the
  differential effectiveness of schools

7
Accounting for variance in pupils KS4 scores
Target 11291
Base0
8
Building the CVA model
9
Building the CVA model
10
Accounting for variance in pupils KS4 scores
Target 11291
PA only 5351
Base0
11
Building the CVA model

• A simple model
• The line has an equation
• Predicted KS4 capped points Intercept
  Co-efficient KS2 APSIntercept
  60Co-efficient 3.5Pupil Value Added Actual
  - Predicted
• 47 of variation in pupils capped KS4 points
  scores can be explained by this equation, i.e.
  47 of 11291 5351

• Improving the model
• Adding additional factors, each with their own
  co-efficient
• This changes the co-efficients of factors already
  in the model and the intercept
• Altering the shape of the line

12
Adding a quadratic line
13
Building the CVA model

• A more elaborate model
• Making the line curved and adding gender as a
  factor
• KS4 capped points Intercept Co-efficient1
  KS2 APS Co-efficient2 KS2 APS squared
  Co-efficient3 femaleIntercept
  -59Co-efficient1 6.6Co-efficient2
  0.2Co-efficient3 23.8
• 48 of variation in pupils capped KS4 points
  scores can be explained by this equation

14
CVA Factors- www.standards.dfes.gov.uk/performance
15
Factors
16
Building the CVA model

• Evaluating the choice of factors
• Do they make sense educationally?
• How much variation do they explain?
• Are they significant?
• What are their effect sizes?

• Effect Sizes
• Measure of the relative importance of a
  co-efficient in a statistical model
• Effect sizes greater than 0.6 are considered
  large
• Effect sizes smaller than 0.2 are considered
  small

17
CVA- the important effects
18
CVA- the less important effects
19
Multi-level modelling Calculation of school lines
20
Variance between schools and between pupils

• Multi-level modelling
• To calculate amount of variance in KS4 points
  between schools (differences between school
  lines)
• To calculate amount of variance in KS4 points
  within schools (scatter of pupils around school
  lines)
• Responsible for the majority of difference
  between SX and CVA
• Shrinkage Factor
• Is a function of variance between schools,
  variance within schools and number of pupils in
  the cohort at a school
• Varies from school to school depending on size of
  cohort
• Is applied to the mean KS4 value added score for
  all pupils at a school to produce the schools
  CVA score and confidence intervals

21
Variance in the CVA model

• Calculating the shrinkage factors, confidence
  intervals and significance

22
Variance in the CVA model

• Calculating the shrinkage factors, confidence
  intervals and significance

23
Calculating the shrinkage factor

• Shrinkage factor
• (342/ (342 (4491/ number of pupils in cohort))).

• School CVA score
• SF mean pupil value added score
• Mean pupil CVA score at the school -9.17
• 195 pupils, SF 0.94
• School CVA score 1000 (0.94-9.17) 991.4
• If 50 pupils, school CVA score 1000
  (0.79-9.17) 992.8

24
Calculating school significance

• Confidence interval (95)
• 1.96 square root of (between school variance
  within school variance) divided by (number of
  pupils between school variance within school
  variance)
• 1.96 sqrt ((341.8737 4490.57)/ (number of
  pupils341.8737 4490.57))
• For 195 pupils 9.1
• CVA score was -8.62
• Lower confidence limit -17.72
• Upper confidence limit 0.49

• Significance
• If lower confidence limit gt0 SIG
• If upper confidence limit lt0 SIG-
• So the example school not significant (just!)-
  but would have been sig- without the shrinkage
  factor

25
Accounting for variance in pupils KS4 scores
Target 11291
CVA 6458
PA only 5351
Base0
26
FFT SX model

• SX models
• Pupils divided into 96 bands and the mean capped
  KS4 points score calculated for each band
• This is then used as the main explanatory factor
• Some small adjustments are made to the line
• School averages
• Separate lines for schools not calculated, i.e.
  no shrinkage
• School score is the simple average (mean) of
  pupils value added scores

27
PA lines in SX and CVA
28
FFT SX the important effects
29
FFT SX the less important effects (1)
30
FFT SX the less important effects (2)
31
CVA and SX comparison of factors

• Common to both models
• Prior attainment, SEN School Action Plus, Joined
  late have similar effect sizes and are gt0.6 in
  both models
• EAL, School Action, Bangladeshi, Chinese, Black
  African, Gypsy/ Roma, Irish heritage Traveller
  have similar effect sizes and are gt0.2
• Differences between models
• Pupil FSM and school GDF rank relatively
  important in FFT
• Interactions of pupil prior attainment with
  school FSM and school mean intake score also
  important
• In care, Indian, Pakistani, Asian Other, Any
  other, Pupil IDACI relatively important in CVA

32
CVA and SX comparison

• Example
• Girl, August born, White British, not FSM, not
  EAL, not SEN, not mobile
• IDACI score of postcode 22
• School FSM rank- 74 GDF (ACORN) rank 67

33
Impact of variables on KS4 points score estimates
34
Impact of variables on KS4 points score estimates
35
Impact of variables on KS4 points score estimates
FFT PA only estimate 364.9
36
Impact of variables on KS4 points score estimates
FFT PA only estimate 364.9
As produced by PAT
37
Issues with CVA

• What are the issues?
• Both explain roughly similar amounts of
  variation in KS4 points scores- 60 (SX) 57
  (CVA)
• CVA used for purposes for which it was not
  designed- i.e. pupil groups
• Schools with better CVA residuals tend to have
• Higher than average proportions of ethnic
  minority pupils
• Higher than average prior-attainment
• Beware large differences in percentile ranks
• Particularly in the 20th to 80th percentile range

38
Issues with CVA
39
PANDA- pupil groups
40
PANDA- pupil groups
41
PANDA- pupil groups
42
PANDA- pupil groups
43
PANDA- pupil groups
44
PANDA- pupil groups
45
Comparison between School CVA and FFT SX
46
Pupil residuals by ethnicity
47
Pupil residuals by SEN stage
48
Improving CVA and SX

• Measurement error
• SEN
• IDACI at pupil level
• Test marks
• PLASC data (e.g. date of joining)
• How can models be improved?
• Use of interaction terms (e.g. ethnicity and FSM)
• Use of random slopes for school lines
• Use of ordinal models for individual subjects
• Including CVA in FFT reports
• Include CVA as well as or instead of SX?
• Report showing schools with significantly
  different scores?
• Report showing pupils with significantly
  different scores?
• SX Ready Reckoner?

49
Improving CVA and SX

• Development of criteria to establish model
  fitness
• Clear statistical principles for potential
  improved models
• Discussed implications of educational
  significant residuals
• Logs of FFT customers views for possible model
  enhancements
• Basis for putting out possible improved models to
  FFT customers
• Management of model change, rules and
  timescales/timetables

50
General conclusions on CVA and SX comparisons

• CVA and SX VA differences at school-level are
  important for national accountability, but few
  schools are really judged different overall
• CVA is a simple multi-level model more
  complexity would help predictions (random slopes
  in prior attainment), and
• CVA lacks sophistication - no interactions
  (which might not be so important if slopes
  varied), and though
• SX has more sophistication but some have little
  impact and some are more difficult to interpret
• CVA has issoos when sliced by pupil type for
  PANDAs and PAT, which FFT seems to avoid .. but
• Neither model MUST be used uncritically
• FFT SX model has some desirable features for
  progress reflection which expanded information
  from CVA could provide
• BOTH systems would gain from a coherent plan for
  model testing and discussion with customers
  that is, with schools and LAs
• Variety is the spice of life provided your life
  doesnt depend on the variety