Topic Note: Evidence from the Evaluation of Aimhigher

1
Topic Note Evidence from the Evaluation of
Aimhigher

• Gareth Griffiths
• HE Analysis

2
What is a topic note?

• Topic papers are internal (usually) DfES
  analytical papers which summarise the evidence on
  a particular policy or policy area
• Topic papers do not conduct new analysis, but
  rather pulls together the work done to date. This
  draws on a series of research and evaluation
  reports developed since Aimhigher Excellence
  Challenge began in 2001
• This topic paper aims to summarise the evaluation
  evidence on Aimhigher that answers key strategic
  questions about its impact
• Has Aimhigher increased aspirations to HE,
  attainment at school and, ultimately,
  participation at university?
• Is Aimhigher cost-effective? This becomes
  particularly important in the context of next
  years comprehensive spending review.
• It does not seek to address other important
  questions relating to delivery and design of
  Aimhigher in the field.

3
The evaluation strategy

• The Aimhigher Evidence Programme comprised five
  strands of activity
• Strand 1 - Tracking survey of young people
• Strand 2 - Data linking and analysis
• Strand 3 - Area studies
• Strand 4 - Survey of educational providers
• Strand 5 - Regional and area monitoring and
  evaluation
• The bulk of this presentation is based upon
  Strand 1, which NfER and the IFS carried out on
  our behalf, though findings from strands 3 and 4
  are also included where they relate to the
  strategic questions outlined above.

4

• Is Aimhigher having an impact on aspirations and
  attainment?

5
Impact on aspirations

• The views of partnerships and young people are
  clear Aimhigher is changing perceptions of HE
  for the better
• they get motivated to want to progress and that
  is what Aimhigher Excellence Challenge does
  really well.
• most partnerships are clear that attitudes are
  changing.
• some of the most important outcomes of AHEC to
  date had been increases in the awareness of
  higher education amongst young people (and their
  teachers), changes in cultural expectations in
  AHEC schools and amongst parents of young people
  in those schools, and greater confidence amongst
  young people that HE might be within their
  grasp.

6
Activities that give young people a direct link
to University and undergraduates seem to be
considered particularly effective

• University Visits give young people the
  opportunity to look outside of their own
  environment. You can forget how difficult it is
  for young people to break away from the norm.
• now, when I think about me going to university,
  I can imagine it, and it seems real because I
  have been there and seen it.
• people who have actually been (to university),
  they know what it is like, so they can tell you.

7
Quantitative evidence of improved aspirations
some evidence of changing minds
8
Summary of impact on aspirations

• The views of partnerships and young people are
  clear Aimhigher is changing perceptions of HE
  for the better
• Aimhigher gives young people the chance to see
  what university is like, and sow the seeds that
  university might be for them
• Interventions strongly associated with increased
  aspirations are
• speaking to HE staff and students about
  university
• attending residential visits at university
• going to an Aimhigher Roadshow

9
Quantitative evidence Differences in differences
analysis

• Aim of Differences in differences analysis is to
  compare changes over time in treatment area with
  changes over time in comparison area, to isolate
  the effect of the policy
• Problems with control group
• Excellence in Cities
• Partnerships for progression
• National roll-out

10
Differences in differences analysis
11
Some limitations to DiD analysis

• As already explained, despite the careful choice
  of comparison areas and strict controls, there
  remains some uncertainty as whether all
  differences between the areas are controlled for
• This evaluation took place just 1 year into
  Aimhigher Excellence Challenge, whereas a full
  intervention would adopt a drip drip approach
  over several years
• There is some overlap between Aimhigher
  Excellence Challenge and Excellence-in-Cities,
  and Partnerships 4 Progression (these effects
  should be relatively minor, and to an extent may
  offset each others)

12
But, while bearing in mind these caveats

• The tentative conclusion is that aspirations to
  HE were 3.9 percentage points higher in Aimhigher
  Excellence Challenge schools than in non-AHEC
  schools.

13
Impact on attainment

• Reports from teachers and young people, together
  with quantitative assessments, show a clear
  association between Aimhigher and stronger GCSE
  performance
• Mentoring (amongst lower performers), visits or
  discussions about HEIs, and Summer Schools are
  also found to be statistically associated with
  greater attainment at GCSE
• But, design of the interventions is crucial
  there must be clear linkages between the activity
  (eg master classes) and the pupils own
  curriculum

14
Qualitative evidence is mixed. For example

• Revision skills classes
• I did a PE test.I used that revision skill and
  I got 70 percent, second highest in the class. So
  I have gone up one grade.
• But they need to be clearly linked with the
  students curriculum and pitched at the right
  level if they are to have the biggest impact
• Mentors
• (my mentor is really helpful because) if I am
  struggling then she can explain it for me.
• But, a lot depends on the relationship between
  the student and their mentor/undergraduate, and
  the overall evidence is mixed.

15
It also needs to be remembered that focusing on
attainment alone will not achieve Aimhighers
objectives

• As one practitioner put it
• study skills have an impact on the way students
  work, revise and take notes and prepare their
  work but students wont say they want to go to
  university (as a result) thats where the trip
  (to a HEI) has most impact.
• So while the qualitative evidence on direct
  attainment-raising activities is mixed, there is
  some evidence that these activities are effective
  under the right circumstances

16
But, quantitative evidence on attainment is
positive

• While bearing in mind caveats of DiD approach
  already outlined above
• pupils in AHEC schools improved by 2.5 GCSE
  points versus those in comparison schools
• average GCSE scores were 0.1 points higher and
  GCSE English scores were 0.2 points higher.
• On average pupils in AH EC schools achieved 0.3
  more grades at GCSE A-C than those in comparison
  schools.
• significant improvement in the proportion of
  young people in AH schools attaining levels 4,5,
  or 6 in the Key Stage 3 tests.
• It is estimated that AHEC led to a 4.6
  percentage point increase in Maths attainment at
  Key Stage 3.

17
Quantitative evidence on attainment
Note Key point here is that impact seems to be
greatest for low attainers at KS3.
18
So, Aimhigher seems to be having a positive
impact on both aspirations and attainment But
what about the ultimate impact?

• The most important question
• Is Aimhigher actually increasing participation
  from under-represented groups???

19
So, Aimhigher seems to be having a positive
impact on both aspirations and attainment But
what about the ultimate impact?

• The most important question
• Is Aimhigher actually increasing participation
  from under-represented groups???
• A We dont know! (yet)

20
Too soon to say cohort structure for evaluation
21
Too soon to say

• We have data for just one cohort of young people
  and only 659 respondents - who had reached
  their first post-18 outcome in the longitudinal
  surveys. Amongst these, a quarter had originally
  intended (at Y11) not to go into HE but who now
  were at university.
• An even greater proportion of those in Yrs 12 and
  13 had changed their minds since Yr 11, but is
  this evidence of increasing effect of AH, or will
  they change mind again when it comes to actually
  applying?
• Institutions believe that AH is having an impact
  on decisions, with a third of HEIs claiming that
  AH has increased applications to their
  institutions, though half said it was still too
  early to say. A third of FE colleges also claimed
  that AH had increased applications to their
  institutions, suggesting that AH is having an
  impact on aspirations and progression at FE as
  well as HE level
• But, despite the positive evaluation evidence,
  and lots of anecdotal evidence of its
  effectiveness, we dont know yet whether
  Aimhigher is succeeding in its ultimate aims.

22
Future plans

• Due to low response rates after 4 years of
  interviews (659), no further surveys are planned
• Will now seek to match survey data to HESA/ILR
  data to see how many AH participants made it to
  University/further study. Ideally match UCAS data
  too, but this may not be possible
• Costs project reporting end-November will allow
  us to complete VfM assessment

23
Finally.does Aimhigher offer value for money?
24
Costs and benefits

• Evidence discussed above indicates tentatively
  that Aimhigher Excellence Challenge may have
  raised aspiration to HE by 3.9 percentage points.
  
• There is also strong evidence on the economic
  benefits to greater HE participation and
  attainment. Analysis of labour market data
  suggests that each extra graduate, on average,
  earns over 100,000 more than just A level
  holders across their lifetime.
• Combining these allows us to estimate, under
  certain key assumptions, a break-even cost for
  Aimhigher. That is, how much Aimhigher would have
  to cost per participant to generate a net benefit
  to society.

25
So what does it cost?

• At the time of writing, there is very little
  intelligence on the total resource costs of
  individual Aimhigher activities
• A study is currently underway to collect the
  resource costs of delivering Aimhigher
  activities, and the results should be available
  in early 2007.
• Early indications are that the cost per
  participant will be well below the amount needed
  for Aimhigher to generate a positive NPV.

26
Extra slide on activities
27
(No Transcript)