Evaluating Aimhigher: the National Picture

1
Evaluating Aimhigherthe National Picture

• Viv Wylie, Aimhigher West Midlands and
• Action on Access

2
National Evidence Programme

• Coherent strategy established in 2005
• Aim to understand the extent to which the aim
  and objectives of the Aimhigher programme are
  being met and to identify what activities work,
  for whom, to what extent and under what
  circumstances
• Objectives re effectiveness cost effectiveness
  additionality closing the social gap equal
  opportunities
• Principles of Aimhigher evaluation (2003)
  robust, support practice, useful to all
  stakeholders build on existing work not top
  down operate at all levels support investment
  case take account of equality and diversity
  planned and coherent whole

3
Programme Strands

• Strand 1 Tracking Survey of Young People (NFER)
  completed
• Strand 2 Data Linking and Analysis
  (HEFCE,DfES,LSC) completes 2007?
• Strand 3 Area Studies (EKOS) completes
  December 2006
• Strand 4 Survey of Educational providers (SHU)
  completed
• Strand 5 Area/Regional Monitoring and Evaluation

4
Research Findings

• Strand 1 Tracking Survey of Young People aged
  16-19 from 2001/2
• Higher than expected levels of attainment at key
  stage 3 and in GCSE associated with designation
  as member of wp/GT cohorts. Increased likelihood
  of 5 A-C GCSEs
• Significant association between HEI visits in
  Year 11 and positive changes of attitude to HE
• Parental impact and experiences/circumstances of
  learner more significant in attitudes to learning
  and attainment than school type.

5
Strand 1 continued

• Decision making process key moments Yr 11 and
  post 16 period
• Most effective activities
• Opportunities to learn about HE, talking to
  students, visits to HEIs Aimhigher roadshow,
  study skills classes. GT related activities
• Most helpful information
• HE courses
• What it is like to be an undergraduate
• Finances

6
Strand 2

• Data Linking and Analysis
• Linking HESA individualised student records to
  LSC individualised learner record (ILR)
• Linking HESA record and ILR to Pupil Level Annual
  School Census PLASC
• No outcomes to date

7
Strand 3

• Area Studies anonymous study of 7 areas looking
  at effectiveness of activities and organisational
  structures
• Coordinated programme of activities more
  effective than one-off events
• Successful activities tended to be larger scale,
  more visible
• Success depended on targeting and engaging
  right constituencies, good communications with
  appropriate contacts at right level, gearing
  activity to age group, follow-up work within
  curriculum.
• Examples one to one - eg.mentoring, sustained
  intensity eg. residentials, master classes
  delivered at appropriate level

8
Strand 3 continued

• Issues - the linking of raised aspiration with
  raised attainment
• - less emphasis on vocational
  routes and WBL
• - more sequencing of activities
  needed
• - more focus on target groups
• Recent work compares centralised and devolved
  structures. Benefits of decentralised working
  with small budgets and local accountability
  though central team can provide broader range of
  activities and brokering.

9
Strand 4

• Survey of Educational Providers involving HEIs
  and FECs in EC areas,and WBL providers
• Overall Aimhigher held in high regard and fits
  well with institutional wp strategies
• Aimhigher had significant impact on wp activities
  in terms of range and partnership
• FE and HE thought Aimhigher impacted positively
  on progression from level 3 to HE
• Most effective activities considered by HEIs to
  be residentials and HE visits/open days,
  mentoring, IAG
• 
  continued

10
Strand 4 Continued

• Work based learning providers thought clear
  progression routes and improved IAG were key .
• Least successful activities
• Road shows (HE and FE)
• Master/revision classes (HE and FE)
• Mapping/apprenticeship routes and skills (HE)
• Links with employers, SSCs and professional
  bodies (HE)

11
Strand 5

• Area/ regional monitoring and evaluation this
  provides local evidence complementary to national
  programme
• - new monitoring template
• - local evaluation strategies peer
  reviews, questionnaires, tracking surveys
  etc
• - data collection and analysis at regional
  level
• Issue how is this evidence to be used at
  national level?

12
Additional Research Findings

• NFER research projects
• Aimhigher/Excellence Challenge A policy
  evaluation using labour force survey
• More individuals from disadvantaged
  backgrounds have benefited from policy and are
  more likely to enter HE
• An economic evaluation of early impact of
  Aimhigher Excellence Challenge on pre16 outcomes
• Aimhigher led to average 4.6 point increase
  in Y11 students expecting to leave education aged
  20 in EiC areas (all 3 phases) Indications of
  GCSE improvement.

13
Cost Study

• Aims to provide evidence of costs (hidden and
  partnership) of activities in different areas,
  and provide toolkit and examples of good
  practice.
• Phase 1 feasibility study in 4 areas, phase 2
  further 10 case studies. Reports November 2006.
• Outcomes to date
• understanding of variables in reporting models
• construction of costing model overheads, core
  delivery costs, additional delivery costs, hidden
  costs/benefits in kind, unit costs (per student)
• some comparative unit costs, VfM model
  and good practice

14
Summary

• Most effective activities
• Residentials (though possibly not for all groups
  without preparation), campus visits/open days,
  mentoring, subject related tasters, IAG
  particularly re HE courses, student life and
  finance in combination to provide coherent
  programmes
• Conversations with students and HE staff valued
• Parental input and support important
• Least effective activities
• one off activities such as road shows

15
Summary Continued

• There is evidence that Aimhigher is well regarded
  by institutions who believe that it has positive
  effects on progression
• NFER research shows association of improved
  attainment and progression rates with Aimhigher
  activities
• DfES topic paper argues from existing evidence
  that Aimhigher is contributing to improved
  progression and attainment.

16
Issues

• Completed research somewhat historical
  encompasses EC and early days of Aimhigher,
  conservative account of Aimhigher activity.
• Still no evidence of discrete impact of Aimhigher
  on improved performance and participation no
  control group
• Not much focus on Aimhighers work on vocational
  routes
• What combinations are most effective?
• How is regional and area work to complement
  national picture
• Need for consistent data collection and analysis
• Further national research?

17
Whither?

• HEFCE to provide
• Ward level patterns of young participation vs.
  range of other area measures from 1991 census
  (end 2006)
• Update to national analysis of participation to
  2003 cohorts of 18 year olds (August 07)
• Revision of local participation information
  2001/2 cohorts
• (August 07)
• Analysis of participation by secondary schools
  and KS4 attainment (December 07-August 08)

18
Whither 2

• Further Survey of educational providers focussing
  on
• Effective identification of Aimhigher learners
• Progressive programmes of activity
• Models of best practice
• Comments on Aimhigher by learners
• Degree to which schools and colleges are engaging
  with Aimhigher

19
Whither 3

• Regional/area evaluation -
• Working group under auspices of evidence groups
  to audit varieties of evaluation strategies in
  areas and regions with view to -
• considering relationship between activity and
  effect at local levels to complement national
  research
• identifying good practice in evaluating impact
• disseminating findings