Update on the REF

1
Update on the REF

• June 2009

Lesley Dinsdale, RED
2
Update on the REF

• How is the framework being developed
• Latest on the likely shape of the REF
• Hefces timetable for implementation
• Key issues still to be resolved
• REF and QR funding formula
• What does it mean for UOB short and longer term

3
Basic Principles Hefce Statement, May 2008
A unified framework to cover all
disciplines Will combine bibliometrics and
other quantitative indicators with light-touch
peer review within a variable geometry of
assessment. Bibliometric indicators will be a
key element in quality assessment wherever this
is appropriate, with light-touch peer review of
research outputs operating where it is not.
4
Basic Principles - continued

• In each discipline, quality to be assessed by
• Bibliometric indicators or expert review of
  outputs, or some combination of these
• Other quantitative indicators (e.g. income,
  students)
• Supplementary qualitative information, including
  assessment of user value and impact
• Expert panels will be established to set criteria
  for their disciplines and to oversee the quality
  assessment

5
Hefce development of the REF

• Use of bibliometrics pilot of 22 HEIs July 08
  April 09, interim report due May 09
• Consulting on all other aspects of the framework
• Expert Advisory Group (c.100 RAE2008 panel
  members and others) and Technical Sounding Board
• Informal meetings, e.g. PVCs Research, Research
  Administrators, etc.
• Other projects the RIN examining publication
  behaviour, Hefce/JISC Group on Systems, PA
  Consulting on burdens of data collection, etc

6
Probable Structure

• Hefces thoughts as of June 2009 - Indications
  are that ideas are now firming up
• Informed by
• Initial outcomes of the bibliometrics pilot
  (April 2009)
• Two rounds of meetings with the Hefce Expert
  Advisory Group
• The shape looks significantly different to that
  originally proposed in the first 2007/8
  consultation!

7
Probable Structure
Outputs
Environment
Impact
Element
Qualitative narrative, case study approach to
exemplars Quantitative indicators?
Qualitative RA5 type narrative Quantitative
indicators income, students
Expert review, informed by bibliometrics where
appropriate
Assessed by
Reported as
Profile against criteria originality,
significance, rigour (as in RAE2008)
Single point or profile?
Single point or profile?
Note Collection and assessment of individual
esteem indicators has been dropped
8
Outputs issues and probable outcomes 1

• Use of bibliometrics vs expert review
• Hefce statement Bibliometrics may not be mature
  enough for us to use formulaically at this stage
  they should inform peer review
• All panels likely to sample read at least some
  outputs
• Expert review will be dominant in many
  disciplines
• All or selected staff submitted?
• No viable alternative to selection of staff by
  institutions
• Strong steer from EAG that the proportion of
  eligible staff submitted should be included, but
  Hefce not keen on this
• Outputs credited to author or institution?
• Author credit (as in RAE2008) preferred

9
Outputs issues and probable outcomes 2

• All outputs submitted or a selection?
• Selected gives better fit with RAE2008 outputs
  profile. Concerns about long 1 (unfunded) tails
  if all submitted. Possibly best 4 but whether to
  allow variation across subjects TBC
• Institutions will therefore have to provide staff
  and outputs data
• Assessment of Significance
• May be to the discipline or users
• Will ask for Other Details statement where
  appropriate
• Issues still to resolve on staff selection and
  outputs
• How will panels combine bibliometrics with peer
  review?
• Assignment of staff and/or outputs to subject
  fields for citation analysis
• Which database will be used for citation
  analysis? could be both WOS and Scopus

10
Outputs issues and probable outcomes 3

• Issues still to resolve on staff selection and
  outputs
• Generation of an outputs quality profile from
  average output type, field and year normalised
  citation counts
• How can expert review be made less burdensome?
• Need for clarity on the definition of eligible
  staff
• Inclusion of Cat C and D staff

11
Environment issues and probable outcomes

• Assessed by qualitative and quantitative
  indicators
• Qualitative RA5 type narrative about the
  department, focusing on future strategy,
  sustainability, training of researchers, support
  for collaboration and interdisciplinary research
• Quantitative will include income and student
  data. Other data?
• Issues still to resolve on Environment
• Only income and students relating to submitted
  staff or all for the whole department?
• Quantitative data collection HESA vs HEIs
  collecting
• Need for clarity on the rubric for the narrative
  consistent across all panels and preference for
  common templates

12
Impact Issues and Probable Outcomes

• Originality and rigour remain essential criteria
  for all research. (But) In REF we are looking for
  more explicit recognition of social, economic and
  public policy impact, as well as academic
  significance. (D. Sweeney, Jan 2009)
• Impact to be assessed by structured
  narrative/case-study approach relating to
  departments or groups, not to individual outputs
• Will be supported by some quantitative indicators
• Further consultation underway with research users
• Issues still to resolve on Impact
• Scope should it include academic impact and
  user engagement?
• Narratives timelags, attribution and
  verification
• Choice of indicators
• Burden of data collection for institutions
• How to involve research users in the design and
  in the process of assessment

13
Constitution and role of the panels

• Number of units of assessment
• Agreed little room for very significant
  reduction Now 50-60? (cf 69 in RAE2008)
• Keep Main Panel/Sub Panel Structure
• Role of panels
• Setting the criteria, choosing indicators
  appropriate for the discipline
• Need for greater consistency of approach
• Setting the relative weightings of each element
  to be given in the overall assessment
• Conducting the assessment, including peer review
• Ensuring assessment is conducted with rigour and
  applying common quality standards

14
Timetable for implementation
15
REF and QR funding

• RAE2008 will continue to determine mainstream
  allocation up to (and including) 2013/14
• Assumptions for REF from 2014/15 onwards
• New funding formula likely to use similar model
  to current one for mainstream allocation
• Volume - Submitted staff?
• Quality profiles
• Subject cost weighting
• The separate charity support fund, research
  degree programme supervision (RDP) fund and
  Business Support Element are likely to continue
• UOB context 09/10 Mainstream 36.1M, Charity
  6.2M, RDP 6.2M, Business Support 1.6M, Total
  50.2M

16
Summary and issues for UOB

• It looks quite like the RAE, but
• We will have to be smarter about collecting and
  measuring our impact (cf research councils and
  other initiatives)
• For subjects where citations will be used to
  inform assessment of outputs we may need to be
  able to collect and analyse citation data
• Income and students are still important measures
  
• Significant implications for systems, but hard to
  plan when details not known
• Timing potentially very little time to get ready
  in current Hefce timetable need to plan for
  resource to be available when required
• The clock started ticking in January 2008!

17
Will it ever happen.

• Hefce keen to get framework agreed before next
  election
• John Denham. Peter Mandelson?
• Dual support secure, but Hefce to consider
  whether a greater proportion of HE funding might
  become contestable in order to promote innovative
  development (letter to Hefce following 2009
  budget)
• Confirmed that selectivity in funding should
  continue
• Next governments policy on dual support funding,
  protecting the science base, role of universities
  in economic recovery of UK??
• Is there an alternative anyway?
• Probably the answer is yes, but further delay in
  the timetable for implementation is possible

18
Further reading

• Hefce REF website
• http//www.hefce.ac.uk/research/ref/
• Structural adjustments, article by Zoe Corbyn
  in the Times Higher Education 14 May 2009
• http//www.timeshighereducation.co.uk/story.asp?se
  ctioncode26storycode406492