A Metric for Academic Research

1
A Metric for Academic Research
Shanghai WCU 2-conference, November 1, 2007
Ulf Sandström Linköping University Sweden
2
Context and Overview

• Performance indicators used for competitive or
  formula-based funding
• Building of a performance model the Waring model
• Model applied to Australian universities

3
Norway, UK, and Australia

• Norway the heterogeneity between disciplines is
  a big challenge
• UK is considering use of metrics
• Australian RQF combines counting of items with
  field normalized citation scores plus peer
  groups.
• Might solve some the heterogeneity problem, but
  the huge costs

4
Formula-based funding

• Raw counting of articles does not work (Butler
  2003)
• A quality dimension should be included (Glänzel
  1990)
• Do not use Journal Impact Factors (Seglen 1997)
• Field normalised citation scores (Leuven Leiden)

5
Two different worlds of Science?
Books
Journals
Medicine Natural Science
Humanities Social Science
6
Four worlds of research
Proceedings
COMP
ENG
SOC
Journals
MED NAT
Reports
HUM
Books
7
(No Transcript)
8
Academic output by field of research and
publication type (from Butler)
9
Proportion of articles in journals indexed by
ISI, by field
10
ICEBERG METAPHOR
CHEM
PHY
MED
AGR
ENG
SOC
HUM
COMP
11
The Bibliometric Turn

• The Budapest group showed (1985, 1989) that
  publication data can be used to estimate research
  personnel per country or by US state
• They corrected UNESCO figures for Eastern Europe.

12
Solving the productivity issue

• Objective find the field normalized
  productivity per authorGeographical area the
  Nordic countries
• Unification of Nordic university addresses
• Unification of all AU names (1st AU and reprint
  author RP) at Nordic univ.
• 51,000 unique authors 2001-2004

13
Clusters of ISI subject categories
14
Skewed distributions
AGR (Nordic data)
15
Frequencies in four classes
16
Left Truncated Mean Values
AGR (Nordic data)
17
Left Truncated Mean Values II
18
The Waring Model

• Produces reference values per macro class
• Taking number of publications per university over
  the reference value gives a Waring value per
  university a number that express the real
  size
• While humanities today produce one article
  chemistry has to produce ten

19
AUSTRALIA 2001-2004 Reference values based on
Australian and Swedish Researchers ISI data
Waringref per 2-year periods
20
AUSTRALIA 2001-2004
Reference values based on Australian and
Swedish Researchers
Waringref per 2-year periods
21
AUSTRALIA 2001-2004
22
Floor funding model

• The Waring value is used as an estimate in order
  to adjust the university output
• (Waring) (field normalized citation score)
• Sum of Production and Quality
• Next slide shows the result for Australia.

23
Go 8 Univ. Proposed Performance Related Fundi
ng Australian Universities 2001-2004
24
(No Transcript)
25
Summa Waring CPP/FCSm
26
Final reflection

• When set in action all performance models are
  normative machines.
• Therefore, it is vital for science that all areas
  are treated fairly.
• Engineering, Environment, Dentistry, Business
  Studies etc. are as important as other areas.

27
Citation Relations between Areas of Research
Physics
Materials
Basic life sciences
Environmental
Engineering
Chemistry
Biology
28
Method for Normalization
journals
CPP
FCS
JCS
A B C D E
Subject category
29
Open citation window 1998-2007 (May)
1998 1999 2000 2001 2002 2003 2004 2005 2006 2007
98 99 00 01 02 03 04 05 06
30
Thank you!
www.forskningspolitik.se