Academic Ranking of World Universities Methodologies and Problems

1
Academic Ranking of World UniversitiesMethodologi
es and Problems
By Professor Nian Cai LIU Center for World-Class
Universities, Graduate School of
Education Shanghai Jiao Tong University, China
August, 2008
2
Outline
3
Purposes
4
Dream of Chinese for WCU

• World-class university (WCU) is a dream for
  generations of Chinese. Its not only for pride,
  but also for the future of China.
• Recently, Chinese government has launched several
  initiatives for research universities. The
  best-known one is specially designed to build WCU
  (985 Project).

5
Goals of Top Chinese Universities

• Many top Chinese universities have setup their
  strategic goals as WCU.
• Most of them have also set time tables for
  achieving the goal of WCU. For example
• 2016 for Peking University
• 2020 for Tsinghua University

6
Questions About WCU

• Is there a clear definition for WCU?
• How many WCU should there be in the world?
• What are the positions of top Chinese
  universities in the world?
• How can Chinese universities improve themselves
  to reach the goal of WCU?

7
Academic Ranking of World Universities

• Our original purpose of doing the Academic
  Ranking of World Universities (ARWU) was to find
  out the position of Chinese universities in the
  world and the gap between them and WCU.
• ARWU was put on the internet upon the
  encouragement of colleagues from all over the
  world.

8
Features of ARWU

• ARWU uses a few carefully selected, objective
  criteria and internationally comparable data that
  everyone could verify in some way.
• It has been carried out by a ranking team of four
  researchers at Shanghai Jiao Tong University for
  their academic interests.
• It has been done independently without any
  financial support from any external sources.

9
Methodologies Results
10
Selection of Universities

• Any university that has any Nobel Laureates,
  Fields Medals, Highly Cited Researchers, or
  papers published in Nature or Science.
• Major universities of every country with
  significant amount of papers indexed by Thomson.
• Number of universities scanned gt2000
• Number of universities actually ranked gt1000
• Number of ranked universities on our web 500

11
Ranking Criteria and Weights
12
Definition of Indicator Alumni

• The total number of the alumni of an institution
  winning Nobel Prizes and Fields Medals.
• Alumni are defined as those who obtain bachelor,
  Masters or doctoral degrees from the
  institution.
• Different weights are set according to the
  periods of obtaining degrees. The weight is 100
  for alumni of 1991-2000, 90 for alumni of
  1981-1990, 80 for alumni of 1971-1980, and so
  on.
• If a person obtains more than one degrees from an
  institution, the institution is considered once
  only.

13
Definition of Indicator Award

• The total number of the staff of an institution
  winning Nobel prizes in physics, chemistry,
  medicine and economics and Fields Medal in
  Mathematics.
• Staff is defined as those who work at an
  institution at the time of winning the prize.
• Different weights are set according to the
  periods of winning the prizes. The weight is
  100 for winners since 2001, 90 for winners in
  1991-2000, 80 for winners in 1981-1990, 70 for
  winners in 1971-1980, and so on.
• If a winner is affiliated with more than one
  institution, each institution is assigned the
  reciprocal of the number of institutions.
• For Nobel prizes, if a prize is shared by more
  than one person, weights are set for winners
  according to their proportion of prize.

14
Definition of Indicator HiCi

• The number of highly cited researchers in 21
  broad subject categories in life sciences,
  medicine, physical sciences, engineering and
  social sciences.
• The definition of categories and detailed
  procedures can be found at the website of
  Institute of Scientific Information.
• The total number of HiCi is about 5000, about
  4000 of which is university staff.

15
Definition of Indicator NS

• The annual average number of articles published
  in Nature and Science in the past five years.
• To distinguish the order of author affiliation, a
  weight of 100 is assigned for corresponding
  author, 50 for first author (second author if
  the first author is the same as corresponding
  author), 25 for the next author, and 10 for
  other authors.
• Only publications of article type are considered.
  

16
Definition of Indicator PUB

• Total number of articles indexed in Science
  Citation Index-expanded (SCIE) and Social Science
  Citation Index (SSCI) in the past year.
• A weight of 2 is assigned to articles indexed in
  SSCI to compensate the bias against humanities
  and social sciences.
• Only publications of article type are considered.
  

17
Definition of Indicator PCP

• The sub-total scores of the above five indicators
  divided by the number of full-time equivalent
  academic staff.
• If the number of academic staff for institutions
  of a country cannot be obtained, the total scores
  of the above five indicators is used.
• For ranking 2005, the number of full-time
  equivalent academic staff is obtained for
  institutions in USA, China, Australia, Italy,
  Netherlands, Sweden, and Belgium etc.

18
Main Sources of Data

• Nobel laureates
• http//www.nobel.se
• Fields Medals
• http//www.mathunion.org/medals/
• Highly-cited researchers
• http//www.isihighlycited.com
• Articles published in Nature and Science
• http//www.isiknowledge.com
• Articles indexed in SCIE and SSCI
• http//www.isiknowledge.com

19
Results of ARWU

• http//www.arwu.org/
• Top 500 universities in the world
• Top 100 universities in North and Latin America
• Top 100 universities in Asia/Oceania
• Top 100 universities in Europe
• Statistics of top universities by region and
  country
• Percentage distribution of top universities by
  country as compared with the share of global
  population and GDP

20
Problems Discussion
21
Methodological Education and Service

• Education is the basic function of any
  university, however, it would be impossible to
  rank the quality of education due to the huge
  differences among the national systems.
• Contribution to the national economic development
  is becoming increasingly important for
  universities, however, it is impossible to
  obtain internationally comparable indicators and
  data.
• The academic or research performance of
  universities, a good indication of their
  reputation, can be ranked internationally.

22
Methodological Humanities Social Sciences

• Many well-known institutions specialized in
  humanities and social sciences are ranked
  relatively low.
• Since 2004, the indicator of NS is not
  considered for institutions specialized in
  humanities and social sciences, its weight is
  relocated to other indicators.
• Since 2005, a weight of 2 for articles indexed by
  SSCI is considered.
• Nevertheless, if a university specialized in
  social sciences and humanities had Nobel
  Laureates in economics and Highly Cited
  Researchers in social sciences, it should have
  good standing.

23
Methodological Language Bias

• English is the language of international academic
  community.
• Any ranking based on academic performance will be
  biased towards institutions in English-speaking
  countries.
• One possible solution papers published in
  non-native languages are offered a special
  weight.
• Another possible solution normalization of total
  articles by the proportion of journal editors of
  each country.

24
Methodological Award and Alumni

• Universities which started after 1911 do not have
  a fair chance.
• Disciplines not related to the awarding fields do
  not have a fair chance. Other important awards
  include Abel, Pulitzer, Turing, Tyler, Pritzker,
  etc.
• Institutions for winning awards and those for
  doing the researches may not be the same.
• Institutions for obtaining degrees and those for
  pursuing the studies may not be the same.
• Postdoctoral training is not considered.

25
Methodological Per Capita Performance

• The weight of the PCP indicator for per capita
  performance is rather low. Large institutions
  have relatively high positions in the ranking.
• However, its very difficult to obtain
  internationally comparable data on the number of
  academic staff.
• The types of academic staff such as purely
  teaching staff, teaching and research staff,
  purely research staff.
• The ranks of academic staff such as professor,
  associate professor, reader, lecturer, research
  scientist etc.

26
Technical Attributions

• Many universities have more than one commonly
  used names such as Virginia Tech and Virginia
  Polytechnic and State University.
• Variations due to translation such as Univ Koln
  and Univ Cologne, Univ Vienna and Univ Wien.
• Abbreviated names such as ETH Zurich for Swiss
  Federal Institute of Technology Zurich.
• Some authors only write their departmental or
  institute names without mentioning their
  university names.

27
Technical Definition of Institution

• University systems such as Univ California
  system, Univ London system.
• Affiliated institutions and research
  organizations such as Ecole Polytechnique
  Montreal (affiliated to University of Montreal),
  CNRS Labs (affiliated to French universities).
• Teaching and affiliated Hospitals complex!
• Our answer according to authors expression.

28
Other Technical Problems

• Merging, splitting, inheriting,
  discontinuing, name-changing of institutions such
  as
• Univ Kwazulu-Natal in South Africa merged from
  Univ Natal and Univ Durban-Westville.
• University of Innsbruck in Austria splitted into
  Univ Innsbruck and Innsbruck Medical Univ.
• Humboldt Univ Berlin and Free Univ Berlin
  inheriting the Nobel Prizes of the Berlin
  University before world war II.

29
Ranking by Broad Subject Fields (ARWU-FIELD)
30
Requests for Ranking of World Universities by

• Broad subject fields or schools, colleges
• and
• Subject fields or programs, departments
• In addition, many top Chinese universities want
  to learn their positions in the world by broad
  subject fields or disciplines.

31
Definition of Broad Subject Fields

• Natural Sciences and Mathematics (SCI)
• Engineering/Technology and Computer Sciences
  (ENG)
• Life and Agriculture Sciences (LIFE)
• Clinical Medicine and Pharmacy (MED)
• Social Sciences (SOC)
• Arts and humanities are not ranked

32
ARWU-FIELD Indicators and Weights
33
Changes in Indicators and Definition

• NS in ARWU is not used in ARWU-FIELD.
• TOP is the percentage of articles published in
  the top 20 journals of each broad subject field.
  
• Fund is the total engineering-related research
  expenditures. Its used only for ENG ranking.
• Alumni and Award since 1951 are used for all
  rankings fields except ENG.
• PUB is the total number of articles indexed by
  Thomson in the past year.

34
Results of ARWU-FIELD

• http//ed.sjtu.edu.cn/ranking.htm
• Top 100 universities in SCI
• Top 100 universities in ENG
• Top 100 universities in LIFE
• Top 100 universities in MED
• Top 100 universities in SOC
• Statistics of top universities by region
  country
• List of top universities by number of top fields

35
Special Problems in ARWU-FIELD

• Its difficult to obtain data on
  engineering-related research expenditures and
  make them comparable. For 2007, Fund was obtained
  only for US and Canadian universities.
• Its difficult to separate the Nobel Laureates in
  Physiology or Medicine. They are used in both
  LIFE and MED ranking.

36
Final Remarks
37
Controversy of Ranking

• Any ranking is controversial and no ranking is
  absolutely objective.
• University rankings become popular in many
  countries. Whether we agree or not, ranking
  systems clearly are here to stay.
• The key issue then becomes how to improve ranking
  systems for the benefits of higher education
  (IREG).

38
Use of Ranking

• Rankings should be used with cautions. Their
  methodologies must be read carefully before
  reporting or using their results.
• Rankings should be used in combination with other
  types of evaluation whenever possible, such as
  bench-marking, peer review, etc.

39
Future Efforts

• Study all the above mentioned problems and
  continuously improve the ranking methodologies.
• Establish more comprehensive databases of WCU.
• Update ARWU (every August) and ARWU-FIELD
  annually (every February).

40
Future Efforts (Conted)

• Provide ranking of universities specialized or
  strong in engineering, medicine, etc. based on
  the classification of world universities.
• Provide ranking of universities with different
  size, history, budget and function etc. once
  internationally comparable data are obtained.

41
Thank You Very Much!
http//gse.sjtu.edu.cn/ http//www.arwu.org