Research Assessment in the Humanities and the ESFERIH project Alain Peyraube CNRS

1
Research Assessment in the Humanities and the
ESF-ERIH project Alain PeyraubeCNRS EHESS,
Paris, FRLondon, HERA Workshop, 25 Jan. 2007
2
The situation in Europe

• Evaluation of researchers, research teams and
  centres generally made at the national level
• ESF workshop on the Evaluation in the Humanities
  Budapest, June 2001. - 29 people from 17
  different countries

3
Budapest workshop (1)Objectives

• Three aims
• Discussion of the different evaluation criteria
  and indicators
• Advantages and shortcomings of the quantitative
  criteria
• How to compile a list of scholarly journals in
  the different fields of the Humanities

4
Budapest workshop (2)Evaluation criteria

• Publications
• Grants obtained
• - from res. councils of other countries
• - from national research councils
• International relations
• - Study grants
• - Guest lectures

5
Budapest workshop (3)Evaluation criteria

• Conferences
• - Convenor/Organiser of a conference
• - Paper delivered at conferences (distinguish
  keynote/invited speakers/ ordinary papers)

6
Budapest workshop (4)Evaluation criteria

• Impact on students supervising PhDs, teaching
  in graduate schools
• Professional activities
• - Editor of periodical/member of editorial board
• - Reviews for profess. journals/books
• - Prof. associations (member of board, etc.)

7
One example from outside Europe Australian case
(1)

• Publications (one book every 5 years, 2 papers in
  ref. int. journals p. year)
• Grants (from overseas, from ARC)
• Conference papers (Keynote/Invited/Ordinary)
• Training PhD students
• New courses initiated

8
One example from outside Europe Australian case
(2)

• Teaching evaluations of the students
• Contribution to the academic community
• Committee membership
• External Consultancies

9
Publications how to evaluate? (1)

• Monograph
• Book chapter
• Edited book
• Refereed journal article
• Critical edition, text edition, annotated edition
• Textbook

10
Publications how to evaluate? (2)

• Entry for encyclopedia
• Conference proceedings article
• Substantial book review (review article)
• Translation
• Audio or Video recording

11
Budapest workshop (5)

• AHCI (ISI-Tomson scientific web of knowledge)
  very unsatisfactory
• Urgent need for a European Reference Index for
  the Humanities (ERIH) as an additional tool for
  research evaluation, and not the exclusive means
• Request the ESF-SCH to go ahead and try to
  compile lists of ref. journals

12
SCH November 2001 March 2003

• Discipline directory 15 disciplines (or subject
  groups)
• Sub-categorization of journals A, B, C
• ESF Mos approached with guidelines to provide
  lists of reference journals discipline by
  discipline

13
ERIH The 15 disciplines/panels (1)

• Anthropology
• Archaeology
• Art and Art history
• Classical Studies
• Gender Studies
• History
• History and Philosophy of Science
• Linguistics

14
ERIH The 15 disciplines/panels (2)

• Literary Studies
• Music Musicology
• Oriental African Studies
• Pedagogical Educational Research
• Philosophy
• Psychology
• Religious Studies Theology

15
ERIH Journal categories

• A category (expected 5-20 of all titles) high
  ranking, intern. level pub., very strong
  reputation among researchers in the field
• B standard int. level pub., good reputation
  among researchers
• C important local/regional level pub.,
  occasionally cited outside the publ. country,
  only European pub. considered

16
ERIH A long process 2004 and 2005

• Spring 2004 Setting-up of a ERIH Steering
  Committee F. Kiefer (HU), A. Mustajoki (FI), A.
  Peyraube (FR), Gudrun Gersmann (DE), Marc
  Waelkens (BE), Michael Worton (UK).
• May 2005 ERIH Coordinator takes up position
  under HERA contract (ERA-NET)
• September 2005 Composition of Expert Panels
  finalised.
• Oct.-Dec. 2005 Expert Panel sessions (2
  postponed until 2006).

17
Further developments in 2006 (1)

• Panel Chairs Meeting Steering Committee Meeting
  (Brussels, 1-2 February 2006)
• Preliminary draft lists sent to Mos in March
  2006. Ask for comments, additions, corrections
• April-May 2006 consultation process began
• June 2006 18 Mos responded. Mos contacted again,
  new inputs received.

18
2006 (2)

• A major concern of the committee was that
  category C might include some residual journals,
  not conforming to the criteria stated. All ERIH
  journals, including C journals, have to be  good
  scientific journals 
• None of the lists should be called  final
  lists  but rather  initial lists 

19
2006 (3)

• Panel chairs contacted by the Steering Committee
  to refine the lists
• Steering Committee meeting in London in September
  2006 8 disciplines/panels considered as
  finalised for approval by the SCH Anthropology,
  Gender Studies, History, History and Philosophy
  of Science, Linguistics, Music and Musicology,
  Philosophy, Religious Studies and Theology

20
Next steps

• Complete and strengthen the robustness of the
  remaining 7 lists (by Expert panels)
• Prepare workplan for inclusion of other
  publications (monographs, etc)
• Prepare an application for infrastructures and
  indicator funding (FP7)
• Publish initial lists after approval

21
ERIH and citation analysis / counting

• Is there a risk that the reference list of
  journals could be used for bibliometric analysis?
• The wide use today of, particularly, AHCI
  (ISI-Tomson) and Google-scholar
• For example Ghent University (BE)
• It is essential for the future of the Humanities
  to furnish themselves with reliable reference
  tools, following the example of other scientific
  disciplines.

22
ERIH and citation analysis / counting (2)

• We can no longer allow ourselves the indulgence
  of attibruting this (the lack of appropriate
  tools) to a peculiarity of the Humanities
• Specialists in the Humanities should take into
  consideration its special characteristics and
  develop the corresponding tools

23

• FIN