HOW DOES PEER REVIEW WORK

1
HOW DOES PEER REVIEW WORK?

• Sven Hemlin
• The Sahlgrenska Academy
• Göteborg University
• Plenary talk for the meeting of research councils
  of the HERA ERA-Net (Humanities in the European
  Research Area), 18 Nov, 2005, Dublin

2
(No Transcript)
3
A CRUCIAL FINDING PEERS OFTEN DISAGREE

• A classic study (Peters Ceci, 1982)
• Low correlations between peer judgements
  (Cicchetti, 1991)
• So, which is best Peer agreement or peer
  disagreement?

4
(No Transcript)
5
TWO PEERS AGREE

• Rational Good
• e.g. based on scientific merits of the
  application
• Not rational Bad
• e.g. based on personal liking of the applicant

6
(No Transcript)
7
TWO PEERS DISAGREE

• Rational Good
• e.g. peers apply different criteria of
  scientific quality
• Not rational Bad
• e.g. one peer supports the application because
  he likes the applicant, and another peer rejects
  it because she dislikes the applicant

8
APPLICATIONS MANUSCRIPTS

• Acceptance Rejection
• Appli- Disagreement (S) Agreement (H/S)
• cation
• Manu- Agreement (SF H/S) Agreement (GD H/S)
• Script Disagreement (GD H/S)

9
FURTHER ISSUES

• Support of conventional research (Campanario,
  1996 Martin Irvine, 1983 Sigelman Whicker,
  1987)
• Researcher status and dept status are influential
  (Cole, Rubin Cole, 1977 Prelli, 1989)
• Cognitive particularism (Travis Collins, 1991
  Hemlin et al., 1995)

10
ETHICAL ISSUES

• Sexism nepotism (Wennerås Wold, 1997)
• Old boys network (Gillespie et al., 1985 Merton,
  1973)

11
IMPROVEMENTS OF PR

• Scientometrics (e.g. publication figures)
• Standardised procedures (e.g. rating scales)
• Quality criteria (e.g. appropriate methods,
  originality in problem statement)
• Peer selection (broad or deep knowledge, junior
  or senior, interdisciplinaryFoss Hansen Borum,
  1999Fuller, 1995)

12
SUGGESTIONS

• Change peers frequently
• Discuss quality criteria, imple-mentation,
  feedback
• Use converging indicators
• Do not standardise procedures rigidly, but allow
  flexibility