Title: Alternative Peer Review: Quality Management for 21st Century Scholarship
1Alternative Peer ReviewQuality Management for
21st Century Scholarship
http//www.public.iastate.edu/gerrymck/APR.ppt
- Gerry McKiernan
- Science and Technology Librarian
- and Bibliographer
- Iowa State University Library
- Ames IA
- USA
gerrymck_at_iastate.edu
2Workshop on Peer Review in the Age of Open
Archives
- International School for Advanced Studies
- Interdisciplinary Laboratory
- Trieste, Italy
- May 23-24, 2003
3THANK YOU!
- Workshop Advisory Board (Marco Fabbrichesi
(INFN/SISSA Italy), Stevan Harnad (University of
Southampton, UK), Stefano Mizzaro (University of
Udine, Italy) and Corrado Pettenati (CERN
Library, Geneva, Switzerland) - Iowa State University, Faculty Senate, Committee
on Recognition and Development - European Commission
- Iowa State University Library
- Heike Kross, Ph.D.
4 DISCLAIMER (1) The screen prints selected for
this presentation are for educational purposes
and their inclusion does not constitute an
endorsement of an associated product, service,
place, or institution.
5 DISCLAIMER (2) The views and opinions expressed
in this presentation are those of the presenter
and do not constitute an endorsement by Iowa
State University or its Library.
6 NOTICE
No editors, authors, or referees were harmed in
the preparation of this presentation.
7http//www.sciencecartoonsplus.com/
8Giuseppe De Nittis (1846-1884) The Macchiaioli /
Italian Impressionists
9Campo di Biche (1875)
10PEER REVIEW DEFINITION
- Peer review is the assessment by an expert of
material submitted for publication.
Carin M. Olson, Peer Review of the Biomedical
Literature, American Journal of Emergency
Medicine 8 no.4 (July 1990) 356-358.
11PEER REVIEW PURPOSES
Peer review helps to ensure that published
research is
Important Original
Timely Technically-reliable
Internally-consistent Well-presented
Benefited from guidance by experts
Carin M. Olson, Peer Review of the Biomedical
Literature, American Journal of Emergency
Medicine 8 no.4 (July 1990) 356-358.
12PEER REVIEW STRENGTHS
The underlying strength of peer review isthe
concerted effort by large numbers of researchers
and scholars who work to assure that valid and
valuable works are published and conversely to
assure that invalid or non-valuable works are not
published .
Anne C. Weller, Editorial Peer Review Its
Strengths and Weaknesses. (Medford, NJ
Information Today, 2001).
13Houston, We Have a Problem!
14PEER REVIEW PROBLEMS
- Subjectivity
- Bias
- Abuse
- Detecting defects
- Fraud and Misconduct
- Delay
Fytton Rowland, The Peer-Review
Process, Learned Publishing 15 no. 4 (October
2002) 247-258.
Report version http//www.jisc.ac.uk/uploaded_doc
uments/rowland.pdf
15SUBJECTIVITY
- Summary rejections by editor without sending the
paper to referees - Choice of referees by the editor (choosing for
example, a known harsh referee for a paper the
editor wishes to see rejected)
16BIAS
- Discrimination against authors because of their
nationality, native language, gender or host
institution - Situations where author and referee are
competitors in some sense, or belong to warring
schools of thought
17ABUSE
- Too many articles out of one piece of research,
or duplicate publication - Intellectual theft omission or downgrading of
junior staff by senior authors - Plagiarism (stealing others yet unpublished work
that has been sent for review) - Delaying publication of potentially competing
research
18DETECTING DEFECTS
- Identification of factual errors within submission
19FRAUD and MISCONDUCT
- Fabrication of results
- Falsification of data
- False claim of authorship for results
20DELAY
There is much muttering about publication delay,
a real enough problem, especially in paper
publication, but peer review itself is often
responsible for as much of the delay as the paper
publication and distribution process
itself. Stevan Harnad
Stevan Harnad, Implementing Peer Review on the
Net Scientific Quality Control in Scholarly
Electronic Journals, in Scholarly Publication
The Electronic Frontier, edited by Robin P. Peek
and Gregory B. Newby (Cambridge MA MIT Press,
1996).
http//www.ecs.soton.ac.uk/harnad/Papers/Harnad/h
arnad96.peer.review.html
21http//www-marketing.wharton.upenn.edu/ideas/pdf/p
eerpaper.pdf
22Peer review is slow, expensive, profligate of
academic time, highly subjective, prone to bias,
easily abused, poor at detecting gross defects,
and almost useless in detecting fraud.
Richard SmithEditor, BMJ
- Richard Smith, Opening Up BMJ Peer Review,
- BMJ 318 (7175) (January 2 1999) 4-5
23Stephen Lock, A Difficult Balance Editorial Peer
Review in Medicine (Philadelphia, PA ISI Press,
1986).
24(No Transcript)
25EXAMPLE
26-
http//archive.salon.com/tech/feature/2002/09/16/p
hysics/
27Science
Nature
http//physicsweb.org/article/news/6/9/15/
28http//agenda.cern.ch/fullAgenda.php?idaa01193
29RECOMMENDATIONSWorkshop on the Open Archives
Initiative (OAI)and Peer Review Journals in
Europe, CERN, Geneva Switzerland, March 22-24,
2001
- The participants were unanimous in their belief
that the certification of scholarly work remains
a fundamental part of a system for scholarly
communication. - It was also generally believed that the
electronic environment allows for novel
approaches to accord a stamp of quality to
scholarly works.
Alison Buckholtz, Raf Dekeyser, Melissa
Hagemann, Thomas Krichel, and Herbert Van de
Sompel, Open Access Restoring Scientific
Communication to Its Rightful Owners, European
Science Foundation Policy Briefing 21 (April
2003) 1-8.
http//www.arl.org/sparc/SPB21_OAI.pdf
30Let us be imaginative in exploring the
remarkable possibilities of this brave new
medium.
- Stevan Harnad, Implementing Peer Review on the
Net Scientific Quality Control in Scholarly
Electronic Journals, in Scholarly Publication
The Electronic Frontier, edited by Robin P. Peek
and Gregory B. Newby (Cambridge MA MIT Press,
1996).
http//www.ecs.soton.ac.uk/harnad/Papers/Harnad/h
arnad96.peer.review.html
31(No Transcript)
32(No Transcript)
33(No Transcript)
34(No Transcript)
35Let us be more imaginative in exploring the
remarkable possibilities of this brave new
medium. With Apologies to Stevan Harnad
36(No Transcript)
37TM
http//lockss.stanford.edu/
38LOCKSS
- For centuries libraries and publishers have had
stable roles publishers produced information
libraries kept it safe for reader access. There
is no fundamental reason for the online
environment to force institutions to abandon
these roles. - The LOCKSS model capitalizes on the traditional
roles of libraries and publishers. LOCKSS
creates low-cost, persistent digital "caches" of
authoritative versions of http-delivered
content.
39LOCKSS
- The LOCKSS software enables institutions to
locally collect, store, preserve, and archive
authorized content thus safeguarding their
community's access to that content. - The LOCKSS model enforces the publisher's access
control systems and, for many publishers, does no
harm to their business models.
40LAMPSS
Lots of Alternative Models Provide Sensible
Solutions
41(No Transcript)
42DISCLAIMER
ON
- The alternative peer review models profiled are
for informational and educational purposes only
and do not necessarily constitute an endorsement.
43ALTERNATIVE PEER REVIEW
- Institution-based
- Citation-based
- Index-based
- Metadata-based
- Computer-assisted
- NO Peer Review
- Moderator-based
- Tier-based
- Neo-Classical
- Certification-based
- Open Peer Review
- Commentary-based
- Collaborately-filtered
44NEO-CLASSICAL PEER REVIEW
45Neo-Classical Peer Review
46Neo-Classical Peer Review
47Neo-Classical Peer Review
48CERTIFICATION-BASED
49Certification-Based
- The process of pre-publication peer review
could be improved and become a more reliable
indicator of manuscript quality if reviewers were
trained in, and subsequently applied
systematically, critical skills and use of a
hierarchy of evidence to classify submitted
articles being reviewed.
Stephen Pritchard , Peer Review - a Proposal for
Change, Paper presented at Thinking Globally -
Acting Locally Medical Libraries at the Turn of
an Era, 8th European Conference of Health and
Medical Libraries, September 16-21, 2002,
Deutsche Zentralbibliothek für Medizin, Köln,
Germany. http//www.zbmed.de/eahil2002/abstracts/p
ritchard.pdf
50OPEN PEER REVIEW
51Open Peer Review
- IDENTIFICATION OF REVIEWERS / SIGNED REVIEWS
- BMJ
- bmj.com
- BioMed Central
- biomedcentral.com
- eMJA (Medical Journal of Australia)
- www.mja.com.au/public/information/project.html
52COMMENTARY-BASED
53Commentary-based
- Readers can comment before and/or after classic
peer review, or instead of classic peer review - Electronic Transactions on Artificial
Intelligence - (www.etaij.org)
- OPEN REVIEW / REFEERING
- Journal of Interactive Media in Education
- (www-jime.open.ac.uk)
- PRE- AND POST- COMMENTARY
- Psycoloquy
- (psycprints.ecs.soton.ac.uk)
- POST PEER REVIEW COMMENTARY)
54Electronic Transactions on Artificial
Intelligence
55Journal of Interactive Media in Education
56http//psycprints.ecs.soton.ac.uk/
57(No Transcript)
58(No Transcript)
59COLLABORATIVELY-FILTERED
60Collaboratively-Filtered
- DEFINITION Guiding people's choices of what
to read, what to look at, what to watch, what to
listen to (the filtering part) and doing that
guidance based on information gathered from some
other people (the collaborative part)." - Paul Resnick
http//www.cni.org/Hforums/cni-announce/1996/0031.
html
61ResearchIndex / CiteSeer
http//www.researchindex.com
62INSTITUTION-BASED
63Institution-Based
- INSTITUTIONAL INITIATIVES
- DSpace (MIT)
- www.dspace.org
- eScholarship (University of California)
- escholarship.cdlib.org
- Glasgow ePrint Service (University of Glasgow)
- eprints.lib.gla.ac.uk
64CITATION-BASED
65Citation-Based
- Citations to Open Access / OAI-compliant
- documents are indicators of document importance
-
http//citebase.eprints.org/
66(No Transcript)
67(No Transcript)
68(No Transcript)
69(No Transcript)
70INDEX-BASED
71Index-Based
- INDEXING OF EPRINTS BY COMMERICAL ABSTRACTING AND
INDEXING SERVICE - Chemical Abstracts (American Chemical Society)
(CAS) indexes select appropriate e-prints from
the arXiv.org eprint server as well from the
Chemical Preprint Service (Elsevier) - Its selection criteria for this kind of
electronic document are essentially the same as
for the traditional printed documents they must
report new information of chemical or
chemistry-related interest and must be original
publications. Also, the electronic publication
must be publicly available and have some relative
permanence . - Eric Shively / Chemical Abstracts Service
-
72Index-Based
- CAB Abstracts doesnt currently include Eprints
or Preprints, but we are looking at the
implications and possible mechanisms for
accessing and indexing Eprints and/or Preprints
related to the applied life sciences. - Tracy Shaw / CAB International
73METADATA-BASED
74Metadata-Based
ltoai-qualitygt ltcategorygtinternallt/categorygt ltproc
essgt peer review lt/processgt ltorganizationgt CERN lt/
organizationgt ltpoliciesgt http//www.cern.ch/polici
es/review.html lt/policiesgt lt/oai-qualitygt
William Y. Arms, Quality Control in Scholarly
Publishing On The Web. What Are the Alternatives
to Peer Review? PowerPoint presentation given at
the Workshop on the Open Archives Initiatives
(OAI) and Peer Review Journals in Europe, March
22-24, 2003, CERN, Geneva, Switzerland. http//www
.cs.cornell.edu/wya/papers/CERN-2001.ppt
75COMPUTER-ASSISTED
76Computer-Assisted (1)
- SOFTWARE THAT ASSISTS IN THE EVALUATION OF A
SUBMITTED MANUSCRIPT - A Software Program to Aid in Peer Review
- Alvar Loria and Gladys Faba
- Objective To characterize a personal
computer-based software program developed as an
aid to peer review of medical papers. - Design The software is a Windows-based
application that records automatically a numeric
score to a series of questions related to 8
sections of scientific papers (introduction,
methods, results, and discussion, plus 4 other
sections). The questions and sections vary
according to type of paper (original reports,
case reports, or reviews), and the final output
is a score with a maximum of 100 for a "perfect"
paper. The software was tested using a single
reviewer to judge 289 papers (169 original
reports, 50 case reports, and 70 reviews) from 44
Mexican medical journals. All statistical
analysis of scores were done with nonparametric
tests.
77Computer-Assisted (2)
- Results The paper scores ranged from 29 to 97
with slightly higher median and less dispersion
of scores for reviews as compared with original
reports and case reports, but these differences
did not reach significance. Two observations
suggest that the software operated reasonably
well a) there were some differences in the
section scores by type of paper that agreed well
with differences in their complexity b) the
journal scores showed an association with their
number of original papers and their percentage of
original papers (Kruskal-Wallis test, P.06 and
0.07, respectively). - Conclusions The software operated reasonably
well when used to compare the relative quality of
289 papers. The validity of the program is
restricted in this study to the experience of 1
reviewer. An analysis of the raw scores helped in
detecting some ambiguous and redundant questions
that have been modified in an improved version.
The program has a potential as a training tool
for inexperienced reviewers or as a scorekeeper
for experienced peer reviewers.
Alvar Loria and Gladys Faba, A Software Program
to Aid in Peer Review, Abstract of paper
presented at the Third International Congress on
Biomedical Peer Review and Global Communications,
September 18-20, 1997, Prague, Czech Republic.
http//www.ama-assn.org/public/peer/arev.htm
78NO PEER REVIEW
79NO Peer Review
http//xxx.arXiv.cornell.edu
80MODERATOR-BASED
81 Moderator-based (1)
- The intent of this model is to allow the widest
range of scientific manuscripts to be archived,
searched, and distributed electronically at the
lowest possible cost. - This would be accomplished through very minimal
filtering and subsequent placement of eprints on
a non-commercial archival server by a
subject-specific Moderator appointed by a society
(or consortia of societies). - A society-appointed Editorial Board (with
double-blind peer review approved by the
non-profit Peer Review Inc. organization) would
then the identify the most important materials
from among these archived items, and the stamp of
approval for these items would be included in a
secondary Virtual Collection.
82Moderator-based (2)
- There are no direct submissions to the Editorial
Board manuscripts would be directed to the
Editorial Board in one of three ways - 1. nominated by the eprint Moderator upon receipt
for the archival server, - 2. notification sent to the Editorial Board when
a threshold number of hits are generated by any
one manuscript on the archive server, and - 3. nominated by readers of material from the
archive this process requires a letter of
support outlining the importance of the work to
the Editorial Board.
83Moderator-based (3)
- The Virtual Collection could be produced as a
variety of products - enhanced abstracts
- email threads (with comments)
- virtual reviews of sub-disciplines
- SDIs (selective dissemination of information)
current awareness tools - This process
- reduces the load on the Editorial Boards, which
results in a faster review process
differentiates those items worthy of higher
recognition from those worthy of archiving,
making it easier for a reader to filter material,
based upon a society and discipline authority
(rather than commercial reasoning) - provides for search/browse/sdi from the Virtual
Collection for filtered info, reducing this more
expensive option for only those items recognized
as of the highest quality.
843
2
1
David Stern, The eprint Moderator Model,
Newsletter on Serials Pricing Issues no. 214
(February 8, 1999). http//www.lib.unc.edu/prices/
1999/PRIC214.HTML214.5
85TIER-BASED
86Tier-based
- Two separate domains
- Standard Tier
- Any and all submissions would be accepted after
a cursory examination of or other pro forma
certification. - The review process could be minimally
labor-intensive, perhaps relying primarily on an
automated check of author institutional
affiliation, prior publication record, research
grant status, or other related background and
involve human labor primarily to adjudicate
incomplete or ambiguous results of an automated
pass. -
87Tier-based
- Upper Tier
- At some later point (which could vary from
article to article, perhaps with no time limit),
a much smaller set of articles would be selected
for the full peer review process. The initial
selection criteria for this smaller set could be
any of a variety of impact measures, to be
determined, and based explicitly on their prior
widespread and systematic availability and
citability e.g., reader nomination or rating,
citation impact, usage statistics, editorial
selection, ... . -
Paul Ginsparg, Can Peer Review be Better
Focused?, Science Technology Libraries 22
No. 3/4 (In press). http//arxiv.org/blurb/pg02pr.
html
88DISCLAIMER
OFF
- Presented for Your Consideration
89FREEDOM OF IDEAS
http//www.nrm.org/exhibits/current/four-freedoms.
html
90FUTURE OF IDEAS
- The explosion of innovation we have seen in
the environment of the Internet was not conjured
from some new, previously unimagined
technological magic instead, it came from an
ideal as old as the nation. Creativity flourished
there because the Internet protected an
innovation commons.
91FUTURE OF IDEAS
The Internets very design built a neutral
platform upon which the widest range of creators
could experiment. The legal architecture
surrounding it protected this free space so that
culture and information the ideas of our
eracould flow freely and inspire an
unprecedented breadth of expression.
Lawrence Lessig, The Future of Ideas The Fate
of the Commons in a Connected World. (New York
Random House, c2001).
92- ITS
- NOT
- ABOUT
- PUBLICATION
- ITS
- ABOUT
- IDEAS.
93The Association of Learned and Professional
Society Publishers (ALPSP) SurveyAuthors and
Electronic Publishing
- Scholarly research communication has seen
far-reaching developments in recent years. - Most journals are now available online as well as
in print, and numerous electronic-only journals
have been launched - the Internet opens up new ways for journals to
operate. - Authors have also become conscious of alternative
ways to communicate their findings, and much has
been written about what they ought to think.
94 ALPSP felt that it would be timely to discover
what they actually thought and what they actually
did. This survey aimed to discover the views of
academics, both as authors and as readers. Some
14,000 scholars were contacted across all
disciplines and all parts of the world, and
nearly 9 responded their detailed comments make
thought-provoking reading.
Alma Swan and Sheridan Brown. Authors and
Electronic Publishing The ALPSP Research Study
on Authors' and Readers Views of Electronic
Research Communication. (West Sussex, UK The
Association of Learned and Professional Society
Publishers, 2002). http//www.alpsp.org/pub5.ht
m
95When asked to predict what would be the most
common form of quality control in five years
time, only a bare majority answered traditional
peer review.Fytton Rowland, The Peer-Review
Process, Learned Publishing 15 no. 4 (October
2002) 247-258.
Report version http//www.jisc.ac.uk/uploaded_doc
uments/rowland.pdf
96FURTHERMORE
- 16 said that the referees would no longer be
anonymous - 27 said that traditional peer review would be
supplemented by post-publication commentary - 45 expected to see some changes in the
peer-review system within the next five years
Fytton Rowland, The Peer-Review Process,
Learned Publishing 15 no. 4 (October 2002)
247-258.
Report version http//www.jisc.ac.uk/uploaded_doc
uments/rowland.pdf
97Importance of the Peer Review Process
http//www.alpsp.org/pub5.ppt
98What is Gray/Grey Literature ?
- Papers are often written to inform funding bodies
about the results of research projects, to
support grant applications, to inform rapidly a
specific scientific community, to present
preliminary results at conferences or as
dissertations. - Such material is disseminated quickly, often in
limited numbers, before or without the formal
publication process. Such documents are called
non-conventional or grey literature.
http//www.kb.nl/infolev/eagle/what_is_gl.htm
99The Value of Grey Literature Grey literature is
really a type of informal communication, which on
a scale of formality, fits in somewhere between
conversation and normal publication. A formal
publication may follow later but in many cases -
contrary to the common assumption - these papers
may not been made publicly available at all.
http//www.kb.nl/infolev/eagle/what_is_gl.htm
100Nevertheless, grey publications may contain
comprehensive, concrete and up-to-date
information on research findings, and
investigations have shown, that even when grey
documents are published officially at a later
stage, detailed information on techniques,
methods, measured values and details of
experiments are frequently omitted. For these
details of importance for further research, the
non-conventional literature is then the first and
only source of information.
http//www.kb.nl/infolev/eagle/what_is_gl.htm
101Veterinary Medicine
- 12 Major Veterinary Medicine Journals
- Overall, 6.38 of cited literature was Gray/Grey
Literature - The figures for individual journals ranged from
about 2.5 to 10 gray/grey literature - Research journals cited a higher percentage of
Gray/Grey Literature than did Clinical titles - William H. Weise and Nancy Pelzer, Bibliometric
Study of Grey Literature in Core Veterinary
Medicine Journals, Journal of the Medical
Library Association 91 no. 4 (October 2003) In
press.
102Indexing and Abstracting Services
- SIGLE System for Information on Grey Literature
- National Technical Information Service (NTIS)
- PsychINFO (Psychological Abstracts)
103SIGLESystem for Information on Grey Literature
- Grey literature documents covered by SIGLE are
technical or research reports, preprints,
committee reports, working papers, dissertations,
conference papers, discussion and policy papers,
government reports, market surveys, etc.
http//www.kb.nl/infolev/eagle/frames.htm
104SIGLESystem for Information on Grey Literature
- No. of Records Category
- 4,158 Aeronautics
- 17,044 Agriculture, plant veterinary sciences
- 17,668 Environmental pollution, protection
control - 256,657 Humanities, psychology social
sciences - 81,269 Biological medical sciences
- 25,089 Chemistry
http//www.kb.nl/infolev/eagle/frames.htm
105NTIS (National Technical Information Service)
The NTIS Database produced by the National
Technical Information Service, is the preeminent
resource for accessing the latest U.S.
government-sponsored research and worldwide
scientific, technical, engineering, and
business-related information.
http//www.csa2.com/csa/factsheets/ntis.shtml
- NTIS Database provides bibliographic data and
abstracts of unclassified and publicly available
information from research reports, journal
articles, data files, computer programs and audio
visual products, from U.S. and non-U.S.
governmental, organizational, and commercial
sources
106Subject Coverage
Administration and Management Aeronautics Aerodynamics Agriculture Behavior Society
Business Chemistry Communications Computer Science
Education Energy Engineering Environmental Sciences
Health Care International Trade Library Information Science Materials Sciences
Mathematical Sciences Natural Resources Earth Sciences Nuclear Science Physics
Regulations Technology Tele-communications Transportation
107PsycINFO
PsycINFO provides access to international
literature in psychology and related disciplines.
Unrivaled in its depth of psychological coverage
and respected worldwide for its high quality, the
database is enriched with literature from an
array of disciplines related to psychology such
as psychiatry, education, business, medicine,
nursing, pharmacology, law, linguistics, and
social work.
http//www.csa2.com/csa/factsheets/psycinfo.shtml
PsycINFO includes psychological research and its
applications the database is of prime relevance
to many industries and research establishments
worldwide. The sources include over 1,400
professional journals, chapters, books, reports,
theses and dissertations, published
internationally.
108Subject Coverage
Applied Psychology Communication Systems Developmental Psychology Educational Psychology
Experimental Psychology Personality Psychological and physical disorders Professional personnel issues
Physiological Psychology and Neuroscience Psychometrics And Statistics Social Psychology Treatment and Prevention
109DATABASE COVERAGE AND SIZE
Database Coverage Size
SIGLE 1976 Present 781,410 records (November 2002)
NTIS 1964 - Present 2,168,400 records (October 2001)
PsycINFO 1872- Present 1,870,180 records (September 2002)
110http//www.neci.nec.com/lawrence/papers/online-na
ture01/
111Conference papers are typical gray/grey
literature!
112EPrints are Gray/Grey Literature
- Daniela Luzi (1998) E-Print Archives a New
Communication Pattern for Grey Literature,
Interlending Document Supply 26 no. 3 (1998)
130-139.
113Gray/Grey Literature
- Its good enough,
- its smart enough,
- and
- doggone it, people use it!
With apologies to Stuart Smalley
114http//www.openarchives.org/Register/BrowseSites.p
l
115(No Transcript)
116(No Transcript)
117(No Transcript)
118June 2003
http//software.eprints.org/
119http//eprints.anu.edu.au/
120http//caltechcstr.library.caltech.edu/
121http//eprints.lub.lu.se/
122http//ndltdpapers.dlib.vt.edu9090/
123http//dspace.org/index.html
124http//rocky.dlib.vt.edu/etdunion/cgi-bin/browse.
pl
125http//www.ncstrl.org/
126(No Transcript)
127Quality Control in Scholarly Publishing on the
Web
- "Most of the high quality materials on the Web
are not peer-reviewed and much of the
peer-reviewed literature is of dubious quality. - William Y. Arms, "What Are the Alternatives to
Peer Review? Quality Control in Scholarly
Publishing On The Web." - Journal of Electronic Publishing, 8 no. 1 (August
2002). - http//www.press.umich.edu/jep/08-01/arms.html
128http//www.update-software.com/Cochrane/MR000016.p
df
129Cochrane Methodology Review
- Despite its widespread use and costs, little hard
evidence exists that peer review improves the
quality of published biomedical research. - There had never even been any consensus on its
aims and that it would be more appropriate to
refer to it as competitive review.
Caroline White, Little Evidence for
Effectiveness of Scientific Peer Review, BMJ
326 (February 1, 2003) 241 http//bmj.com/cgi/rep
rint/326/7383/241/a.pdf
130Cochrane Methodology Review
- On the basis of the current evidence, the
practice of peer review is based on faith in its
effects, rather than on facts,' state the
authors, who call for large, government funded
research programmes to test the effectiveness of
the classic peer review system and investigate
possible alternatives.
Caroline White, Little Evidence for
Effectiveness of Scientific Peer Review, BMJ
326 (February 1, 2003) 241 http//bmj.com/cgi/rep
rint/326/7383/241/a.pdf
131Cochrane Methodology Review
- The use of peer-review is usually assumed to
raise the quality of the end-product (i.e. the
journal or scientific meeting) and to provide a
mechanism for rational, fair and objective
decision-making. However, these assumptions have
rarely been tested.
Tom O. Jefferson, Phil Alderson, Frank Davidoff,
and Elizabeth Wager, Editorial Peer-review for
Improving the Quality of Reports of Biomedical
Studies. (Middle Way, Oxford Update Software
Ltd, 2003). http//www.update-software.com/Cochran
e/MR000016.pdf
132Cochrane Methodology Review
- The available research has not clearly identified
or assessed the impact of peer-review on the more
important outcomes (importance, usefulness,
relevance, and quality of published reports) - Given the widespread use of peer-review and
its importance, it is surprising that so little
is known of its effects
Tom O. Jefferson, Phil Alderson,Frank Davidoff,
and Elizabeth Wager, Editorial Peer-review for
Improving the Quality of Reports of Biomedical
Studies. (Middle Way, Oxford Update Software
Ltd, 2003). http//www.update-software.com/Cochran
e/MR000016.pdf
133Royal Society
http//www.biomedcentral.com/news/20030203/04/
134UNCITEDNESS
David P. Hamilton, "Publishing by and for? --
the numbers, Science (New Series) 250 (4986)
(December 7 1990) 1331-1332. http//www.garfield.
library.upenn.edu/papers/hamilton1.html
135David P. Hamilton,Research Papers Whos Uncited
Now?, Science (New Series) 251 (4989) (January
4, 1991) 25 http//garfield.library.upenn.edu/pap
ers/hamilton2.html
136David P. Hamilton,Research Papers Whos Uncited
Now?, Science (New Series) 251 (4989) (January
4, 1991) 25 http//garfield.library.upenn.edu/pap
ers/hamilton2.html
137REJECTED CITATION CLASSICS
- NOBEL PRIZE RESEARCH
- Severo Ochoa
- Polynucleotide phosphorylase
- Hans Krebs
- Citric acid cycle
- Rosalind Yalow
- Radioimmunoassay
- Harmut Michel
- Photosynthetic processes
Juan Miguel Campanario, Commentary On
Influential Books and Journal Articles
Initially Rejected Because of Negative Referees
Evaluations, Science Communication 16 no. 3
(March 1995) 306-325
138PEER REVIEW PURPOSES
Peer review helps to ensure that published
research is
Important ? Original ?
Timely ? Technically-reliable ?
Internally consistent ? Well-presented ?
Benefited from guidance by experts ?
Carin M. Olson, Peer Review of the Biomedical
Literature, American Journal of Emergency
Medicine 8 no.4 (July 1990) 356-368.
139FILTERING (1)
Researchers look at certain types of
electronic publications because, despite being
tentative, may be relevant to their work.
Researchers are expected to do their own
downstream-filtering of relevant information,
which in the electronic world can be facilitated
by providing meta-information.
140FILTERING (2)
- Some have expressed the concern that having
non-peer reviewed documents with peer-reviewed
documents on the same server would contaminate
the latter and compromise its quality - Readers could have trouble in distinguishing the
different sections - Making non-peer-reviewed as well as
peer-reviewed material will confuse both
scientists and the public .
141FILTERING (3)
- UpStream / DownStream
- However, this perhaps belittles the ability of
scientists to recognize different levels of
evidence and to be able to interpret quality
labels that could make clear that certain
materials is non-peer-reviewed content after all
this is the age of transparency rather than
paternalism .
Gunther Essenbach, The Impact of Preprint
Servers and Electronic Publishing on Biomedical
Research, Current Opinion in Immunology 12 no.
5 (October 2000) 499-503.
http//yi.com/home/EysenbachGunther/scans/Eysenbac
h2000e_CurrOpImmunol_preprint_servers.pdf
142INVISIBLE HAND OF CLASSICAL PEER REVIEW
- The refereed journal literature needs to be
freed from both paper and its costs, but not from
peer review, whose invisible hand is what
maintains its quality. - Stevan Harnad
http//www.presidentmoron.com
143INVISIBLE HAND OF CLASSICAL PEER REVIEW
144INVISIBLE HAND OF CLASSICAL PEER REVIEW
Human nature being what it is, it cannot be
altogether relied upon to police itself.
Individual exceptions there may be, but to treat
them as the rule would be to underestimate the
degree to which our potential unruliness is
vetted by collective constraints, implemented
formally.
Stevan Harnad, The Invisible Hand of Peer
Review, Exploit Interactive no. 5 (April 2000).
http//www.exploit-lib.org/issue5/peer-review/
145INVISIBLE HAND OF CLASSICAL PEER REVIEW
The system is not perfect, but it is what has
vouchsafed us our refereed journal literature to
date, such as it is, and so far no one has
demonstrated any viable alternative to having
experts judge the work of their peers, let alone
one that is at least as effective in maintaining
the quality of the literature as the present
imperfect one is.
146INVISIBLE HAND OF CLASSICAL PEER REVIEW
Remove that invisible constraint -- let the
authors be answerable to no one but the general
users of the Archive arXiv.org (or even its
self-appointed "commentators") -- and watch human
nature take its natural course, standards eroding
as the Archive devolves toward the canonical
state of unconstrained postings the free-for-all
chat-groups of Usenet , that Global Graffiti
Board for Trivial Pursuit -- until someone
re-invents peer review and quality control.
Stevan Harnad, The Invisible Hand of Peer
Review, Exploit Interactive no. 5 (April
2000). http//www.exploit-lib.org/issue5/peer-revi
ew/
147INVISIBLE HANDS
148INVISIBLE HANDS
- Personal reputation
- Institutional reputation
- Pride
- Self-respect
- Professional respect
- Peer pressure
- Critical Peer Response
- Invisible College
- Self-Archiving-Process-Itself
- Open access
- Common Sense
- Self-correcting dynamics
149RECOMMENDATIONSWorkshop on the Open Archives
Initiative (OAI)and Peer Review Journals in
Europe, CERN, Geneva Switzerland,March 22-24,
2001
- It was also generally believed that the
electronic environment allows for novel
approaches to accord a stamp of quality to
scholarly works.
Alison Buckholtz, Raf Dekeyser, Melissa
Hagemann, Thomas Krichel, and Herbert Van de
Sompel, Open Access Restoring Scientific
Communication to Its Rightful Owners, European
Science Foundation Policy Briefing 21 (April
2003) 1-8.
http//www.arl.org/sparc/SPB21_OAI.pdf
150Examples of new metrics that can be extracted
from a fully electronic communication system are
- Usage counts of a work
- Automatically extracted citation information with
a scope beyond the ISI- core journals - Amount of discussion generated by a paper
submitted in a system with open peer review and
peer comment - Etc.
Alison Buckholtz, Raf Dekeyser, Melissa
Hagemann, Thomas Krichel, and Herbert Van de
Sompel, Open Access Restoring Scientific
Communication to Its Rightful Owners, European
Science Foundation Policy Briefing 21 (April
2003) 1-8.
http//www.arl.org/sparc/SPB21_OAI.pdf
151Scientific Publishing as Rhetoric
- The problems with peer review become evident
once the fact that science has a rhetorical
element is accepted. - On the one hand, the traditional mode of peer
review obscures the problems of reference and the
rhetorical dimension of science. The rhetorical
process which is at the heart of science and peer
review conveniently disappears with the final
publication of the manuscript. In its place is an
ideal typical representation (the scientific
paper) of the realist assumptions about empirical
reference. All the academic world sees is a
polished manuscript where the personal
involvement of the researcher and reviewers has
been systematically eliminated.
Mike Sosteric, Interactive Peer Review A
Research Note, Electronic Journal of Sociology 2
no. 1 (1996). http//socserv.socsci.mcmaster.ca/EJ
S/vol002.001/SostericNote.vol002.001.html
152IDEAL SPEECH SITUATION
Jürgen Habermas
- A theoretical construct that describes the ideal
type of interpersonal interaction that should
exist in a rhetorical situation.
153- IDEAL SPEECH SITUATION
- the ideal speech situation permits each
interlocutor an equal opportunity to initiate
speech - there is mutual understanding between
interlocutors - there is space for clarification
- all interlocutors are equally free to use of any
speech act - there is equal power over the exchange.
- Applied in the context of peer, the Ideal Speech
Situation would permit unimpeded authorial
initiative, endless rounds of give and take,
and unchecked openness among authors, editors,
and referees.
Mike Sosteric, Interactive Peer Review A
Research Note, Electronic Journal of Sociology 2
no. 1 (1996). http//socserv.socsci.mcmaster.ca/EJ
S/vol002.001/SostericNote.vol002.001.html
154CONTINUA
- Continuum of PUBLICATION
- (Scholarly Skywriting)
-
- WEAK
-
- MEDIUM
- Continuum of REVIEW
- (Scholarly Skyreading)
STRONG
155OPEN ACCESS and OPEN RETRIEVAL without OPEN
USE
- Incongruent
- Contradictory
156Three-Legged Stool
- ACCESS -- RETRIEVAL -- USE
157(No Transcript)
158ACCESS
159RETRIEVAL
- OPEN ARCHIVES INITIATIVE FOR METADATA HARVESTING
160USE
161- ACCESS -- RETRIEVAL -- USE
162Lots of Alternative ModelsProvide Sensible
Solutions
163Four-Legged Stool
- ACCESS -- RETRIEVAL -- USE -- NAVIGATION
164INFORMATION OVERLOAD
165OAIster
- A search engine for freely available,
difficult-to-access, academically-oriented
digital resources that are OAI -compliant
http//oaister.umdl.umich.edu/
166(No Transcript)
167University of Michigan Digital Library
Production Service
- institutional repositories
- departmental repositories
- e-Journal collections
- technical reports
- dissertations and theses
- discipline eprint collections
- working papers
- Internet resources
- audio
- video
- images
168(No Transcript)
169Cognitive Psychology
170(No Transcript)
171Open Archives Initiative Protocol for Metadata
Harvesting(OAI-PMH)
ADD
METADATA ELEMENT for QUALITY
172Within the framework of OAI, there is a need for
a new protocol for certification. There was
strong support for the extension of the usage of
the OAI protocol beyond discovery-related metadata
. Given the focus of the 1st OAI workshop on
peer review, concrete actions were suggested to
address the exchange of certification-related meta
data using the OAI protocol in a trusted
environment.
Alison Buckholtz, Raf Dekeyser, Melissa
Hagemann, Thomas Krichel, and Herbert Van de
Sompel, Open Access Restoring Scientific
Communication to Its Rightful Owners, European
Science Foundation Policy Briefing 21 (April
2003) 1-8.
http//www.arl.org/sparc/SPB21_OAI.pdf
173QUALITY METADATA (1)
ltoai-qualitygt ltcategorygtinternallt/categorygt ltproc
essgt peer review lt/processgt ltorganizationgt CERN lt/
organizationgt ltpoliciesgt http//www.cern.ch/polici
es/review.html lt/policiesgt lt/oai-qualitygt
William Y. Arms, Quality Control in Scholarly
Publishing On The Web. What Are the Alternatives
to Peer Review? PowerPoint presentation given at
Workshop on the Open Archives Initiatives (OAI)
and Peer Review Journals in Europe, March 22-24,
2003, CERN, Geneva, Switzerland http//www.cs.corn
ell.edu/wya/papers/CERN-2001.ppt
174QUALITY METADATA (2)
ltoai-qualitygt ltcategorygtinternallt/categorygt ltrati
nggt lt/ratinggt ltorganizationgt 42 lt/organizati
ongt ltpoliciesgt http//www.cern.ch/policies/review.
html lt/policiesgt lt/oai-qualitygt
175(No Transcript)
176CERTIFICATION SERVICES
Faculty of 1,000,000
177CERTICATION SERVICES
- New roles for Indexing and Abstracting Services
- Expanded Role for Learned and Professional
Societies - Establishment of Formal/Commercial Reviewing
Services
178PEER REVIEW STRENGTHS
The underlying strength of peer review isthe
concerted effort by large numbers of researchers
and scholars who work to assure that valid and
valuable works are published and conversely to
assure that invalid or non-valuable works are not
published .
Anne C. Weller, Editorial Peer Review Its
Strengths and Weaknesses. (Medford, NJ
Information Today, 2001).
179Faculty of 1000 / BioMed Central
- BioMed Central (biomedcentral.com) publishes
Faculty of 1000 (F1000), the leading literature
evaluation service and new online research tool
that highlights the most interesting papers in
biology, based on the recommendations of over
1000 leading scientists. F1000 is managed by
scientists for scientists . and provides a
rapidly updated consensus map of the important
papers and trends across biology.
www.facultyof1000.com
180Faculty of 1000 / BioMed Central
- Among its many benefits, F1000
- systematically organizes and evaluates the mass
of information within scientific literature - provides scientists with a continuously updated
insider's guide to the most important papers
within any given field of research - highlights papers on the basis of their
scientific merit rather than the journal in which
they appear - offers the researcher a consensus of
recommendations from well over 1000 leading
scientists and, - offers an immediate rating of individual papers
by the authors' peers, and an important
complement to the indirect assessment provided by
the journal impact factor.
181Faculty of 1000 / BioMed Central
- Within the F1000, the entire field of biology
is divided into 16 subject areas
(Faculties)(e.g., Biochemistry,, Cell
Biology, Microbiology). Each Faculty is
subdivided into three (3) to twelve (12)
Sections, (e.g., Biochemistry Biocatalysis,
Molecular evolution, Protein folding), with each
section comprised of between 10 to 50 faculty
members. F1000 seeks to invite the best
internationally known scientists in each
represented field and to involve both experienced
and younger investigators.
182Peer review is a quality-control and
certification (QC/C) filter necessitated by the
vast scale of learned research today. Without it,
no one would know where to start reading in the
welter of new work reported every day, nor what
was worth reading, and believing, and trying to
build ones own further research upon.
- Stevan Harnad, Free at Last The Future of
Peer-Reviewed Journals, - D-Lib Magazine 5 no. 12 (December 1999)
- http//www.dlib.org/dlib/december99/12contents.htm
l
183SEIZE THE E!
- Embrace the potential of the digital environment
to facilitate access, retrieval, use, and
navigation of electronic scholarship.
184OPEN NAVIGATION
185New Age Navigation
- Innovative Interfaces for Electronic Journals
- Gerry McKiernan
- The Serials Librarian
- Fall 2003
186SUMMARY. While it is typical for electronic
journals to offer conventional search features
similar to those provided by electronic
databases, a select number of e-journals have
also made available higher-level access options
as well. In this article, we review several novel
technologies and implementations that creatively
exploit the inherent potential of the digital
environment to further facilitate use of
e-collections.
- Gerry McKiernan, New Age Navigation Innovative
Interfaces for Electronic Journals, - The Serials Librarian, Fall 2003.
http//www.coleonline.us/serialslibrarian/
187http//www.highwire.org
188Topic Map
189Topic Map
190Topic Map
191Topic Map
192Topic Map
Secondary Screen
193Topic Map
194Topic Map
TopicMap is based on the Hyperbolic Tree SDK for
Java, licensed from Inxight Software, Inc., a
spin-off company from Xerox PARC, and leading
provider of Unstructured Data Management
solutions for accessing, analyzing and delivering
information.
http//www.inxight.com
195Semio
Automated categorization software technology
http//www.entrieva.com/
196Kohonen Self-Organizing Maps
http//websom.hut.fi/websom/milliondemo/html/root.
html
197Kohonen Self-Organizing Maps
http//websom.hut.fi/websom/milliondemo/html/1_cx5
.html
198Kohonen Self-Organizing Maps
http//websom.hut.fi/websom/milliondemo/html/2_gx1
0.html
199Kohonen Self-Organizing Maps
http//websom.hut.fi/websom/milliondemo/getnd.cgi?
32323
200Kohonen Self-Organizing Maps
http//websom.hut.fi/websom/cgi-bin/getfile.cgi/co
mp.ai/39340
201http//www.springer.de/books/toc/3540679219-c.pdf
202There are some excuses, but at the bottom it
will be seen to be the sluggishness of human
nature and its superstitious cleavage to old
habits.
- Stevan Harnad, Free at Last The Future of
Peer-Reviewed Journals, - D-Lib Magazine 5 no. 12 (December 1999)
- http//www.dlib.org/dlib/december99/12contents.htm
l
203 In the digital world, the evaluation process
stands ready to be reinvented in a clear,
rational way by the relevant research communities
themselves.
- Jean-Claude Guédon,
- In Oldenburgs Long Shadow Librarians, Research
Scientists, Publishers, and the Control of
Scientific Publishing. - (Washington, D.C. Association of Research
Libraries, 2001), 54.
http//www.arl.org/arl/proceedings/138/guedon.html
204lt/ENDQUOTEgt
- The Medium is the Message
- And
- the Method.
- With apologies to Marshall McLuhen
205 THANK YOU FOR YOUR ATTENTION!
206Alternative Peer ReviewQuality Management for
21st Century Scholarship
- Gerry McKiernan
- Science and Technology Librarian
- and Bibliographer
- Iowa State University Library
- Ames IA
- USA
gerrymck_at_iastate.edu
207OPEN MIND
208REVISED VERSION 1.01