Evaluation of Process, Value, and Influence of Scholarly Book Reviews in LIS Journals

1
Evaluation of Process, Value, and Influenceof
Scholarly Book Reviews in LIS Journals

• Jeppe Nicolaisen
• jni_at_db.dk
• Department of Information Studies
• Royal School of Library and Information Science

2
Agenda

• Motivations and objectives of the research
  project
• Research questions
• Organizational framework
• The scholarliness of published peer reviews
• Scholarly communication and strategic research
  materials
• Results of a bibliometric study of book reviews
  in selected social
• science fields

3
Motivations and objectives

• The secrecy involved in so much
• of the peer review process
• precludes a comprehensive,
• systematic study of the subject.
• Gordon Moran (1998)

4
Motivations and objectives (2)

• Peer review is the process whereby authorities in
  a given field determine the validity and assess
  the relative significance of a particular
  contribution of a scholar or scientist within
  that field.
• Editorial refereeing
• Evaluation of grant applications
• Awards
• Promotions
• Hiring
• Book reviews
• etc.

5
Motivations and objectives (3)

• The peer review process is criticized for lack of
  reliability and validity
• The process is influenced by socio-psychological
  mechanisms, which make the process unreliable.
• Predicting the eventual importance of scientific
  work would require one to predict an
  unpredictable future nothing forces the
  conclusion that it would be the least bit foolish
  to make many of our choices by drawing lots.

6
Motivations and objectives (4)

• A large-scale investigation of the peer review
  process in LIS.
• Reliability
• Validity
• Scholarly book reviews Published peer reviews.
• Published peer reviews ? The peer review process.
• Scholarly book reviews in LIS journals ? The peer
  review process in LIS.

7
Motivations and objectives (5)

• Bornsteins hypothesis
• If a relationship between citation frequency and
  research quality does exist, this relationship is
  not likely to be linear. The relationship between
  research quality and citation frequency probably
  takes the form of a J-shaped curve, with
  exceedingly bad research cited more frequently
  than mediocre research (e.g. as an example of an
  idea or line of research that turned out to be a
  blind alley, or as an example of what not to do
  in a particular area).
• Bornstein (1991)

8
Citation- frequency
Peer review
9
Research questions

• To what extent is the evaluative content of
  scholarly book reviews influenced by
  socio-psychological mechanisms?
• To what extent is the evaluative content of
  scholarly book reviews valid?
• What is the relationship between peer review and
  citations?
• How, and to what extent does scholarly criticism
  contribute to shape the relationship between peer
  review and citations?

10
Organizational framework

• 6 research papers
• The J-shaped distribution of citedness, Journal
  of Documentation, Vol. 58 No. 4, in press.
• Structure-based interpretation of scholarly book
  reviews a new research technique, Proceedings
  of the Fourth International Conference on
  Conceptions of Library and Information Science,
  in press.
• The scholarliness of published peer reviews a
  bibliometric study of book reviews in selected
  social science fields.

11
Organizational framework (2)

• The reliability of book reviews in LIS journals
• The validity of book reviews in LIS journals
• The relationship between peer review and
  citations
• Final thesis

12
The scholarliness of published peer reviews

• Book reviews serve a number of vital functions
• They are widely used to help estimate the
  quality and importance
• of books published by academicians and thus
  to help make
• decisions about hiring, promotions and salary
  increases (Glenn,
• 1978)
• Developing library and information center
  collections
• Allow members of academic communities to keep
  up with the
• latest professional progress despite of the
  eternal growth and
• dissemination of recorded knowledge

13
The scholarliness of published peer reviews (2)

• Book reviews differ in regard to their
  prototypicality
• Some are short, while others are longer. Some are
  more focused on providing a general view of how
  the book is organized rather than an actual
  evaluation. Some reviewers insert the book in the
  field while others concentrate on making topic
  generalizations or inform about the author or
  potential readership.
• What are the characteristics of a scholarly
  book review?

14
Scholarly communication

• Scholarly communication is a communicative
  practice anchored in three dimensions (Kling
  McKim, 1999)
• Publicity
• Accessibility
• Trustworthiness

15
Scholarly communication (2)

• A trustworthy book review
• should describe and characterize not only the
  book in question,
• but also the subject with which it is
  dealing.
• should answer the question is the book a real
  addition to our
• knowledge, and if so, what exactly has been
  added?.
• should position the book in the literature
  devoted to the same
• subject and evaluate the new contribution in
  relation to related
• works.
• George Sarton (1960)

16
Strategic research materials

• are the empirical materials that exhibit the
  phenomenon to be explained or interpreted to such
  advantage and in such accessible form that it
  enables the fruitful investigation of previously
  stubborn problems and the discovery of new
  problems for further inquiry
• (Merton, 1987)
• Publication counts
• Citation counts
• Textual indicators
• Colonic titles
• References

17
Strategic research materials(2)

• The practice of citing other works is second
  nature to anyone
• writing a scholarly or scientific paper
  (Kaplan, 1965)
• Explicit references are essential in order to
  communicate
• effectively and intelligently about
  scientific and technical subjects
• (Garfield, 1977)
• The practice of footnoting is as old as
  scholarship itself
• (Price, 1970)

18
Results of a bibliometric study

• Table 1. Definition of six social science fields
  based on ISIs Subject Category assignments

Modified from Glänzel (1996)
19
Results of a bibliometric study (2)

• Table 2. SSCI Search for scholarly book reviews
  in the field of Economics, 1997-2001

20
Results of a bibliometric study (3)

• Table 3. Selected social science fields Number
  of book reviews in six equal sized periods
  1972-2001, number of scholarly book reviews,
  percentage of scholarly book reviews

21
Results of a bibliometric study (4)

• Figure 2. Selected social science fields
  Percentage of scholarly book reviews in six equal
  sized periods
• 1972-1976, 1977-1981, 1982-1986, 1987-1991,
  1992-1996, 1997-2001

22
Results of a bibliometric study (5)

• Table 4. LIS core journals Number of book
  reviews in six equal sized periods 1972-2001,
  number of scholarly book reviews, percentage of
  scholarly book reviews

(White McCain, 1998)
23
Results of a bibliometric study (6)

• Figure 3. LIS journals and LIS core journals
  Percentage of scholarly book reviews in six equal
  sized periods 1972-2001

24
Results of a bibliometric study (7)

• Price (1963) maintains that if any sufficiently
  large segment of science is measured in any
  reasonable way, the normal mode of growth is
  exponential. He even stated that he had no
  hesitation in suggesting it as the fundamental
  law of any analysis of science.
• If science in fact is growing exponentially, then
  we are able to predict the future growth of
  scholarly book reviews in the six fields under
  study.
• Mathematically, the law of exponential growth
  follows from the simple condition that at any
  time the rate of growth is proportional to the
  size of the population or to the total magnitude
  already achieved.

25
Results of a bibliometric study (8)

• Figure 4. Selected social science fields The
  expected exponential growth of scholarly book
  reviews 2002-2086

26
References

• Bornstein, R.F. (1991), The predictive validity
  of peer review a neglected issue, Behavioral
  and Brain Sciences, Vol 14 No. 1, pp. 138-139.
• Garfield, E. (1977), To cite or not to cite a
  note of annoyance, Current Contents, Vol. 35
  (August 29.), pp. 5-8.
• Glänzel, W. (1996), A bibliometric approach to
  social sciences. National research performance in
  6 selected social science areas, 1990-1992,
  Scientometrics, Vol. 35 No. 3, pp.291-307.
• Glenn, N. (1978), On the misuse of book
  reviews, Contemporary Sociology, Vol. 7 No. 3,
  pp.254-255.

27
References (2)

• Kaplan, N. (1965), The norms of citation
  behavior prolegomena to the footnote, American
  Documentation, Vol. 16 No 3, pp.179-184.
• Kling, R.K. McKim, G. (1999), Scholarly
  communication and the continuum of electronic
  publishing, Journal of the American Society for
  information Science, Vol. 50 No. 10, pp. 890-906.
• Merton, R.K. (1987), Three fragments from a
  sociologists notebooks establishing the
  phenomenon, specified ignorance, and strategic
  research materials, Annual Review of Sociology,
  Vol. 13, pp. 1-28.
• Moran, G. (1998), Peer review and academic
  paradigms, Journal of Information Ethics, Vol. 7
  No. 2, pp. 19-29.

28
References (3)

• Price, D.J.S. (1963), Little Science Big Science,
  Colombia University Press, New York, NY.
• Price, D.J.S. (1970), Citation measures of hard
  science, soft science, technology, and
  nonscience, in Nelson, C.E. Pollock, D.K.
  (Eds) Communication Among Scientists and
  Engineers, Heath, Lexington, MA, pp.3-22.
• Sarton, G. (1960), Notes on the reviewing of
  learned books, Science, Vol. 131 (April 22.),
  pp. 1182-1187.
• White, H.D. McCain, K.W. (1998), Visualizing a
  discipline an author co-citation analysis of
  Information Science, 1972-1995, Journal of the
  American Society for Information Science, Vol. 49
  No. 4, pp. 327-355.