Ginny Barbour

1
Measuring impact
Ginny Barbour Chief Editor PLoS
Medicine vbarbour_at_plos.org www.plosmedicine.org
2
Outline

• The impact factor
• and all its problems
• Other measures of impact
• What next?

3
Published articles serve two purposes

• Dissemination of research results
• The key measure of research output

4
Who cares about impact?
5
Types of impact

• New knowledge
• Opening new research agenda
• insight
• method
• Practical applications
• medicine
• agriculture
• conservation
• Journals can have other types of impact

6
How can impact be measured?

• Where the work is published
• Citations
• Web usage
• Expert rating
• Community rating
• Media/blog coverage
• Policy development

7
How is impact measured?
The Impact factor
8
The impact factor

• Introduced in 1950s by Eugene Garfield

Number of citations in 07 to articles published
in 05/06
Number of (some) articles published in 05/06
9
The impact factor
citations
articles
2006 and 2005
2007
IF5 Articles published in 05/06 were cited an
average of 5 times in 07.
10
Whats the problem?
Over-interpretation We judge the worth of a
paper on the basis of the impact factor of the
journal in which it was published.
11
Perverse effects

• Publishing is inefficient
• researchers compete for places in the elite
  journals
• Editorial policy is influenced by IF
• journals compete for the most citable articles
• other strategies to boost IF
• Distortion of research
• selection for certain types of researchers
  (Lawrence)

12
More problemsThe denominator problem

• Journals publish many types of articles
• Thomson counts citations to all articles in a
  journal
• But only some articles are considered citable
• Thomson does not explain the basis for this
  decision
• Example PLoS Medicine

13
More problems (contd)

• Citation behaviour in different fields
• Thomson index part of literature
• English language
• decisions can take ages
• Skewed distribution
• 89 of Natures 2004 IF generated by 25 of
  articles (Campbell)
• Problems with the data
• ill-defined and manifestly unscientific number
  (Rossner et al)

14

• Unreliable dataFlawed methodologyOver-interpre
  tation

15
Source Thomson, 2007 JCR Science Edition
16
Source Thomson, 2007 JCR Science Edition
17
Source Thomson, 2007 JCR Science Edition
18
Source Thomson, 2007 JCR Science Edition
19
Source Thomson, 2007 JCR Science Edition
20
Source Thomson, 2007 JCR Science Edition
21
Article-level metrics
22
How can impact be measured?

• Where the work is published
• Citations
• Web usage
• Expert rating
• Community rating
• Media/blog coverage
• Policy development

23
(No Transcript)
24

• Citation counts
• 36 in CrossRef
• 65 in Scopus
• 74 in Web of Science

25

• Why more?
• Journals not indexed by other services
• Books, theses, grey literature
• Errors

26
Web usage statistics

• What to measure?
• downloads
• data standards (COUNTER)
• Who is the user?
• machines or people
• How to capture all usage
• proxies, caching
• multiple sites (Pubmed Central, institutions)
• Gaming

27
(No Transcript)
28
(No Transcript)
29
(No Transcript)
30
(No Transcript)
31
(No Transcript)
32
(No Transcript)
33
(No Transcript)
34
(No Transcript)
35

• Article Scorecard (whats this?)
• Formal Citations
• Google Scholar (56)
• Scopus (23)
• Web usage (data)
• Moderate High Extreme
• Expert rating
• F1000 Factor 3.0
• User rating -
• Science Media/blog coverage
• Google News (5)
• Trackbacks (1)

Source Thomson, 2007 JCR Science Edition
36
Authority 3.0

• Prestige of the publisher (if any).
• Prestige of commenters/users
• Percentage of a document quoted in other
  documents.
• Raw links to the document.
• Valued links
• Obvious attention discussions in blogspace,
  comments etc
• Language in comments positive, negative,
  clarified, reinterpreted.
• Quality of author's institutional affiliation(s).
  
• Significance of author's other work.
• Amount of author's participation in other valued
  projects.
• Reference network the significance of all the
  texts cited.
• Length of time a document has existed.
• Inclusion of a document in lists of "best of," in
  syllabi, indexes, etc
• Types of tags assigned to it
• Authority of the taggers, the authority of the
  tagging system.

Michael Jensen
37
What do we need to develop?

• Better ways of measuring article-level indicators
• Citations
• Usage in its widest meaning

38
Citations

• Desired characteristics
• Accuracy, comprehensiveness, timeliness,
  objectivity, simplicity, transparency
• Ideally from an open dataset
• Google Scholar
• CrossRef
• Pubmed Central
• ?a new one

39
Usage

• COUNTER Journal Reports
• established standards for journal-level reports
• COUNTER Article-level usage
• developing a code of practice
• combining usage at publisher and PMC

40
What other measures are there?

• H-factor
• Scimago

41
References

• Campbell, P. Escape from the impact factor.
  Ethics in Science and Environmental Politics. Doi
  10.3354/esep00078
• Adler, R., Ewing, J. Taylor, P. Citation
  statistics. A report from the International
  Mathematical Union. http//www.mathunion.org/publi
  cations/report/citationstatistics/
• Rossner, M., van Epps, H., Hill, E. Show me the
  data. JCB 179, 1091-1092
• Lawrence, P. Lost in publication how measurement
  harms science. Ethics in Science and
  Environmental Politics. Doi 10.3354/esep00079
• Jensen, M. Chronicle of Higher Education. The
  New Metrics of Scholarly Authority. Volume 53,
  Issue 41, Page B6
• Taraborelli, D. Soft peer review Social software
  and distributed scientific evaluation.
  Proceedings of the 8th International Conference
  on the Design of Cooperative Systems (COOP 08),
  Carry-Le-Rouet, May 20-23, 2008
• The PLoS Medicine Editors The Impact Factor Game
  PLoS Medicine Vol. 3, No. 6, e291
  doi10.1371/journal.pmed.0030291