History of Priority Scores at the NIH

1
History of Priority Scores at the NIH

• Michael R. Martin, Ph.D.
• Director, Division of Physiology and Pathology
• Center for Scientific Review
• National Institutes of Health
• June 30, 2005

2
(No Transcript)
3
In the Beginning August 1946

• DRG panels patterned on Office of Scientific
  Research and Development /Committee on Medical
  Research review panels
• Panel votes to either approve or disapprove

4
And Then January 1949

• A substantial increase in the number of
  applications and approval ratings leads to a
  priority rating system
• Use a 1 to 5 scale in whole digit intervals,
  along with rating criteria to establish order of
  payment.
• Scores were averaged and multiplied by 100.
• Rating criteria standardized

5
1960s

• 1962 Institute review panels established
• B/I/Ds tested a variety of procedures for rating
  including 3 and 10 point scales, separate ratings
  for scientific merit and rank order, and tenth
  point intervals in priority scores.
• Different DRG study sections began to show
  different scoring patterns, raising questions
  about comparability

6
(No Transcript)
7
1971 Executive Committee on Extramural
Activities Priority Score Review Committee

• Recommend retaining priority score
• Recommend one standard procedure for adjusting
  score suggest normalization or percentiles
• 1972 one year pilot of normalization

8
1973 Normalization Evaluation Committee

• Evaluation finds that normalization is broadly
  accepted, but three ICs object
• Normalization discontinued

9
Grants Peer Review Committee 1976

• Some ICs continue to use normalization and some
  raw scores.
• Recommended a single notation and score
  convention

10
1979 NIH Committee to Study Priority Scores

• Recommended
• Displaying only the normalized score
• Reviewer should use 0.1 point intervals
  primarily because of reviewer insistence and the
  fact it wouldn't make any difference to the
  outcome
• Score should continue to be displayed to three
  digits in part because two would cause
  confusion
• Two ICs object to normalization on the grounds
  that normalization would obliterate the
  distinctions between the more innovative fields
  and those which are less innovative.

11
1980

• Frederickson mandated the use of raw priority
  scores but allowed ICs to develop alternatives.
  Tenth point intervals accepted.
• For FY 1980 Congress mandated a payline of 212,
  thus adjusting downward the budget of some ICs
  that used only normalized scores.

12
Priority Scores in the Early 1980s

• The pressure between number of applications and
  appropriation levels increases.
• Major priority score aberrations begin
• Mean priority score dropped from 250 in 1960s to
  below 200 in 1985
• A and B study sections have very different
  scoring patterns even though they review the same
  types of applications

13
(No Transcript)
14
EPMC Working Group on the Movement of Priority
Scores, 1987

• Recommendation 1 adopt percentiling system wide
• Recommendation 2 return to the convention of
  using 9 steps in the 1 to 5 scale steps of 0.5
• Recommended that two digits be displayed instead
  of three, or three digits with the last digit
  rounded to zero

15
(No Transcript)
16
Committee on Rating of Grant Applications (1996)

• Recommendations
• Applications evaluated on three criterion
• Score on each Criterion
• Reviewers use an eight step scale (0-7) scale
• Scores are averaged and reported as two digits

17
Committee on Rating of Grant Applications (1996)

• PROG and DRGAC discussion
• Propose four criteria instead of three
• Criteria should be specifically addressed in
  critique
• Written critique should drive message, not scores
  on individual criteria

18
Committee on Rating of Grant Applications (1996)

• PROG and DRGAC discussion
• Criterion scores not supported
• Considerable overlap or lack of distinction?
  between the individual criteria
• Correlation between criterion would be very high,
  diminishing the value of an individual score
• Giving individual criteria scores would detract
  from the message in the critique. 

19
Committee on Rating of Grant Applications (1996)

• PROG and DRGAC discussion
• Global score should be retained
• Taps scientific expertise and judgment of
  reviewer
• An algorithm would limit reviewers flexibility
• Criterion weights would change project by project

20
Committee on Rating of Grant Applications (1996)

• PROG and DRGAC discussion
• Cool to reduced rating scale
• Current scheme accepted and understood
• Unscored range reduces the number of bins to less
  than 20
• No evidence that experts cannot distinguish
  between more than 11 bins

21
Committee on Rating of Grant Applications (1996)

• Approved
• Five criterion
• Single global score (1-5 scale in tenths)
• Criterion-based discussion and critiques
• Not Approved
• Scoring by criterion
• Eight interval scale and two integer average

22
Other Studies

• 1974 RAND Corporation study shows that funded
  applications with good scores produced more
  highly cited publications
• 1988 DRG study comparing half point to one tenth
  point priority score intervals with verbal
  descriptors suggests that there is little value
  in the half point scale compared to the tenth
  point scale. Reviewers react very negatively to
  the more course scale.

23
Other Studies

• 1991 DRG study of re-ranking applications at the
  end of a meeting shows little impact on outcome
• 1995 NIGMS begins using rounded priority score.
  Stops after two years because of Advisory
  Committee concerns that valuable information is
  lost.

24
Observations

• Utility and value of priority score method is
  widely recognized.
• Reviewers report being very comfortable with 41
  interval scale.
• Priority scores are reliable indicators of
  relative scientific merit and rank order within a
  study section over time but not between study
  sections.
• Percentile is a valuable tool that has allowed
  cross study section comparisons.

25
(No Transcript)
26
History of Priority Scores at the NIH

• Michael R. Martin, Ph.D.
• Director, Division of Physiology and Pathology
• Center for Scientific Review
• National Institutes of Health
• June 30, 2005