NIH Peer Review: Continuity and Change

1
NIH Peer Review Continuity and Change
Toni Scarpa
Center for Scientific Review National Institutes
of Health Department of Health and Human Services
APS/Cardiovascular Section Salt Francisco, April
2006
2
Peer Review An N.I.H. Conception

• Is the heart and soul of NIH
• Has produced an effective partnership between the
  federal government and research institutions
• Has created the best academic medical centers,
  the best biomedical/behavioral research and
  biotechnology
• Has made possible the best cures and the best
  prevention
• Has been admired and imitated here and abroad
• Has protected NIH against outside influence

3
Center for Scientific Review
4
Applications received for all of NIH and
applications referred for CSR review, FY 1998-2004
80,000
60,000
Number of applications
40,000
20,000
0
1998
2000
2002
2004
Fiscal year
Applications received for all of NIH
Applications assigned for review by CSR
5
CSR 4 Review Divisions 23 IRGs
Division of Biologic Basis of Disease Elliot
Postow, Ph.D.
Division of Molecular and Cellular
Mechanisms Donald Schneider, Ph.D.
Division of Clinical and Population-Based
Studies Anita Miller Sostek, Ph.D
Division of Physiology and Pathology Michael
Martin, Ph.D.
Bioengineering Sciences and Technologies IRG
(BST) Sally Amero, Ph.D.
Cardiovascular Sciences IRG (CVS) Joyce Gibson,
D.Sc.
Biology of Development and and Aging
(BDA) Sherry Dupere, Ph.D.
Digestive Sciences IRG (DIG) Mushtaq Khan,
Ph.D., DVM
Brain Disorders and Clinical Neuroscience IRG
(BDCN) Dana Plude, Ph.D.
Biological Chemistry and Macromolecular
Biophysics IRG (BCMB) John Bowers, Ph.D.
Hematology IRG (HEME) Joyce Gibson, D.Sc.
Health of the Population IRG (HOP) Robert
Weller, Ph.D.
Integrative, Functional and Cognitive
Neuroscience IRG (IFCN) Christine Melchior, Ph.D.
Cell Biology IRG (CB) Marcia Steinberg, Ph.D.
Risk, Prevention, and Health Behavior IRG
(RPHB) Michael Micklin, Ph.D.
Musculoskeletal, Oral, and Skin Sciences IRG
(MOSS) Daniel McDonald, Ph.D.
Genes, Genomes, and Genetics IRG (GGG) Richard
Panniers, Ph.D.
Surgery, Radiology, and Bioengineering IRG
(SRB) Eileen Bradley, D.Sc.
Renal and Urological Sciences IRG (RUS) Daniel
McDonald, Ph.D.
Molecular, Cellular and Developmental Neuroscienc
e IRG (MDCN) Carole Jelsema, Ph.D.
Respiratory Sciences IRG (RES) Mushtaq Khan,
Ph.D., DVM
6
CSR Mission Statement

• To see that NIH grant applications receive fair,
  independent, expert, and timely reviews -- free
  from inappropriate influences -- so NIH can fund
  the most promising research.

7
Necessary Changes in CSR Peer Review Operations
CSR Operations Current Systems
New Systems?
Complexity and Impact
Time
8
Changes in CSR Operations

• Increase communications between CSR, the ICs, our
  reviewers and applicants
• Increase uniformity
• Increase efficiency
• Facilitate work of IC program staff

9
Potential of Knowledge Management Tools for Peer
Review

• Collexis Software or Others
• Knowledge management solutions
• Fingerprinting and text retrieving
• Disease coding
• Benefits for Peer Review
• Assigning applications to Integrated Review
  Groups or Study Sections
• Selecting reviewers (one application, multiple
  applications)
• Nine pilots are underway to begin to assess these
  benefits

10
Study Section Realignment

• Review of one IRG every month
• Total review every 2 years

11
Required Changes in Current Systems

• Shorten the review cycle

12
This is Not an Ford Assembly Line
Evaluate Scientific Merit of Applications
Receipt
Refer
EnterpriseArchitecture_at_mail.nih.gov
13
Shortening the NIH Review Cycle, Initial Steps

• For most research grants, we are posting summary
  statements within one month after the study
  section meeting instead of two to three months
  after the meeting (effective Oct 05)
• We are conducting a pilot study to speed the
  review process for new investigators so they may
  revise and resubmit for the very next review
  cycle 4 months earlier than before (effective Feb
  06)

14
Desirable Changes in CSR Review

• Shorten the review cycle
• Address concern that clinical research is not
  properly evaluated
• Improve the assessment of innovative, high-
  risk/high-reward research

15
The judging of grants has become a charade.
The American Society for Cell Biology
The judging of grants has become a charade. To
be funded, the experimental plan has become a
litany of experiments already accomplished so
that everything is feasible. When grants come
back with unfundable scores, new investigators
may not have sufficient resources to do the
experiments that show feasibility. Zena
Werb President, ASCB
Newsletter August 2005
16
Possible Changes in Current Systems

• Shorten the review cycle
• Address concern that clinical research is not
  properly evaluated
• Improve the assessment of innovative, high-
  risk/high-reward research
• Do more to recruit and retain more high-quality
  reviewers

17
Expanding Peer Reviews Platforms

• Electronic Reviews
• Telephone Enhanced Discussions
• Video Enhanced Discussions
• Asynchronous Electronic Discussions

Study Sections
Necessity ? Clinical
reviewers Preference ?
Physicists, computational biologists New
Opportunities ? Fogarty, International
Reviewers
18
The First NIH Study Section
1945
19
Possible New Systems
If we didnt have any peer-review system and we
had to design one from scratch, what would it
look like?
20
(No Transcript)
21
QuestionsApplications

• Should we have 3 or 365 deadlines for most
  applications?

• Should applications (Rs) be shortened? Should
  appendices be eliminated or reduced in size?

• Is there more value in having 2-3 reviewers
  reading 25-page applications or 10-15 reviewers
  reading 5-page applications?

22
QuestionsStudy Section Meetings

• What is the ideal number of members to have serve
  on a study section?

• Is one study section with 50-70 reviewers
  efficient?

• What is the intellectual contribution of
  individual reviewers in large study sections?

23
QuestionsScoring

• Is it proper or desirable to have 50-70 reviewers
  voting on priority scores for each application
  referred to their study section?

• Is consensus always good? Or should we focus on
  score variance?

24
Applications Received for All of NIH FY 1998-2004
80,000
60,000
Number of applications
40,000
20,000
0
1998
2000
2002
2004
Fiscal year
25
Number of Research Grant Applications/Applican
t
26
CSR Applications Reviewed, Regular and SEP
May Council Only
27
Study Section Application/Reviewer
Ratio October Council Only
28
Two groups of challenges/opportunities

• Reviewers
• Decrease the number of reviewers and increase
  the quality
• Increase the number of applications reviewed
  without extra workload
• Recruit and retain the best reviewers
• SRA
• Increase efficiency
• Recruit and train

29
Possible Short Term Approaches for Increasing
Efficiency for Reviewers and CSR

• Replace Many SEPs with Smaller Parallel Study
  Sections
• Enlarge Study Section Membership and Decrease
  Frequency of Participation
• Pre Meeting Streamlining
• Various Review Platforms
• Hybrid Review Platforms
• Staggering Application Deadlines
• 2 instead of 3 reviews
• Shorten Applications
• More Structured Applications and Reviews

30
Size of Grant Applications

• RO1
• Will increase number of applications reviewed by
  reviewers
• Will decrease the number of reviewers in a study
  section
• May be combined with a change in format of the
  application, more consonant to review criteria
• May be combined with scoring individual criteria
  (i.e. relevance, innovation, etc
• Strong support by councils and scientific
  leadership
• May result in more innovation

31
This is CSR
32
Coronary Heart Disease Age-Adjusted Death Rates
in U.S. Actual (blue) vs Expected (yellow)
33
The True Value of Peer Review