The NIH Peer Review Process

1
The NIH Peer Review Process

• Sally A. Amero, Ph.D.
• NIH Review Policy Officer
• Office of Extramural Research

2008 NIH Regional Seminars
2
The NIH Peer Review Process
National Institutes of Health

• ?Primary Federal agency in the US for conducting
• and supporting medical research
• Executive Branch, White House
• Department of Health and Human Services
• 27 Institutes Centers (ICs)
• Extramural and intramural
• programs
• 24 have funding authority
• Over 18,000 employees

3
The NIH Peer Review Process
NIH Peer Review System

• Two-tiered
• Initial peer review (SRGs, IRGs, study
• sections)
• I/C Advisory Council or Board (Council)
• Mandated by Public Health Service Act

• Per year
• Nearly 80,000 applications
• Over 18,000 reviewers

4
The NIH Peer Review Process
Division of Receipt and Referral (DRR), Center
for Scientific Review (CSR)
Application received by CSR DRR Assignments
made ? ? Initial peer review
Funding considerations SRG (CSR or IC)
IC(s) duals possible ?
? Second level of review National Advisory
Council/Board ? IC Director (funding decisions)
5
The NIH Peer Review Process
NIH Institutes and Centers (ICs)
Office of the Director
National Institute on Alcohol Abuse and Alcoholism
National Institute of Arthritis
and Musculoskeletal and Skin Diseases
National Cancer Institute
National Institute on Aging
National Institute of Child Health and
Human Development
National Institute of Allergy and Infectious
Diseases
National Institute of Diabetes and Digestive
and Kidney Diseases
National Institute of Dental and Craniofacial Rese
arch
National Institute on Drug Abuse
National Institute of Environmental Health
Sciences
National Institute on Deafness and
Other Communication Disorders
National Eye Institute
National Human Genome Research Institute
National Heart, Lung, and Blood Institute
National Institute of Mental Health
National Institute of Neurological Disorders
and Stroke
National Institute of General Medical Sciences
National Institute of Nursing Research
National Library of Medicine
National Center for Complementary and
Alternative Medicine
National Center on Minority Health and Health
Disparities
National Institute of Biomedical Imaging and
Bioengineering
Fogarty International Center
National Center for Research Resources
Division of Receipt and Referral
Lack funding authority
Center for Information Technology
Center for Scientific Review
Clinical Center
6
The NIH Peer Review Process
Writing the Application

• Start early
• Seek advice from colleagues
• Start with a good idea
• Talk to your NIH Program Official(s)
• Use the NIH webpage (www.nih.gov)
• Remember review criteria
• Follow instructions carefully

Transition to Electronic Submission
(http//era.nih.gov/ElectronicReceipt/)
7
The NIH Peer Review Process
Remember Review Criteria

• Presented in the Funding
• Opportunity Announcement (FOA)
• Sent to reviewers with applications
• Guide discussion at the meeting
• Format for critiques in summary statement
• Standardized across NIH by type of
• mechanism

8
The NIH Peer Review Process
Review Criteria

• Standard criteria Factored into priority score
• Significance Research projects and contracts
  
• Approach
• Innovation
• Investigator(s)
• Environment
• Protection of human subjects
• from research risks
• Vertebrate animal welfare
• Biohazards

9
The NIH Peer Review Process
Review Criteria

• Additional considerations
• Not factored into priority score
• Budget
• Foreign institutions
• Resource Sharing

Establishment of Multiple Principal Investigator
Awards for the Support of Team Science
Projects (http//grants.nih.gov/grants/guide/notic
e-files/NOT-OD-07-017.html)
10
The NIH Peer Review Process
Significance
Does this study address an important problem?
If the aims of the application are achieved,
how will scientific knowledge or clinical
practice be advanced? What will be the effect
of these studies on the concepts, methods,
technologies, treatments, services, or
preventative interventions that drive this
field?
11
The NIH Peer Review Process
Approach
Are the conceptual or clinical framework,
design, methods, and analyses adequately
developed, well integrated, well reasoned, and
appropriate to the aims of the project? Does
the applicant acknowledge potential problem
areas and consider alternative tactics? For
applications designating multiple PD/PIs, is
the leadership approach, including the
designated roles and responsibilities,
governance and organizational structure
consistent with and justified by the aims of
the project and the expertise of each of the
PD/PIs?
12
The NIH Peer Review Process
Innovation
Is the project original and innovative? For
example Does the project challenge existing
paradigms or clinical practice address an
innovative hypothesis or critical barrier to
progress in the field? Does the project develop
or employ novel concepts, approaches,
methodologies, tools, or technologies for
this area?
13
The NIH Peer Review Process
Investigator(s)
Are the PD/PI(s) and other key personnel
appropriately trained and well suited to
carry out this work? Is the work proposed
appropriate to the experience level of the
PD/PI(s) and other researchers? Do the
PD/PI(s) and the investigative team bring
complementary and integrated expertise to the
project (if applicable)?
14
The NIH Peer Review Process
Environment
Do(es) the scientific environment(s) in which
the work will be done contribute to the
probability of success? Do the proposed
studies benefit from unique features of the
scientific environment(s), or subject
populations, or employ useful
collaborative arrangements? Is
there evidence of institutional support?
15
The NIH Peer Review Process
Research Plans

• Introduction to Application
• (resubmission or revision only)
• Specific Aims
• Background and Significance
• Preliminary Studies/Progress Report

• Research Design and Methods
• Inclusion Enrollment Report
• (renewal or revision only)
• Bibliography and References Cited

16
The NIH Peer Review Process
Research Plans - continued

• Protection of Human Subjects
• Inclusion of Women and Minorities
• Targeted/Planned Enrollment Table
• Inclusion of Children
• Vertebrate Animals
• Select Agent Research
• Multiple PD/PI Leadership Plan
• Consortium/Contractual Arrangements
• Letters of Support
• Resource Sharing Plans

17
The NIH Peer Review Process
Writing the Application

• Be realistic about project scope
• For resubmissions, prepare
• thoughtful response to critiques
• Use legible figures/legends
• Summarize experimental details
• Outline alternative approaches
• Cite work of others appropriately
• Have letters, supporting documents in
• place

18
The NIH Peer Review Process
Submitting the Application

• Standard Receipt Dates
• (http//grants.nih.gov/grants/funding/submissio
  nschedule.htm)
• Special Receipt Dates provided in
• funding announcement
• Dont be late!

19
The NIH Peer Review Process
Grants.gov U.S. Organizations

• Obtain Employer Identification
• Number (EIN) - IRS
• Request DUNS -
• Dun Bradstreet
• Register with CCR -
• Central Contractor Registry
• - identify the eBiz Point of Contact (POC)
• Register the Authorized Organization Reps
• (AORs), who sign and submit the application
• (http//www.grants.gov/applicants/get_register
  ed.jsp)

20
The NIH Peer Review Process
eRA Commons

• Required
• One-time registration - organization
• Registration - Signing Officials (SOs)
• SOs register themselves
• SOs create separate accounts for the Project
  Directors/Principal Investigators (PDs/PIs)

Allow 2 - 4 weeks to complete!
Register (https//commons.era.nih.gov/commons/re
gistration/registrationInstructions.jsp)
21
The NIH Peer Review Process
eRA Commons Registration of PD/PI

• PD/PI works with SO
• to become registered
• PD/PI and SO require
• separate accounts
• SOs create separate accounts for the Project
• Directors/Principal Investigators (PDs/PIs)

Allow 4 - 5 business days to complete!
Register (https//commons.era.nih.gov/commons/re
gistration/registrationInstructions.jsp)
22
The NIH Peer Review Process
Due Dates

• Paper Submission or postmark date
• Electronic 500 PM local time
• on due date
• RFAs/PARs with special receipt dates Receipt
  date
• Permission to submit late not granted in
  advance

Modified Application Submission, Referral and
Review for Appointed NIH Study Section Members
(http//grants.nih.gov/grants/guide/notice-files/
NOT-OD-08-026.html)
23
The NIH Peer Review Process
Division of Receipt and Referral, CSR

• Checks for completeness
• Determines area of research
• Assigns an identification number
• Assigns a grant number
• Assigns application to specific
• NIH IC for possible funding
• Assigns a Scientific Review Group

NIH receives 80,000 applications per year
Central receiving point for all competing
applications
24
The NIH Peer Review Process
Receipt and Referral

• CSR Review
• Most R01s, fellowships, and small business
  applications
• Some Program Announcements (PAs, PARs), Requests
  for Applications (RFAs)

• Institute/Center Review
• IC-specific features
• Program projects
• Training grants
• Career development awards
• RFAs

25
The NIH Peer Review Process
Cover Letter of Application

• Application title
• FOA and title
• Request for assignment to particular Scientific
• Review Group (SRG) or study section
• Request for assignment to particular IC
• Disciplines involved, if multidisciplinary
• Explanation for late application

Not all requests can be honored.
26
The NIH Peer Review Process
Cover Letter of Application

• SRG information
• CSR
• http//cms.csr.nih.gov/PeerReviewMeetings/CSRIRGDe
  scription/
• http//www.csr.nih.gov/committees/rosterindex.asp
• ICs http//era.nih.gov/roster/index.cfm
• Areas of IC interest
• IC websites http//www.nih.gov/icd/index.html

27
The NIH Peer Review Process
Cover Letter of Application

• Suggested format
• List one request per line
• Place SRG IC review requests on separate
  lines
• Place positive negative requests on separate
  lines
• Include name of IC or SRG, followed by a dash
• and acronym
• Provide explanations
• for each request
• in a separate paragraph

28
The NIH Peer Review Process
Initial Peer Review

• Scientific Review Officer (SRO)
• Designated Federal Official
• Extramural scientist
• Identifies and recruits reviewers
• Manages conflicts of interest
• Oversees arrangements for review meetings
• Presides at review committee meetings
• Prepares and releases summary statements

29
The NIH Peer Review Process
Scientific Review Group (SRG)

• Study Section
• Expertise
• Stature in field
• Mature judgment
• Impartiality
• Geographical balance
• Diversity

Chartered Special Emphasis
Permanent or Panel (SEP) membership
ad hoc membership
30
The NIH Peer Review Process
Reviewer Assignments

• Three qualified reviewers (2 1)
• Based on scientific content of
• application
• Expertise of reviewer
• Suggestions from PI on types of
• expertise not names!
• Suggestions from Program staff
• Conflicts of interest

Telephone reviewers discuss and score Mail-in
reviewers do not discuss, do not score
31
The NIH Peer Review Process
Conflicts of Interest (COI)

• Financial
• Employment
• Personal
• Professional
• SRG membership
• Other interests

• Two COI vouchers signed by each reviewer
• Pre-meeting
• Post-meeting

32
The NIH Peer Review Process
Scientific Review Groups (SRGs)

• Make recommendations
• Scientific and technical merit
• Budget and project duration
• Bars to award human subjects,
• vertebrate animals, biohazards
• Resource Sharing Plans
• Other administrative factors
• Priority scores
• Written critiques
• (summary statements)

Study Sections do not make funding decisions!
33
The NIH Peer Review Process
Scientific Review Groups (SRGs)

• Also factored into priority scores
• Protection of human subjects
• from research risks
• Inclusion plans
• Vertebrate animal welfare

34
The NIH Peer Review Process
Scientific Review Groups (SRGs)

• Confidentiality
• All materials, discussions, documents
• (except those in the public domain)
• Reviewers sent guidance with applications
• All questions referred to SRO
• Closed to the public
• Program staff may observe

Reviewers sign two Confidentiality
Certifications!
35
The NIH Peer Review Process
NIH Scoring System

• Private scoring at SRG meeting
• Numerical scores
• 1.0 (most meritorious) to 5.0 (least
  meritorious)
• Final priority score average of individual
• scores
• Ranked by percentile for certain mechanisms
• Streamlining (UN)
• Other designations (NR, DF, AB, NP, etc.)

36
The NIH Peer Review Process
Streamlining

• Unscoring
• Focus on more meritorious applications
• Research projects 50
• Shared instrumentation 40
• Fellowship applications 30
• RFAs pre - arranged limits
• Requires full concurrence of SRG
• Not discussed at SRG meeting
• Not assigned numerical scores
• Reviewer critiques in summary statement

37
The NIH Peer Review Process
SRG Procedures

• Call to Order - Chairperson
• Policy and instructions - SRO
• Streamlining
• Discuss each application
• Scoring
• Discuss other considerations
• Resource Sharing Plans
• Foreign institutions

38
The NIH Peer Review Process
SRG Procedures

• Discussion format
• Members with conflicts excused
• Initial levels of enthusiasm
• (assigned reviewers)
• Primary reviewer explains project, strengths,
• weaknesses
• Other assigned reviewers follow
• Open discussion (full panel)
• Levels of enthusiasm (assigned reviewers)

39
The NIH Peer Review Process
SRG Procedures

• Reviewer workload
• 6 8 as reviewer
• 2 3 as discussant
• Clarity and brevity critical
• Dont assume reviewers will
• See the unstated
• Grasp nuances
• Understand lingo
• Look things up
• Read your mind!

40
The NIH Peer Review Process
SRG Procedures

• If 60 applications/SRG meeting
• 50 streamlined
• 30 applications to discuss and score
• If 9 hour SRG meeting (800 AM 500 PM)
• ½ hour introduction, streamlining
• 1 hour lunch, 2 x 15 minute breaks

Leaves 14 minutes/application 3 -
4 minutes/reviewer
Clarity and brevity are essential!
41
The NIH Peer Review Process
SRG Procedures

• Internet Assisted Review
• Reviewer critiques/preliminary scores
• Due several days before SRG meeting
• Acceptance of supplementary
• material at discretion of SRO
• Correcting errors or omissions
• New data or newly accepted
• publications
• Additional letters of commitment
• Cannot modify application

42
The NIH Peer Review Process
Alternate Styles of Review

• Teleconferences
• Editorial-style review
• Video-enhanced discussions
• Asynchronous electronic
• discussions

Final report on Enhancing Peer Review coming
soon!
43
The NIH Peer Review Process
eRA Commons

• Priority Score
• Three days after conclusion of SRG meeting
• Summary statement
• 4 8 weeks after conclusion of SRG meeting
• Available to Program Officers at that time
• Confidential document
• Available to
• PD/PIs
• NIH officials
• Advisory Council members

44
The NIH Peer Review Process
Summary Statement

• First page
• Program Officer (upper left corner)
• Name
• Contact information
• Priority Score
• Percentile (if applicable)
• Codes
• Human subjects
• Vertebrate animals
• Inclusion plans

45
The NIH Peer Review Process
Summary Statement - continued

• Subsequent pages
• Description (provided by applicant)
• Resumé and Summary of Discussion
• Reviewer critiques essentially unedited
• Committee Recommendations
• Budget
• Human subjects
• Vertebrate animals
• Administrative Notes

46
The NIH Peer Review Process
After the Review

• Program Officer Point of Contact
• Wait for summary statement
• Read summary statement
• carefully

A favorable score is not a guarantee of funding!
47
The NIH Peer Review Process
After the Review

• Consult Program Officer
• Consider options if outcome
• unfavorable
• Revise and resubmit application
• Consider critiques in summary statement
• Address critiques in introduction and text
• Appeal review outcome
• Procedural deficiencies
• Factual errors
• May result in re-review of same application by
• different SRG

48
The NIH Peer Review Process
Advisory Council/Board

• Second level of review
• Advisory to IC Director
• Rosters http//www1.od.nih.gov/cmo/committee/ind
  ex.html
• Schedule http//www1.od.nih.gov/cmo/committee/in
  dex.html

49
The NIH Peer Review Process
Advisory Council/Board

• Research priority areas
• Policy
• Appeals
• Funding
• Quality of SRG review
• Concur with SRG recommendations
• Modify SRG recommendations
• Deferral for re-review
• Cannot change priority score from SRG

50
The NIH Peer Review Process
Funding Considerations

• Scientific and technical merit
• (initial peer review)
• Council recommendation
• Relevance to
• program priorities in IC
• Availability of funds

51
The NIH Peer Review Process
Additional Information

• Enhancing Peer Review Initiative
• http//enhancing-peer-review.nih.gov/
• Office of Extramural Research
• Peer Review Process
• http//grants.nih.gov/grants/peer_review_proc
  ess.htm
• Peer Review Policies Practices
• http//grants.nih.gov/grants/peer/peer.htm
• Center for Scientific Review
• http//cms.csr.nih.gov/AboutCSR/WelcometoCSR/
  

52
The NIH Peer Review Process
Contact Information
Sally Amero, Ph.D. NIH Review Policy
Officer Extramural Research Integrity Liaison
Officer Office of Extramural Programs Office of
Extramural Research National Institutes of
Health 301-435-1418 ameros_at_od.nih.gov