The NIH Peer Review Process

1
The NIH Peer Review Process

• Sally A. Amero, Ph.D.
• NIH Review Policy Officer
• Office of Extramural Research

2008 NIH Regional Seminars
2
The NIH Peer Review Process
Project Director/Principal Investigator (PD/PI)
Writing the Application
Signing Official (SO) or PD/PI
Submitting the Application
Receipt and Referral
Center for Scientific Review
Scientific Review Officer (SRO)/Study Section
Initial Peer Review
Council/Board Review
NIH Institutes and Centers
3
The NIH Peer Review Process
Writing the Application
Start early Seek advice from colleagues Start
with a good idea Talk to your NIH Program
Official(s) Use the NIH webpage
(www.nih.gov) Remember review criteria Follow
instructions carefully
Transition to Electronic Submission
(http//era.nih.gov/ElectronicReceipt/)
4
The NIH Peer Review Process
Remember Review Criteria
Standard criteria Factored into priority score
Significance Protection of human subjects
Approach from research risks
Innovation Vertebrate animal welfare
Investigator(s) Biohazards Environment
Research projects and contracts
Additional considerations Not factored into
priority score Budget Resource Sharing
Foreign institutions
Establishment of Multiple Principal Investigator
Awards for the Support of Team Science
Projects (http//grants.nih.gov/grants/guide/notic
e-files/NOT-OD-07-017.html)
5
The NIH Peer Review Process
Significance
Does this study address an important problem?
If the aims of the application are achieved,
how will scientific knowledge or clinical
practice be advanced? What will be the effect
of these studies on the concepts, methods,
technologies, treatments, services, or
preventative interventions that drive this
field?
6
The NIH Peer Review Process
Approach
Are the conceptual or clinical framework,
design, methods, and analyses adequately
developed, well integrated, well reasoned, and
appropriate to the aims of the project? Does
the applicant acknowledge potential problem
areas and consider alternative tactics? For
applications designating multiple PD/PIs, is
the leadership approach, including the
designated roles and responsibilities,
governance and organizational structure
consistent with and justified by the aims of
the project and the expertise of each of the
PD/PIs?
7
The NIH Peer Review Process
Innovation
Is the project original and innovative? For
example Does the project challenge existing
paradigms or clinical practice address an
innovative hypothesis or critical barrier to
progress in the field? Does the project develop
or employ novel concepts, approaches,
methodologies, tools, or technologies for
this area?
8
The NIH Peer Review Process
Investigator(s)
Are the PD/PI(s) and other key personnel
appropriately trained and well suited to
carry out this work? Is the work proposed
appropriate to the experience level of the
PD/PI(s) and other researchers? Do the
PD/PI(s) and the investigative team bring
complementary and integrated expertise to the
project (if applicable).
9
The NIH Peer Review Process
Environment
Do(es) the scientific environment(s) in which
the work will be done contribute to the
probability of success? Do the proposed
studies benefit from unique features of the
scientific environment(s), or subject
populations, or employ useful collaborative
arrangements? Is there evidence of
institutional support?
10
The NIH Peer Review Process
Research Plans
Introduction to Application (resubmission or
revision only) Specific Aims Background and
Significance Preliminary Studies/Progress
Report Research Design and Methods Inclusion
Enrollment Report (renewal or revision
only) Bibliography and References Cited

11
The NIH Peer Review Process
Research Plans
Protection of Human Subjects Inclusion of Women
and Minorities Targeted/Planned Enrollment
Table Inclusion of Children Vertebrate
Animals Select Agent Research Multiple PD/PI
Leadership Plan Consortium/Contractual
Arrangements Letters of Support Resource Sharing
Plans

12
The NIH Peer Review Process
Human Subjects Research
If not exempt, HS Section should address Human
subjects involvement and characteristics Risks
to human subjects Sources of materials,
Potential risks, Recruitment Adequacy of
protection against risks Recruitment and
informed consent Protections against risks
Potential benefits of the proposed research
Importance of the knowledge to be gained
If Clinical Trial Data and Safety Monitoring
Plan
13
The NIH Peer Review Process
Human Subjects Research
Or request exemption and justify Exemption areas
(E1, E2, E3) 1. Established educational
settings 2. Educational tests, survey
procedures, interview procedures, or
observation of public behavior if Subjects
not identified Subjects privacy rights
protected 3. Educational tests, survey
procedures, interview procedures, or
observation of public behavior if Subjects
are public officials or candidates Without
exception, confidentiality required
14
The NIH Peer Review Process
Human Subjects Research
Exemption areas (E4, E5, E6) 4.
Collection/study of existing data, documents,
records, specimens if Sources are
publicly available or Subjects cannot be
identified 5. Research and demonstration
projects Certain public benefit
Service programs 6.
Taste and food quality evaluation and
consumer acceptance Foods without
additives U.S. Government approved
ingredient
15
The NIH Peer Review Process
Inclusion of Women and Minorities
Inclusion Plan should provide Distribution of
subjects Subject selection criteria and
rationale Rationale for proposed exclusion
Proposed outreach programs
Additional instructions NIH-Defined Phase III
Clinical Trials Targeted/Planned Enrollment
Tables for Reporting Race and Ethnicity Data
for Subjects in Clinical Research
16
The NIH Peer Review Process
Inclusion of Children
Child defined as an individual under the age
of 21 years Inclusion Plan should provide
Rationale for selecting specific
age range Description of expertise of
investigative team
17
The NIH Peer Review Process
Human Subjects Research
IRB approval Just in Time Education required
in protection of human subjects Protection of
human subjects, and inclusion plans, can be
factored into priority scores!
18
The NIH Peer Review Process
Vertebrate Animal Research
VA Section should address Proposed use of
animals Species, strains, ages, sex, numbers
Justification for use of animals, choice of
species, numbers of animals Veterinary care
Procedures for ensuring that discomfort,
distress, pain, injury limited to that which
is unavoidable Method of euthanasia to be used
and reasons
19
The NIH Peer Review Process
Vertebrate Animal Research
Provide Animal Welfare Assurance IACUC Approval
Just-in-Time Consideration of vertebrate animal
welfare can be factored into the priority score!
20
The NIH Peer Review Process
Grants.gov U.S. Organizations
Obtain Employer Identification Number (EIN) -
IRS Request DUNS - Dun Bradstreet Register
with CCR - Central Contractor Registry -
identify the eBiz Point of Contact (POC) Register
the Authorized Organization Reps (AORs), who sign
and submit the application http//www.grants.gov/a
pplicants/get_registered.jsp
21
The NIH Peer Review Process
eRA Commons

• Required
• One-time registration - organization
• Registration - Signing Officials (SOs)
• SOs register themselves
• SOs create separate accounts for the Project
  Directors/Principal Investigators (PDs/PIs)

Allow 2 - 4 weeks to complete!
Register (https//commons.era.nih.gov/commons/re
gistration/registrationInstructions.jsp)
22
The NIH Peer Review Process
eRA Commons Registration of PD/PI
PD/PI works with SO to become registered PD/PI
and SO require separate accounts SOs create
separate accounts for the Project
Directors/Principal Investigators (PDs/PIs)
Allow 4 - 5 business days to complete!
Register (https//commons.era.nih.gov/commons/re
gistration/registrationInstructions.jsp)
23
The NIH Peer Review Process
PHS 398 Application Forms (http//grants.nih.gov/
grants/funding/phs398/phs398.html)
Paper format Signature of institutional official
required Mechanisms Program Project Grants
(Ps) Center Grants (Ps) Institutional
NRSA Awards (Ts) Other Training
Grants Complex Mechanisms (D71/UR2, G12,
M01, R10/U10, R24/U24, S06, U19, U45, U54,
U56) Cooperative Agreements
(U01s) Research Career Dev. (Ks)
24
The NIH Peer Review Process
Electronic Submission
SF424 (RR) Forms (http//era.nih.gov/ElectronicR
eceipt/) All submissions respond to a Funding
Opportunity Announcement (FOA) Search NIH Guide
for Grants Contracts (http//grants.nih.gov/gran
ts/guide/index.html) Find Grant Opportunities
(www.Grants.gov) Use Parent Announcements for
program of interest (http//grants.nih.gov/grants
/guide/parent_announcements.htm)
25
The NIH Peer Review Process
Electronic Submission

• Grants.gov provides
• Tracking Number
• Date/Time stamp
• Initial submission
• Timestamp before 500 PM local time submitting
  
• organization) on due date
• Applicants have 2 weekdays to view assembled
  application
• Errors must be corrected within 5 business days

26
The NIH Peer Review Process
Writing the Application
Be realistic about project scope For
resubmissions, prepare thoughtful response to
critiques Use legible figures/legends Summarize
experimental details Outline alternative
approaches Cite work of others appropriately Get
letters, supporting documents in place
27
The NIH Peer Review Process
Submitting the Application
Standard Receipt Dates (http//grants.nih.gov/gran
ts/funding/submissionschedule.htm) Special
Receipt Dates provided in funding
announcement Dont be late!
28
The NIH Peer Review Process
Submitting the Application

• NIH Policy on Late Submission of Grant
  Applications
• (http//grants.nih.gov/grants/guide/notice-files/N
  OT-OD-08-027.html)
• Paper Submission or postmark date
• Electronic 500 PM local time on due date
• RFAs or PARs with special receipt dates
  Receipt date
• Permission to submit late not granted in
  advance
• Modified Application Submission, Referral and
• Review for Appointed NIH Study Section Members
• (http//grants.nih.gov/grants/guide/notice-files/N
  OT-OD-08-026.html)

29
The NIH Peer Review Process
Due Dates Selected mechanisms
30
The NIH Peer Review Process
Submitting the Application
31
The NIH Peer Review Process
Due Dates Selected Mechanisms
32
The NIH Peer Review Process
Due Dates Selected Mechanisms
33
The NIH Peer Review Process
Due Dates Selected Mechanisms
PHS 416-1 Form http//grants.nih.gov/grants/fundi
ng/416/phs416.htm
34
The NIH Peer Review Process
NIH Institutes and Centers (ICs)
Office of the Director
National Institute on Alcohol Abuse and Alcoholism
National Institute of Arthritis
and Musculoskeletal and Skin Diseases
National Cancer Institute
National Institute on Aging
National Institute of Child Health and
Human Development
National Institute of Allergy and Infectious
Diseases
National Institute of Diabetes and Digestive
and Kidney Diseases
National Institute of Dental and Craniofacial Rese
arch
National Institute on Drug Abuse
National Institute of Environmental Health
Sciences
National Institute on Deafness and
Other Communication Disorders
National Eye Institute
National Human Genome Research Institute
National Heart, Lung, and Blood Institute
National Institute of Mental Health
National Institute of Neurological Disorders
and Stroke
National Institute of General Medical Sciences
National Institute of Nursing Research
National Library of Medicine
National Center for Complementary and
Alternative Medicine
National Center on Minority Health and Health
Disparities
National Institute of Biomedical Imaging and
Bioengineering
Fogarty International Center
National Center for Research Resources
Lack funding authority
Center for Information Technology
Center for Scientific Review
Clinical Center
35
The NIH Peer Review Process
Division of Receipt and Referral, CSR
Application received by CSR DRR
Assignments ? Initial peer review Funding
considerations SRG (CSR or IC) IC(s)
duals possible ?
? Second level of review National Advisory
Council/Board ? IC Director (funding decisions)
36
The NIH Peer Review Process
Division of Receipt and Referral, CSR
Checks for completeness Determines area of
research Assigns an identification
number Assigns a grant number Assigns application
to specific NIH IC for possible
funding Assigns a Scientific Review Group
NIH receives 80,000 applications per year
Central receiving point for all competing
applications
http//cms.csr.nih.gov/ResourcesforApplicants/Subm
issionAndAssignmentProcess.htm
37
The NIH Peer Review Process
Receipt and Referral

• CSR Review
• Most R01s, fellowships, and small business
  applications
• Some PAs, PARs, RFAs
• http//cms.csr.nih.gov/

• Institute/Center Review
• IC-specific features
• Program projects
• Training grants
• Career development awards
• RFAs

38
The NIH Peer Review Process
Grant Number
1 R01 CA xxxxxx - 01 A1 Type (1 New, 2
Competing renewal, etc.) Mechanism (R01, F32,
P01, U54, etc.) Funding IC (AI, CA, GM, HG,
etc.) Number (Sequential by IC) Project
Year Submission
39
The NIH Peer Review Process
Cover Letter of Application
Application title FOA and title Request for
assignment to particular SRG CSR
http//cms.csr.nih.gov/PeerReviewMeetings/CSRIRGDe
scription/ http//www.csr.nih.gov/committees/ro
sterindex.asp ICs http//era.nih.gov/roster/i
ndex.cfm Request for assignment to particular IC
IC websites http//www.nih.gov/icd/index.html Di
sciplines involved, if multidisciplinary Explanati
on for late applications
Not always possible to honor requests
40
The NIH Peer Review Process
Cover Letter of Application
Suggested format List one request per line
Place SRG IC review requests on separate lines
Place positive negative requests on separate
lines Include name of IC or SRG, followed by a
dash and acronym Provide explanations for
each request in a separate paragraph
41
The NIH Peer Review Process
Initial Peer Review
Scientific Review Officer (SRO) Designated
Federal Official Extramural scientist
Identifies and recruits reviewers Manages
conflicts of interest Oversees arrangements
for review meetings Presides at review
committee meetings Prepares and releases
summary statements
42
The NIH Peer Review Process
Scientific Review Group (SRG)
Study Section Expertise Stature in
field Mature judgment Impartiality Geographical
balance Diversity Chartered Special
Emphasis Permanent or Panel (SEP)
membership ad hoc membership
43
The NIH Peer Review Process
Reviewer Assignments
At least three qualified reviewers per
application (2 1) Based on scientific content
of application Expertise of reviewer Suggestions
from PI on types of expertise not
names! Suggestions from Program staff
44
The NIH Peer Review Process
Reviewer Assignments
Management of conflict of interest Financial
Professional Employment Other interests
Personal SRG membership Two COI vouchers
signed by each reviewer
May include Telephone reviewers discuss and
score Mail-in reviewers do not discuss, do
not score
45
The NIH Peer Review Process
Scientific Review Groups (SRGs)
Recommendations Scientific and technical
merit - standard criteria
Significance Protection of human subjects
Approach Vertebrate animals
Innovation Biohazards
Investigator(s) Environment Budget
and project duration Bars to award human
subjects, biohazards Resource Sharing Plans
Other administrative factors for research
projects
46
The NIH Peer Review Process
Scientific Review Groups (SRGs)
Protection of human subjects from research
risks, Inclusion Plans, and Vertebrate Animal
Welfare can be factored into priority scores!
47
The NIH Peer Review Process
Scientific Review Groups (SRGs)
Confidentiality All materials, discussions,
documents (except those in the public domain)
Reviewers sign confidentiality voucher All
questions referred to SRO Closed to the
public Program staff may attend, observe

48
The NIH Peer Review Process
NIH Scoring System
Private scoring at SRG meeting Numerical scores
1.0 (most meritorious) to 5.0 (least
meritorious) Final priority score
average of individual scores Ranked
by percentile for certain mechanisms Streamlining
(UN) Other designations (NR, DF, AB, NP, etc.)
49
The NIH Peer Review Process
Streamlining
Unscoring Focus on more meritorious
applications Research projects 50
Shared instrumentation 40 Fellowship
applications 30 RFAs pre-arranged
Requires full concurrence of SRG Not discussed
at SRG meeting Not assigned numerical scores
Reviewer critiques in summary statement
50
The NIH Peer Review Process
SRG Procedures
Roll call and introductions - Chair Policy and
instructions - SRO Streamlining Discuss each
application Initial levels of enthusiasm
Members with conflicts excused
Primary reviewer explains project,
strengths, weaknesses other assigned
reviewers follow Open discussion/ revisit
enthusiasm Scoring

51
The NIH Peer Review Process
SRG Procedures
If 60 applications/SRG meeting 50
streamlined 30 applications to discuss and
score If 9 hour SRG meeting (800 AM 500 PM)
½ hour introduction, streamlining 1 hour
lunch, 2 x 15 minute breaks

Leaves 14 minutes/application 3 - 4
minutes/assigned reviewer
52
The NIH Peer Review Process
SRG Procedures
Reviewer workload 6 8 as reviewer
2 3 as discussant Clarity and
brevity critical Cant assume reviewers will
see the unstated grasp nuances
understand lingo look things up

53
The NIH Peer Review Process
SRG Procedures
Internet Assisted Review Reviewer
critiques/preliminary scores Due several days
before SRG meeting Acceptance of
supplementary material at discretion of SRO
Correcting errors or omissions New data or
newly accepted publications Additional
letters of commitment Cannot modify application
54
The NIH Peer Review Process
Alternate Styles of Review
Teleconferences, telephone reviewers Mail-in
reviewers Video-enhanced discussions Asynchronous
electronic discussions
55
The NIH Peer Review Process
eRA Commons
Priority Score Three days after conclusion of
SRG meeting Summary statement 4 8 weeks
after conclusion of SRG meeting Available to
Program Officers at that time Confidential
document PD/PI NIH officials
Advisory Council members

56
The NIH Peer Review Process
Summary Statement
Program Officer name, contact information
Priority Score Percentile (if applicable) Codes
human subjects, vertebrate animals Description
(provided by applicant) Resumé and Summary of
Discussion (except UNs) Reviewer critiques
essentially unedited Committee
Recommendations Budget, Human subjects,
Vertebrate animals Administrative Notes
57
The NIH Peer Review Process
After the Review

Program Officer Point of Contact Wait for
summary statement Read summary statement
carefully A favorable score is not a guarantee
of funding!
58
The NIH Peer Review Process
After the Review

Consult Program Officer Consider options if
outcome not favorable Revise and resubmit
application Consider critiques in summary
statement Address critiques in introduction
and text Appeal review outcome Procedural
deficiencies Factual errors May
result in re-review of same application by
different SRG
59
The NIH Peer Review Process
Advisory Council/Board
Second level of review Advisory to IC
Director Rosters http//www1.od.nih.gov/cm
o/committee/index.html Schedule
http//www1.od.nih.gov/cmo/committee/index.html
60
The NIH Peer Review Process
Advisory Council/Board
Research priority areas Policy Appeals
Funding Quality of SRG review Concur
with SRG recommendations Modify SRG
recommendations Deferral for re-review
Cannot change priority score from SRG

61
The NIH Peer Review Process
Funding Considerations
Scientific and technical merit (initial peer
review) Council recommendation Relevance to
program priorities in IC Availability of funds
62
The NIH Peer Review Process
Enhancing Peer Review Initiative http//enhancing-
peer-review.nih.gov/ Office of Extramural
Research Peer Review Process
http//grants.nih.gov/grants/peer_review_process.h
tm Peer Review Policies Practices
http//grants.nih.gov/grants/peer/peer.htm Center
for Scientific Review http//cms.csr.nih.gov/Abou
tCSR/WelcometoCSR/
63
The NIH Peer Review Process
Contact Information
Sally Amero NIH Review Policy Officer Office of
Extramural Research 301-435-1418
ameros_at_csr.nih.gov