Peer Review of NIH Research Grant Applications

1
Peer Review of NIH Research Grant Applications
Center for Scientific Review National Institute
of Diabetes and Digestive and Kidney
Diseases National Institutes of Health
2
U. S. Department of Health and Human Services
The Secretary Deputy Secretary
Administration for Children and Families (ACF)
Food and Drug Administration (FDA)
Health Resources and Services Administration (HRSA
)
Administration on Aging (AoA)
Health Care Financing Administration (HCFA)
Indian Health Services (IHS)
National Institutes of Health (NIH)
Agency for Health Care Policy and
Research (AHCPR)
Centers for Disease Control and Prevention (CDC)
Substance Abuse and Mental Health
Services Administration (SAMHSA)
Program Support Center (PSC)
Agency for Toxic Substances and Disease
Registry (ATSDR)
3
National Institutes of Health

• Much of the biomedical research in the united
  states is supported by the federal government,
  primarily the National Institutes of Health (NIH)

4
National Institutes of Health
Office of the Director
National Institute on Alcohol Abuse and Alcoholism
National Institute of Arthritis
and Musculoskeletal and Skin Diseases
National Cancer Institute
National Institute on Aging
National Institute of Child Health and
Human Development
National Institute of Allergy and Infectious
Diseases
National Institute of Diabetes and Digestive
and Kidney Diseases
National Institute of Dental and Craniofacial Rese
arch
National Institute on Drug Abuse
National Institute of Environmental Health
Sciences
National Institute on Deafness and
Other Communication Disorders
National Eye Institute
National Human Genome Research Institute
National Heart, Lung, and Blood Institute
National Institute of Mental Health
National Institute of Neurological Disorders
and Stroke
National Institute of General Medical Sciences
National Institute of Nursing Research
National Library of Medicine
National Center for Complementary and
Alternative Medicine
National Center on Minority Health and Health
Disparities
National Institute of Biomedical Imaging and
Bioengineering
Fogarty International Center
National Center for Research Resources
Center for Information Technology
Center for Scientific Review
Clinical Center
5
A Typical Institute/center
Office of the IC Director
National Advisory Council
Board of Scientific Counselors
Extramural
Intramural
Scientific Programs
Laboratory Studies
Clinical Studies
Grants
Contracts
6
NIH Extramural Program Mission
Manage a portfolio of investments to improve
health through science
Identify scientific opportunities
Foster the best science
Promote effective communication
Ensure proper stewardship
Exemplify and promote the highest level of
scientific integrity, public accountability, and
responsibility in the conduct of science
Promote effective ways to communicate scientific
information to scientists, health practitioners,
and the public
7
NIH Extramural Program

• Grant patron
• (Assistance, encouragement)
• Cooperative partner
• Agreement (assistance but substantial
• Program involvement)
• Contract purchaser
• (Procurement)

8
Overall Peer Review Process
9
Dual Review System for Grant Applications

• First Level of Review
• Scientific Review Group (SRG)
• Provides Initial Scientific Merit
• Review of Grant Applications
• Rates Applications and Makes
  Recommendations for Appropriate Level of Support
  and Duration of Award

• Second Level of Review
• Council
• Assesses Quality of SRG
• Review of Grant Applications
• Makes Recommendation to
• Institute Staff on Funding
• Evaluates Program Priorities
• and Relevance
• Advises on Policy

10
Review Process for a Research Grant
National Institutes of Health
Research Grant Application
School or Other Research Center
Center for Scientific Review
Assigns to IRG/Study Section IC
Study Section
Initiates Research Idea
Submits Application
Evaluates for Scientific Merit
Institute
Evaluates for Program Relevance
Advisory Councils and Boards
Allocates Funds
Conducts Research
Recommends Action
Institute Director
Takes final action for NIH Director
11
Overall Timeframe From Submission to Award

• There are three overlapping cycles per year

JAN FEB MAR APR MAY JUN
JUL AUG SEP OCT NOV DEC
JAN FEB MAR APR MAY JUN
JUL
Cycle 1
Review
Council
Receipt
Award
Referral
12
Typical Timeline for a New Individual Research
Project Grant Application (R01)

• There are three overlapping cycles per year

• Submit in February (June, October)
• Review in June (October, February)
• Council in September (January, May)
• Earliest award in December (April, July)

• Cycle 1----
• Cycle 2----
• Cycle 3----

13
Types of Scientific Review GroupsWhere Are
Applications Reviewed?
GROUPS
APPLICATIONS REVIEWED
Research Projects
CSR IRGs
Academic Research

Study Sections
Enhancement Awards
Postdoctoral Fellowships
Special Emphasis Panels
Small Business Innovation
Research
Shared Instrumentation
INSTITUTES
Program Projects
Scientific Review Groups
Centers
Institutional Training Grants
Conference Grants
Career Awards
Small Grants
RFAs (R01, R21, R03, U01)
Contract Review Committees
Contracts
14
Peer Review of NIH Support Mechanisms
CSR
Institutes

• Research project grant (R01) Program Project
  grant (P01)
• Postdoctoral fellowship (F32) Center grant
  (P30, P50, P60)
• Senior fellowship (F32) Institutional
  fellowship (T32)
• Fogarty international center Academic career
  award (K07)
• Fellowship (F05, F06) Mentored clinical
  scientist
• Short-term training (T35) development award
  (K08)
• Small business grants (R41, R42 Conference
  grant (R13)
• R43, R44) MARC fellowships (F34, F36,
  T34)
• Academic Research Enhancement Minority
  Biomedical Support
• Award (R15) Grant (S06)
• Biomedical Research Support Resource grant
  (P40, P41, R24,
• Shared Instrumentation R26, R28)
• Grant (S10) Contract

Reviewed by CSR upon request
15
Center for Scientific Review (CSR)

• Focal point for initial review at NIH

• Central receipt point for PHS applications
• Referral to Institutes and to IRGs and Study
  Sections
• Review of most research and research training
  applications ..for scientific merit

16
Grant Application Receipt and Assignment
17
Applications Submitted to NIH

• Approximately 46,000 grant applications are
  submitted to NIH each year, of which 25-30 are
  funded
• Competing grant applications are received for
  three review cycles per year

18
CSR Receipt and Referral Central
Receipt Point for Applications
Submitted to the Public Health Service
National Institutes of Health
Substance Abuse and Mental Health
Services Administration
CSR Receipt Referral
Agency for Health Care Policy Research
Centers for Disease Control
Office of Assistant Secretary for Health
Food Drug
19
Competing Applications Reviewed By
NOTE Starting in FY93, NIDA, NIAAA, and NIMH are
included in NIH totals
20
Receipt Dates
Depend on the Type of Application

• Jan, May, Sept 10 institutional training grant
  applications
• Jan, May, Sept 25 academic research enhancement
  award
• Feb, June, Oct 1 new research applications
• Mar, Jul, Nov 1 revised, competing
  continuations, and

Supplemental Applications
and Small Business Innovation Research
Applications
21
Flow of a Competing Grant Application
Through the CSR Referral Section
Project Control Unit Receipt Record
Group (Process, Enter Skeletal Data into
Computer, Print Status)
Mailroom (Receive, Date Stamp, Preliminary Sort,
Add Bar Code)
Application (Original 5 Copies)
Assignment Unit (Assign to Scientific Review
Group and Awarding Organization)
(Advance Copy) Scientific Review Group
Project Control Unit Review Control Group (Add
Labels, Send Data to IMPAC, Print Mailers,
Distribute)
(2 Copies) Scientific Review Group
(Original) Print Shop
(1 copy) OPERA EIS,DMC
(1 copy) Project Control Files
(35 copies) SRG
(20 copies) Original Awarding Organization
(1 copy) Project Control
(1 copy) Research Documents (IC only)
22
Applications are Assigned by

• Referral Officers
• Professional scientists, most of whom also serve
  as scientific review administrators of CSR study
  sections

23
Applications Are Assigned To

• Scientific review groups based on
• Specific review guidelines for each scientific
  review group
• Institutes based on
• Overall mission of the Institute
• Specific programmatic mandates and interests of
  the Institute

24
Assignment to Institutes

• Applications are referred to an institute or
  center as the potential funding component
• This assignment is based on a match between the
  research proposed and the overall mission of the
  institute or center
• Where applications are appropriate for more than
  one institute or center, multiple assignments are
  made

25
Sample Application Number

• Individual serial
  amended
• Research number
  
• Grant
• 1 R01 DK 12345 01 A1
• New national
  grant
• Application cancer
  support
• Institute
  year

26
Assignment of Applications
Review Group
Institute
Form Approved Through 9-30-97 OMB No. 0925 0001
AA
Department of Health and Human Services Public
Health Service GRANT APPLICATION Follow
instructions carefully. Do not exceed
character length restrictions indicated on sample.
Leave Blank--for PHS Use Only Type 1
Activity R21 Number A154321-01 Review Group
ZDK1 Formerly Council/Board
(Month,Year) Date Received 09/01
07-26-01
1. Title of Project (Do not exceed 56
characters, including spaces and punctuation.)
Planning Grant for Diabetes Sc Ed in Tribal
Schools
2. Response to Specific Request for Applications
or Program Announcement Number DK 01-033
Title Planning Grant for Diabetes and Science
Education
Yes (if Yes state number and title)
X
x
No

3. Principal Investigator/Program Director
3a. Name (Last, first, middle)
3b.
Degree(s) 3c. Social Security No. Smith,
Brian E.
Ph.D.
27
Assignment Notification Letter

• Dear Dr. Smith
• Your grant application entitled PLANNING GRANT
  FOR DIABETES AND SCIENCE EDUCATION IN TRIBAL
  SCHOOLS has been received by the National
  Institutes of Health and assigned to a scientific
  review group (SRG) for scientific merit
  evaluation and to an institute/center for funding
  consideration. Specific information about your
  assignment is given below. The initial peer
  review should be completed by AUGUST, 2001, and a
  funding decision made shortly after the
  appropriate National Advisory Group meets in
  SEPTEMBER, 2001. Questions about the assignment
  should be directed to the scientific review
  administrator (SRA) or the Division of Receipt
  and Referral, Center for Scientific Review
  (formerly Division of Research Grants) at (301)
  435-0715. Other questions prior to review should
  be directed to the scientific review
  administrator and questions after the review to
  the program staff in the institute/center.

28
Assignment Notification Letter (continued)

• Assignment Number 1 R01 DK12345 - 01
• Dual Assignment NS
• Scientific Review Group
• ZDK1 GRB-5 S1 S
• Information about this IRG may be found on the
  NIDDK Home page (http//www.niddk.nih.gov)
• Scientific Review Administrator
• DR. FRANCISCO CALVO, SRA
• NATIONAL INSTITUTE OF DIABETES AND
• DIGESTIVE AND KIDNEY DISEASES
• 6707 DEMOCRACY BLVD, ROOM 752
• BETHESDA MD 20892-5452
• [phone]

29
Assignment Notification Letter (Continued)

• Institute/Center
• NATIONAL INSTITUTE OF DIABETES AND DIGESTIVE AND
  KIDNEY DISEASES
• DIV/EXTRAMURAL ACTIVITIES
• 6707 DEMOCRACY BLVD, ROOM 752
• NATIONAL INSTITUTES OF HEALTH
• BETHESDA, MD 20892
• [phone]

30
Initial Review in IC
31
Peer Review

• CSR and IC Study Sections are managed by a
  Scientific Review Administrator (SRA) who is a
  scientist, usually at the Ph.D. level, whose
  scientific background is close to the expertise
  of the study section
• Each CSR/IC standing study section has 12-24
  members who are primarily from academia
• Institute review is also different in that
  Special Emphasis Panels are constituted
  specifically to review applications that have
  been received in response to an RFA or other
  Institute specific grant mechanisms
• As many as 60-100 applications are reviewed at
  each study section meeting, again this depends on
  the complexity of the review

32
Scientific Review Administrator
Designated Federal official with overall
responsibility for the review process, including

• Performing administrative and technical review of
  applications to ensure completeness and accuracy
• Selecting reviewers based on broad input
• Managing study section meetings
• Preparing summary statements
• Providing any requested information about study
  section recommendations to institutes and
  national advisory councils/boards

33
Selection of Peer Reviewers
Active and Productive Researchers
Research Capability
Non-Research
Non-Doctoral
Scientific Community
34
Criteria for Selection of Peer Reviewers
Active Productive Researchers
Meet Internal Administrative Considerations

• Geography

Doctoral or Equivalent Degree
Research Capability

• Institutional

Interest in Serving

• Affiliation

• Non-Federal

• Employment

Non-Research

• Gender

Expertise in Discipline of Review Group and
Specialization Needed
Ethnic
Status
Non-Doctoral
Scientific Community
Active and Productive Researchers
35
Study Section Meeting
36
Certification of No Conflict of Interest

• This will certify that in the review of
  applications and proposals by (study section) on
  (date), I did not participate in the evaluation
  of any grant or fellowship applications from (1)
  any organization, institution or university
  system in which a financial interest exists to
  myself, spouse, parent,child, or collaborating
  investigators (2) any organization in which I
  serve as officer, director, trustee, employee or
  collaborating investigator or (3) any
  organization which I am negotiating or have any
  arrangements concerning prospective employment or
  other such associations.
• __________________ __________________
• __________________ __________________
• __________________ __________________
• __________________ __________________

SIGNATURES
37
Confidentiality

• Review materials and proceedings of review
  meetings represent privileged information to be
  used only by consultants and NIH staff.
• At the conclusion of each meeting, consultants
  will be asked to destroy or return all
  review-related material.
• Consultants should not discuss review proceedings
  with anyone except the SRA.
• Questions concerning review proceedings should be
  referred to the SRA.

K185pp.46
38
Review of Research Grants

• REVIEW CRITERIA
• Significance
• Approach
• Innovation
• Investigator
• Environment
• Overall Evaluation Score Reflects Impact on
  Field

39
Review Criteria (Continued)

• Significance Does the study address an important
  problem? How will scientific knowledge be
  advanced?
• Approach Are design and methods well-developed
  and appropriate? Are problem areas addressed?
• Innovation Are there novel concepts or
  approaches? Are the aims original and
  innovative?
• Investigator Is the investigator appropriately
  trained?
• Environment Does the scientific environment
  contribute to the probability of success? Are
  there unique features of the scientific
  environment?

40
Specific Review Criteria RFA DK-01-033

• Administrative, leadership qualifications and
  experience of the Program Director.
• Adequacy and availability of any necessary
  institutional facilities and resources.
• Adequacy of the plan for developing a diabetes
  science education program, including use of
  existing resources, building partnerships with
  other institutions, use of consultants, and plans
  for critical review of the curriculum.
• Documentation of potential participants for the
  science education program.

41
Research Involving Human Subjects
42
Research Involving Human Subjects
Areas exempt from human subject review

• Education research
• Normal educational practices
• Educational tests, survey or interview
  procedures, or observation of public behavior
• Subjects not identified
• Subjects privacy rights protected
• Educational tests, survey or interview
  procedures, or observation of public behavior not
  exempt in previous category if
• Subjects are public officials or public office
  candidates
• Federal statute requires confidentiality without
  exception

43
Research Involving Human Subjects (Continued)
Areas exempt from human subject review

• Collection or study of existing data, documents,
  records, pathological specimens
• Information publicly available
• Subjects not identified
• Research and demonstration projects regarding
  certain public benefit or service programs
• Taste and food quality evaluation and consumer
  acceptance studies using
• Foods without additives
• U.S. Government approved food ingredient

44
Research Involving Children
Children must be considered for inclusion in all
human subject research supported by NIH
45
Scientific Review Group or Study Section Actions

• Scored, Scientific Merit Rating (priority scores
  and percentiles)
• Unscored (lower half)
• Deferral

46
Action

• Scored -- Scientific Merit Rating 1.0 to
  approximately 3.0
• Based on the relevant review criteria, the
  application is judged to be in the upper half of
  applications reviewed by the study section or
  scientific review group. The recommendation can
  be for the requested time and amount or for an
  adjusted time and amount. A priority score is
  provided, and a summary statement prepared that
  incorporates the written critiques plus a resume
  and summary of the discussion.

47
Action

• Unscored
• Application is unanimously judged to be in the
  lower half of applications reviewed by the study
  section or scientific review group. No priority
  score is assigned. The summary statement
  provided to the applicant is a compilation of
  reviewers comments prepared prior to the
  meeting.

48
Action

• Deferral
• The study section cannot make a recommendation
  without additional information. This information
  may be obtained by a project site visit or by
  submission of additional material by the
  applicant.

49
Post Scientific Review Group Actions

• Calculations of priority scores and percentile
  rankings
• Preparation of summary statements
• Removal of applications from national advisory
  council / board consideration

50
Summary Statement

• Once applications are reviewed, the results are
  documented by the SRA in a summary statement and
  forwarded to the Institute (and the PI) where a
  funding decision is made
• The summary statement contains
• Overall Resume and Summary of Review Discussion
• Essentially Unedited Critiques
• Priority Score and Percentile Ranking
• Budget Recommendations
• Administrative Notes

51
National Advisory Council or Board Review
52
Council Actions

• Concurrence with study section action
• Modification of study section action
• Deferral for re-review

53
What Determines Which Awards Are Made?

• Scientific merit
• Program considerations
• Availability of funds

54
Mechanisms for Preliminary Studies

• Small grants (R03)
• Feasibility/new technology/innovative high risk
  ideas
• Short term, usually 25,000 - 50,000
• Extremely variable in detail - example
• NCRR 1 year, 35,000, non-renewable
• NIDDK 3-6 months, 12,500 - 25,000
• NCI 2 years, 50,000
• Institute review
• Exploratory/developmental grants (R21)
• Feasibility (for those without preliminary data)
• 100,000/2 years (for this RFA it is 75,000/1
  year)
• Increasing use
• CSR review (RFA review performed by NIDDK)

55
Preparation of an Application
56
PHS Research Grant Application Kit (form PHS
398)
Mail Completed Forms To CENTER FOR SCIENTIFIC
REVIEW NATIONAL INSTITUTES OF HEALTH ROCKLEDGE II
ROOM 1040 MSC-7710 BETHESDA MD 20892-7710
57
When Preparing an Application

• Read instructions
• Never assume that reviewers will know what you
  mean
• Refer to literature thoroughly
• State rationale of proposed investigation
• Include well-designed tables and figures
• Present an organized, lucid write-up
• Obtain pre-review from faculty at your
  institution

58
Common Problems in Applications

• Lack of new or original ideas
• Absence of an acceptable scientific rationale
• Lack of experience in the essential methodology
• Questionable reasoning in experimental approach
• Uncritical approach
• Diffuse, superficial, or unfocused research plan
• Lack of sufficient experimental detail
• Lack of knowledge of published relevant work
• Unrealistically large amount of work
• Uncertainty concerning future directions

59
for
GrantsNIH GUIDE and
ContractsU.S. Department of Health and Human
Services

• Announces NIH Scientific Initiatives
• Provides NIH Policy and Administrative
  Information
• Available on the NIH Web Site
  http//www.nih.gov

60
Program Announcement

• Invites grant applications in a given research
  area
• May describe new or expanded interest in a
  particular extramural program
• May be a reminder of a continuing interest in a
  particular extramural program
• Generally has no funds set aside
• Applications reviewed in CSR along with
  unsolicited grant applications

61
Requests for Applications (RFA)

• Announcement describing an institute initiative
  in a well-defined scientific area
• Invitation to the field to submit research grant
  applications for a one-time competition
• Set-aside of funds for a certain number of awards
• Applications generally reviewed within the
  issuing institute, in this case the NIDDK

62
NIH Information Sources
63
Information on the World Wide WebSelected Sites
of Interest

• National Institutes of Health (http//www.nih.gov)
  
• Office of Extramural Research (http//www.nih.gov/
  grants/oer.htm)
• Grants Policy (http//www.nih.gov/grants/policy/po
  licy.htm)
• Center for Scientific Review (http//www.csr.nih.g
  ov)
• Referral and Review (http//www.csr.nih.gov/refrev
  .htm)
• Overview of Peer Review Process (http//www.
  csr.nih.gov/review/peerrev.htm)
• CSR Study Section Rosters (http//www.csr.nih.gov/
  committees/rosterindex.asp)
• NIH Peer Review Notes (http//www.csr.nih.gov/prn
  otes/prnotes.htm)

64
Office of Extramural Research

• Handles requests for grant applications, program
  guidelines, and general information regarding
  grant applications
• Office of Extramural Research
• National Institutes of Health
• 6701 Rockledge drive, suite 6095
• Bethesda, Maryland 20892-7910
• Phone [phone]
• Fax [phone]
• E-mail grantsinfo_at_nih.Gov

65
There is no grantsmanship that will turn a bad
idea into a good one, but..There are many
ways to disguise a good one.

• Dr. William Raub,
• Former Deputy Director, NIH