NIH Peer Review in the Population and Social Sciences

1
NIH Peer Review in the Population and Social
Sciences
Center for Scientific Review National Institutes
of Health U.S. Department of Health and Human
Services
2
Charles N. Rafferty, Ph.D.
Scientific Review Administrator and Referral
Officer Center for Scientific Review National
Institutes of Health

• Study Sections
• Health Services Organization and Delivery (HSOD)
  reviews R01, R21, and R03 applications
• Occupational Health and Safety SBIR/STTR
• Referral IRGs
• HOP, RPHB, BST, and DIG

3
Outline

• Overview of NIH and Peer Review
• Application Receipt and Referral
• Initial Peer Review Process, The Study Section
• Grantsmanship

4
Premise

• Understanding the peer review process will help
  you prepare a successful grant application.
• Success Award

5
Dr. Brent Stanfield

• Our work is critical because we know the result
  of peer review is the primary factor determining
  which research NIH funds."

6
National Institutes of Health

• Much of the biomedical research in the United
  States is supported by the Federal Government,
  primarily the National Institutes of Health (NIH)

7
NIH Extramural Awarding Components

• National Cancer Institute (NCI)
• National Heart, Lung, and Blood Institute (NHLBI)
• National Institute of Diabetes and Digestive and
  Kidney Diseases (NIDDK)
• National Library of Medicine (NLM)
• National Institute of Child Health and Human
  Development (NICHD)
• National Institute of Deafness and Other
  Communication Disorders (NIDCD)
• National Institute of Environmental Health
  Sciences (NIEHS)
• National Institute of Neurological Disorders and
  Stroke (NINDS)
• National Institute on Aging (NIA)
• National Institute of Allergy and Infectious
  Diseases (NIAID)
• National Institute of Arthritis and
  Musculoskeletal and Skin Diseases (NIAMS)
• National Institute of Dental and Craniofacial
  Research (NIDCR)
• National Eye Institute (NEI)
• National Institute of General Medical Sciences
  (NIGMS)
• National Institute of Mental Health (NIMH)
• National Institute for Nursing Research (NINR)
• National Institute on Alcohol Abuse and
  Alcoholism (NIAAA)
• National Institute on Drug Abuse (NIDA)
• National Center for Complementary and Alternative
  Medicine (NCCAM)

8
A Typical Institute/Center
Office of the IC Director
National Advisory Council
Board of Scientific Counselors
Extramural
Intramural
Scientific Programs
Laboratory Studies
Clinical Studies
Grants
Contracts
9
NIH Extramural Program

• Grant Patron
• (assistance, encouragement)
• Cooperative Partner
• Agreement (assistance but substantial
• program involvement)
• Contract Purchaser
• (procurement)

10
NIH Funding in FY 2004 By MechanismTotal 27B
Over 80 of NIH funds support extramural
research.
11
NIH Peer Review

• Process of evaluation of NIH grant applications
  for scientific and program merit
• NIH uses dual review system
• Scientific Review Group or Study Section
• Institute/Center Program Review

12
Dual Review System for Grant Applications

• First Level of Review
• Scientific Review Group (SRG)
• Provides Initial Scientific Merit
• Review of Grant Applications
• Rates Applications and Makes
  Recommendations for Appropriate Level of Support
  and Duration of Award

• Second Level of Review
• Council
• Assesses Quality of SRG
• Review of Grant Applications
• Makes Recommendation to
• Institute Staff on Funding
• Evaluates Program Priorities
• and Relevance
• Advises on Policy

13
Review Process for a Research Grant
National Institutes of Health
School or Other Research Center
Research Grant Application
Center for Scientific Review
Assigns to IC IRG/ Study Section

Initiates Research Idea
Study Section
Submits Application
Reviews for Scientific Merit
Institute
Evaluates for Relevance
Advisory Councils and Boards
Allocates Funds
Conducts Research
Recommends Action
Institute Director
Takes final action
14
Types of Scientific Review GroupsWhere are
Applications Reviewed?
GROUPS
APPLICATIONS REVIEWED
Research Projects
CSR IRGs
Academic Research

Study Sections
Enhancement Awards
Postdoctoral Fellowships
Special Emphasis Panels
Small Business Innovation
Research
Shared Instrumentation
INSTITUTES
Program Projects
Centers
Scientific Review Groups
Institutional Training Grants
Conference Grants
Career Awards
Small Grants
Contract Review Committees
RFAs
Contracts
15
Center for Scientific Review (CSR)

• Focal Point for Initial Review at NIH

• Central receipt point for PHS applications
• Referral to Institutes and to IRGs and Study
  Sections
• Review of most research and research training
  applications ..for scientific merit

16
Center for Scientific Review

• Referral
• Central Receipt Point for most PHS Grant
  Applications
• Institute Assignment (Potential Funding
  Component)
• Assignment to Scientific Review Group in CSR or
  in an Institute
• Scientific Review
• More than 200 chartered study sections and
  regularly recurring special emphasis panels that
  review
• Research Grant Applications
• Postdoctoral Fellowship Application
• Academic Research Enhancement Award Applications
• Small Business Innovation Research Applications

17
Overall Timeframe from Submission to Award

• There are three overlapping cycles per year

JAN FEB MAR APR MAY JUN
JUL AUG SEP OCT NOV DEC
JAN FEB MAR APR MAY JUN
JUL
Cycle 1
Review
Council
Receipt
Award
Referral
18
Grant Application Receipt and Assignment
19
Applications Submitted to NIH

• Over 60,000 grant applications are submitted to
  NIH each year, of which 25-30 are funded
• Competing grant applications are received for
  three review cycles per year

20
CSR Receipt and Referral Central
Receipt Point for Applications
submitted to the Public Health Service
National Institutes of Health
Substance Abuse and Mental Health
Services Administration
CSR Receipt Referral
Agency for Health Care Policy Research
Centers for Disease Control
Office of Assistant Secretary for Health
Food Drug
21
Applications are Assigned by

• Referral Officers
• Professional scientists, most of whom also serve
  as scientific review administrators of CSR study
  sections

22
Applications are Assigned to

• Scientific review groups based on
• Specific review guidelines for each scientific
  review group
• Institutes based on
• Overall mission of the Institute
• Specific programmatic mandates and interests of
  the Institute

23
Assignment to Institutes

• Applications are referred to an Institute or
  Center as the potential funding component
• This assignment is based on a match between the
  research proposed and the overall mission of the
  Institute or Center
• Where applications are appropriate for more than
  one Institute or Center, multiple assignments are
  made

24
Sample Application Number

• Individual Serial
  Amended
• Research Number
  
• Grant
• 1 R01 CA 123456 01 A1
• New National
  Grant
• Application Cancer
  Support
• Institute
  Year

25
Special Referral Issues

• Investigators should write a cover letter for
  their applications!
• Referral Officers almost always honor
  investigator requests for Institute assignments
  (funding) and CSR study section assignments
  (review)
• NIMH, NIAA, and NIDA review all health services
  and treatment research applications assigned to
  them for funding
• All other investigator-initiated health services
  and treatment research applications are reviewed
  by CSR

26
Initial Review in CSR
27
CSR Study Sections

• Each CSR standing study section has 12-24 members
  who are primarily from academia
• CSR standing study sections convene face-to-face
  meetings
• As many as 60-100 applications are reviewed by
  each study section
• Each study section is managed by a Scientific
  Review Administrator

28
CSR Review Divisions
Division of Biologic Basis of Disease Elliot
Postow, Ph.D.
Division of Molecular and Cellular
Mechanisms Donald Schneider, Ph.D.
Division of Physiology and Pathology Michael
Martin, Ph.D.
AIDS and Related Research IRG (AARR) Ranga V.
Srinivas, Ph.D.
Biochemical Sciences IRG (BCS) Zakir Bengali,
Ph.D.
Cardiovascular Sciences IRG (CVS) Joyce Gibson,
D.Sc.
Bioengineering Sciences and Technologies IRG
(BST) Sally Amero, Ph.D.
Digestive Sciences IRG (DIG) Mushtaq Khan, Ph.D.
Biology of Development and and Aging IRG
(BDA) Sherry Dupere, Ph.D.
Hematology IRG (HEME) Joyce Gibson, D.Sc.
Immunology IRG (IMM) Calbert Laing, Ph.D.
Biophysical and Chemical Sciences IRG (BPC) John
Bowers, Ph.D.
Integrative, Functional and Cognitive
Neuroscience IRG (IFCN) Christine Melchior, Ph.D.
Cell Development and Function IRG (CDF) Marcia
Steinberg, Ph.D.
Musculoskeletal, Oral, and Skin Sciences IRG
(MOSS) Daniel McDonald, Ph.D.
Genetic Sciences IRG (GNS) Richard Panniers,
Ph.D.
Renal and Urological Sciences IRG (RUS) Daniel
McDonald, Ph.D.
Molecular, Cellular and Developmental Neuroscienc
e IRG (MDCN) Carole Jelsema, Ph.D.
Respiratory Sciences IRG (RES) Mushtaq Khan, Ph.D.
29
Biobehavioral and Behavioral Processes IRG (BBBP)
Study Sections

• BRLE Biobehavioral Regulation, Learning and
  Ethology
• MESH Biobehavioral Mechanisms of Emotion,
  Stress and Health
• LCOM Language and Communication
• CP Cognition and Perception
• APDA Adult Psychopathology and Disorders of
  Aging
• CPDD Child Psychopathology and Developmental
  Disabilities
• MFSR Motor Function, Speech, and Rehabilitation
• BBBP Small Business Activities

30
Health of the Population IRG (HOP) Study Sections

• CLHP Community-Level Health Promotion
• BGES Behavioral Genetics and Epidemiology
• SSPS - Social Sciences and Population Studies
• HSOD Health Services Organization and Delivery
• BMRD - Biostatistical Methods and Research
• ECD - Epidemiology of Chronic Disease
• EPIC Epidemiology of Cancer
• ECDA Epidemiology of Clinical Disorders and
  Aging
• NSCF Nursing Science Children and Families
• NSAA Nursing Science Adults and Older Adults
• HOP Small Business Activities

31
Risk Prevention and Health Behavior IRG (RPHB)
Study Sections

• PDRP Psychosocial Development, Risk and
  Prevention
• PRDP Psychosocial Risk and Disease Prevention
• BMIO Behavioral Medicine Interventions and
  Outcomes
• SPIP - Social Psychology, Personality and
  Interpersonal Processes
• RPHB Small Business Activities

32
Scientific Review Administrator

• Designated Federal Official
• Performs administrative and technical review of
  applications
• Selects reviewers
• Manages study sections
• Prepares summary statements
• Provides requested information about study
  section recommendations to Institutes and
  National Advisory Councils/Boards

33
Selection of Peer Reviewers
Active and Productive Researchers
Research Capability
Non-Research
Non-Doctoral
Scientific Community
34
Criteria For Selection of Peer Reviewers

• Demonstrated Scientific Expertise
• Doctoral Degree or Equivalent
• Mature Judgment
• Work Effectively in a Group Context
• Breadth of Perspective
• Impartiality
• Interest in Serving
• Adequate Representation of Women and Minority
  Scientists

35
Review of Research Grants

• REVIEW CRITERIA
• Significance
• Approach
• Innovation
• Investigator
• Environment
• _________
• Protection of Human Subjects
• Inclusion of Women, Minorities, and Children
• Animal Welfare/Biohazards
• Overall Evaluation Score Reflects Impact on
  Field

36
Review Criteria (continued)

• Significance Does the study address an important
  problem? How will scientific knowledge be
  advanced? What are the societal benefits?
• Approach Are design and methods well-developed
  and appropriate? Are problem areas addressed?
• Innovation Are there novel concepts or
  approaches? Are the aims original and
  innovative?
• Investigator Is the investigator appropriately
  trained?
• Environment Does the scientific environment
  contribute to the probability of success? Are
  there unique features of the scientific
  environment?

37
Additional Review Criteria Amended
Applications (Most Mechanisms)

• Adequacy of response to the previous review
• Degree of overall improvement of the revised
  application

38
Scientific Review Group or Study Section Actions

• Scored, Scientific Merit Rating (priority scores
  and percentiles)
• Unscored (lower half)
• Deferral
• Not recommended for further consideration

39
Priority Scores/Percentile Rank

• For each study section, applications in the upper
  half are scored from 1.0-3.0, with 1.0 the best
  score
• Individual scores are averaged and multiplied by
  100 to give the final priority score

• Percentile ranking is calculated based on
  results of current

• plus past two meetings

40
Summary Statement

• Once applications are reviewed, the results are
  documented by the SRA in a summary statement and
  forwarded to the Institute (and the PI) where a
  funding decision is made
• The summary statement contains
• Overall Resume and Summary of Review Discussion
• Essentially Unedited Critiques
• Priority Score and Percentile Ranking
• Budget Recommendations
• Administrative Notes

41
What Determines Which Awards Are Made?

• Scientific merit
• Program considerations
• Availability of funds

42
Grantsmanship

• Steps in preparing a successful grant application

43
There is no grantsmanship that will turn a bad
idea into a good one, but..There are many
ways to disguise a good one.

• William Raub, Past Deputy Director, NIH

44
The NIH Grant Culture

• Bio-Medical Model
• Randomized Clinical Trial as gold standard
• Evidence based
• Underlying Conceptual Model
• Emphasis on Outcomes
• Need for Measurement

45
Step One Scoping

• Identify possible research projects
• Use web-based NIH data-bases and resources
• Identify candidate NIH Institutes/Centers
• Identify candidate NIH grant initiatives
• Program announcement (PA)
• Request for applications (RFA)
• Investigator initiated application
• Review NIH grant application procedures PHS 398
  Instructions

46
Step Two Make NIH Contacts

• Confer with NIH Program Directors
• Assess the fit to the Institute/Center
• Find out whats new PAs and RFAs
• Decide on mechanism e.g., RO1, R03, R21
• Find collaborators
• Identify review issues Dos and Donts
• Define product and focus application

47
Types of Grants

• R01 Research Projects
• R03 Small Research Grants
• R21 Exploratory/Developmental Grants
• R13 Conference Grants
• R41/R42 Small Business Technology Transfer
  Grants Phase I/II
• K-Awards Career Development
• F-Awards - Fellowships
• P01 Research Program Projects
• http//deainfo.nci.nih.gov/flash/awards.htm

48
NIH Grant Mechanisms

• R01 Traditional investigator-initiated grant
• lt 500K/yr, 3-5 yrs. Need approval if more than
  500K for any year of the grant
• R03 Small Grant
• lt 100K for 2 yrs
• R21 (NCI) Exploratory/Developmental Grant
• lt 275K for 2 yrs
• R13 Conference Grants
• amount dependent on score, timeliness, budget,
  NIH interest

49
NIH Opportunities for Young
Investigators

• National Research Service Individual Fellowship
  (F32)
• Mentored Research Scientist Development Award
  (K01)
• Independent Scientist Award (K02)
• Mentored Clinical Scientist Development Award
  (K08)
• Small Grant (R03)
• Academic Research Enhancement Award (R15)
• Exploratory/Developmental Grant (R21)

50
Step 3 Develop Your Idea

• Review literature
• Generate preliminary data
• Enlist collaborators, include letters of
  commitment
• Review successful grant applications of other
  colleagues

51
The Top Ten List

• Read and re-read the program announcement
• Assemble a strong research team
• Use the strongest study design possible
• If you have not been on a study section, confer
  with someone who has
• Be sure to document the innovations(s)
• Document strong access to the study population
• Make sure the writing, organization, grammar
  are as tight as possible (write, re-writeread,
  re-read)
• Seek reviews from experienced peers before
  submission
• Make careful use of the summary statement
• Persevere and dont take rejection personally

(Source Ross Brownson 1/13/2004)
52
Step 4 Writing the Application

• Clear, concise writing style
• Be focused
• Dont rush
• Critique, critique, and critique again
• Follow up with NIH program directors before and
  after review

53
Step 5 Preparing the Application

• Follow instructions PHS 398
• Never assume that reviewers know what you mean
• Refer to literature thoroughly
• Present a clear rationale for the proposed work
• Make sure that the experimental approach is
  thorough and detailed
• Include well-designed tables and figures
• Anticipate human subject issues

54
General Design Issues

• Will it work?
• Supporting preliminary data
• Valid Instruments
• Pilot data
• Reality check subject burden
• Will compliance rate(s) be adequate

55
Methodological Issues

• Sampling Methods
• Power Calculations
• Theoretical-based Intervention
• Compliances
• Data Acquisition and Management
• Participant Training and Monitoring
• Data Analysis

56
Human Subjects Issues

• Four criteria
• Risks
• Protections
• Benefits to subjects and others
• Importance of knowledge
• Data Safety and Monitoring Plan for clinical
  trials
• Exemptions applicable
• Inclusion plans
• Minorities, women, children,
• Risks include the possibility of physical,
  psychological, or social injury resulting from
  research.

57
More Human Subject Issues

• Recruitment and informed consent
• Vulnerable populations
• Incentives
• Informed Consent
• Participation
• Use of information
• Future analysis

http//ohrp.osophs.dhhs.gov/humansubjects/guidance
/45cfr46.htm
58
Step 6 Submit the application

• Include cover letter
• Request Institute assignment for funding
• Request study section assignment for review
• Indicate potential conflicts
• Suggest expertise but not reviewers by name
• Multiple Institute assignments acceptable
• Institute or CSR review predetermined (you cant
  choose)
• Meet submission deadlines
• If late, ask for exception and provide reason
• Weather, health, study section activities
• Exceptions never granted prior to submission

59
Step 7 Monitor Review Process

• Contact Scientific Review Administrator for
  information and to express any concerns
• Timing
• Institute assignments
• Study section assignment
• Provide input about needed expertise - Do Not
  Suggest Reviewers by Name!
• Identify possible conflicts of study section
  reviewers e.g., professional, personal,
  financial, institutional
• Be mindful that NIH review administrators are
  typically managing multiple meetings involving
  about 100 applications per round

60
Step 8 Post Review Followup

• Contact Program Director for information and
  guidance
• Discuss outcome of merit peer review
• Review summary statement
• What the scores mean (Institute ranking)
• Strengths and weaknesses
• Recommendations for improvement
• Discuss Institute program priorities
• Likelihood of funding
• Next steps

61
Most Common Problems

• Lack of new or original ideas
• Absence of an acceptable scientific rationale
• Lack of experience in the essential methodology
• Questionable reasoning in experimental approach
• Uncritical approach
• Diffuse, superficial, or unfocused research plan
• Lack of sufficient methodological detail
• Lack of knowledge of published relevant work
• Unrealistically large amount of work
• Uncertainty concerning future directions

62
Summary

63
Keys To Success

• Find collaborators and mentors who are
  experienced in writing and winning NIH grants
• Make contact with NIH scientific staff at
  appropriate stages of the review/award cycle
• Institute/Center Program Administrators
• Scientific Review Administrators
• Recognize that NIH peer review has a special
  culture based on standing study sections composed
  of senior academic researchers with long
  histories of service and expectations of style,
  academic rigor, and hypothesis-based research

64
Good Luck!

• If at first you dont succeed
• Revise and resubmit

65
Contact Information

• Charles N. Rafferty, Ph.D.
• Scientific Review Administrator
• Health of the Population IRG
• Center for Scientific Review
• National Institutes of Health
• 6701 Rockledge Drive, MSC 7770, Room 3172
• Bethesda, MD 20892 (20817 for overnight mail)
• 301-435-3562
• 301-480-3962 fax
• raffertc_at_csr.nih.gov
• http//www.csr.nih.gov

66
NIH Information Sources
67
Information on the World Wide WebSelected Sites
of Interest

• National Institutes of Health (http//www.nih.gov)
  
• Office of Extramural Research (http//www.nih.gov/
  grants/oer.htm)
• Grants Policy (http//www.nih.gov/grants/policy/po
  licy.htm)
• Center for Scientific Review (http//www.csr.nih.g
  ov)
• Referral and Review (http//www.csr.nih.gov/refrev
  .htm)
• CSR Study Section Rosters (http//www.csr.nih.gov/
  committees/rosterindex.asp)
• Review Group Meeting Dates
  (http//www. csr.nih.gov/committe
  es/meetings/ssmeet1.asp)
• CSR Reorganization News (http//www.csr.nih.gov/r
  eview/reorgact.asp)

68
CSR Web Site http//www.csr.nih.gov

• News and Events
• Resources for Applicants
• Study Section Information
• Employment Opportunities
• Contact Information

69
END