CSR

1
CSRs Mission and Function and Whats New in Peer
Review
Martha M. Faraday, Ph.D. Scientific Review
Officer Division of AIDS, Behavioral Population
Sciences Risk Prevention Health Behavior
IRG Psychosocial Risk Disease Prevention Study
Section
Date April 22, 2009
National Institutes of HealthU.S. Department of
Health and Human Services
2
National Institutes of Health
Office of the Director
National Institute on Alcohol Abuse and Alcoholism
National Institute of Arthritis
and Musculoskeletal and Skin Diseases
National Cancer Institute
National Institute on Aging
National Institute of Child Health and
Human Development
National Institute of Allergy and Infectious
Diseases
National Institute on Drug Abuse
National Institute of Environmental Health
Sciences
National Institute on Deafness and
Other Communication Disorders
National Eye Institute
National Institute of Dental and Craniofacial Rese
arch
National Institute of Diabetes and Digestive
and Kidney Diseases
National Human Genome Research Institute
National Institute of Mental Health
National Institute of Neurological Disorders
and Stroke
National Institute of General Medical Sciences
National Institute of Nursing Research
National Heart, Lung, and Blood Institute
National Library of Medicine
National Center for Complementary and
Alternative Medicine
National Center on Minority Health and Health
Disparities
National Institute of Biomedical Imaging and
Bioengineering
Fogarty International Center
National Center for Research Resources
Center for Scientific Review
Clinical Center
Center for Information Technology
3
CSR Mission Statement

• To see that NIH grant applications receive
• fair, independent, expert, and timely reviews -
• free from inappropriate influences - so the
• Institutes and Centers within the NIH can fund
• the most promising research.

4
CSR Peer Review 2008 Statistics

• 77,000 applications received
• 56,000 applications reviewed
• 16,000 reviewers
• 240 Scientific Review Officers
• 1,600 review meetings

5
Scientific Review Process
Dual Review System for Grant Applications
First Level of Review CSR/Institute
Review Scientific Review Group (SRG) (Study
Section)
Second Level of Review NIH Institute/Center
Council
6
CSR Review Divisions
7
Assignment to CSR Review Groups
Within an IRG, applications are assigned for
review to

• Standing Study Sections when the subject matter
  of the application matches the referral
  guidelines for the study section
• Special Emphasis Panels (SEPs) when the subject
  matter does not fit into any study section, or
  when assignment of an application to the most
  appropriate study section would create a conflict
  of interest. Also used for special mechanisms
  (e.g., fellowships, SBIRs, AREAs)

8
When Preparing Your Application

• Read the instructions
• Never assume that reviewers will know what you
  mean
• Refer to the literature thoroughly
• State rationale of proposed investigation
• Include well-designed tables and figures
• Present an organized, lucid write-up
• Remember to address human subjects, vertebrate
  animals, potential biohazards these could affect
  your score
• Obtain pre-review from faculty at your
  institution
• NIH Grant Writing Tips http//grants.nih.gov/gran
  ts/grant_tips.htm

9
Directing Your Application to a Specific Study
Section

• Peruse CSR Study Section Guidelines to Identify
  a Possible Home for Your Application
• http//csr.nih.gov/
• Recently revised alternative study sections
  listed in approximate order of degree of overlap
• Submit a Cover Letter

10
CSR Web Site http//www.csr.nih.gov

• About CSR
• News and Reports
• Peer Review Meetings
• Resources for Applicants

Study Section Descriptions Rosters
11
Role of Scientific Review Officer (SRO)
Designated Federal official with overall
responsibility for the review process

• Performs administrative and technical review of
  applications to ensure completeness and accuracy
• Selects reviewers based on broad input
• Manages study section meetings
• Prepares summary statements
• Provides any requested information about study
  section recommendations to Institutes/Centers and
  National Advisory Councils/Boards

12
WHOM DO I CONTACT?

• Before review, contact the Scientific Review
  Officer in CSR
• After review, contact your Program Officer in the
  NIH funding institute or center

13
Pre-Meeting Review Process

• Appropriate reviewers recruited by SRO minimum
  of 3 interactive reviewers per application
• Conflicts of interest identified
• Applications made available to reviewers 6 weeks
  prior to meeting
• Critiques and preliminary scores posted by
  assigned reviewers on NIH web site at least 2-3
  days prior to meeting
• Critiques and preliminary scores (excluding
  conflicts) available to review group prior to
  meeting

14
Where Do We Find Reviewers?

• National Registry for Society-Recommended
  Reviewers
• Successful applicants
• Word of mouth
• Recommendations from study section members
• Recommendation from NIH IC staff
• CRISP (crisp.cit.nih.gov)
• PubMed
• Scientific Conferences

15
Traditional Review Meeting Process

• Upper half applications discussed
• Reviewers are guided by specific review criteria
  
• Protections for Humans, Vertebrate Animals,
  Environment (Biohazard) may affect final score
• Assigned reviewers recommend scores for each
  application in upper half all members not in
  conflict vote their conscience (outlier score
  policy pertains)
• Other considerations not affecting final score
  are discussed (e.g., budget, foreign applicants,
  resource sharing plans)
• Lower half applications not discussed, not
  assigned an overall score
• Aspects of this process will change in May,
  2009
• http//enhancing-peer-review.nih.gov

16
Post Meeting Review Process

• Scores are provided to investigators within 3
  working days
• Summary Statements for discussed and scored
  applications include Resume Summary of
  Discussion, (largely unedited) critiques, and
  other recommendations (e.g., Budget)
• Summary Statements for lower half (Not Discussed)
  applications receive (largely unedited) critiques
  and review criteria scores but no overall impact
  scores
• All Summary Statements are made available within
  30 days of meeting (10 days for new
  investigators R01s)

17
Whats New in Peer Review?
18
2008 The Year of Peer Review
Enhancing Peer Review Fund the best science, by
the best scientists, with the least
administrative burden Elias Zerhouni, MD,
Former Director, NIH
19
Recommendations
20
Amended Applications

• To speed the funding of meritorious science and
  minimize reviewer burden
• As of January 25, 2009, all original new
  applications (i.e., never submitted) and
  competing renewal applications will be permitted
  only a single amendment (A1).

21
Whats New in Peer Review

• New Investigators/Early Stage Investigators
• Enhanced Review Criteria
• Template-Based Critiques
• Scoring Scale (9 point scale)
• Criterion Scoring
• Overall Impact Score

22
New Investigators/Early Stage Investigators

• New Investigator (NI)
• PD/PI who has not yet competed successfully for a
  substantial NIH research grant
• For multiple PD/PIs-all PD/PIs must meet
  requirements for NI status
• Early Stage Investigator (ESI)
• PD/PI who qualifies as a New Investigator AND is
  within 10 years of completing the terminal
  research degree or is within 10 years of
  completing medical residency (or equivalent)
• Applies only to R01 applications
• New Investigators/Early Stage Investigators will
  be clustered together for review

23
Enhanced Review Criteria

• Overall Impact
• Assessment of the likelihood for the project to
  exert a sustained, powerful influence on the
  research field(s) involved
• New Core Criteria Order
• Significance
• Investigator(s)
• Innovation
• Approach
• Environment
• Review criteria enhanced and expanded

24
Critiques
Template-Based Critiques

• Goal To improve the quality of the critiques and
  to focus reviewer attention on the review
  criteria
• Provide clear, concise, and explicit information
• Aid in identifying the strengths and weaknesses
  of each criterion

25
Template-Based Critiques

• Critique template contains a total of 18 boxes
• Reviewers should provide text for only those
  criteria that are applicable.

1. Significance 6. Resubmission 13. Overall Impact
2. Investigator(s) 7. Renewal 14. Budget and Period of Support
3. Innovation 8. Revision 15. Select Agents
4. Approach 9. Protection of Human Subjects 16. Applications from Foreign Organization
5. Environment 10. Inclusion of Women, Minorities, and Children 17. Resource Sharing Plan
11 Vertebrate Animals 18. Additional Comments to Applicant
12. Biohazards
26
Template-Based Critiques

• Goal is to write evaluative statements and to
  discourage summarizing the application
• Comments should be in the form of bullet points
  or if necessary short narratives
• Do not record scores on the critique template
• The entire template is uploaded to IAR to become
  part of the summary statement.

1. Significance  Please limit text to ¼ page
Strengths Weaknesses Strengths Weaknesses
27
Scoring 9 Point Scale

• Goal To improve the transparency of the scoring
  process
• Score applications on five review criteria using
  a scale of 1-9.
• Preliminary overall impact score using 1-9
  scale.
• Should not be the average of the criterion
  scores.
• Not Discussed applications will receive initial
  criterion scores from the three assigned
  reviewers

28
Scoring Descriptions
Impact Score Descriptor Strengths/Weaknesses
High Impact 1 Exceptional   Weaknesses
High Impact 2 Outstanding   Weaknesses
High Impact 3 Excellent   Weaknesses
Moderate Impact 4 Very Good   Weaknesses
Moderate Impact 5 Good   Weaknesses
Moderate Impact 6 Satisfactory   Weaknesses
Low Impact 7 Fair   Weaknesses
Low Impact 8 Marginal   Weaknesses
Low Impact 9 Poor   Weaknesses
29
Clustering

• NI/ESI R01 applications will be clustered
  together in review.
• ESI applications will not be separately
  clustered within the NI\ESI group.
• NI/ESI applications will be identified for
  reviewers so there can be appropriate review in
  context of career stage.
• Expectations of preliminary data and publication
  track record less than for established
  investigators.

30
Order of Review

• Goal Discuss applications in order of average
  preliminary score.
• Why
• Concern - variation of scores during different
  times of the meeting.
• One recommendation was to recalibrate scores at
  the end of the meeting .
• Solution
• Recalibrate dynamically throughout meeting.

31
Order of Review

• For calibration purposes
• Begin meeting by discussing the best scored
  application (any activity code)
• NI/ESI R01s clustered beginning of meeting
• All other activity codes clustered if feasible
  (if at least 10 discussed (may include R03, R15,
  and R21s as a group that can be clustered)

32
Order of Review

• Summary
• Discussion order is based on the average of the
  impact scores from assigned reviewers
• Final scores of discussed applications may differ
  from preliminary scores as re-calibration happens
  dynamically

33
Not Discussed

• Discuss 50-60 of applications
• SRO will then ask if there are any other
  applications that panel wishes to discuss
• The remaining applications will not be discussed
• (applications receive criterion scores only)
• Same after review of 60 of SBIR applications

34
Final Scores

• Discussed applications will receive an overall
  score from each eligible (i.e., without conflicts
  of interest) panel member and these scores will
  be averaged to one decimal place, and multiplied
  by 10. The 81 possible priority scores will thus
  range from 10-90.
• Percentiles will be reported in whole numbers.

35
Summary Statements

• Summary statement will be shorter and more
  focused.
• Discussed applications will also have a summary
  of the panels discussion at the meeting.
• ALL applications will be scored.
• Not discussed applications will receive criterion
  scores only.

36
Recruiting the Best Reviewers

• Move a meeting a year to the West Coast
• Additional review platforms
• Develop a national registry of volunteer
  reviewers
• Searchable database with 4,000 reviewers
• Provide tangible rewards for reviewers
• No submission deadlines for chartered members
• of study sections (effective February 2008).
• 1574 chartered members used flexible deadlines
• during the last 6 months
• Provide flexible time for reviewers
• Choice of 3 times/year for 4 years or
• 2 times/year for 6 years

37

• THANK YOU!

This is CSR