NIH Peer Review Where Should We Go From Here McGowan

1
NIH Peer Review Where Should We Go From Here
John J. McGowan, Ph.D. Director, Division of
Extramural Activities Acting Associate Director
for Management and Operations National Institute
of Allergy and Infectious Diseases
2
Objectives this Morning

• Putting the discussion in context
• Past, current, and future attempts to expedite
  the award cycle
• Discuss ideas, answer questions, listen to your
  comments

3
Stress in the System

• People
• Systems
• Resources

Flat budgets increase the pressure
4
Expectations from the Research Community

• Scientist as applicant
• Will I get funded?
• How quickly can I get feedback to resubmit?
• Scientist as reviewer
• Is my time being well spent Is outstanding
  science being funded?
• Can I help advise the applicant?

5
Expectations within NIH

• NIH science administrator for review
• Was best science identified using the criteria to
  assess scientific and technical merit and
  feasibility?
• Did reviewer comments get captured on recombinant
  DNA research, human subjects, research animals,
  and select agents, and more?
• NIH science administrator for program
• How does the science identified map to my
  portfolio
• What insights did I gain from the discussion to
  help advise
• me for those who are getting funded?
• the applicant that is not funded?

6
Core Values

• Increase Quality
• Improve Efficiency
• Provide Increased Flexibility
• Help Prioritize Work

7
Steps in the Process
Adminreview
Meeting preparation
Pre-award activities
Pre-Council activities
Post-rev docs
Receipt
Peer reviewmeeting
Advisory Council
Reviewers assigned,materials distributed
Grantaward
Scores and summarystatements released
8
Two Options to Speed Up Referral and Assignment

• Add additional staff to speed up CSR processing
  time
• Streamline steps in the current CSR process
• Use scanned in applications and pilot prototype
  technology with re-engineered processes to
• Find potential reviewers
• Identify conflicts of interest
• Make study sections assignments
• Refer to the appropriate IC

9
Adding Both Could Cut 2 months
Process Step Months

• Receipt
• IC Study Section Assignment
• Administrative Review

0 2 3

• NIH Enterprise level
• Knowledge management technology - integration
• IC local solutions
• Local software
• Macro that crosschecks conflict of interest, PIs,
  and reviewers.
• Use of an Excel workbook which has
• Administrative codes put in during administrative
  review
• Established boilerplate language to merge with
  summary statements
• Ease production of summary statements.

10
Steps in the Process
Adminreview
Meeting preparation
Pre-award activities
Pre-Council activities
Post-rev docs
Receipt
Peer reviewmeeting
Advisory Council
Reviewers assigned,materials distributed
Grantaward
Scores and summarystatements released
11
Pilots on Meeting Preparation

• Reviewers select applications that they would
  review

12
Pilot Possibilities Meeting Preparation

• SRA moderates, approves choices, eliminates
  potential conflicts of interest
• Allow reviewers to choose applications that are
  closest to their interest and field of expertise

13
Pilots on Meeting Preparation

• Reviewers select applications that they would
  review
• Internet submission of critiques and scores prior
  to the meeting

14
Pilot Possibilities Meeting Preparation
Internet Submissions Prior to the Meeting
Pilot to Implementation Takes Time
Initiatives Pilots
Impact
Year

• 1995
• 1996
• 1998
• 2001

NIAID Electronic assisted peer review Upload
Critiques/Scores/ Review prior to the
meeting Expand pilots to ICs CSR NIAID
Service CSR ICs NIH Internet assisted peer
review

• Improve quality of review
• More informed discussion
• More effective use of time

Confirm initial results NIH adopts Expedited
awards
NIH incorporates into IMPAC 2
Takes Time -
- Commitment
15
Internet Assisted Review (IAR)

• Scores and critiques loaded viewed on a secure
  Internet
• Contents can made available to reviewers for
  phone secure web blog or email for discussion.
  Thus far has worked best when there are a limited
  number of
• Applications
• Reviewers (for example, highly specialized
  individuals and with clinical investigators)

16
Evaluation Comments from Reviewers

• Allowed a more thoughtful consideration of
  difficult or controversial points Dyann Wirth

17
Electronic Initial Review Pilots Reviewers
Feedback

• Improved the quality of review
• Assist the giving feedback to applicants in the
  gray zone
• Permit more effective use of reviewers time at
  review meetings
• May decrease the time required in a specific
  review meeting

focus groups with 168 reviewers from 32
separate review meetings
18
Reviewers Like It !!!!
Did access to critiques prior to review result in
a more informed discussion ?
Did the availability of the critiques alter how
you scheduled your time for reviews ?
45
Yes
Would you recommend that this system be extended
to other study sections ?
19
Rank OrderAs a Result of Viewing Comments
Scores
100
80
68
50
20
Pilots on Meeting Preparation

• Reviewers select applications that they would
  review
• Internet submission of critiques and scores prior
  to the meeting
• Virtual meetings prior to or in place of a review
  meeting

21
Pilot Possibilities Virtual Meetings

• Cost-benefit ratio for concept of not having
  standard in-person meetings?
• Increased flexibility
• More communication options make it easier for
  reviewers.
• Also makes it easier to recruit top-quality
  reviewers
• Decreased committee management costs.
• But, virtual meetings reduce the mentoring and
  networking aspects of review meetings.

22
Adding Both Could Cut 2 months
Change
Process Step Months
Receipt IC Study Section Assignment
Administrative Review Reviewers Assigned /
Materials Distributed Peer Review Meeting
0 1 2 4
Technology pilots
0 2 3 3-5 4-6
Reviewers need 6-8 weeks read and critique
applications

• Internet Assisted Review
• Technology and tools in place and have improved
  quality and hasten summary statement generation
• Enabled flexibility for reviewers in their
  participation in the review process
• Being used with other methods on small scale for
  certain mechanisms or when limited numbers of
  applications are involved
• What pros and cons for not having a review
  meeting for selected to all mechanisms?
• Knowledge management technology - integration
• IC local solutions
• Local software
• Macro that crosschecks conflict of interest, PIs,
  and reviewers.
• Use of an Excel workbook which has
• Administrative codes put in during administrative
  review
• Established boilerplate language to merge with
  summary statements
• Ease production of summary statements.

23
Steps in the Process
Adminreview
Meeting preparation
Pre-award activities
Pre-Council activities
Post-rev docs
Receipt
Peer reviewmeeting
Advisory Council
Reviewers assigned,materials distributed
Grantaward
Scores and summarystatements released
24
Potential Pilots on How Review is Done

• Virtual meetings NSF model of only 3 reviewers
• Pilot selective mechanisms R03, R21, SBIR?

25
Potential Pilots on How Review is Done

• Virtual meetings NSF model of only 3 reviewers
• Changing the triage threshold

26
Potential Pilots on How Review is Done

• Virtual meetings NSF model of only 3 reviewers
• Changing the triage threshold
• Different methods to group applications

27
Pilot PossibilitiesGrouping Applications

• Sort applications into bins High, Medium, and
  Low.
• Medium full summary statement.
• Summary statements for others are just critiques.
• This saves time normally spent preparing
  additional summary statements.

28
Potential Pilots on How Review is Done

• Virtual meetings NSF model of only 3 reviewers
• Changing the triage threshold
• Different methods to group applications
• Selective discussion
• Changes in the review meeting
• Production aspects
• Different Outcomes

29
Pilot PossibilitiesSelective Discussion

• Up to 80 to 90 percent concurrence on scores on
  most applications as scored in IAR
• Focus on reviewing the applications where there
  is disagreement.
• Requires consensus on the definition of
  concurrence by the reviewers.

30
Potential Pilots on How Review is Done

• Virtual meetings NSF model of only 3 reviewers
• Changing the triage threshold
• Different methods to group applications
• Selective discussion
• Changes in the review meeting
• Production aspects
• Different Outcomes

31
Production Changes at the Review Meeting

• Score at the meeting.
• DEAS staff calculate scores and enter data into
  IMPAC by the end of each meeting day and
  validation by the SRA within 3 days.
• Primary reviewer creates written summary after
  discussion during the meeting.
• Primary reviewers already provide an oral summary
  at the meeting.
• Record oral summary for on-site transcription.
• Immediate confirmation of transcribed text by
  reviewers results in the written summary.
• Recording can then be erased.
• Used to make the resume.

32
Potential Pilots on How Review is Done

• Virtual meetings NSF model of only 3 reviewers
• Changing the triage threshold
• Different methods to group applications
• Selective discussion
• Changes in the review meeting
• Production aspects
• Different Outcomes abbreviated amendments

33
Different OutcomesExpedited Re-Review of Amended
Applications
34
Expectations from the Research Community

• Scientist as applicant
• Will I get funded?
• How quickly can I get feedback to resubmit?
• Scientist as reviewer
• Is my time being well spent Is outstanding
  science being funded?
• Can I help advise the applicant?

35
Conventional Submission and Review of Amended
Applications

• The time between initial submission and award for
  amended applications is very long
• Preparation of complete resubmissions requires
  substantial time and effort
• High impact issue in FY95, 60 of applications
  funded by NIAID were amended

36
Expediting Re-ReviewAbbreviated Amendments

• Study Section members identify meritorious
  applications suitable for abbreviated process
• Applicants chooses to resubmit a revised
  application or respond to the issues raised by
  study section
• If applicant chose the abbreviated process the
  SRA generates a letter with concerns from the
  community
• Applicant submits a response back to the SRA
  within 4-6 wks later than usual for revised
  applications
• Study section reads the response and scores the
  application.

37
Abbreviated Amendments

• Tropical Medicine and Parasitology (TMP)
• Jean Hickman, SRA
• average 50 applications/cycle
• gt98 of applications assigned to NIAID
• 18 out of 19 of reviewers have Internet access

38
Tropical Medicine Pilot

• Instead of revising and resubmitting applications
  in the potential gray zone, this pilot allowed
  applicants to address points in a letter.
• This took pressure off of all parties compared to
  a resubmission.
• Review staff had to coordinate the letters and
  responses.
• Lack of standard processes, systems and
  technology made it difficult to sustain the pilot

39
Unlinking the Council Meeting from Council Review
Adminreview
Meeting preparation
Pre-award activities
Pre-Council activities
Post-rev docs
Receipt
Peer reviewmeeting
Advisory Council
Reviewers assigned,materials distributed
Grantaward
Scores and summarystatements released
40
Expectations from the Research Community

• Scientist as applicant
• Will I get funded?
• How quickly can I get feedback to resubmit?
• Scientist as reviewer
• Is my time being well spent Is outstanding
  science being funded?
• Can I help advise the applicant?

41
Expedited Council Review Saves 3 months
Process Step Months
Receipt IC Study Section Assignment
Administrative Review Reviewers Assigned /
Materials Distributed Peer Review
Meeting Post-Review Activities Official
Scores Summary statements released Pre-Council
Activities Council Meeting Pre-award
activities Grant Award
0 2 3 3-5 4-6 6-7 8-9
42
Examples of Prior Experiments
Initiatives Pilots
Impact
Year

• 1994
• 1995
• 1996
• 1998
• 1999
• 2001

Expedited Awards Internet assisted peer review
Upload Critiques/Scores/ Review prior to
the meeting Expand pilots to ICs CSR NIAID
Service CSR ICs Hyper-accelerated
Review Internet assisted peer review

• Awards made 1-3 months earlier
• Improve quality of review
• More informed discussion
• More effective use of time

Confirm initial results NIH adopts Expedited
awards Receipt to award in 3-4 months grants
doing ancillary studies to ongoing clinical trials
NIH incorporates into IMPAC 2
43
Features of Hyper-accelerated Review

• Designed for additional research studies to an
  ongoing clinical trial
• Monthly processing dates (after the 9th)
• 10 page limitation on the research plan
• Budgets limited to 250K and below per year
• Reviews done using IAR with three reviewers and
  if needed a follow-up discussion.
• Only 1 amended application is possible
• Option is made available only by invitation from
  NIH staff
• limited to 5 pages
• Must directly addresses the questions and
  concerns raised in the initial review.
• Awards made in 3 to 4 months
• Resource intensive

44
Hyper-accelerated Review
45
Stress in the System

• People
• Systems
• Resources

Flat budgets increase the pressure
46
(No Transcript)
47
Questions, Comments, Discussion
Sought

• Most things are rarely as easy as they
  seem-especially when youre the one trying to do
  them
• 
  Juli Baldwin