Peer Review of ARS Research Projects

1
Peer Reviewof ARS Research Projects
A Brief Overview of the Process

• Office of Scientific Quality Review
• Steven Huber, Scientific Quality Review Officer

2
OSQR Web Sitehttp//www.osqr.ars.usda.gov

• National Programs
• Schedules of Reviews
• Forms and Resources (TIPS!)
• Reviewer Information
• Peer Review status
• Manual (online comments soon)
• Staff

3
Topics to Cover

• Background
• Role of the Research Leader
• The Prospectus Project Plan
• General writing tips
• The panel process
• Outcomes

4
New Mandate for this process

• The Agricultural Research, Extension and
  Education Reform Act of 1998 (public law
  105-185)
• ARS projects be evaluated by panels composed
  predominantly of non-ARS scientists
• With oversight from the REE Advisory Committee

5
Intent of Peer Review

• To obtain constructive, independent, expert
  feedback and advice on ways to improve the
  scientific and technical merit of each project
  plan
• NOT funding issues

6
Fundamental Questions

• Panelists are asked to evaluate
• MERIT AND SIGNIFICANCE
• (Relevance to NP)
• APPROACHES AND PROCEDURES
• LIKELIHOOD OF SUCCESS
• (feasibility)
• Suggestions for improvements or solutions to
  problems are solicited

7
Steps in the Process
16 wk (24 wk)
RL and Scientists develop Prospectus
NPS
RLs Prepare Project Plan with LSs Submit Plan for
approval
AD
18 wk (14 wk)
OSQR
6 wk
Peer Review
Response and Revision Submit revised project plan
6 wk
AD/NPS
6 wk
OSQR
Re-review?
8
Scientists Responsibilities

• Communicate with the NP Team.
• Scope of the project.
• Coordination with other projects.
• QUALITY of the Project Plan.
• Many are poorly written.

9
Project Plan Template

• Title and Investigator(s)
• TOC (coming soon!)
• Project summary
• Objectives (lt ½ p)
• Need for research (1p)
• Scientific Background (5-6 p)
• Approach Procedures (6-12 p)
• Milestones Outcomes (? 1 p)
• Literature Cited
• Past Accomplishments
• Issues of Concern statements
• Appendices

15 - 25 pages
10
Scientific Background

• Focus on the pertinent literature
• Show linkage/coordination with other similar
  projects in the NP (or other NPs) and with
  minority-coded projects in the MU.

11
GENERAL TIP

• Use illustrations (figures, schemes, etc) to
  enhance the Plan. (Up to 2 pages does not count
  against page limit).
• Preliminary data
• How your project fits with others
• Working models
• Experimental design/treatments

12
Approach and ProceduresTips

• Lack of necessary detail is the most common
  criticism.
• Tell what will be done, by whom, and what will
  result. Identify staff involved with key portions
  of the project.
• If an objective is supported by a grant, mention
  that.

13
The Panel
Panelists Primary reviewers Secondary reviewers
OSQR Officer Provides orientation
Observes Debriefs Receives results
Panel Chair
Ad-hoc reviewers may be used
14
Panel Chair Selection
SUGGESTIONS for Non-ARS Panel Chair from NPS,
ADs, etc.
OSQR Officer Selects Chair
OSQR Officer Consults inside and outside ARS
15
Panelist Selection
SUGGESTIONS for Panelists from NPS, ADs, RLs,
Lead Scientists, University Scientists,
Customers, etc.
OSQR Officer Approves Panel
Panel Chair (PC) Makes recommendations to OSQR
Officer
Based on PC recommendations and diversity
requirements
16
Suggested Panelists
Use this form!
17
The Panel Meeting Process
Panel Discussion Primary reviewer presents the
plan input from secondary reviewer, panelists,
and ad hoc reviewer(s).
? 1 hour
Turn in to SQR Officer
Panelists and Panel Chair assign an action class
to the project.
Turn in to SQR Officer
Primary reviewer modifies the review
recommendation on disk
Panel Chair Approval
On breaks, evenings
18
Action Classes

• Each panelist individually provides overall
  Action Class assignment
• No revision required. Minor revision might be
  made.
• Minor revision required. Objectives fit the NP
  action plan approaches to all objectives are
  sound. Project is feasible.
• Moderate revision required. Moderate revision of
  an objective and/or one approach needed. Project
  is feasible.
• Major revision required. Should be sound and
  feasible after major revision. (May be
  re-reviewed).
• Not feasible. Deficiencies in expertise or
  facilities or major flaw.

Action Class Matrix will be provided
19
After the Panel Meeting

• If the composite Action Class is
• No revision required
• Minor revision required
• Moderate revision required
• Major revision required
• Not feasible

Scientists solicit comments and revise the Plan
within 6 weeks
Revised Plans will be revised AND re-reviewed
Revise for immediate re-review delay revision
until major resource changes (e.g., personnel)
are made or terminate
20
Responding to the Review Recommendations
Project Title____________________________________
_________________________ CRIS __________________
_____ National Program_________________________
_ Lead Scientist_________________________________
Reviewer _________________ PEER REVIEW OF ARS
RESEARCH PROJECT 2. Adequacy of Approach and
Procedures Please comment on the following Are
the hypotheses and/or plan of work well
conceived? Are the experiments, analytical
methods, and approaches and procedures
appropriate and sufficient to accomplish the
objectives? How could the approach or research
procedures be improved? Please list and number
each significant recommendation being made. Be
sure to briefly state the rational or basis for
suggestions made or questions raised. Each
recommendation can include several specific
questions you believe should be addressed by the
lead scientist. 1. Objective 1--The hypothesis
being tested is only one of several that should
be considered. For example ARS Response
21
ARS Responses to Review Recommendations

• ARS scientists must respond to each major
  recommendation.
• TONE of the response should be receptive, not
  defensive or condescending.
• CONTENT make all reasonable efforts to
  accommodate suggestions made. Lack of
  adoption must be justified. Should stand
  alone.

The completed forms go back to the panelists!
22
Re-Review of Project Plans

• Purpose
• 1) independent evaluation of response, and
• 2) maintain credibility with
• community.

23
NP108, NP302, NP202 Panel Statistics

• 7 Panel Chairs and 44 Panelists (51 total)
• 1 ARS employee
• 33 employed by universities
• 7 employed by private industry, trades groups, or
  self-employed
• 10 Other federal agencies (U.S. and Canada)

24
NP108, NP302, NP202 Panel results
Number of Projects
Action Class
25
Comments from Panelists

• Think the peer review process is important for
  the agency
• Appreciated the positive approach of the ARS peer
  review process.
• Enjoyed the experience (opportunity for them to
  discuss science with peers).
• Hope the review recommendations will be useful
  and incorporated.

26
Benefits of Peer Review

• Better short- and long-term thinking and planning
  of research (if done properly!).
• Increased communication among scientists, ADs and
  NPLs.
• Increased collaboration (especially among ARS
  labs).
• Increased qualityperspectives from peer
  reviewers.
• Greater ability to compete for competitive funds.

27
OSQR Web Sitehttp//osqr.ars.usda.gov/

• Peer Review Status
• Manual
• Staff

• National Programs
• Schedules of Reviews
• Forms and Resources
• Reviewer Information