Lessons Learned From Peer Reviews of Other National Programs

1
Lessons Learned From Peer Reviews of Other
National Programs

• John W. Radin
• National Program Leader
• Plant Physiology and Cotton

2
Plant Biological and Molecular Processes

• ARS National Program 302
• John W. Radin and Kay W. Simmons, Co-Leaders

33.9 Million Annual Research Budget 99
Scientists (FTEs) 37 Locations
Hypoallergenic soybeans Good for humans and pigs
Late blight-resistant potatoes through genetic
engineering
3
Plant Biological and Molecular ProcessesLaborator
y Locations
4
Plant Biological and Molecular ProcessesTwo-Year
Initial Timeline for Program Establishment
1999 June Customer Workshop I Sept. Customer
Workshop II Dec. Draft Action Plan Written and
Made Available
2000 Feb. Customer Feedback Incorporated into
Action Plan Sept. Project Plans Submitted for
Peer Review Nov. Plans Reviewed by External
Peer Panels
2001 Mar. Revised Plans Submitted as
Needed June Plans Approved and Activated
5
Plant Biological and Molecular Processes Peer
Review Results
Results after revisions, June 2001 (35 projects)
Results before revisions, Nov. 2000 (36 projects)
6
Plant Biological and Molecular ProcessesPeer
Review Follow-up

• Of 36 initial projects, 13 were rated Needs
  Major Revision and 1 was rated Unfeasible.
  Both categories require resubmission to the
  panel.
• One sub-standard project was discontinued, and
  the personnel and resources were reassigned. The
  others were extensively revised with much closer
  oversight from the Area Directors and NPS.
• On second review, 1 project was rated Needs
  Major Revision. It underwent a second round of
  restructuring, with even closer oversight.
• The last project was approved in Jan. 2002, 14
  months after the peer panel met.

Moral Take the process seriously
7
The Top 5 Causes of Failure

• Contingencies are addressed carelessly if
  considered at all. Few research programs
  progress exactly as planned. What will you do if
  the research plan does not work?

Example of poor contingency plan If we are
unable to demonstrate potential for effective
biological control with this predator species, we
will work with other species.
Needed (1) Which new insect species will be
evaluated? (2) Why are they expected to be
useful? (3) What will be the sources for the new
species? (4) Do new handling and rearing
procedures need to be developed?
8
The Top 5 Causes of Failure

• 4 Cooperators are almost always needed because
  they bring in specialized skills, but the plans
  ignore or underplay that need. Panels look for
  specific documentation of collaborations, i.e.,
  letters with a description of what the cooperator
  will actually do.

Example of poorly documented cooperation Allellop
athic effects of cover crops will be evaluated in
the field by Dr. John Smith.
Needed Details of Dr. Smiths contribution (1)
What crops/rotations? (2) What cover crops? (3)
Which weeds? Dr. Smith must also supply a letter
describing his input.
9
The Top 5 Causes of Failure

• 3 Plans are unfocused, or objectives are
  unrelated to each other. Panels look for
  objectives that clearly build upon each other. Do
  not combine unrelated objectives just because
  they seem interesting or convenient for
  management purposes.

Reviewers evaluate whether the planned work will
reach the stated goal. This includes (1)
technical quality and (2) whether the information
acquired will be interpretable and applicable.
The latter usually requires a multi-faceted
coordinated approach. Both are equally important.
10
The Top 5 Causes of Failure

• 2 Too many objectives dilute resources and
  minimize the likelihood of success. Describe
  only what you expect to accomplish during the
  5-year life of the project. Provide preliminary
  data, if you have it, to show feasibility.

Example of over-written plan We will genetically
transform insects with a conditional lethal gene,
evaluate its stability, inheritance, expression,
and phenotypic effects, obtain permits for
release of the transgenic insects, and monitor
their effectiveness in field trials.
Needed A better grip on reality!
11
The Number 1 Cause of Failure

• Plans are written without acknowledging parallel
  work elsewhere in ARS, and without building
  multi-laboratory teams to take advantage of
  additional resources. This is a major reason for
  the workshop. Identify and make those
  connections while you are here.

The panels have a unique broad view of research
across ARS. Most have been greatly distressed
when they see failure to communicate, cooperate,
and coordinate. If other scientists are
collecting relevant data, plan and work jointly
with them.
12
What Has Peer Review Accomplished So Far?

• ARS research is given much higher credibility
  than previously (both by groups evaluating the
  process and by peer scientists who participate in
  panels).
• Panels do not move or redirect funds, which
  Congress has appropriated for specific locations
  and objectives. Instead, they identify the
  poorest projects for special attention and
  upgrading (unlike a competitive grants panel,
  which concentrates on only the best proposals).
  Without question, this has been good for ARS
  without altering priorities.
• The best projects have little trouble with peer
  review. (See Top-5 list of problems).
• The system is working as intended.