Patricia M' Dehmer

1
Performance Assessment in DOEsOffice of Science
Current Practices
Patricia M. Dehmer Associate Director of Science
for Basic Energy Sciences 24 January 2002
2
The Government Performance and Results Act
The Government Performance and Results Act of
1993 (GPRA or the Results Act) holds agencies
accountable for program performance by requiring
that they think strategically and that they set,
measure, and report on goals annually, usually in
budget documents.
3
Goals for Federally Funded Science
SCIENCE, TECHNOLOGY, AND THE FEDERAL GOVERNMENT
National Goals for a New Era (COSEPUP, 1993)

• Summary of Science Goals
• The first goal is that the U.S. should be among
  the world leaders in all major areas of science.
• The second goal is that the U.S. should maintain
  clear leadership in some major areas of science.
  The decision to select a field for leadership
  would be based on national objectives and other
  criteria external to the field of research.
• The comparative performance of U.S. research in a
  major field would be assessed by independent
  panels of experts from within and outside the
  field.

4
Evaluation Criteria for Federally Funded Science
EVALUATING FEDERAL RESEARCH PROGRAMS Research
and the Government Performance and Results Act
(COSEPUP, 1999)

• Among the Six Conclusions
• Both applied research and basic research programs
  can be evaluated meaningfully on a regular basis.
• The most effective means of evaluating federally
  funded research programs is expert review, which
  includes quality review, relevance review, and
  benchmarking.
• Agencies must evaluate their research programs by
  using measurements that match the character of
  the research.
• The development of effective methods for
  evaluating and reporting performance requires the
  participation of the scientific and engineering
  community.

5
The challenge is to integrate GPRA, COSEPUP,
and our own goals and measures
SCs immediate tasks Craft the details to be
set down in our budget documents of long-term
goals and annual targets to enable the U.S. to be
among the world leaders in each area of science
supported by SC. Assess progress toward the
goals.
6
Performance Assessment Current Practices in
DOEs Office of Science
What is assessed? The obligatory SC budget
pie How are assessments made? Assessment methods
for the different parts of the pie
7
Office of Science
Director James F. Decker (Acting) Principal
Deputy Director Milton D. Johnson
(Acting) Deputy Director for Operations James
Turi (Acting)
ASCAC BESAC BERAC FESAC HEPAP NSAC
Program FY01 Funding
BES 973.8M
BER 514.1M
FES 241.9M
ASCR 161.3M
HEP 695.9M
NP 351.8M
Office of Advanced Scientific Computing
Res. Associate Director C. Edward Oliver
Office of Basic Energy Sciences Associate
Director Patricia Dehmer
Office of Biological and Environmental
Res. Associate Director Aristides Patrinos
Office of Fusion Energy Sciences Associate
Director N. Anne Davies
Office of High Energy and Nuclear
Physics Associate Director S. Peter Rosen
Chicago Operations Office Manager Marvin E.
Gunn, Jr.
Oak Ridge Operations Office Manager G. Leah
Dever
Berkeley Site Office Director Richard H. Nolan
Stanford Site Office Director John S.
Muhlestein
Office of Resource Management Associate
Director John Rodney Clark
Office of Planning and Analysis Director William
J. Valdez
Office of Laboratory Policy Director Antoinette
Joseph
Office of Lab. Operations and ESH Associate
Director James Turi
135.4M
0.9M
61.3M
Science Education 4.5M
NOTE Director of Science equivalent to
Assistant Secretary position and filled by
Presidential Appointment (Senate confirmed)
Principal Deputy Director equivalent to Principal
Deputy Assistant Secretary Associate Directors
equivalent to Deputy Assistant Secretaries.
Appropriation after reductions for the General
Reduction.
8
Budget Elements of the Office of Science
Program Direction 139.9M
Safeguards Security 34.4M
FY 2001 Budget 3,140.9 M
AIP 28.4M
Construction 315.5M
Research (Universities) 709.2M
GPP GPE 50.7M
Capital Equipment 224.8M
Research (Laboratories) 785.8M
Major User Facilities 852.2M
Includes the funding for not-for-profits,
other agencies, and private institutions
9
FY 2001 University Research Programs
10
Research Areas

• Chemical Sciences
• Analytical Chemistry
• Atomic, Molecular Optical Sciences
• Chemical Kinetics
• Chemical Physics
• Catalysis
• Combustion Dynamics
• Electrochemistry
• Heavy Element Chemistry
• Interfacial Chemistry
• Organometallic Chemistry
• Photochemistry
• Photosynthetic Mechanisms
• Radiation Chemistry
• Separations Science
• Solar Energy Conversion
• Theory, Modeling, Computer Simulation
• Thermophysical Properties

• Materials Sciences and Engineering
• Catalysis
• Ceramics
• Condensed Matter Physics
• Corrosion
• Electronic Properties of Materials
• Experimental Techniques Instrument Devel.
• Fluid Dynamics and Heat Flow
• Intermetallic Alloys
• Magnetism and Magnetic Materials
• Materials Physics and Chemistry
• Mechanical, Physical, and Structural Properties
• Metallic Glasses
• Metallurgy, Metal Forming, Welding Joining
• Nano- and Microsystems Engineering
• Neutron and Photon Scattering
• Nondestructive Evaluation
• Photovoltaics
• Polymer Science

• Life Sciences
• Human Genome
• Structural Biology
• Microbial Genome
• Low Dose Radiation Research
• Functional Genomics
• Human Subjects in Research
• Structural Biology Facilities
• Genome Instrumentation
• Computational Structural Biology
• Medical Sciences
• Molecular Radiopharmaceutical Development
• Boron Neutron Capture Therapy
• Molecular Nuclear Medical Imaging
• Imaging Gene Expression
• Biomedical Engineering
• Environmental Sciences

• BES - Basic Energy Sciences
• HENP - High Energy and Nuclear Physics
• FES - Fusion Energy Sciences
• BER - Biological Environmental Research
• ASCR - Advanced Scientific Computing Research

11
Review of University Research

• SCs Merit Review System guidelines, published
  in the Federal Register, and 10 CFR 605 set forth
  the policy and procedures applicable to the award
  and administration of grants and cooperative
  agreements by SC. http//www.er.doe.gov/produ
  ction/grants/merit.html http//www.er.doe.gov
  /production/grants/605index.html
• 3 independent reviewers for initiation of an
  award and for award renewals (generally every 3-4
  years). Mail review, site visits, panel reviews
  all acceptable.
• Evaluation criteria(1) Scientific and/or
  technical merit or the educational benefits of
  the project(2) Appropriateness of the proposed
  method or approach(3) Competency of
  applicant's personnel and adequacy of proposed
  resources(4) Reasonableness and
  appropriateness of the proposed budget and(5)
  Other appropriate factors, established and set
  forth by SC in a notice of availability or
  in a specific solicitation.
• Selection of applications for award is based upon
  the findings of the technical evaluations, the
  importance and relevance of the proposed
  application to SC's mission, and funding
  availability.

12
FY 2001 Laboratory Programs
13
DOE Laboratories (SC, DP, EM, FE, EE)
Pacific Northwest National Laboratory
Ames Laboratory
Argonne National Laboratory
Idaho National Environmental and Engineering
Laboratory
Brookhaven National Laboratory
Fermi National Accelerator Laboratory
National Energy Technology Laboratory
Lawrence Berkeley National Laboratory
Princeton Plasma Physics Laboratory
Stanford Linear Accelerator Center
Thomas Jefferson National Accelerator Facility
Lawrence Livermore National Laboratory
Oak Ridge National Laboratory
Sandia National Laboratories, CA
Los Alamos National Laboratory
Sandia National Laboratories, AL
National Renewable Energy Laboratory
14
Review of DOE Laboratory Research

• Laboratory programs are merit reviewed in a
  manner similar to that of university programs but
  using procedures appropriate to laboratories,
  e.g., site visits for multiple-PI activities.
  All lab programs are reviewed regularly using
  methods that match the character of the
  laboratory programs.
• Example -- Basic Energy Sciences.http//www.er.do
  e.gov/production/bes/labreview.htmlThe document
  Merit Review Procedures for Basic Energy
  Sciences Projects at the Department of Energy
  Laboratories sets forth the procedures for merit
  review of research projects funded at these
  institutions. Mail review and visiting
  committees are used depending on the size of the
  program.
• Example HEP and NP.Single-purpose labs are
  reviewed annually by visiting committees. HEPAP
  and NSAC review lab programs on a rotating basis.

15
Major Scientific User Facilities
Facility
Site
Fac Ops ( in M)
BES
ASCR
HENP
BER
FES
Total 852.2
16
ASCR User Facilities NERSC
Seaborg IBM SP 5TFlops
National Energy Research Scientific Computing
Center
Oakland Scientific Facility
17
ASCR User Facilities Energy Sciences Network
18
BES X-ray and Neutron Scattering Facilities
Advanced Photon Source
Intense Pulsed Neutron Source
Advanced Light Source
National Synchrotron Light Source
Spallation Neutron Source
High-Flux Isotope Reactor
Manuel Lujan Jr. Neutron Scattering Center
19
BES User Facilities
Advanced Photon Source
Electron Microscopy Center for Materials Research
The worlds largest collection of scientific user
facilities for exploring the atomic world
operated and funded by a single program
organization
Center for Microanalysis of Materials
Materials Preparation Center
Intense Pulsed Neutron Source
National Synchrotron Light Source
Advanced Light Source
National Center for Electron Microscopy
Spallation Neutron Source
Center for Nanophase Materials Sciences
Molecular Foundry
Surface Modification Characterization Center
Stanford Synchrotron Radiation Lab
Los Alamos Neutron Science Center
Shared Research Equipment Program
Linac Coherent Light Source
High-Flux Isotope Reactor
Combustion Research Facility
Center for Integrated Nanotechnologies
James R. MacDonald Lab
Pulse Radiolysis Facility

• 4 Synchrotron Radiation Light Sources
• Linac Coherent Light Source (CD0 approved)
• 4 High-Flux Neutron Sources (SNS under
  construction)
• 4 Electron Beam Microcharacterization Centers
• 5 Special Purpose Centers
• 3 Nanoscale Science Research Centers (CD0s
  approved)

Under construction In design/engineering In
design/engineering


20
BER User Facilities
Structural Biology User Facilities
(synchrotron radiation) ANL, BNL, LBNL, SSRL
Structural Biology User Facilities
(neutron radiation) ORNL, LANL
Field Research Center for Natural and Accelerated
Bioremediation Research
Throughfall Displacement Experiment
Laboratory for Comparative and Functional
Genomics (under construction)
Atmospheric Radiation Measurement (ARM) Site
(plus Alaska and Tropical Western Pacific)
21
FES User Facilities
Alcator C-Mod at Massachusetts Institute of
Technology
DIII-D Tokamak at General Atomics
National Spherical Torus Experiment at Princeton
Plasma Physics Laboratory
22
High Energy Physics User Facilities
Tevatron (with D-Zero and CDF detectors) _at_ FNAL
B-factory (with BaBar detector) _at_ SLAC
Alternating Gradient Synchrotron _at_ BNL
23
Nuclear Physics User Facilities
Argonne Tandem Linac Accelerator System _at_ ANL
88-Inch Cyclotron _at_ LBNL
Relativistic Heavy Ion Collider AGS _at_ BNL
Holifield Radioactive Ion Beam Facility _at_ ORNL
William H. Bates Linear Accelerator _at_ MIT
Continuous Electron Beam Accelerator Facility _at_
TJNAF
24
Review of Scientific User Facilities
Advanced Scientific Computing Research
(ASCR)NERSC was peer reviewed in FY 2001 using a
panel of independent experts (8 computational
scientists, 6 administrators of similar centers,
1 industry representative). The findings were
that NERSC has a sound strategy for providing
high-performance scientific computing hardware
and services and a commitment to excellence to
provide comprehensive services to a broad base of
scientific users Basic Energy Sciences (BES)?
Since 1997, approximately 10 BESAC subpanels have
been charged with reviewing the major facilities
including the four light sources, the three
neutron scattering facilities, and the four
electron beam microcharacterization centers. In
addition to reviewing the impact of the science,
the quality of operations, and the service to
users, these reviews addressed the impacts of the
shutdown of the HFBR, provided the basis for
long-range planning for all the facilities, and
addressed the need for new facilities. The
reviews resulted in significant changes in DOE
and MO contractor management, including funding
adjustments, management changes at the contractor
laboratories and at the facilities themselves,
and changes in how the facilities interact with
users. ? Another round of reviews of these
facilities, patterned after those led by BESAC
subpanels, began in 2001 using panels charged by
the BES program. ? BESAC has also been closely
involved in the planning of the Linac Coherent
Light Source and the Nanoscale Science Research
Centers.
25
Review of Scientific User Facilities
Charge to the BESAC subpanel for the review of
the BES synchrotron radiation light sources 1.
What has been the scientific impact of
synchrotron radiation based research during the
past decade, and what is it expected to be during
the next decade? 2. What is the scientific and
technological demand for synchrotron radiation?
From what fields and sectors? Who are the
newcomers? How has the demand changed since the
1984 Seitz-Eastman report, and how might it
change in the future? Please provide
quantitative information whenever possible, e.g.,
how has structural biology or geosciences or
environmental sciences or x-ray microscopy
changed during the past decade at the various
light sources? 3. What is the user demand at
each of the DOE synchrotron light sources? What
is the distribution of users? Are there special
needs served (e.g., scientific, industrial,
geographical) at the different light sources,
and, if so, are these needs growing or
declining? 4. What is the expected future
capability of each synchrotron light source over
time? How do the capabilities complement one
another? 5. What does each light source see as
its own vision of the future? How do the visions
complement each other? How well do the visions
accommodate potential changes found in item 2?
6. In a constant budget scenario, what is the
appropriate level of research and development
(RD) funding for efforts related to continuously
improving current facility operations such as
accelerator RD, the design of insertion devices,
the design of advanced instrumentation, etc.?
How should these funds be apportioned between the
facilities themselves and the user community
including the broader accelerator RD community?
What is the priority between support for such RD
and direct support for users? 7. In a constant
budget scenario, what level of investment should
DOE/ BES make in RD for 4th generation
synchrotron sources and how should this effort be
distributed among the facilities and other
research sectors? 8. In a constant budget
scenario, is the level of DOE/BES support of
synchrotron radiation related research for users
and user-controlled beamlines appropriate and, if
not, how should it be changed? 9. If additional
funds were available to DOE/BES, should they be
invested in items 6, 7, and 8 and, if so, what
should the priority be among them? 10. What would
be the consequences of the shutdown of one or
more of the four DOE/BES synchrotron light
sources?
26
Review of Scientific User Facilities, Cont.
Biological and Environmental Research (BER)? The
EMSL was reviewed in November 2001 to assess
scientific impact, operations, user access, user
satisfaction, laboratory management, ESH, and
cyber (and other) security activities. ? The
Joint Genome Institute Production Genomics
Facility was reviewed in July 2000 to assess
technical, cost, schedule, and management issues.
A key element of this review was to evaluate
progress in correcting deficiencies identified at
a similar review held in November 1998. ? Each
structural biology facility is reviewed every 4-5
years by groups that include experts in the
facilitys technology and experts in the
applications of the technology. The results are
evaluated by BER program managers in consultation
with managers in DOE-BES, DOE-NNSA for LANSCE
and in other agencies, including NIH, NSF and
NIST. Fusion Energy Sciences (FES) ? The major
scientific user facilities are DIII-D at General
Atomics, Alcator C-Mod at MIT, and National
Spherical Torus Experiment at Princeton. Each is
reviewed by peers in conjunction with contract or
grant renewal every 5 years. OFES is arranging
for similar reviews at mid-term. ? Each facility
submits an annual work proposal that is reviewed
in a community-wide meeting and by OFES staff. ?
Each facility has its own Program Advisory
Committee providing advice on the basis of
technical review. ? Each facility sponsors an
annual planning meeting in which previous results
are reviewed and new ideas for research are
proposed by the community.
27
Review of Scientific User Facilities, Cont.
High-Energy Physics (HEP) ? Annual reviews are
conducted at HEPs five major facilities
Stanford Linear Accelerator Center (SLAC),
Fermilab, Argonne National Laboratory (ANL),
Brookhaven National Laboratory (BNL), and
Lawrence Berkeley National Laboratory (LBNL). ?
HEP Program Managers attend Program Advisory
Committee (PAC) meetings to assess the quality of
research, relevance and effectiveness of user
facility research programs. ? The High Energy
Physics Advisory Panel (HEPAP) periodically
reviews research activities and capabilities in
the High Energy Physics program. ? HEPAP is
charged periodically to review research
activities and capabilities of the HEP program.
Nuclear Physics (NP)? Annual reviews are
conducted at NP's two major facilities RHIC and
TJNAF. ? Annual on-site Program Manager
Reviews are conducted at NP's smaller user
facilities. ? NP Program Managers attend Program
Advisory Committee (PAC) meetings to assess the
research quality, relevance and effectiveness of
user facility research programs. ? Focused
bottoms-up operation reviews are conducted
periodically. ? NSAC is charged periodically to
review research activities and capabilities in
the Nuclear Physics program.
28
Annual Reporting by Facilities
PART 1 -- USER SATISFACTION MINI SURVEY 1.
How satisfied were you with the fraction of the
year that the facility operates? 2. How
satisfied were you with the schedule or service
(i.e., was the time or service delivered on
schedule and was downtime kept to a
minimum)? 3. How satisfied were you with the
performance (i.e., was beam or service maintained
close to specifications)? 4a. How satisfied were
you with the support for users provided by the
facility staff? 4b. How satisfied were you with
the support for users provided by the PRT/CAT
staff? 5. Which of the following benefits have
resulted or will soon result from your use of
this facility this year? (circle all that
apply) 1. a. Created new fundamental
knowledge (i.e., basic research) b.
Discovered other new knowledge (i.e., applied
research) c. Developed a new or improved
product, process or technology (i.e,
development) 2. a. Dissemination of new
knowledge via publication in peer-reviewed open
literature b. Dissemination of new
knowledge via presentations at professional
society meetings c. Dissemination of new
knowledge via other presentations d.
Acquired a patent 3. Furthered Department of
Energy mission(s) 4. Obtained access to unique
capabilities not available elsewhere (e.g.,
forefront experiments one-of-a-kind
instruments distinctive materials or
services) 5. Facilitated collaborative
interactions (e.g., stimulated new ideas for
future experiments, increased
multidisciplinary workenabled a new approach
within your discipline) 6. Trained students
(undergraduate, graduate or postdoctoral
associate) 7. Other benefit(s) please specify

6. Are
the training and safety procedures appropriate?
If not, how would you change them? 7. What
would you like this facility to do differently?
8. Other comments.
The complete questionnaire can be downloaded
from http//www.sc.doe.gov/production/bes/download
/BESfiles.html
29
Annual Reporting by Facilities, cont.
PART 2 -- QUANTITATIVE DATA Questions 1-9
User demographics 1.,2. Numbers and types of
users 3. Numbers of co-proposers 4. Employer /
U.S. or foreign 5. Employment level 6. Age (not
required) 7. Facility association (member of a
PRT/CAT or general user) 8. Source of support 9.
Topic of the experiment (area of
science) Questions 10-16 Facility budget and
operations data 10. Annual budget 11. Sources
of funding for annual budget 12. Replacement
cost 13. Hours of operation (maximum, scheduled,
delivered) 14. Fraction of facility in use 15.
Numbers of beam ports, beam lines, end
stations 16. Beam line statistics for each beam
line
The complete questionnaire can be downloaded
from http//www.sc.doe.gov/production/bes/download
/BESfiles.html
30
Synchrotron Light Sources The Success
StoryFrom the Province of Specialists in the
1980s to a Widely Used Tool in the 21st Century
The number of researchers using the synchrotron
radiation light sources is expected to reach
11,000 annually when beamlines are fully
instrumented.
Who funds the light sources? The Basic Energy
Sciences program provides the complete support
for the operations of these facilities.
Furthermore, BES continues as the dominant
supporter of research in the physical sciences,
providing as much as 85 of all federal funds for
beamlines, instruments, and PI support. Many
other agencies, industries, and private sponsors
provide support for instrumentation and research
in specialized areas such as protein
crystallography.
Example of quantitative data
31
Recent ASCAC Studies

• Review of facilities and the computational side
  of biotechnology (Report due January 2002)
• Assessment of the high-performance computational
  needs and capabilities throughout the Office of
  Science (Report due September 2002)

32
Recent BESAC Studies

• Neutron Source Upgrades and the Specifications
  for the SNS (1996)
• Research Reactor Upgrades (Robert Birgeneau,
  Chair)
• Spallation Neutron Source Upgrades (Gabriel
  Aeppli, Chair)
• Technical Specifications for the Next Generation
  Spallation Source (Thomas Russell, Chair)
• DOE Synchrotron Radiation Sources and Science
  (November, 1997 Robert Birgeneau, Chair and
  Z.-X. Shen, Vice Chair)
• Review of the High Flux Isotope Reactor Upgrade
  and User Program (October, 1998 Jack Crow,
  Chair)
• Novel, Coherent Light Sources (January, 1999
  Steve Leone, Chair)
• Complex and Collective Systems (August, 1999)
• Review of the Advanced Light Source (February,
  2000 Yves Petroff, Chair)
• Review of the Electron Beam Microcharacterization
  Centers (February, 2000 John Stringer, Chair)
• Neutron Scattering (February, 2000 Martin Blume,
  Chair)
• Review of IPNS/LANSCE (December, 2000 Ward
  Plummer, Chair)
• Committee of Visitors, Chemical Sciences Research
  Activities (Report due 2002 Carl Lineberger,
  Chair)
• Biomolecular Materials (Report due 2002 Mark
  Alper and Sam Stupp, Cochairs)
• Performance Measurement in the Office of Science
  (Report due 2002 John Stringer, Chair)
• Challenges in Catalysis (Report due 2002)

33
Recent BERAC Studies

• Medical Sciences Instrumentation Research Program
  (11/95)
• Nuclear Medicine Program (11/95)
• Health Effects Research Program (11/95)
• Global Change Research Program (2/96)
• Human Genome Program - Joint Genome Institute
  (8/96)
• Global Change Research Program Integrated
  Assessment Research (10/96)
• Human Genome Program Priorities (3/98)
• Structural Biology Research Program -
  Macromolecular crystallographic use of
  synchrotrons for the Interagency Structural
  biology Working Group (7/98)
• Structural Biology Research Program
  Technologies and Discoveries (8/98)
• Global Change Research Program (10/98)
• Low Dose Radiation Research Program (12/98)
• Boron Neutron Capture Therapy Program (11/99)
• Global Change Research Program - National
  Assessment (3/00)
• Biological and Environmental Research Program
  Facilities (3/00)
• Opportunities in Genomics Genomes to Life
  (6/00)
• Natural and Accelerated Bioremediation Research
  Program (NABIR) Data Management (4/00)
• NABIR Bacterial Transport Research (10/00)
• Global Change Research Program (3/01)
• Biological and Environmental Research Portfolio
  (3/01)

34
Recent FESAC and Other FES Studies

• Energy RD Shaping our Nations Future in a
  Competitive World, SEAB (1995)
• The U.S. Program of Fusion Energy Research and
  Development, PCAST (1995)
• A Restructured Fusion Energy Sciences Program,
  FEAC (1996)
• Federal Energy Research and Development for the
  Challenges of the 21st Century, PCAST (1997)
• Report on the Nature and Level of U.S.
  Participation in Possible ITER Activities, FESAC
  (1998)
• Realizing the Promise of Fusion Energy, Final
  Report of Task Force, SEAB (1999)
• Fusion Science Assessment Committee, Interim
  Assessment, NRC (1999)
• Priorities and Balance within the Fusion Energy
  Sciences Program, FESAC (1999)
• FUSAC Final Report, NRC (2001)
• Review of the Fusion Theory and Computing
  Program, FESAC (2001)
• Review of Burning Plasma Physics, FESAC (2001)

35
Recent HEPAP Studies

• Vision for the Future of High-Energy Physics (May
  1994 Sid Drell, Chair)
• Accelerator-Based Neutrino Oscillation
  Experiments (September 1995 Frank Sciulli,
  Chair)
• Assessment of the Status of Accelerator Physics
  and Technology (May 1996 Jay Marx, Chair)
• Planning for the Future of High-Energy Physics
  (February 1998 Fred Gilman, Chair)
• White Paper on Planning for U.S. High-Energy
  Physics (October 2000 Fred Gilman, Chair)
• Long-Range Planning for U.S. High-Energy Physics
  (due January 2002 Jonathan Bagger and Barry
  Barish, Co-chairs)
• At least once a year, a HEPAP meeting is
  conducted at one of the laboratories.
  Approximately half of this meeting is devoted to
  reviewing the Laboratorys programs and projects.
  After the meeting, the Chair of HEPAP, in the
  letter report to DOE, addresses the Laboratorys
  program.

36
Recent NSAC Studies

• Implementation of the 1989 Long Range Plan (April
  1992 John Schiffer, Chair)
• National Nuclear Data Needs of the 1990's (April
  1992 Wayne Cassatt, Chair)
• Steady State Operating Costs for CEBAF (July
  1992 Derek Lowenstein, Chair)
• DOE Nuclear Physics Program Review (needs and
  priorities) (May 1994 Walter Henning, Chair)
• RHIC Experimental Equipment Review (April 1995
  Konrad Gelbke, Chair)
• NSAC Long Range Plan Working Group (February
  1996 Ernest Moniz, Chair)
• RHIC Operating Budget Review (August 1996
  Richard Orr, Chair)
• DOE Medium Energy Program Review (September 1998
  James Symons, Chair)
• ISOL Taskforce (November 1999 Hermann Grunder,
  Chair)
• RIA Costing Review (August 2001 Michael
  Harrison, Chair)
• NSAC Long Range Plan Working Group (Report due
  2002 James Symons, Chair)
• DOE Low Energy Program Review (Report due 2002
  Brad Filippone, Chair)

37
GAO Findings on SC Merit Review

• Following the GAO report "Federal Research --
  Peer Review Practices at Federal Science Agencies
  Vary" (GAO/RCED-99-99), the House Committee on
  Science requested a follow-up study at DOE, which
  included an audit of the peer review procedures
  of BES.
• GAO randomly sampled 100 BES research projects
  from a database of 1,298 projects funded in FY
  1998. The sample included 75 grants and 25
  projects funded at 9 DOE laboratories.
• The resulting report, "Federal Research DOE Is
  Providing Independent Review of the Scientific
  Merit of Its Research" (GAO/RCED-00-109, April
  2000) found "On the basis of our review of
  available documentation from program and project
  files for fiscal years 1998 and 1999, the Office
  of Basic Energy Sciences ... followed the merit
  review procedures they have established...and
  are performing merit reviews on projects or
  programs, are selecting reviewers with the
  requisite knowledge of the research, are
  requiring those reviewers to apply appropriate
  criteria in making their evaluations, and are
  using the merit review evaluations in making
  award decisions" (page 15).
• Both GAO reports are available athttp//www.sc.d
  oe.gov/production/bes/peerreview.html

38
FY 2001 Construction Projects
39
The Spallation Neutron Source
40
Review of Construction Projects

• SCs Construction Management Support Division
  conducts independent technical, cost, schedule,
  and management peer reviews of SC construction
  projects and large experimental equipment. These
  reviews are known as Lehman Reviews after the
  Division Director, Dan Lehman.
• Lehman Reviews are widely known in DOE, other
  agencies, and abroad. Dan Lehman has briefed OMB
  and other agencies on the process, which is now
  being adopted by other parts of DOE.
• A primary responsibility is conducting reviews of
  major projects, which are held typically twice
  yearly and may include 30-40 independent
  technical experts, who are divided into 6-8
  subpanels during the review to investigate all
  aspects of the project.
• Reviews can result in modifications to the
  project, work stoppage, and management changes.

41
Office of Science
Details of SNS Project reporting relationships
from the the Office of Science to the SNS Project
Office
NOTE Director of Science equivalent to
Assistant Secretary position and filled by
Presidential Appointment (Senate confirmed)
Principal Deputy Director equivalent to
Principal Deputy Assistant Secretary Associate
Directors equivalent to Deputy Assistant
Secretaries.
Director James F. Decker (Acting) Principal
Deputy Director Milton D. Johnson
(Acting) Deputy Director for Operations G. Leah
Dever (Acting)
Office of Basic Energy Sciences Associate
Director Patricia Dehmer Deputy Associate
Director Iran L. Thomas
Office of High Energy and Nuclear Physics
Office of Biological Environmental Research As
sociate Director Aristides Patrinos
Office of Advanced Scientific Computing Research
Associate Director C. Edward Oliver
Office of Planning and Analysis Director
William J. Valdez
Office of Laboratory Policy Director Antionette
Joseph
Office of Laboratory Operations
and Environment, Safety and Health Associate
Director James A. Turi
Office of Resource Management Associate
Director John Rodney Clark
Office of Fusion Energy Sciences Associate
Director N. Anne Davies Director of
International Activities Michael Roberts
Associate Director S. Peter Rosen Deputy
Associate Director Robin Staffin
Report of review results
Request for SNS review
High Energy Physics Division Director John R.
OFallon
Management Analysis and Human
Resources Division Director Myrna Vallette
Construction Management Support
Division Director Daniel Lehman
Mathematical, Information and Computational Scienc
es Division Director (Acting) Walter Polansky
Life Sciences Division Director Marvin E. Frazier
Materials Sciences and Engineering
Division Director Iran Thomas
Research Division Director John W. Willis
Technology Research Division Director
(Acting) Sam Barish
Chemical Sciences, Geosciences, and
Biosciences Division Director (Acting) William
Millman
Laboratory Infrastructure Division Director John
Yates
Financial Management Division Director Ralph De
Lorenzo
Facilities and Enabling Technologies Division Dir
ector Michael Roberts
Medical Sciences Division Director Michael V.
Viola
Nuclear Physics Division Director Dennis G. Kovar
Info/Coordination
Office of Scientific and Technical Information Di
rector Walter Warnick
Environment, Safety and Health Division Director
Van Nguyen
Grants and Contracts Division Director John
Alleva
Environmental Sciences Division Director Jerry
Elwood
Spallation Neutron Source
Jeffrey Hoy
SNS Program Manager
DOE Headquarters
DOE Field Contractor
Chicago Operations Office Manager Marvin E.
Gunn, Jr.
Oak Ridge Operations Office Manager Leah Dever
Berkeley Site Office Director Richard H. Nolan
Stanford Site Office Director John S. Muhlestein
DOE SNS Project Manager

ORNL Site Office
SNS Project Office
UT/Battelle
ORNL SNS
Les Price
42
FY 2001 Infrastructure Projects
43
Physical Infrastructure at the DOE SC Labs
The condition of the infrastructure at the DOE
Science labs ranges from superb ... to ugly to
worse than ugly.
44
Landlord Reviews of SC Multiprogram
Laboratories

• Charge to the Lehman Review Committees
• Achieve an understanding of the availability of
  funding sources (i.e., GPP, GPE, MEL-FS, Waste
  Management, overhead, space charge, etc.) and
  evaluate their allocations in support of landlord
  activities.
• Evaluate the overall effectiveness of the
  implementation of landlord responsibilities
  consistent with the Office of Sciencess
  high-level expectations (SC-1 memorandum of
  7/13/98) in the areas of Environment, Safety
  Health (ESH), Infrastructure, and Stakeholder
  Relations. In addition, review the status of
  on-going and planned ESH and waste management,
  infrastructure management and planning, and
  integrated safeguards and security, including
  cyber security. Particular attention should be
  given to understanding whether unfunded risks are
  acceptable.

45
DOE MO Contractor Appraisals

• Contractor appraisals are performed annually by
  DOE
• Laboratory peer reviews and landlord reviews
  provide input for these appraisals. Overall
  appraisals are rolled up from individual
  laboratory reviews for all programs.
• Ratings are given for the following performance
  measures
• Research quality
• Relevance to mission
• Research facilities
• Research management
• Contractor appraisals affect performance fees and
  contract renewals.

46
FY 2001 Program Direction
47
Example of Review of Program ManagementCommittee
of Visitors
Charge to BES Committee of Visitors for chemistry
programs 1. For both the DOE laboratory
projects and the university projects, assess the
efficacy and quality of the processes used
to (a) solicit, review, recommend, and document
proposal actions and (b) monitor active project
and programs. 2. Within the boundaries defined
by DOE missions and available funding, comment on
how the award process has affected (a) the
breadth and depth of portfolio elements, and (b)
the national and international standing of the
portfolio elements. 3. Comment on future
directions proposed by the Division and BES
management and on opportunities that might not
have been presented. 4. Comment on how the
process for these reviews might be improved.
48
Summary of Review Types
MO Contractor Assessments Lehman Reviews
FY 2001 Budget 3,140.9 M
Lehman Reviews of Cost, Schedule, Scope, and
Technical Management
COVs
10 CFR 605
SS
Reviewed as part of facilities
Prog Dir
Research (Universities)
Construction
Lehman Landlord Reviews
AIP
GPP,GPE
Reviewed as part of university or lab programs
Capital Equipment
10 CFR 605, formally adapted to labs
Research (Laboratories)
Major User Facilities
10 CFR 605, formally adapted to labs annual
reporting of operating statistics
Laboratory MO contractors undergo annual
appraisal of overall performance, based on
laboratory reviews of components
Includes the funding for not-for-profits,
other agencies, and private institutions
49
Summary of Review Mechanisms

• Scientific peer review of research and facility
  operations by mail, site visits, and panels as
  appropriate
• Detailed reporting of statistics for facility
  operations
• Construction project management reviews
• Infrastructure maintenance reviews
• Annual evaluation of MO contractors
• Committees of Visitors to assess program
  management
• Advisory Committee subpanels for programmatic,
  facility, laboratory, and relevance reviews
• Occasionally, NRC, COSEPUP, The Washington
  Advisory Group, JASON, etc.

50
Membership of Review Panels and Advisory
Committees

• Scientific experts from the academic research
  community, the private sector, government
  laboratories, other government agencies, other
  offices in DOE, and the international community
• Scientific and technical experts who use the
  results of research, e.g. NERSC reviews include
  materials scientists, chemists, biologists, etc.
• Industrial and DOE technology office managers who
  are downstream from the immediate output of
  basic research

51
Measure All the Right ThingsQuality (?),
Relevance (?), Leadership (?), Management (?)
Experiments in International Benchmarking of US
Research FieldsNAS, NAE, IOM Committee on
Science, Engineering, and Public Policy
(COSEPUP), 2000 International Benchmarking of
US Materials Science and Engineering
Research Summary and Conclusions The United
States is among the worlds leaders in all
subfields, and it is the leader in some. The US
is currently among world leaders in all of the
subfields of materials science and engineering,
and currently it enjoys a clear lead in
biomaterials. The US is expected to maintain its
lead in metals and electronic-photonic materials
because of their large US industrial base.
Erosion of US leadership is expected in the
subfields of composites, catalysts, polymers, and
biomaterials because of the high priority being
given to these subfields by other countries.
Current US weakness in materials synthesis and
processing relative to Europe and Japan is
especially highlighted in the panels assessment.
Subfields assessed biomaterials,
ceramics, composites, magnetic materials, metals,
electronic and optical-photon materials,
superconducting materials, polymers, and
catalysts
52
Measure Only the Right ThingsWhat You Measure is
What You Get