How CSREES Uses Your Reporting

1
How CSREES Uses Your Reporting

• 2007 CSREES Administrative Officers Conference
• Planning Evaluation
• Mini-Conference
• April 24, 2007

2
Overview of the Budget-Performance Cycle

• Bob MacDonald
• Director
• Office of Planning and Accountability

3
Budget-Performance Cycle
Performance-Based Budget Request Performance-Based Budget Request
Proposals for Increases Impacts Performance Measures PART results
CSREES Strategic Budget Planning
Guidance Portfolio Evaluations Stakeholder Input Administration Congress
Partners Plans Results Partners Plans Results
Projects Formula
Proposals Plans of Work
Progress Reports Annual Report
Portfolio Evaluation Portfolio Evaluation
Internal Self-Assessment (Annual) Portfolio Review Expert Panel (PREP) (Every 5 Years)
OMB Evaluation
Program Assessment Rating Tool (PART) (Every 5 Years)
4
Portfolio Review Expert Process
(PREP)Retrospective Reporting and Forward
Planning

• Henry Doan
• Planning and
• Evaluation Leader

5
The Portfolio Review Expert Panel (PREP) Process

• OMB PART/PMA BPI led to development of new
  portfolio assessment tool and measures
• Uses OMB RD criteria (relevance, quality, and
  performance)
• Portfolio analysis used to assess progress toward
  goals and guide announcements for grants

6
The PREP Process (contd)

• Focuses on Outcomes rather than processes
• Level of analysis is a portfolio identified via
  KAs
• Expert Panels score portfolio progress and
  provide recommendations for CSREES
• Expert panelists are asked to systematically
  assess distinct dimensions of the three OMB RD
  criteria. The scoring process is standardized
  across portfolios, therefore, able to provide
  quantitative assessment of portfolios

7
Number of Portfolios Assessed

1. Agricultural markets and trade
2. International economic development
3. Agricultural and food processing / bio-based
   products
4. Structures of the agricultural sector and farm
   management
5. Plant production
6. Animal production
7. Expand economic opportunities through economic
   and business decision-making
8. Informed decisions affecting quality of life in
   rural areas
9. Food safety
10. Plant protection
11. Animal protection
12. Nutrition
13. Forest and rangelands
14. Soil, air, and water

8
Phases of the PREP Process

1. Identify/Select Expert Panels
2. Develop Self-Study Report
3. Compile Evidentiary Materials
4. Self-Score Prior to Panel Meeting
5. Convene Expert Panels
6. Oral Presentations to Panels
7. Panel Exit Debriefing
8. Beyond PREP

9
1. Identify and Select Expert Panels

• Potential panelists names were suggested by
  Deputy Administrators and NPLs.
• Selection of high-level panelists with broad
  experience in topic area after careful reviews
  for absence of conflict of interest. Panel
  membership included University Vice-presidents,
  Deans/Associate Deans Industry experts (company
  vice presidents, etc.) Evaluation Experts, and
  Experts from other federal agencies.

10
Phases of the PREP Process

1. Identify/Select Expert Panels
2. Develop Self-Study Report
3. Compile Evidentiary Materials
4. Self-Score Prior to Panel Meeting
5. Convene Expert Panels
6. Oral Presentations to Panels
7. Panel Exit Debriefing
8. Beyond PREP

11
2. Develop Self-Study Reports

• NPLs search CRIS, Science Education Impacts,
  Research and Extension Impacts manually review
  plans of work, annual reports, and termination
  reports to search for nuggets of important
  information (e.g. success stories, etc.) to
  present in self-study reports

12
Examples of Information Reported

• From searching the CRIS system
• From reviewing Plan of Work, Annual Reports, and
  Termination Reports

13
Dietary Patterning and Obesity Through
Adolescence (P. Crawford University of
California, Berkeley) Funding National
Research Initiative Competitive Grants
Program Award 2001-35200-10658
14
Using Stage-Based Interventions to Increase Fruit
and Vegetable Intake in Young Adults (S. Nitzke
University of Wisconsin) Funding Initiative
for Future Agriculture and Food Systems Award
2001-52102-11226 Using the Stages of Change
Model to Increase Fruit and Vegetable Intake (J.
Ruud University of Nebraska, Lincoln) Funding
National Research Initiative Competitive Grants
Program Award 2001-35200-10638
15
Using Sheep for Controlling Weeds and Planting
Perennial Plant Species (University of
Nevada) Funding State AES and CES CSREES
Source Science and Education Impact Report
2005-03-11. CRIS Accession No 0186922
16
Environmentally Friendly Disease Control
(Tennessee State University) Funding
Evans-Allen, 1890 Institutions Capacity
Building Source Science and Education Impact
Report 2005-03-01 CRIS Accession No 0180643
17
Hawaii Community Resource Obesity Project (S.
Beckham, Waianae Coast Comprehensive Health
Center) Funding National Research Initiative
Competitive Grants Program Award
2004-35215-14238
18
Function of Vitamin A in Quail Embryogenesis (M.
Zile, Michigan State University)
19
Phases of the PREP Process

1. Identify/Select Expert Panels
2. Develop Self-Study Report
3. Compile Evidentiary Materials
4. Self-Score Prior to Panel Meeting
5. Convene Expert Panels
6. Oral Presentations to Panels
7. Panel Exit Debriefing
8. Beyond PREP

20
3. Compile Evidentiary Materials

• NPLs compile materials submitted or published by
  partners (e.g. newsletters, project descriptions,
  publications, printed materials, etc.) and make
  them available to panels for inspection during
  panel deliberations
• NPLs may have called partners for additional
  materials to be included in the evidence

21
Examples of Materials Included

• Partners publications that were submitted to
  NPLs
• Results from searching the Internet

22
(No Transcript)
23
(No Transcript)
24
(No Transcript)
25
(No Transcript)
26
(No Transcript)
27
(No Transcript)
28
(No Transcript)
29
(No Transcript)
30
Phases of the PREP Process

1. Identify/Select Expert Panels
2. Develop Self-Study Report
3. Compile Evidentiary Materials
4. Self-Score Prior to Panel Meeting
5. Convene Expert Panels
6. Oral Presentations to Panels
7. Panel Exit Debriefing
8. Beyond PREP

31
4. Self-Score

• Based on self-studies and evidentiary materials,
  NPL teams self-score using a score sheet.
• The score sheet was based on OMB RD Investment
  Criteria quality, relevance, and performance.

32
RD Criteria Dimensions

• Relevance
• . Scope
• . Focus
• . Contemporary/emerging issues
• . Integration
• . Multi-discipline balance
• Quality
• . Significance of findings
• . Stakeholder inputs
• . Alignment with current state of science
• . Appropriate/cutting edge methodology
• Performance
• . Portfolio productivity
• . Portfolio comprehensiveness
• . Portfolio timeliness
• . Agency guidance
• . Portfolio accountability

33
Phases of the PREP Process

1. Identify/Select Expert Panels
2. Develop Self-Study Report
3. Compile Evidentiary Materials
4. Self-Score Prior to Panel Meeting
5. Convene Expert Panels
6. Oral Presentations to Panels
7. Panel Exit Debriefing
8. Beyond PREP

34
5. Convene Expert Panels

• Panels meet in Washington for 2.5 days
• Day 1 Orientation and oral presentations by
  NPLs
• Day 2 Deliberations and scoring
• Day 3 Finalizing scores and draft panel
  reports providing exit interviews

35
Phases of the PREP Process

1. Identify/Select Expert Panels
2. Develop Self-Study Report
3. Compile Evidentiary Materials
4. Self-Score Prior to Panel Meeting
5. Convene Expert Panels
6. Oral Presentations to Panels
7. Panel Exit Debriefing
8. Beyond PREP

36
6. Oral Presentations to Panels

• NPLs focus on partners work (beyond self-study
  reports) in oral presentations to panels

37
Highlight 1 Zile, Function of Vitamin A in
Quail Embryogenesis
Objective Elucidate the function of vitamin A in
the formation of blood vessels in early embryonic
development

• Approach
• Compare vascular morphogenesis of sinus venosus
  from precursor endothelial cells in vitamin A
  deficient and normal quail embryos
• Rescue vitamin A deficient embryos using retinol
  and retinoic acid and examine vascular morphology
• Examine expression of vasculogenesis-specific
  genes in vitamin A deficient normal embryos

Impact This project received renewal in FY
2004 Results will inform revisions of Dietary
Reference Intakes for vitamin A for pregnancy,
which in turn impact food assistance programs
(e.g. WIC) goal is to reduce incidence of heart
defects in newborns
38
Highlight 2 R.S. Chapkin, Nutritional
Modulation of Immunity Regulation of Th1/Th2
Responses

• Objectives
• n-3 PUFA impact on T-cell subset activation
• n-3 PUFA effect on Th1/Th2 ratio response to
  antigen

• Approach
• Mice fed corn oil, fish oil or DHA supplemented
  corn oil diets
• Measured T-cell microdomain lipid composition
  membrane subdomain distribution of signal
  transducing molecules before and after stimulation

• Impact
• First information about how dietary n-3 PUFA
  suppress inflammatory response
• Information can be used to plan interventions to
  reduce risk for chronic disease (e.g., CVD,
  arthritis)

39
Highlight 3 Ahluwalia, Iron Status and Immune
Response in Homebound Older Women
Objective Determine effects of Fe deficiency on
measures of immune function in homebound older
women Evaluate functional benefits associated w/
improvement in Fe status

• Approach
• Select apparently healthy, homebound women 60-85
  yo collect 2 blood samples to evaluate immune
  function and nutritional status collect diet
  recalls
• Provide supplements to Fe-deficient subjects for
  2 months collect samples and assess status as
  above

Impact Apparently healthy women were found to
have impaired immune function and quality of life
(shortness of breath, attention problems)
resulting from Fe deficiency Results will impact
Dietary Reference Intakes and dietary guidance
for older women
40
Highlight 4 Kant, Consumption of Energy-Dense,
Nutrient-Poor Foods by American Children
Objective To determine the proportion of daily
energy and macronutrients from energy-dense,
nutrient-poor (EDNP) foods in the diets of US
children (8-18 year olds)

• Approach
• Secondary analysis of NHANES III, 1988-1994 for
  4800 children
• Analysis to determine proportion of daily energy
  and macronutrients from EDNP foods association
  of consumption of EDNP foods with adequacy of
  intake of protein and micronutrients, assoc of
  consumption of EDNP foods with health biomarkers,
  etc.

Impact EDNP foods are widely consumed and
associated with reduced intakes of
micronutrients Children who consumed school lunch
were less likely to consume EDNP foods Provides
data to support policy decisions
41
Highlight 5 Tanumihardjo, Promotion of High
VegetableConsumption as a Weight-Loss Strategy
and General Well-Being
Objective Conduct clinical trial with vegetable
consumption as an intervention Promote
consumption of vegetables at state (WI) and local
levels

• Approach
• Wt. loss intervention in 2 groups of obese
  adults diet rich in vegetables (8 svgs/day) vs.
  500 kcal reduction, low fat diet
• Use serum carotenoids as biomarkers to measure
  compliance
• Develop lesson plans, educational materials,
  recipes for outreach via CES

Impact Project began in April, 2004
Intervention being conducted now. If successful,
serve as model for interventions that require
little training and medical supervision
42
Highlight 6 Beckham, Hawaii Community Resource
Obesity Project

• Objective
• Reduce prevalence of obesity among Native
  Hawaiians
• Increase socio-economic viability of the Waianae
  Coast

• Impact
• Increased awareness of healthy eating, physical
  activity production of local healthful foods.
• Decreased prevalence of obesity among Native
  Hawaiians.
• Increased socio-economic viability of Waianae
  Coast will be increased
• Data to support policy decisions

• Approach
• Pre post weight, BMI, body fat girth among 3
  cohorts diet alone, clinic alone, diet plus
  clinic
• Pre post sales of native foods following
  establishment of community network

43
Highlight 7
Fleming, Randomized Controlled Community
Intervention to Reduce the Risk of Type 2
Diabetes in Overweight African American Children

• Approach
• Intervention among 9-10 yo children, Oakland, CA
  2 wk summer camp, 2 yrs of weekly unstructured
  and monthly reinforcement sessions
• Healthy eating, physical activity, self-esteem
  and self-efficacy
• Measure responses at 3 months, and 1 and 2 years
  post-intervention

Preliminary results Improved insulin
sensitivity, blood glucose parameters in
intervention group Improved relationships with
peers and adults
44
Highlight 8
Boushey, Student-Centered Web-based Communities
Multidisciplinary Approach for Adolescent Obesity
Prevention

• Goals
• Improve diet quality
• Increase physical activity
• Decrease sedentary behaviors

Preliminary results Baseline measures on 642
students Ht., wt., Bioelectrical impedence
analysis (body composition), food intake,
sedentary behaviors Final measures 595 students
(93 retention) same as baseline plus DXA (dual
energy x-ray absorptiometry (DXA)
Approach 6th grade classroom intervention in
IN, AZ intervention and control schools.
45
Specific Research Leading to Education and
Behavior Change

• Premature Birth and Docosahexaenoic Enriched
  Functional Foods.
• FY 2000, a four-year IFAFS (Initiative for
  Future Ag. Food System) grant

46
Distribution of Childrens Nutrition Research
Center (Houston, TX)

• Nutrition and Your Child
• Electronic distribution gt10,000 subscribers
• Hard copy distribution gt2500

47
Community Food Projects Competitive Grant Program

• 2000-2004 170 grants active
• List of all is on the web www.worldhungeryear.or
  g
• Farmers Markets
• Community Gardens
• School Garden
• Farm to School

48
Community Food Projects Competitive Grant Program
Farmers Markets

• Farmers Market Funding
• Farmers Markets as

49
Community Food Projects Competitive Grant Program
Community Gardens

• Community Gardens

50
Community Food Projects Competitive Grant Program
School Garden

• School Gardens

51
Farm to School
52
Phases of the PREP Process

1. Identify/Select Expert Panels
2. Develop Self-Study Report
3. Compile Evidentiary Materials
4. Self-Score Prior to Panel Meeting
5. Convene Expert Panels
6. Oral Presentations to Panels
7. Panel Exit Debriefing
8. Beyond PREP

53
7. Panel Exit Debriefing

• Panel prepared for Debriefing and drafted Panel
  Report
• Panel provided Debriefing to CSREES
• Panel commented on the portfolio based on their
  deliberations and discussed recommendations
• Panel provided portfolio scores
• Q/A

54
Selected Comments from External Panels

• Lack of an adequate data system
• Disappointment with the evidence provided (by the
  self-study reports, oral presentations, etc.)
• Failure to present a complete picture of
  portfolio inputs, activities, outputs, and
  outcomes
• Need to demonstrate measurable outcomes
  (throughout the system)

55
Further Comments

• Need to solidify or make clearer relationships
  with partners.
• Need more details and discussion on extension
  outcomes --more details, more outcome examples
  based upon extension activities. Need
  documentation of specific results achieved by
  these activities.
• Need more systematic and comprehensive
  documentation of program/CSREES impacts.
• Use of logic models

56
Phases of the PREP Process

1. Identify/Select Expert Panels
2. Develop Self-Study Report
3. Compile Evidentiary Materials
4. Self-Score Prior to Panel Meeting
5. Convene Expert Panels
6. Oral Presentations to Panels
7. Panel Exit Debriefing
8. Beyond PREP

57
8. Beyond PREP

• Review panel reports
• Respond to panel recommendations
• Plan for implementation of recommendations
• Implementation of recommendations
• Feedback

58
Annual Self-Studies

• Portfolio self-studies are conducted internally
  every year by NPLs
• Report on implementations of panel
  recommendations for the year
• Report on new changes/improvements in portfolios
• Self-score annual self-studies by NPLs

59
What has been done at the Agency level ?

• 1Solution project
• Redesign CRIS report to make it possible to
  report extension budget figures
• Build POW and standardize annual reports to make
  it easy for partners to report outcomes following
  a logic model format
• Develop management dashboard to enable the Agency
  to search for outcomes submitted in annual
  reports

60
What has been implemented invarious Units (as
responses to comments/recommendations)?

• Activities in the Natural Resource Unit --
  retreat for planning hiring an NPL for needed
  areas as identified by the panel, etc.
• Activities in the Animal System retreat for
  road map development, etc.
• Activities in the Plant System hiring an NPL
  for needed area as identified by the panel, etc.
• Activities in Family, 4H, and Nutrition Unit
  retreat for planning, etc.

61
How Will Things Be Done Differently?

• Improved relationships with partners
• Redesigned report system making it easier for
  both NPLs and partners to be able to report and
  retrieve data, particularly outcomes
• Redesigned data system making it easier to report
  accountability data
• When can one expect the new system to be in
  effect? April 2008!

62
Summary

• Using the PREP process, CSREES has convened
  external panels to assess 14 portfolios,
  supporting all five Agency strategic goals.
• A new paradigm shift, destruction of silos.
• The PREP process provides opportunities to
  solidify CSREES-institution partnerships
• Panelists have provided comments
  recommendations useful for CSREES to improve upon
  program management and communications with
  partners -- benefiting both partners.

63
Summary (contd)

• CSREES has been responding to panel comments and
  recommendations to improve upon the way to do
  business building the new POW, standardizing
  annual reports, 1Solution, etc.
• April 2008

64
Budget-Performance Cycle
Performance-Based Budget Request Performance-Based Budget Request
Proposals for Increases Impacts Performance Measures PART results
CSREES Strategic Budget Planning
Guidance Portfolio Evaluations Stakeholder Input Administration Congress
Partners Plans Results Partners Plans Results
Projects Formula
Proposals Plans of Work
Progress Reports Annual Report
Portfolio Evaluation Portfolio Evaluation
Internal Self-Assessment (Annual) Portfolio Review Expert Panel (PREP) (Every 5 Years)
OMB Evaluation
Program Assessment Rating Tool (PART) (Every 5 Years)
65
CSREES OPA Web Pages from CSREES web site
(www.csrees.usda.gov), click on About Us, then on
Strategic Planning and Accountability
66
For Further Information

• Bob MacDonald
• Director, Office of Planning and Accountability
• rmacdonald_at_csrees.usda.gov
• [phone]
• Henry Doan
• Planning and Evaluation Leader
• hdoan_at_csrees.usda.gov
• [phone]