Program Performance and Evaluation: Policymaker Expectations

1
Program Performance and Evaluation Policymaker
Expectations

• 2009 International Education Programs Service
• Technical Assistance Workshop

Eleanor Briscoe February 2, 2009
2
Who Are Our Policymakers?

• Congress
• The White House
• Office of Management and Budget
• Department of Education

3
Presentation Objectives

• Government Performance and Results Act of 1993
• Higher Education Opportunity Act (Title VI data
  collection and evaluation activities)
• National Research Councils 2007 Review of
  Title VI/Fulbright-Hays programs
• FY 2008 House Appropriations Committee Report
  (directive addressing management and program
  needs)
• Future of the PART Process
• Expectations of our Policymakers

4
ED/Budget Service

• The Department of Educations Budget Service has
  lead responsibility for
• developing and implementing the Departments
  annual budget,
• formulating budget and related legislative
  policies
• presenting the budget and related policy
  proposals to the public and to Congress
• establishing and maintaining a Department wide
  performance-based budget formulation, execution,
  and management system and
• reviewing and analyzing Department program
  operations.

5

• To that end, we as budget analysts
• Develop all Department-level budget documents.
• Work with the Departments offices to assure that
  performance measures ensure the effectiveness of
  the Department programs.
• Maintain liaison with OMB, congressional
  committees CBO GAO and other organizations on
  the formulation, presentation, and execution of
  the Departments budget.

6
GPRA

• Holds Federal agencies accountable for achieving
  program results
• Requires setting goals and public reporting
• Seeks to reduce waste and inefficiency
• Helps identify what works and what does not

7
How Budget Service presents your data?

• The Budget Service reports what you do as part of
  the GPRA process to Congress and OMB
• through technical assistance to congressional
  staffers on behalf of their representatives,
• at OMB and congressional hearings, and
• as part of the annual budget submission to OMB
  and Congress.

8
Overview of Higher Education Opportunity Act
(Title VI data collection and evaluation
activities)

• Requires the Secretary to assist grantees to
  develop a student survey to determine
  postgraduate employment, education, and training.
• Requires that the survey be administered once
  every two years.
• Specifies that up to 1 percent of authorized
  funds may be used to support Title VI program
  evaluations.

9
National Research Councils 2007 Review of
Title VI/Fulbright-Hays programs

• Study concluded
• that the Department should commission independent
  outcome and impact evaluations of all programs
  every 4 to 5 years
• a need for better and more reliable data
• a lack of rigorous, reliable information
  available on Title VI program performance.

10
FY 2008 House Appropriations Committee Report
Directive (addressing management and program
needs)

• Urged the Department to improve IRIS to ensure
  that
• data collected contain performance outputs and
  outcomes that are relevant to program monitoring
  and improvement
• data system provide greater standardization,
  allow comparison across years and across
  programs, and provide information to all grantees
  and to the public.

11

• Actions taken by the Department to improve
  management practices and performance
• Developing and implementing a strategy to use
  efficiency measures to improve cost
  effectiveness.
• Analyzing data for annual and long-term
  performance measures including looking at
  effective ways to measure post graduation data.
• Developing and implementing a strategy for
  conducting independent, rigorous, periodic,
  meaningful evaluations of the outcomes and impact
  of the international education programs.
• Developing a measure to track language skill
  changes through the use of reliable assessment
  tools.
• Making grantee performance data available to the
  public in a transparent and meaningful manner.

12
The PART Process

• Developed to assess the effectiveness of federal
  programs and to help inform management action,
  budget requests, and legislative proposals
  directed at achieving results.
• Examined various factors that contribute to the
  effectiveness of a program and required that
  conclusions be explained and substantiated with
  evidence.
• Assessed if and how program evaluation is used to
  inform program planning and to corroborate
  program results.
• Consisted of four critical areas of
  assessmentpurpose and design, strategic
  planning, management, and results and
  accountability.

13
Future of the PART Process

• Future of PART uncertain.
• President Obama created a new White House
  positionChief Performance Officeraimed at
  eliminating government waste and improving
  efficiency.
• The office will work with federal agencies to set
  performance standards and hold agency managers
  accountable for progress.

14
What do our policymakers and the public expect?

• Current, useful information on what the Title VI
  programs undertake and what they accomplish.
• Data on postgraduate employment, training, and
  education.
• Evidence that the Title VI programs are actually
  producing experts in areas of vital interest to
  the United States.

15
Outcomes vs. Outputs

• Outcomes Events or conditions external to the
  program and of direct importance to the public,
  beneficiaries and/or customers. They relate to
  the programs mission, purpose and strategic
  goals.
• Outputs Internal program activities products
  and services delivered to the public,
  beneficiaries.

16
Outcomes vs. Outputs (contd)

• Examples of Title VI outputs
• number of foreign language course offerings by
  Title VI institutions
• number of comprehensive instructional resources
  (assessments, publications, curricular materials,
  etc.) produced at Title VI-supported institutions
• number of students served/teachers trained
• Examples of Title VI outcomes
• percentage of graduates who not only find
  employment, but are utilizing their language and
  area skills
• improved outreach and dissemination of
  information about foreign languages and area
  studies
• improved instructional materials
• decreased shortage of foreign language/area
  studies experts in public service and academia

17

• What is the primary purpose of Title VI?

18
Title VI Program Purpose

• Higher Education Act (Title VI, Section 601(b)
• (b) PURPOSES- The purposes of this part are--
• (1) (A) to support centers, programs, and
  fellowships in institutions of higher education
  in the United States for producing increased
  numbers of trained personnel and research in
  foreign languages, area studies, and other
  international studies
• (B) to develop a pool of international experts
  to meet national needs
• (C) to develop and validate specialized
  materials and techniques for foreign language
  acquisition and fluency, emphasizing (but not
  limited to) the less commonly taught languages
• (D) to promote access to research and training
  overseas and
• (E) to advance the internationalization of a
  variety of disciplines throughout undergraduate
  and graduate education

19
Title VI Program Purpose (contd)

• Higher Education Act (Title VI, Section
  601(b)(contd)
• to support cooperative efforts promoting access
  to and the dissemination of international and
  foreign language knowledge, teaching materials,
  and research, throughout education, government,
  business, civic, and nonprofit sectors in the
  United States, through the use of advanced
  technologies and
• to coordinate the programs of the Federal
  Government in the areas of foreign language, area
  studies, and other international studies,
  including professional international affairs
  education and research.

20
Conclusion

• Satisfying our policymakers include
• Managing your program to its maximum
  effectiveness
• Telling your story effectively
• Sustaining your project