Does Government Work Evaluating Programs and Policies

1
Does Government Work?Evaluating Programs and
Policies

• Katherine C. Naff, Ph.D.
• January 22, 2002

2
Policies Often Respond to Problems

• Second hand smoke harms peoples health
• People are living in poverty because they dont
  have the education required for a job
• Cities are plagued by jobless gang members and
  antisocial youth

3
Programs developed to achieve policy objectives

• Objective Reduce second-hand smoke
• Program Smoking cessation program
• Objective Ensure all have access to education
• Program Provide support to college students at
  risk for dropping out
• Objective Help prepare gang members and
  antisocial youth for jobs
• Program Finance tattoo removal

4
Government Performance and Results Act (GPRA)

• Requires federal agencies to
• Set strategic goals
• Measure performance toward achieving those goals
• Publicly report on their progress

5
GPRA is to

• improve congressional decisionmaking by
  providing more objective information on achieving
  statutory objectives, and on the relative
  effectiveness and efficiency of federal programs
  and spending.
• Answer the defining question Do the programs
  contribute to a real difference in the quality
  and security of citizens lives?
• Shift focus from preoccupation with staffing and
  activity levels to a broader focus on the results
  or outcomes of federal programs
• Source David Walker

6
Traditional performance measures

• Amount of money spent to support program
• Number of staff employed
• Number of tasks completed

• GPRA measures
• Outcomes or results

7
Example OSHA

• Activity or Output
• Inspect 2000 plants this year
• Outcome
• Reduce workplace injuries by 20

8
Example Welfare Reform

• Activity or Output
• Enroll 50 percent of welfare recipients in
  training programs
• Outcome
• 50 of welfare recipients are in full-time jobs

9
Performance Measurement vs. Program Evaluation

• Performance measurement asks, Are we achieving
  the goals we set, as shown by measurable
  outcomes?
• Program evaluation asks, Is the program
  responsible for the measurable outcomes we
  observe?

10
Example SSS-TRIO program

• Objective To help at risk students to stay in
  college until they earn their Bachelors degree
• Activities orientation, tutoring, skills
  workshops, mandatory advising, guaranteed
  registration in remedial classes, financial aid

low income, first generation college, or
individual with disabilities
11
Program theory

• Engaging students in this set of activities will
  increase the likelihood that they will graduate
  with a Bachelors degree, and that degree will
  enable them to participate more fully in
  Americas economic and social life.

12
Outcome Line
Activity
Activity
Activity
Ultimate Outcome (happiness)
Subobjective
Subobjective
Outcome of interest
As described by Lawrence Mohr
13
SSS-TRIO Outcome Line
Orientation
Tutoring
Study Skills Workshops
Commitment to program
Improved grasp of concepts
Improved study habits
Fuller participation in America
Graduate
Retained
Better Grades
14
Is SSS-TRIO achieving its purpose?

• Process evaluation Is the program being
  implemented as intended?
• Outcome (impact) evaluation Is the program
  achieving specified outcomes?

15
Process Measures

• of students who attended orientation
• of students who received tutoring
• of students who participated in workshops

16
Outcome measures

• Subobjectives
• of program students who maintained GPA of 2.0
  or better each year
• of program students with GPAs under 3.0 who
  increased their GPAs by .2 after 4 semesters
• of program students retained each year
• Outcome of interest
• of program students who graduate

17
Expected Outcomes

• A minimum of 80 of participants will maintain a
  GPA of 2.0 or better each year.
• 75 of program participants with GPAs under 3.0
  will increase their GPAs by .2.
• A minimum of 65 of program participants will be
  retained each year.
• A minimum of 45 of participants will graduate
  each year.

18
Quantifying Program Impact

• Program Impact Results Counterfactual
• Counterfactual Estimate of what results would
  have been if program had not been implemented

19
Example SSS-TRI

• Program Impact 45 graduation rate - 25
  graduation rate
• Program Impact increase in graduation rate by
  20 percentage points

20
Estimating the counterfactual

• Previous experience
• External events
• Attrition
• Hawthorne effect
• Control group
• Selection bias
• Effects resulting from knowledge about other
  group

21
Other Questions

• Are we addressing the right problem?
• Are there other intervening variables?
• Are there alternative explanations?
• Is the program theory, as specified in the
  outcome line, correct?

22
Examining Linkages (formative evaluation)
Skills workshop
Improved study habits
Graduation
Graduation
Improved study habits
Skills workshop
23
How does evaluation compare to other forms of
research?

• Both require
• Rigorous design
• Similar techniques for data collection and
  analysis
• Concern for issues of validity and reliability
• Competence and objectivity

24
How does evaluation compare to other forms of
research?

• Evaluation
• Has real world consequences
• Takes place in a political environment
• Must deal with the march of time
• Often faces data and other resource constraints
• Is supposed to improve programs as well as assess
  their outcomes.

25
Summing up

• Programs are intended to achieve policy
  objectives, often to solve a problem
• Performance measurement tells us if we are
  getting results
• Program evaluation tells us if the program is
  giving us those results
• Evaluators must consider issues that might
  undermine our conclusions
• Program evaluation is rigorous research in a real
  world setting.

26
Excellent Sources

• Program Evaluation
• Lawrence Mohr, Impact Analysis for Program
  Evaluation (2nd edition, 1995)
• Carol Weiss, Evaluation (2nd edition, 1998)
• Steven Isaac and William B. Michael, Handbook in
  Research and Evaluation (3rd edition, 1997)
• Performance Measurement
• Harry Hatry, et al. Urban Institute Press
• General Accounting Office (www.gao.gov)