Program Evaluation

1
Program Evaluation
2
Lecture Overview

• Program evaluation and program development
• Logic of program evaluation (Program theory)
• Four-Step Model
• Comprehensive Quality Programming
• Empowerment Evaluation
• Group Activity

3
Program Development

• refers to the planning and implementation of a
  program determining its purposes and setting
  goals and outcomes, evaluating program process,
  and evaluating attainment of desired program
  outcomes and wider impacts.

4
History of Program Evaluation

• 1930s human service field grew (post depression)
• WWII programs were launched to meet the needs for
  urban development and housing, technological and
  cultural education, occupation training, and
  preventive health activities.
• By the end of the 1950s program evaluation was
  commonplace
• 1960s increased papers on evaluation research
• 1970s, evaluation was a distinct field
• 1980s benefits and costs of programs
• 1990s new social problems and booming economy

5
Program Evaluation

• Refers to the collection of evidence to judge the
  effectiveness of a community programs or other
  community interventions (com psych definition)
• The use of social research procedures to
  systematically investigate the effectiveness of
  social intervention programs (research
  definition).

6
Project Drug Abuse Resistance Education D.A.R.E.

• Most popular school based intervention
• Little evidence on its effectiveness
• Limited positive influence on psychological
  factors (self-esteem)
• How can this lack of measurable outcomes in a
  popular intervention be explained?

7
Reasons Why Programs May Be Unsuccessful

• Theory Failure concerns program theory which is
  the rationale for why a particular intervention
  is considered appropriate for a particular
  problem with a specific target population
• Implementation Failure concerns quality of
  program implementation (ie program may not be
  generalizable)

8
Effects do not occur when.

• The underlying assumptions of program theory are
  incorrect.
• The program, even if implemented well, doesnt
  affect the variables specified by program theory.
• Or the activity or program is not implemented
  adequately

9
Logic Model

• Graphic representation of the program.
• Shows the logical connections between the
  conditions that contribute to the need for a
  program in a community, the activities aimed at
  addressing these conditions, and the outcomes and
  impacts expected to result from the activities.

10
Evaluation Terminology

• Stakeholders
• Fiduciary
• Funding sources
• Need Assessment
• Evaluation Instruments

11
Four Step Model of Program Evaluation (Lindsay
Wandersman, 1991)

• Step 1 Identify Goals and Desired Outcomes
• Step 2 Process Evaluation
• Step 3 Outcome Evaluation
• Step 4 Impact Evaluation

12
STEP 1) Identifying Goals and Desired Outcomes

• What is your project striving for?
• Goals set a framework for specific outcomes.
• Program Developers MUST describe
• The primary goals of the program
• Target population
• Outcomes desired

13
STEP 2) Process Evaluation

• Monitoring program activities helps organize
  program efforts
• Provides information to help manage the program
  accountability that the program is conducting
  what was done and who was reached
• After an evaluation of outcomes and impacts , the
  evaluation can provide information about why the
  program worked or did not work
• Can help decide whether or not you are ready to
  assess the effects of your program
• What were the intended and actual activities of
  the program/ Did you learn anything.

14
STEP 3) Outcome/Impact Evaluation

• Assess the immediate effects of a program the
  bottom line
• Looks at the desired outcomes defined in Step 1
  and seeks evidence regarding the extent to which
  those outcomes were achieved
• Should be closely linked to goals, but more
  specific

15
STEP 4) Impact Evaluation

• Concerned with the ultimate effects of the
  program
• Impacts are ultimate or longer-term effects of
  the program

16
Comprehensive Quality Programming 9 Strategies

• Why is the intervention or program needed?
• What are the programs goals, target populations
  and desired outcomes?
• How does your program use the scientific
  knowledge and best practice of what works?
• How will this new program fit in with existing
  programs
• How will you carry out the program
• How well was the program carried out
• How did the program work?
• What can you do to improve the program the next
  time you do it?
• If the program or parts of the program were
  effective, what are you doing to continue
  institutionalize the program

17
Accountability Questions Corresponding CQP
Strategies

• Why is the intervention or program needed?
• What are the programs goals, target populations
  and desired outcomes?
• How does your program use the scientific
  knowledge and best practice of what works?
• How will this new program fit in with existing
  programs
• How will you carry out the program
• How well was the program carried out
• How did the program work?
• What can you do to improve the program the next
  time you do it?
• If the program or parts of the program were
  effective, what are you doing to continue
  institutionalize the program

• Needs and assets assessments
• Identifying goals and desired outcomes
• Review research literature and best practices
• Community feedback and planning
• Planning, implementation
• Process evaluation
• Outcome and impact evaluations
• Continuous program improvement
• Institutionalization strategies

18
Empowerment Evaluation

• Program planners and developers learn the basics
  of program evaluation in ongoing program
  improvement.
• Help program developers and staff to achieve
  their program goals, by providing the with the
  tools for assessing and improving the planning,
  implementation, and results of their program.

19
Limitations of Evaluation

• Use of an outside evaluator
• Evaluation feedback is provided at the end of the
  program
• Evaluation findings are perceived as being too
  complex

20
Conclusions

• Program evaluation is necessary to find out
  whether a program is working
• Evaluation can be used for program planning and
  program implementation
• As society becomes more concerned about
  accountability and results , evaluation will
  become more necessary.

21
Group Activity

• Apply 4 step model in terms of your group
  project.
• Answer CQP questions