Introduction to program evaluation

1
Introduction to program evaluation

• Steven M. Culver
• Assistant Director, Academic Assessment
• sculver_at_vt.edu

2
Discussion points regarding program evaluation

• Evaluation should not disrupt your program or its
  participants
• Assess completely and fairly
• The transparency of evaluation methods and
  procedures
• Use practices to ensure valid and reliable
  (credible and trustworthy) information
• Focus on practicality

3
Assumptions

• Evaluation is an evolving process, not a static,
  one-time action
• Evaluation is more than data collection and
  reporting
• Evaluation makes it possible for stakeholders to
  utilize information to continuously improve
  programs and policies.

4
Types of evaluation

• Needs assessment help in designing a new
  program or in justifying an existing program
• Process evaluation understanding and
  documenting program implementation
• Outcome evaluations what happened to program
  participants
• Formative evaluation improvement
• Summative evaluation - accountability

5
Needed steps for successful program evaluation

• Clear focus (for the program and the evaluation)
• Collecting data
• Analyzing interpreting data
• Using the information

6
Focus

• In one sentence, describe the purpose of your
  program.
• What problem(s) are you trying to address?
• Who is served by your program?
• If you were to consider your program
  successful, what would that mean?
• What will be the lasting benefits of your program?

7
Focus on activities and participation

• What do we do?
• Conduct meetings, workshops
• Deliver services
• Develop curriculum, resources
• Train
• Provide counseling, assistance
• Who do we do it with?
• Participants, clients
• Agencies
• Customers

8
Take a moment, and consider your program
9
Collecting data

• Dont necessarily use data because you have it.
• First What do I measure?
• Second How do I measure this?
• What tools do I use?
• When do I measure?
• How often?
• Whom?

10
Example of measurement

• Question How do we know that tutoring in our
  program improves writing ability?
• So, we should measure writing ability, or at
  least indicators of writing ability (think
  vocabulary, grammar, etc.)
• Measures to use writing essay evaluated
  holistically by raters, a cloze test, vocabulary
  test, grammar quiz, tutor perceptions, student
  perceptions (survey, interview, focus group)

11
Measurement opportunities

• Survey
• Focus group, interviews
• Case study
• Observation
• Expert/peer review
• Portfolio reviews
• Tests
• Diaries/journals
• photos/videos
• Logs
• Document analysis
• Unobtrusive measures

12
Measurement issues

• IRB
• Dont lead people toward what you want to hear
• Dont collect more information than you need
• Think of the measurement process not just the
  actual measurement to ensure quality data.

13
Take a moment, and consider your program

• What is it we should measure?
• How can we measure it? (what do we have in place
  now and what could we easily add)

14
Analyze Interpret Data

• Appropriate procedures for processing and
  analyzing quantitative and qualitative data
• Interpretation of results more than one set of
  eyes involvement of stakeholders
• What can we say with some confidence?
• What limitations must be addressed.
• If expected changes are not apparent, what will
  be your response?

15
Using the information

• Make the evaluation come full circle use the
  information for program improvement and for
  improvement of the evaluation process
• Review with stakeholder groups.
• Use for next steps

16
Useful sources

• Chen, H. (2005). Practical program evaluation
  Assessing and improving planning, implementation,
  and effectiveness. Sage.
• McDavid, J., Hawthorn, L. (2006). Program
  evaluation and performance measurement An
  introduction to practice. Sage.
• McNeil, K., Newman, I., Steinhauser, J. (2005).
  How to be involved in program evaluation What
  every administrator needs to know. Lanham, MD
  Scarecrow Education.
• Stufflebeam, D., Shinkfield, A. (2007).
  Evaluation theory, models, and applications.
  Jossey-Bass.