Progress in Process Evaluation

1
Progress in Process Evaluation

• AEA Conference
• November 9, 2007
• Baltimore, Md.

2
Focus on Process Evaluation
I
Needs Assessment
II
Monitoring and Accountability
III
Quality Review Program Clarification
IV
Achieving Outcomes
Process Evaluation
V
Establishing Impact
Recognizes that programs are at different levels
of development
Organizes evaluation activities around these
developmental levels to support program
enhancement
Adapted from Jacobs, F., Kaouscik, J. Williams,
P., Kates, E. (2000). Making it Count
Evaluating Family Preservation Services. Tufts
University Press.
3
Process Data

• Process evaluation collects data to describe a
  program
• When it was delivered
• Where it is delivered
• What is delivered
• How it is delivered
• Who delivered it
• Who received it
• How it fits into the big picture

4
What Can Process Information Do?

• Assures accountability to program stakeholders
• Assesses whether activities are fulfilling
  program goals and objectives
• Monitors program performance
• activities are carried out as intended
• activities are based on standards
• activities follow established procedures
• Assesses quality and consistency of program
• Provides information to funders and staff for
  program improvement

5
Comprehensive Tobacco Control Program
Outcomes
State and Federal Inputs
6
Why Use Process Evaluation in Tobacco Control
(and Other Program Areas)?

• To unpack the comprehensive program black box
• Facilitates intra-state and interstate
  communication
• To clarify program planning
• Supports clearer thinking about goals and how
  attempting to reach them
• Provides a baseline for program and evaluation
  evolution
• To encourage better understanding of how programs
  are linked to outcome
• By being able to examine outcomes at the same
  level as programming

7
The Arizona Tobacco Control Program

• Fifteen county projects
• Inter-tribal Council of Arizona
• Seven American Indian tribes
• Three urban American Indian centers
• Five community partnerships
• Six statewide programs
• Marketing contracts
• Evaluation, Research and Development Unit

8
Process Evaluation Components

• Develop measures of program quantity
• Develop measures of program quality
• Coordinate with other state and national efforts
• Promote comparisons across state
• Develop system with an eye towards linking to
  outcome

9
Process Evaluation Selling the Idea
10
Implementing the Systems
Reports Evaluation
Program Staff
Program Planning
Special Analyses
Process Evaluation Data Bases
Program Activities
Evaluation Unit
Data Edit, Integration
Media Activity
Policy Activity
Direct Service Events
Adult Classes Follow Up
School-based Interventions
11
System Characteristics

• Standardized forms and definitions.
• Statewide training on data reporting.
• Centralized databases.
• Web data entry.
• Web data viewing and download (controlled by
  individual permissions).
• Statewide training on data use and
  interpretation.
• Code framework which is portable.

12
Event Form Timeline
Decision Point
?
Evaluation
Monitoring
Evaluation?
Collaborate
Monitoring?
Timeline
Fall 2007
13
(No Transcript)
14
Target Audience 2 Specific
15
Activity Type 1
16
Activity Type 2 Business Education
17
Event Summary
18
Event List 3 Listing
19
Process Evaluation Collaboration (internal)
Evaluation
Monitoring
Scope of Work
Goals/ Workplan
Actual Audience
Contract
Partners
Comments
Staff Effort
Location
Financial award
Contract managers
Target Audience
Administrators
Program staff
Evaluators
20
Process Evaluation ToolsOpen Source
Collaboration
Other Evaluation, Program or IT Staff
21
FY 2007 Types of Service Events - Prevention
22
FY 2007 Types of Service Events - Cessation
23
FY 2007 Type of Service Event Number of Events
24
FY 2007 Type of Service Event Number of
Participants
25
Merged Map and 2004-2006 Schools Data
Map drawn from 2000 US Census data and 2004-06
schools data
26
Summary of Schools, Classes, and Students
27
Cessation Clients Served, Compared to Population
of Arizona Smokers
28
(No Transcript)