Evaluating Community Interventions

1
Evaluating Community Interventions

• Sarah Levin Martin, PhD
• CDCs Physical Activity and Health Branch
• SLMartin_at_cdc.gov

2
Program Evaluation is

• The systematic collection of information about
  the activities, characteristics, and outcomes of
  programs to make judgments about the program,
  improve program effectiveness, and/or inform
  decisions about future program development
• (Michael Quinn Patton)

3
What is

• Formative assessment
• Process evaluation
• Impact / Outcome evaluation

4
Before, During, and After
5
Intervention Stage of Development (and
corresponding evaluation)
Planning stage (formative assessment) - needs
and resource assessment Implementation stage
(process / impact) - process measures and early
outcomes Maintenance stage (process /
outcome) - process measures and intermediate and
long-term outcomes
6
Things to Consider

• Guide to Community Preventive Services
• Socio Ecological Model
• National Objectives

7
Evaluation Basics (1)

• Evaluation questions
• Indicators
• Data sources
• Performance measures

8
Evaluation Grid- Process Eval
9
Evaluation Grid Outcome Eval
10
Key Evaluation Questions

• Implementation (Process)
• Documents extent to which the program is
  operating as expected
• Effectiveness (Outcome)
• Examines the extent to which a program meets its
  defined short-, mid-, or long-term
  outcomes/effects
• Efficiency (Input-Output relationship)
• Introduces the element of cost
• Causal Attribution
• Examines if progress on any of above is due to
  your program

11
Indicators (Physical Activity)
12
Data Sources

• Existing Data
• Agency records
• Census
• Prospective Data
• Surveys
• Key Informants
• Focus Groups
• Community Forums
• Ethnographic approaches (observation)

13
More on Data

• Quantity
• Quality
• Triangulation

14
Re-Cap of Basics (1) applied to SEM
15
Evaluation Basics (2)

• Evaluation standards
• Evaluation designs

16
Evaluation Standards

• Utility Who needs the info from this evaluation
  and what info do they need?
• Feasibility How much money, time, and effort
  can we put into this?
• Accuracy What design and methods will lead to
  accurate information?
• Propriety Who needs to be involved in the
  evaluation to be ethical?

17
Evaluation Design Continuum
Non Experimental
Quasi Experimental
Experimental
X O O
X O (R) O (R)
X O
O X O (R) O O (R)
O X O O O
O X O
O O O X O O O O O O O
O O
O O O X O O O(R) O O O
O O O(R)
18
Tools

• Warning There are many more caveats than tools!

19
Resources Physical Activity

• Physical Activity Evaluation Handbook
  http//www.cdc.gov/nccdphp/dnpa/physical/handbook/
  pdf/handbook.pdf
• Previous Day Physical Activity Recall
  http//cparg.sph.sc.edu/ tools.asp

• International Physical Activity Questionnaire
• http//www.ipaq.ki.se/
• Jim Salliss website
• http//www-rohan.sdsu.edu/faculty/
  sallis/measures.html
• A Collection of Physical Activity Questionnaires
• MSSE, 29(6), June 1997

NOTE These are just a sampling of readily
available tools CDC does not endorse
these tools other than HHS docs.
20
Resources Planning and Evaluation

• CDC Evaluation Working Group http//www.cdc.gov/ev
  al/index.htm
• Harvard Family Research Project
  http//www.gse.harvard.edu/hfrp/
• Kellogg Foundation Logic Model Development Guide
  www.wkkf.org
• University of Wisconsin-Extension
  http//www1.uwex.edu/ces/lmcourse

21
Resources Other

• SMART BRFSS
• http//apps.nccd.cdc.gov/brfss-smart/index.asp

• Tools for Measuring Environmental Change
• http//socialmarketing-nutrition.ucdavis.edu/Tools
  /somarktoolsEnv.php

NOTE These are just a sampling of readily
available tools CDC does not endorse
these tools other than HHS docs.