Does Your Program Work? Evaluating Suicide Prevention Programs for Evidence of Effectiveness

1
Does Your Program Work? Evaluating Suicide
Prevention Programs for Evidence of Effectiveness
Philip Rodgers, Ph.D. Evaluation
Scientist American Foundation for Suicide
Prevention Pre-Conference Workshop 43rd Annual
Conference of the American Association of
Suicidology, Orlando, FL, April 20-24, 2010
2
Administration of Workshop Pretest
3
Acknowledgements
U.S. Department of Health Human Services
Substance Abuse and Mental Health Services Agency
Howard Sudak, MD AFSP Linda Langford, ScD SPRC
This project is supported by a grant (1 U79
SM57392-05) from the Substance Abuse and Mental
Health Services Administration (SAMHSA), U.S.
Department of Health and Human Services (DHHS).
No official endorsement by SAMHSA or DHHS for the
information in this presentation is intended or
should be inferred.
4
www.sprc.org
5
What is the workshop purpose?

• The purpose of the workshop is to increase
  participants knowledge and use of sound
  evaluation techniques in the field of suicide
  prevention, particularly as relate to standards
  of the National Registry of Evidence-based
  Programs and Practices (NREPP).
• The workshop is intended for those with limited
  or no evaluation experience.
• The workshop is guided by four goals.

6
What are workshop goals?

• Participants will know three types of evaluation
  (process, outcome, impact)
• Participants will know how to use a program logic
  model to guide program evaluation planning
• Participants will have increased knowledge of
  different types of evaluation measures
• Participants will have increased knowledge of
  requirements for application to NREPP

7
How will these goals be achieved?

• Workshop Outline
• Purpose and goals of the workshop
• Critical definitions
• Logic Models and Evaluation
• How to Develop Evaluation Questions
• How to Answer Evaluation Questions
• How to Develop an Evaluation Plan
• Evaluation and NREPP

8
Defining Evaluation
9
What is evaluation?

• 1. Evaluation is concerned with either the
  development, operations, or the outcomes of a
  program or both.
• 2. Evaluation compares a program to a set of
  standards.
• 3. Evaluation is systematic. It is conducted with
  rigor and thoroughness.
• 4. Evaluation is purposeful. It is designed to
  provide information that can improve a program or
  document the effects of one or more aspects of
  it.

10
Why is evaluation important?

• Required by funding agencies.
• Improves performance.
• Demonstrates effectiveness.
• Advances knowledge.

As not everything can be done, there must be a
basis for deciding which things are worth doing.
Enter evaluation. M. Q. Patton
Source for Patton Quote U.S. Department of
Health and Human Services, P. H. S. (2001).
National Strategy for Suicide Prevention Goals
and Objectives for Action. Rockville, MD U.S.
Department of Health and Human Services, Public
Health Service.
11
What are types of evaluation?

• Participatory evaluation
• Formative evaluation
• Summative evaluation
• Responsive evaluation
• Goal-free evaluation
• Empowerment evaluation

• Advisory evaluation
• Accreditation evaluation
• Adversary evaluation
• Utilization evaluation
• Consumer evaluation
• Theory-driven evaluation

We will address Process, Outcome, and Impact
Evaluations
12
What is process evaluation?

• Process evaluation assesses the extent to which
  a program is operating as it was intended.

Source General Accounting Office. (2005).
Performance Measurement and Evaluation
Definitions and Relationships (Vol.
GAO-05-739SP). Washington DC United States
Government Accounting Office.
13
What is outcome evaluation?

• Outcome evaluation assesses the extent to which
  a program achieves its outcome-oriented
  objectives.

Source General Accounting Office. (2005).
Performance Measurement and Evaluation
Definitions and Relationships (Vol.
GAO-05-739SP). Washington DC United States
Government Accounting Office.
14
What is impact evaluation?

• Impact evaluationassesses the net effect of a
  program by comparing program outcomes with an
  estimate of what would have happened in the
  absence of the program.

Source General Accounting Office. (2005).
Performance Measurement and Evaluation
Definitions and Relationships (Vol.
GAO-05-739SP). Washington DC United States
Government Accounting Office.
15
I. Logic Models Evaluation
16
It is important to consider evaluation during the
strategic planning phase

• SPRC Strategic Planning Model
• Describe the problem and its context.
• Identify priorities and long-range goals.
• Consult the science identify strategies.
• Select or develop interventions.
• Develop an evaluation plan.
• Create and implement an action plan.

Source SPRC Strategic Planning Model
17
Logic models can drive evaluations
Generic Gatekeeper Training Logic Model
Outcome Evaluation
Process Evaluation
Impact Evaluation
vs. Control Group
18
Introducing the evaluand
19
Logic Model Activity Worksheet 1

• Form into small groups select a recorder
• For TangleTherapy, complete worksheet by listing
• Guiding assumptions
• Inputs
• Activities
• Outputs
• Short-term Outcomes
• Long-term Outcomes

Time is Up
20
II. How to Develop Evaluation Questions
21
Where do evaluation questions come from?

• Generally, from the goals listed in a logic
  model.
• More specifically, defined by stakeholders
  through a collaborative process.
• Depending upon circumstances, stakeholders can be
  funders, participants, trainers, evaluators, and
  others, or a combination of these.

22
Divergent, convergent process

• Stakeholders meet with relevant materials (grant
  application, logic model, etc.).
• After a review of materials, engage in divergent
  processa free association of evaluation
  questions
• After the divergent process, stakeholders
  collectively narrow list of questions to
  manageable proportions through a convergent
  process.

23
Divergent Convergent ProcessWorksheets 2 and
3
24
Divergent Question Activity Worksheet 2

• Form into small groups select a recorder
• For the TangleTherapy, list any questions you
  would have about the program process and outcomes.

Time is Up
25
Convergent Question Activity Worksheet 3

• Form into small groups select a recorder.
• Using the list of questions derived from the
  previous activity, fill-in the Convergent Phase
  by
• Agreeing amongst yourselves which are the most
  important questions
• Listing questions in the appropriate process,
  short-term, long-term slot.

Time is Up
26
III. How to Answer Evaluation Questions
27
What is measurement?

• Measurement is the means you use to collect data.
• It includes how you collect data and what data
  you collect (and how well you collect data).

28
How will you collect data?

• Questionnaires (in-person, mail, email, phone)
• Psychological Tests
• Interviews
• Health Records
• Health Statistics
• Observations
• Logs

29
Where do you find measures?

• Create your own (pilot test!)
• Borrow from other evaluations (with permission!)
• Search the literature (see Additional Resources)
• Use standardized measures (may cost)
• Brown (adult) and Goldston (adolescent) reviews
• Use existing data sources and records

30
IV. How to Develop an Evaluation Plan
31
What evaluation design do you need?

• What is your purpose?
• Performance assessment?
• Evidence of concept?
• Evidence of effectiveness?

32
There are four basic evaluation designs

• Posttest Only X O
• Pre- and Posttest O X O
• Posttest Only w/control X O
• O
• Pre- and Posttest w/control O X O
• O O

33
Additional data collection points can be added to
the basic desgins

• Posttest Only X O O
• Pre- and Posttest O X O O
• Posttest Only w/control X O O
• O O
• Pre- and Posttest w/control O X O O
• O O O

34
Best evidence comes when subjects are randomly
assigned to groups
x
Exp. Group
Exp. Group
Pool of Subjects or Groups
Con. Group
Con. Group
Random assignment increases the likelihood that
subjects in both groups are equivalent in regards
to factors that may be related to outcomes.
35
If random assignment is not possible

• Compare groups that are similar.
• Use a pretest so that group differencesto some
  extentare accounted for.

36
A fifth type of design is longitudinal
Knox, K. L., Litts, D. A., Talcott, G. W., Feig,
J. C., and Caine, E. D. (2003). Risk of suicide
and related adverse outcomes after exposure to a
suicide prevention programme in the US Air Force
Cohort study. BMJ, 327, 1376.
37
Why are sufficient numbers needed?
Small numbers are unreliable.
Source CDC.
38
Systematically developing an evaluation plan.

• Developing evaluation questions.
• Determining measurement.
• Planning for the evaluation

39
Evaluation Plan Activity Worksheet 4

• Form into small groups select a recorder.
• Use the type of program previously selected.
• For that type of program, fill-in the worksheet
  by listing
• Evaluation Question
• Data Source
• How Collected
• When Collected

Time is Up
40
Evaluation and NREPP
41
Is there an evidence-based registry for suicide
prevention?
Source www.nrepp.samhsa.gov
42
NREPP is different from other registries

• NREPP reviews evidence for specific program
  outcomes.
• NREPP publishes average scores for each of
  criterion.
• NREPP does not categorize programs into levels of
  evidence.
• NREPP casts itself as a decision support tool.

43
NREPP has a limited submission window

• Historically from October 1 through December 31.
• However, due to a large volume of applications,
  the 2008 submission window was cancelled.

44
NREPPs three minimum requirements.

• The intervention demonstrates one or more
  positive outcomes (p .05) in mental health
  and/or substance use behavior among individuals,
  communities, or populations.
• Intervention results have been published in a
  peer-reviewed publication or documented in a
  comprehensive evaluation report.
• Documentation of the intervention and its proper
  implementation (e.g., manuals, process guides,
  tools, training materials) is available to the
  public to facilitate dissemination.

45
NREPP Readiness for Dissemination Criteria

• Availability of implementation materials
• Availability of training and support resources
• Availability of quality assurance procedures

Scored on a 0-4 Scale.
46
NREPP Strength of Evidence Criteria

• Validity of Measurement
• Reliability of Measurement
• Fidelity
• Missing Data
• Confounding Variables
• Appropriateness of Analysis

Scored on a 0-4 Scale.
47
Criterion 1 Validity (Measurement)

• Does the instrument measure what it purports to
  measure?

48
Do adolescents think a lot about suicide?
For high school students during the past year
Died by Suicide 1 in 12,000
Source Centers for Disease Control and Prevention
49
Why are sufficient numbers needed?
Small numbers are unreliable.
Source CDC.
50
Criterion 2 Reliability (measurement)

• Is the instrument consistent?

51
Is this measure valid and reliable?
52
Is this measure valid and reliable?
53
Is this measure valid and reliable?
54
What is the difference in these two scores?
Dialectical Behavior Therapy
Reliability Validity of Measures of
Measures DBT Suicide Attempts 3.8 3.8 SOS
Suicide Attempts 0.8 2.0
SOS Signs of Suicide
55
Criterion 3 Fidelity

• Definition Was the program implemented the way
  it was intended to be implemented?
• What evidence can be collected and provided to
  demonstrate that the program was implemented with
  fidelity?
• Tools that can help increase fidelity are
  trainer checklists, observations (live or
  videotaped), and participant questions.

56
Criterion 4 Missing Data/Attrition

• If a significant number of subjects drop-out
  prior to or are unavailable for data collection,
  results may be biased in either positive or
  negative direction.
• Do all you can to help ensure that all
  participants are available for data collection.
• Conduct a non-respondent bias check.
• Other statistical methods are available.

57
Confounding Variables

• Confounding variables are alternative
  explanations to observed outcomes (also referred
  to as threats to internal validity).
• Some confounding variables can be controlled
  for by evaluation design and analysis.
• Three of the most important confounders are
• Attrition (already addressed)
• Selection
• Regression to the Mean
• History

58
What is selection?

• In order for comparisons to be valid, groups need
  to be equivalent on important variables.
• Equivalence is best achieved through random
  assignment.
• If random assignment is not possible, groups
  should be matched on important variables.

59
What is regression to the mean?

• Groups selected based upon an extreme baseline
  score will be less extreme when scored at
  follow-up.

60
Regression to the mean?
Source Grenyer, B. F. S. et al. (2007). Fish oil
supplementation in the treatment of major
depression A randomised double blind
placebo-controlled trial. Progress in
Neuro-Psychopharmacoogy Biological Psychiatry,
31, 1393-1396.
61
Criterion 6 Data Analysis

• NREPP application requires that evaluation
  results be statistically significant at p .05.
• Generally, the analysis can represent differences
  between pre- and posttest scores, between program
  and control groups, or a combination of the two.
• More complicated analyses may be required.
• It is important to consult with a statistician at
  the start of the evaluation planning process and
  to have a statistician do the analysis after data
  collection.

62
Questions?
63
Administration of Workshop Posttest
64
Process Evaluation Results

• Participants reported that Agreement
• The workshop met stated objectives 90
• Will use what was learned in workshop 95
• Would recommend workshop to others 83
• The handouts were useful 88
• Additional information reported
• Number of people trained. 41
• Content review by trained observer. 94 of
  critical
• content covered

65
Outcome Evaluation Results

• Participants correctly identified Pretest Posttes
  t Difference
• 3 types of evaluation 63 92 29
• 6 logic model phases 71 87 16
• 3 types of data collection 48 95 47
• Best evaluation design for evidence 78 98 20
• 6 NREPP criteria 28 83 60

66
Impact Evaluation Results

• Pretest Posttest Difference
• Identification of evaluation types
• Training Group 63 92 29
• Comparison Group 50 54 04
• Phases of logic models
• Training Group 71 87 16
• Comparison Group 67 64 -05
• NREPP criteria identification
• Training Group 28 83 60
• Comparison Group 10 12 02