Selecting EvidenceBased Strategies: Letting the Data Lead the Way

1
Selecting Evidence-Based Strategies Letting
the Data Lead the Way

• Presented by
• Kerrilyn Scott-Nakai
• The Center for Applied Research Solutions (CARS)
• Sponsored by Western CAPT, a contractor for the
  U.S. Center for Substance Abuse Prevention,
  SAMHSA

1
2
Webinar Session Agenda
2
3
Polling Question 1

• Who is in the audience?
• a) State representatives
• b) Managers/directors
• c) Direct service providers
• d) Evaluators/researchers

4
Polling Question 2

• How would you rate your knowledge/experience
  level regarding this topic?
• a) Beginning/Introductory
• b) Intermediate
• c) Advanced

5
Part I.

• The Context An Overview of the Strategic
  Prevention Framework (SPF)

6
SAMHSAs SPF
Assessment Profile population needs, resources,
and readiness to address needs and gaps
Capacity Mobilize and/or build capacity to
address needs
Planning Develop a Comprehensive Strategic Plan
Implementation Implement evidence-based
prevention programs and activities
Evaluation Monitor, evaluate, sustain, and
improve or replace those that fail
7
The Strategic Prevention Framework
7
8
Strategic Prevention Framework

• Consists of 5 basic planning steps or principles
• Focus on systematic, data-driven, evidence-based,
  and outcome-based planning
• Supports accountability, capacity, and
  effectiveness
• Allows for course corrections along the way
• The process should be aligned along the way
  (similar to a logic model)
• The term Steps is misleading--it is a dynamic
  and iterative process

9
SPF Cornerstones

• Cultural competence and sustainability are
  central to SPF
• They are integral to each step/phase of the
  framework

10
Part II.

• Needs Assessment

11
The Strategic Prevention Framework
11
12
Needs Assessment

• Activities
• Data Collection
• Assessment of Data
• Prioritization of Issues
• Development of Problem Statements

13
Raise of Hands

• How many have been involved with conducting a
  needs assessment?

14
Needs Assessment

• Assessment of
• substance use (consumption) and related problems
  (consequences)
• needs, resources, and gaps in services
• community readiness
• Yields recommendations regarding community
  priorities
• Rank orders AOD problems based on data
• Shares with/engages community stakeholders

15
Needs Assessment

• Runs parallel with the SPF Capacity Building
  process
• Results in the development of Problem Statements
• Clearly links to the Goals, Objectives, and
  selected strategies.
• Connects to the Evaluation process
• may establish a baseline dataset

16
Sizing Up Available Data

• Review the data sources and findings
• Articulate the following
• the major problems documented in the assessment,
• the criteria for identifying priorities, and
• the strengths and weaknesses of the evidence
  base.
• Identify data that was not collected in the needs
  assessment, is relevant and available, and should
  be considered.

16
17
Polling Question 3

• What are examples of common sets of data that you
  utilize?

18
Sizing Up Available Data
18
19
Examples
19
20
Polling Question 4

• What level data do you usually utilize?
• State Level
• County Level
• School District Level
• School Level
• Local Community Level

21
Asset-Related Data
21
22
Asset-Related Data
22
23
Prioritization is Key!

• Many factors involved in substance use problems
• economic conditions,
• availability of alcohol and drugs,
• the quality of schools, and
• community cohesion
• The key is to focus on factors that are feasible
  to address.

23
24
Data Driven Decision Making

• Actual data is used as the central test for
  identifying priorities
• Focuses on those problems that most need
  attention
• Minimizes disagreement based on past commitments,
  personal preferences or even self interest

24
25
Questions to Help Reach Consensus

• Immediate?
• Specific and measurable?
• Solvable in a reasonable amount of time?
• Able to result in real improvement in peoples
  lives?
• Widely and deeply felt?
• Non-divisive and consistent with the groups
  values?
• Resonate strongly?
• Anticipated barriers and resistance?

25
26
Part III

• Planning
• Selecting Evidence-Based Strategies

27
The Strategic Prevention Framework
27
28
SPF Links Planning

• Driven by the Needs Assessment findings and
  Problem Statements
• Engages community members and organizations
  identified through the Capacity Building phase
• Establishes the most appropriate evidence-based
  prevention approaches for Implementation
• Outlines the link between planned objectives and
  methods for measuring progress in the Evaluation
  phase.

29
Planning
29
30
Three Phases of Prevention Planning
30
31
Goals

• Goal are measurable statements of desired
  longer-term, global impact of the prevention
  program.
• They reflect the longer-term outcomes the
  services are intended to have.
• AOD prevention goals typically address changes in
  use or incidence of harmful consequences.

31
32
Objectives

• Objective are more specific measurable statements
  which reflect more immediate or direct outcomes
  of the services/programwhich directly support
  the goal.
• Objectives typically reflect changes in
  participant behaviors or attitudes that occur as
  a result of the prevention services/strategies.
• They may also focus on altering consequences
  experienced by the user or the behavior of users
  that affect others.

32
33
S.M.A.R.T. Objectives

• Specific
• Measurable
• Appropriate
• Realistic
• Time-bound

33
34
Sample Formula for Developing Objectives

• How much of what change will occur to whom by
  when as measured by what?
• Of the ________________________ (state the
  addressed population), ______ participants in
  prevention services will show ____________________
  (decreases or increases), in the__________
  (insert specific indicator) as measured by
  __________________ (state the evaluation tool)
  over the next ____________(identify timeframe in
  years or months).

34
35
The Nature of Evidenceis continuous and
contextual

• Strength of evidence is determined by
• Rigor of the study design
• Rigor and appropriateness of methods used to
  collect and analyze the data
• The extent to which findings can be generalized
  to similar populations and settings.

36
Practical Steps for Selecting Evidence-Based
Policies, Programs, and Practices
Effective?
Appropriate?
Adapted from Identifying Selecting
Evidence-Based Interventions, SAMHSA, January 2007
Relevant?
36
37
Determining Population-Based Service Levels

• Universal Prevention
• Address the entire population.
• Aim is prevent/delay use of ATOD. Deter onset
  through a variety of broad level approaches
• Selective Prevention
• Addresses needs of subsets of population
  considered at risk by virtue of their membership
  in a particular subgroup.
• Targets the entire subgroup regardless of the
  degree of risk in the group.
• Indicated Prevention
• Focuses on individuals who are exhibiting early
  signs or consequences of substance abuse.
• Aim is reduction in first-time substance abuse,
  length of time the signs continue delay of onset
  of substance abuse, and/or reduction in the
  severity of substance abuse

38
Polling Question 5

• What population level do you typically target?
• Universal
• Selective
• Indicated
• All of the above

39
Integration of CSAP 6 Strategies

• 1. Information Dissemination
• 2. Prevention Education
• 3. Alternative Activities
• 4. Community-Based Processes
• 5. Environmental Approaches
• 6. Problem Identification and Referral

Comprehensive approaches using multiple
strategies in combination to address identified
problems and local conditions that contribute to
problems.
40
Utility Check

• Appropriate for the priority population.
• Has the intervention been implemented
  successfully with the same or a similar
  population?
• Are the population differences likely to
  compromise the results?
• Delivered in a setting similar to the one planned
  by the community.
• In what ways is the context different?
• Are differences likely to compromise the
  interventions effectiveness?

41
Utility Check

• Culturally appropriate.
• Did members of the cultural group participate in
  developing it?
• Were materials adapted to the culturally
  identified group?
• Are implementation materials available to guide
  intervention implementation?
• Are training and technical assistance available
  to support implementation?
• Are monitoring or evaluation tools available to
  help track implementation quality?

42
Feasibility Check

• Culturally feasible
• given the values of the community?
• Politically feasible
• given the local power structure and priorities?
• Match with mission, vision, and culture?

43
Feasibility Check

• Administratively feasible
• given the policies and procedures?
• Technically feasible
• given staff capabilities and time commitments and
  program resources?
• Financially feasible
• given the estimated costs of implementation
• including costs for purchase of implementation
  materials and specialized training or technical
  assistance?

44
What Types of Interventions are Appropriate?
44
45
What is the Evidence of Effectiveness?
45
46
Resources for Finding Evidence-Based Interventions

• Sponsored lists or registries
• Research articles and reports in peer-reviewed
  journals
• Documented effectiveness

46
47
Raise of Hands

• Question 1 How many are familiar with NREPP?
• Question 2 How many have utilized CSAP
  Prevention Platform website?

48
Sponsored Lists or Registries

• Advantages
• Interventions have met specified criteria for
  evidence
• Interventions are sufficiently and clearly
  articulated and described to support replication

• Disadvantages
• Interventions are mostly amenable to standard
  research design and proof (e.g., direct
  interventions with individuals are
  over-represented, environmental and
  community-based interventions are
  under-represented)
• Limited in the number of intervention options
• Relatively weak in evidence concerning their
  local applicability

48
49
National Registry of Evidence-Based Programs and
Practices (NREPP)

• Evolved over time to include substance abuse
  prevention, treatment and mental health promotion
  interventions
• Evolved to a continuum of evidence approach
• The previous categories (e.g. model, promising)
  are no longer used
• Seen as a selection tool
• States are able to create their own
  definitions/standards

50
Research Articles and Reports in Peer-Reviewed
Journals

• Advantages
• Uses a variety of research methods that provide
  valid information beyond what the standard lists
  recognize as rigorous
• Provides multi-site studies and reviews that
  focus on specific core practices, skills, or
  principles of effective prevention rather than
  whole programs
• Provides a large information resource that allows
  planners to identify information relevant to
  their particular context, e.g., cultural
  group(s), community characteristics

• Disadvantages
• Requires greater expertise to interpret its
  relevance and applicability to the planning
  process
• Challenges
• Assessing the strength of evidence (how effective
  is a particular practice?)
• Teasing the specific design of policies, programs
  and practices, often generalized in brief
  research reports and journal articles

50
51
Documented Effectiveness

• Advantages
• Grounds the selection process in the reality of
  each community
• Reinforces the critical planning steps of
  developing a clear conceptual framework for
  interventions (e.g., developing a logic model)
• Involves community members and stakeholders in a
  systematic, evidence-based decision process

• Disadvantages
• Requires extensive decision making and
  documentation that create resource demands beyond
  those that are readily available to the community
  
• Requires a willingness to systematically engage
  the realities of the decision process and the
  complexity of fitting interventions to community
  capacity
• Places a premium on effective leadership to bring
  the process to fruition

51
52
Selecting an Intervention Strategy
52
53
Selecting an Intervention Strategy
53
54
The Role of a Strategic Plan

• Develops a comprehensive, logical, and data
  driven strategic plan to address the problems
  identified in Step 1 using the capacity built or
  mobilized in Step 2
• The plan includes Strategic Goals, Objectives,
  and Strategies that address the Problem Statements

55
Components of a Strategic Plan

• Lays the groundwork for
• Implementation activities
• The identification of strategies
• The selection of evidence-based programs,
  policies, and practices to be implemented
• The evaluation plan

56
In Summary

• We reviewed the role of the needs assessment
  process and the relationship with data-driven
  decision making.
• We reviewed the role of relevance,
  appropriateness, and evidence for selection.
• We reviewed sources for evidence-based
  strategies, programs, and practices.

56
57
Q and A
58
Citation/Acknowledgement

• The materials in slides 31 through 42 were
  adapted from
• Identifying Selecting Evidence-Based
  Interventions, SAMHSA, January 2009

59
Resources

• CSAP Prevention Platform

60
Contact Information

• Presenter Information
• Kerrilyn Scott-Nakai
• Center for Applied Research Solutions (CARS)
• 923 College Avenue Santa Rosa, CA 95404
• [phone] Phone
• www.cars-rp.org
• kerrilyn_at_cars-rp.org