IMPLEMENTING COMPLEX EVIDENCEBASED PROGRAMS

1
IMPLEMENTING COMPLEX EVIDENCE-BASED PROGRAMS

• FROM SERVICE TO SCIENCE AND BACK AGAIN
• REAP Conference
• Santa Fe, NM March 20, 2008

2
STUDY DETAILS

• RESEARCH TEAM
• Susan Carter, PI
• susancarter_at_hughes.net
• Gladys Levis-Pilz
• cowalden_at_yahoo.com
• Heather Yazzie
• heatheryazzie_at_yahoo.com

• Support provided by a FIRE grant from the Office
  of Juvenile Justice and Delinquency Prevention,
  US DOJ

3
THE PROBLEM

• We are faced with the paradox of
  non-evidence-based implementation of
  evidence-based programs
• (Drake, Gorman, Torrey, 2002)
• Great emphasis on identifying evidence-based
  programs, but little attention paid to their
  implementation beyond fidelity discussions,
  especially for complex programs

4
RESEARCH QUESTIONS(After Fixen et al, NIRN, 2005)

• BASED ON STAGES OF PROGRAM IMPLEMENTATION
• Exploration and Adoption
• Program Installation
• Initial Implementation
• Full Operation
• Innovation and Sustainability

5
DESIGN AND METHODS

• BASED ON NIYLPS PROJECT VENTURE
• Created by McClellan Hall, Founder of National
  Indian Youth Leadership Project, Gallup, NM
• Grew out of camp program into year round model in
  1990
• Named by NREPP as a Model Program in 2004
• Re-reviewed by NREPP, designated evidence-based
  program in 2007
• Universal prevention model based on
  experiential, youth development principles for
  Native American mid-school youth

6
Project Venture 3 levels of non-negotiables
Populationserved
Programcomponents
Program delivery

• Positive approach
• Outdoor adventure and experiential challenge
• Experiential education
• Wilderness outdoor adventure
• Physical education
• Nature challenge
• Service learning
• Active learning
• Academics in action
• School or community service
• Parent involvement
• Native culture based leadership
• Culture and values
• Language and history
• Cross-cultural exchange and awareness
• Elders

• 200 hours of contact annually
• Risk management principles followed, including
  sequential progression of elements including
• Physical safety
• Mental health safety
• Youth to follow four levels of progression
  through program
• In school (highly recommended)
• After school
• Weekend and overnight
• Intensive camp experience
• Outcomes measured

• Focus on Native youth
• Middle school aged youth
• High school aged Service Staff
• Mix of risk levels (program should not have
  stigma of prevention program)

Bold Basic non-negotiable (element must be in
place, even from beginning) Regular Phased-in
non-negotiable (element must be in place, but may
be phased in over time) Italic Recommendation
(element is recommended, but not
required) Modification allowed (element may
differ if program is appropriately modified)
7
DESIGN AND METHODS

• Phase 1 In-depth Case Studies of a few PV
  programs
• Identify all known PV adoption sites in US
• Select 8 diverse sites considering rural/urban,
  new/experienced agency, agency type, etc.
• Send invitation letter
• Conduct on-site program observations
• Conduct semi-structured interviews
• Review documents related to implementation
• Complete PV Replication Instrument
• Complete saw/read/heard about it table

8
Project Venture currently 36 replication sites
PV replication site
There are approximately 36 replication sites
across 17 states
9
Interview Guide Questions

• Exploration and Adoption
• How did you hear about PV in the first place?
• How/why was the decision made to select?
• What are the components of your program?
• Is the programming running fully or partially?
• Community Support?
• School Support?
• Who makes the decisions for PV?
• Is it important that your facilitators be
  American Indian?
• Program Installation
• How did you prepare to implement PV?
• Who made those decisions?
• Staffing? Financing? Restructuring?
• What about now, how is PV implemented?
• Staffing? Financing? Restructuring?
• Initial Implementation
• Participants, who?

• Full Operation
• How did you implement PV components? Did you
  begin with all components or phase them in?
• Who benefits?
• How do you feel about PV?
• What kinds of out comes have you had?
• Are you measuring outcomes? Tell me about this.
  Would you share data with NIYLP?
• Do you use it to improve programs?
• Does your program hold true to the vision of PV?
  (go through Program Implementation instrument
  with them)
• Innovation and Sustainability
• How do you manage change in population,
  financing, community political/tribal, drift/
  fidelity?
• Any plans to sustain? How?

10
(No Transcript)
11
DESIGN AND METHODS

• Phase 2 Broad study of all PV programs
• Develop survey based on Phase 1 findings
• Administer via web, mail, phone to all known PV
  sites

12
PRELIMINARY FINDINGS FROM CASE STUDIES

• From earlier Project Venture studies, Blooms
  Taxonomy seemed a useful framework upon which to
  hang observational data
• Life lessons available to individual youth
  increased through exposure to other facilitators
  and participants and these lessons also promoted
  higher levels of cognitive and affective learning
  (as detailed below).

13
CASE STUDY FINDINGS (CONT.)

• The researchers attention kept returning to an
  idea of evolution or levels resulting from
  provider-participant and participant-participant
  interactions. These thoughts in turn led to a
  review of the educational psychologist Benjamin
  Blooms taxonomy of levels of learning (Bloom,
  B. (ed.) Taxonomy of Educational Objectives The
  Classification of Educational Goals,1956, New
  York Longmans Green).

14
CASE STUDY FINDINGS (CONT.)

• This classic theory provides a classification of
  levels of thinking and feeling behaviors thought
  to be important in the process of learning. The
  taxonomies are both descriptive and prescriptive
  as it orders categories of cognitive, affective,
  and psychomotor skills. From basic/low levels
  to ever more abstract/higher levels of
  learning.

15

• While experimental studies are crucial for
  advancing our understanding of what works in
  adventure programming, the qualitative
  perspective has the power to show us how
  programming works. Intentionally designed and
  executed qualitative research enriches our
  understanding of what goes on in the black box
  between pre and post tests and provides us with
  the stories to tell and prescriptions for action.

16
Blooms Taxonomy A Promising Analytic Framework

• Cognitive Skills (from less to more complex)
• Knowledge
• Comprehension
• Application
• Analysis
• Synthesis
• Evaluation

17
Blooms (cont.)

• Affective Skills (from less to more complex)
• Receiving phenomena
• Responding to phenomena
• Valuing
• Organization
• Internalizing values