The Case for Program Evaluation

1
The Case for Program Evaluation
Lifting Teen Families Strengthening Programs
for Pregnant and Parenting Teens Summit

• Kayan L. Lewis PhD
• Brian C. Castrucci MA
• M. Aaron Sayegh PhD MPH
• October 4-5, 2007
• Austin TX

2
Overview

• What basic components need to be incorporated in
  the evaluation of programs for pregnant and
  parenting teens?
• What are evidence-based programs and what is
  the evidence for them?
• Which evaluation metrics are realistic to request
  of program staff without being overly burdensome?
  
• How should program officers incorporate
  evaluation metrics into grantmaking?
• Which important aspects of evaluation are most
  often overlooked?

3
Basic Components of an Evaluation for Pregnant
Parenting Teen Program

• Create evaluation plan for grant application
• Search the literature for important factors
  associated with being a pregnant and/or parenting
  teen.
• Consider a theoretical orientation
• Identify essential components of the program and
  evaluate them
• Common components are change in knowledge,
  attitudes, beliefs, and behaviors.
• Determine when you will measure each component
• Will be based on timing of intervention and how
  long it takes for change to occur

4
What are evidence-based programs

• Different registries of evidence-based programs
• National Registry of Evidence Based Programs and
  Practices. http//nrepp.samhsa.gov/index.htm
• CDCs HIV/AIDS Prevention Research Synthesis
  Project. http//www.cdc.gov/hiv/topics/research/pr
  s/evidence-based-interventions.htm
• Systematic determination of different levels of
  evidence of effectiveness
• http//www.thecommunityguide.org/methods/methods-a
  jpm-developing-guide.pdf
• At least one statistically significant,
  positive outcome and no negative effect (p.05)
• Publication in a peer-reviewed journal or
  documented in comprehensive report
• Appropriate follow-up (3- to 6-months) and
  retention rate (70)

5
Realistic Evaluation Metrics

• The ones that measure the outcomes with the
  fewest questions and forms
• Find a balance between service delivery and the
  need for sufficient evaluation
• Budget technology to alleviate data collection
  burden
• Hand-held PDA, laptops, scannable forms and
  support software
• Ask evaluator to decide what would be nice to
  have vs. necessary to measure outcome

6
Incorporating evaluation metrics into grant
announcements

• Investigate the literature to determine
  standardized outcomes for the projects.
• Propose a minimum requirement for evaluation
• Follow-up, basic demographics, basic outcomes
• Re-work evaluation plan from application before
  the intervention starts
• Require unified reporting of results and
  analyses.

7
What important aspects of evaluation are most
often overlooked?

• Preparing evaluation plan/ identifying the
  evaluator as part of the application
• Requiring evaluator presence at program-related
  meetings
• Writing measurable outcomes reporting on
  outcomes
• Going back to original application when reporting
  results
• Administrative work
• Retention strategies, procedures for
  consenting/follow-up, data analysis plan, data
  entry, storage and security of files.

8
Sample Survey Items

• Parental stress scale http//www.personal.utulsa.
  edu/judy-berry/parent2.htm
• Beck Depression Inventory http//www.gender-id.co
  m/TSInfo/BecksScale.htm
• Word substance use as number of times of use in
  the past 30 days
• Measuring religiosity http//www.teenpregnancy.or
  g/works/pdf/Science_Says_20_Religion.pdf

9
Thank you

• Kayan L. Lewis, Ph.D.
• Kayan.Lewis_at_dshs.state.tx.us
• 512-458-7111 X3208