Objectives for Session Eleven

1
Objectives for Session Eleven

• Turn in Data Assignment
• Discuss Hypothetical Formative Evaluation Design
• Analyzing Qualitative Data
• Report Writing, Organization, and Utilization
• Strengths and Weaknesses of Different Data
  Collection Strategies

2
What Findings or Patterns Emerged from Your
Analyses?
3
STOP-DWI in NY State

• STOP-DWI is implemented in NY to reduce
  drunken-driving and alcohol-related accidents
• Program theory
• Implementation/Service Delivery
• Checkpoints, tougher penalties, public service
  announcements, and education programs
• How would we know if it was being implemented
  well?

4
Evaluation Design Proposal for STOP-DWI in NY
State

• Program Description
• Who, where, what, how
• Steps in delivery, logic model, resources/inputs
• Evaluation Questions and Justification
• Data Collection Strategy
• Focus on 4 stages checkpoints, adjudication in
  courts, advertisements, and education programs
• Select methods that are appropriate for given
  stage
• Mix of methods quantitative and qualitative
• Validity and reliability
• Sampling

5
Analyzing Data

• Approach to Quantitative Analysis
• Missing data and Outliers
• Description of Respondents
• Dimensions of Interest/Relationships among Key
  Variables move to more sophisticated analysis
• Approach to Qualitative Analysis
• Confidence in your data?
• What do you know about the program?
• What questions or relationships are you most
  interested in? What does your data tell you
  about these questions or relationships?

6
Strengths and Weaknesses of Different Qualitative
Techniques

• Differences across observation techniques
• Differences between focus groups and interviews
• Document review highly variable
• Note that a variety of techniques are useful in a
  formative evaluation setting

7
Integrating Qualitative and Quantitative Data

• New programs
• Program effects or measures are hard to quantify
• Qualitative data to provide insight into or
  corroborate quantitative findings
• Demands of client and context

8
Report Writing

• Description of the Program
• Presentation of Analysis
• Implemented as planned or intended?
• What affected implementation?
• Interpretation of the Findings
• What do your data suggest or mean for the
  stakeholders?
• Justify and support with data
• Conclusions and Recommendations
• Prioritize, feasibility

9
Factors Undermining Utility of Evaluation Reports

• Poor quality
• Lack generalizability
• Lack program description
• Mismatch between research questions and questions
  data can answer
• Not completed on time
• Characteristics of commitment of organization

10
Readings for Next Time

• Leon, Dziegielewski, and Tubiak. A Program
  Evaluation Of A Juvenile Halfway House
  Considerations For Strengthening Program
  Components.
• DEmidio-Caston and Brown. The Other Side Of
  The Story Student Narratives on the California
  Drug, Alcohol, and Tobacco Education Programs.