Outcome Evaluation Plan Overview

1
Outcome Evaluation Plan Overview

• Program goals
• Methodology
• Implementation Plan
• Roles of Sites and Central Office
• Lessons learned from pilots in Illinois,
  Michigan, Minnesota
• Preliminary Findings
• Next Steps

2
Program goals

• MATEC goal Assess impact of Level II III
  programs on providers practices
• HRSA goal MATEC will administer an outcome
  evaluation for 25 of Level II and all Level III
  programs

3
Methodology

• Baseline
• Potential change in clinical practice
• Reinforcement of current clinical practice
• Follow up (4-6 wks, 12-16 wks)
• What changes took place?
• What practices have been reinforced?
• What happened with barrier/obstacles to
  alter/reinforce practice?

4
Implementation Plan

• 1. Select Level II and III programs (Sites)
• 2. During program (Sites)
• Explain purpose and goals, elicit cooperation
• Participants complete outcome form confirm
  contact info.

5
Implementation Plan (cont.)

• 3. Post program 4-6 weeks (Central Office)
• interview participants
• practice changes
• reinforcement good practices
• barriers/actions affect HIV/AIDS management
• 4. Post program 12-16 weeks (Central Office)
• repeat interviews
• 5. Data analysis and reports (Central Office)

6
Implementation Plan Roles of LPSs and Central
Office

• LPS role
• Select programs to be evaluated
• Collect accurate contact information and consent
• Collect completed Outcome Evaluation Form
  (baseline data from program)
• Send data to Central Office

7
Implementation Plan Roles of LPSs and Central
Office

• Central Office role
• Contact participants for follow-up interviews
• Complete 4-6 week and 12-16 week interviews
• Data analysis and reports to LPS and HRSA

8
Implementation Plan Selection of Programs for
Evaluation

• Level II or Level III classification
• How long is training program?
• Who is target audience?
• Is the training designed to impact clinical
  practice?

9
Lessons learned from pilot studies in IL, MI, MN

• Follow-up interviews take 10-15 minutes
• Difficult to contact participants
• Phone, e-mail not always best contact numbers
• Requires connection from LPS to improve response
• Interviews
• Once contacted, easy to interview

10
Lessons learned from pilot studies in IL, MI, MN

• Recommended strategies
• Introduce form at beginning of program, rather
  than at end (use slide and script)
• Allow participant time to be thoughtful
• Do not hand out as leaving program
• Have participants use form throughout program
• Get accurate contact information and help contact
  when needed

11
Sample Presentation to Training Audience

• Why are we doing this?
• Our funders (HRSA) are interested in knowing how
  this program helped you as a practitioner or
  service provider.
• How are we doing this?
• By tracking your responses to this program and
  the longitudinal effects of this program on your
  practice.
• Your role in this effort
• Completing the outcome evaluation form today, and
  agreeing to speak to our evaluation staff one
  month this program for 5-10 minutes about the
  effects of this program on your practice.

12
Reports from Central Office

• Reporting of aggregate data, outcome evaluation
  reports
• For Central Office Data will be analyzed by
  state, by discipline, by program type, and by
  program level
• For LPS ?????

13
Preliminary Findings

• Follow-up rates
• Program 1 33 (2/6)
• Program 2 32 (6/19)
• Program 3 9 (1/11)
• Program 4 57 (4/7)

14
Preliminary FindingsCategorizing responses (PR
areas)

15
Preliminary Findings F/U Data

• Any successful changes in practice?
• More systematic adherence evaluation, asking
  everyone, every time.
• More HIV screening with Asian immigrant
  populations.
• Any unexpected changes?
• I may end up doing what I think is best for the
  patient, rather than consulting with one of the
  ID docs.

16
Next Steps

• LPS
• Select upcoming programs
• Communicate program plan to CO
• Central Office
• Develop reporting template(s)