Use of Fidelity Measurement in Statewide Technical Assistance Activities

1
Use of Fidelity Measurement in Statewide
Technical Assistance Activities

• Gary Bond Festschrift Conference
• Indianapolis, IN
• September 24, 2009
• Angela L. Rollins, Ph.D.
• ACT Center of Indiana
• IUPUI Psychology Department

2
Overview

• Background on fidelity use in general
• Use for fidelity in technical assistance (TA)
  efforts across various EBPs
• Some new directions
• Further questions

3
Uses of Fidelity Research

• Internal validity examine model adherence/drift
• Facilitating communication in literature
• Synthesizing a body of research
• Identifying critical ingredients of program
  models
• Defining model adaptations

4
Uses of Fidelity TA

• Benchmarks to add perspective
• Over time
• Comparison to other programs
• Comparison to norms
• Coaching
• Written report of strengths, weaknesses,
  recommendations for improvement
• Teaching moments within the assessment day itself

5
Uses of Fidelity TA

• Defining program standards/critical ingredients
• Monitoring adherence to program standards
  tricky new area
• Springboard for other TA activities
• Data gathering (Indiana)
• Outcomes data reporting (Kansas)

6
Indiana ACT Fidelity Approach

• 2 assessors ? 1
• Semi-annual ? annual after first year
• Even with these reductions, as teams grow and
  consultants remains constant, fidelity visits
  take a bigger piece of the overall consulting pie

7
Indiana ACT Fidelity Approach

• Testing phone fidelity measures (see McGrew talk
  tomorrow)
• Level of burden has reduced a little over time,
  even for onsite assessment
• Team leaders more familiar with requests
• Agencies create reports over time (trust?)
• Assessors more adept as well - completing faster
  than in initial years
• Still need ongoing support - reduce rater drift,
  feedback on scoring decisions

8
Multiple Uses of the Visit

• Data gathering
• Admission and discharge data for tracking
• Tracking raw data used to compute fidelity scores
• raw turnover rates
• actual FTE and caseload size
• actual frequency and intensity
• Teaching moments most extensive contact with
  site for the year in most cases

9
Multiple Uses of the Visit

• Report teams scores over time in each report
  (examine annual progress)
• Additional monitoring of state standards
  compliance
• Additional information added to report where
  DACTS and state standards differ
• Noncompliance reported to team initially, to
  state if not remediated within 90 days
• Research on real-world data, but problems with
  restriction in range (state standards dictate
  high fidelity)

10
Indiana DACTS data
11
Fidelity Monitoring IMR

• Semi-annual visits
• Baseline after 3-6 months is more meaningful
  for program development
• Emphasis on qualitative feedback rather than
  numeric score
• Data collection via observation of IMR sessions
  is vital
• Immediate feedback at the end of visit important
  for QI function of fidelity
• In progress fidelity via audiotaped sessions
  based on Muesers work

12
Across EBPs

• Role of fidelity may vary in emphasis across EBPs
• Example Does fidelity measurement of IMR play as
  large a role in implementation as it does for
  ACT?
• Clinical competency needs pronounced for IMR,
  IDDT (see Mueser talk tomorrow)
• Program structural needs for ACT very pronounced

13
Future Directions

• Increasing calls to decrease the burden of
  fidelity measurement (e.g., We want to use one
  FTE or less for monitoring our whole state)
• Phone/remote, less intensive version (see McGrew)
• Lower frequency of assessments
• Screening approach
• key items
• follow-up with full visit if flagged during screen

14
Future Directions

• Ohio developing The Evaluation Database TED
• Online data entry by consultant after visit
• Perhaps during onsite visit with wireless card
• More automated consensus and report function
• Track when consultants enter data and when
  reports go out QA for the TA
• Raw data elements as well for fuller data
  gathering
• IDDT and (SE?) so far, may collaborate to tailor
  to ACT
• Seeking funding to tie in consumer outcomes as
  well
• Contact debra.hrouda_at_case.edu

15
Questions to Ponder

• Are statewide fidelity scores a good measure of
  the TA centers effectiveness?
• Would outside fidelity assessors (outside the
  state/entity) produce similar results?
• Beyond basic requirements for certification, what
  motivates teams to score high on fidelity scale?
• Can we marry research and practice when practice
  yields a restriction in range on fidelity scores?

16
Thank you

• Charles Boyle
• Michelle Salyers
• Lia Hicks
• Deb Hrouda