Improving Practice to Wraparound Through Use of Fidelity Measures

1
Improving Practice to Wraparound Through Use of
Fidelity Measures

• Enrica Bertoldo, Quality Support Manager
• Eleanor Castillo, Director of Outcomes Quality
  Assurance
• Mary Ann Wong, Research Specialist
• Veronica Padilla, Manager of Outcomes
  Evaluations
• 26th Annual California Mental Health Advocates
  for Children Youth Conference, Asilomar,
  California
• May 4, 2006

2
EMQ Children Family Services

• Our Mission
• To work with children and their families to
  transform their lives, build emotional, social
  and familial well-being, and to transform the
  systems that serve them.
• Our Philosophy
• Family Voice ? Team Based
• Collaborative/Integrative ? Community-Based
• Culturally Competent ? Individualized
• Strength-Based ? Natural Supports
• Persistence ? Outcomes-Based

3
Purpose

• Given the complexity of Wraparound,
  existing fidelity measures are utilized to
  improve practice.
• This workshop will share successes and challenges
  in the implementation of fidelity measures
  (WFI-3.0 WOF-2.0) that affect practice and
  supervision to Wraparound.
• A pilot in utilizing a Wraparound supervisory
  adherence measure (W-SAM) will also be discussed.

4
What is Wraparound?

• Team-based, individualized service planning
  process
• Principles of community-based, strength-based,
  family-centered, and culturally competent
• Wraparound is described as a promising practice
  (Burns, Hoagwood, Maultsby, 1998)

5
11 Core Elements of Wraparound

• Voice and Choice
• Youth/Family Team
• Community-Based Services/Supports
• Cultural Competence
• Individualized Services/Supports
• Strengths-Based Services/Supports

• Natural Supports
• 8. Continuation of
  Services/Supports
• Collaboration
• Flexible Resources/ Funding
• Outcome-Based Services/Supports

6
Essential Elements of Wraparound

• Element 1 Voice and Choice
• Families must be full and active partners at
  every level of the wraparound process. If the
  team cannot reach consensus, the final decision
  should be up to the caregiver.
• Element 2 Youth and Family Team
• Wraparound is a team-driven process involving
  caregivers, youth, natural supports, and
  community services working together to develop,
  implement, and evaluate the individualized plan.
• Element 3 Community-based Services and Supports
• Services and supports that the youth and family
  receive should be based in their community. The
  family should not have to leave their community
  if more restrictive services are necessary.
• Element 4 Cultural Competence
• The team should not only be respectful of the
  familys beliefs and traditions, but also
  actively seek to under-stand the familys unique
  perspectives and convey them to others.

7

• Element 5 Individualized Services
• This means that services and supports are
  tailored to the unique situation, strengths, and
  needs of teach individual, and may involve
  existing categorical services and informal
  sup-ports modifying existing services and
  supports and or creating new services and
  supports. Further, the team should create a
  specific plan to meet the familys goals and
  crisis/ safety plan to manage potential
  emergencies.
• Element 6 Strengths-Based Services
• The focus of the team should be on what is
  working and going well for the family. While
  goals may be drawn up based on the familys
  needs, the plan should capitalize on the familys
  positive abilities and characteristics.
• Element 7 Natural Supports
• Services and supports should reflect a balance
  of formal and informal community and family
  supports rather than a reliance on formal
  professional services.
• Element 8 Continuation of Care
• Services and supports must be provided
  unconditionally. In a crisis, services and
  supports should be added rather than placing the
  youth with a new provider.

8

• Element 9 Collaboration
• The team should coordinate services and supports
  so they seem seamless to the family rather than
  disjointed.
• Element 10 Flexible Funding and Resources
• Successful wraparound teams are creative in
  their approach to service delivery and have
  access to flexible funds and resources to
  implement their ideas.
• Element 11 Outcome-Based Services
• Specific, measurable out-comes should be
  monitored to assess the youth and familys
  progress toward goals.
• The description of each element is taken
  directly from Wraparound Fidelity Index 3.0
  (2002), produced by the Wraparound Evaluation and
  Research Team at the University of Vermont.

9
What is Fidelity to Wraparound?

• Treatment Fidelity The degree to which a
  program is implemented as intended (Rast Bruns,
  2003 Moncher Prinz, 1991)
• Adherence to the 11 Core Wraparound Elements
  (e.g., WFI-3.0 Suter et al., 2002)
• Adherence to the Child and Family Team Process
  (e.g., WOF-2.0 Epstein et al., 2002)

10
Why Measure Fidelity?

• Measuring fidelity is essential to families,
  providers, policy makers and researchers
• Without measuring fidelity, how do you ensure the
  Wraparound process is occurring?
• Without outcomes, the Wraparound process is just
  one more fad
• High fidelity has been associated with positive
  outcomes in some studies
• It improves quality assurance
• It helps agencies secure more funding by proving
  outcomes
• It can even help create legislation on how
  families and kids receive services

11
More Whys..

• All nationally recognized Wraparound programs
  have extensive fidelity measurements in place.
• Why?
• Without fidelity measurement standards firmly in
  place, Wraparound programs and team members have
  difficulty knowing what they are doing well and
  what they need to improve in order to achieve the
  ideal as presented in the Wraparound vision.

12
Fidelity Implementation Measures

• Wraparound Fidelity Index-3.0 (WFI-3.0)
• Measures fidelity to Wraparound principles
  through structured interviews
• Administered to Youth, Caregiver, Resource
  Facilitator
• Higher Score Greater Fidelity
• Wraparound Observation Form-2.0 (WOF-2.0)
• Measures fidelity to Wraparound through
  observation of the Child and Family Team meetings
• Higher Score Greater Fidelity

13
Fidelity to Wraparound

• EMQ Sacramento has collected the Wraparound
  Fidelity Index (WFI) since 2001
• Data from WFI-3.0
  January 2003 September 2005

14

• A Comparison of EMQ Data National Data

15
WFI Total Scores by Respondent EMQ National
Data
National WFI-3 Dataset came from the Wraparound
Evaluation and Research Team presentation for the
18th Annual Research Conference on Systems of
Care and Children's Mental Health in Tampa on
3/7/05 by Eric J. Bruns, Ph.D. for "Is it
Wraparound Yet?" - Bootstrapping wraparound
fidelity standards using the WFI .
16
WFI Elements EMQ National Data
National WFI-3 Dataset came from the Wraparound
Evaluation and Research Team presentation for the
Technical Assistance Partnership Webinar on
4/5/04 by Eric J. Bruns, Ph.D. for "Ensuring
High-Quality Wraparound"
17
WFI Elements EMQ National Data
National WFI-3 Dataset came from the Wraparound
Evaluation and Research Team presentation for the
Technical Assistance Partnership Webinar on
4/5/04 by Eric J. Bruns, Ph.D. for "Ensuring
High-Quality Wraparound"
18

• The Relationship Between Fidelity Outcomes

19
The Relationship Between Fidelity Outcomes

• Previous research has found an association
  between greater Wraparound fidelity and better
  child and family outcomes (Bruns, 2004), but the
  relationship has not been clearly understood
• EMQ (2005) completed a study to further
  understand the relationship between fidelity to
  the 11 core elements of Wraparound and treatment
  outcomes

20
Demographics

• Average Age at Admission 14 years
• 63 Male 37 Female
• 62 Caucasian 26 African-American 8 Latino
  2 Asian/Pacific Islander 1 Native-American
  1 Other
• Average Length of Stay 15 months

21
Fidelity to Wraparound

• Implementation Measure
• Wraparound Fidelity Index 3.0 (WFI-3.0)
• Collection since 2003
• Sample in study included the following
• 146 WFIs collected from Youth
• 124 WFIs collected from Caregivers
• 183 WFIs collected from Resource Facilitators

22
Outcome Measures at Discharge

• Child and Adolescent Functional Assessment Scale
• (CAFAS Hodges, 2000)
• Measures youths level of functioning completed
  by Resource Facilitator
• Lower Score Greater Functioning
• Exit Total CAFAS Score
• Child Behavior Checklist (CBCL Achenbach, 2001)
• Measures youths level of behavior problems
  administered to Caregiver
• Lower Score Lower Behavior Problems
• Exit Total, Externalizing, Internalizing CBCL
  Scores
• Living Arrangement at Discharge Community or
  Facility

23
Correlations Among WFI Scores and Outcomes at
Discharge
Note p
24
Significant Results
Note p

Greater community-based services is related to
lower impairment on the CBCL and CAFAS at
discharge

Greater community-based services is related to
community living arrangement at discharge

25
Results

• Greater fidelity to the provision of Community
  Services/Supports in Wraparound is related to
  positive outcomes
• No other elements had as significant correlations
  to outcomes as Community Services/Supports
• No significant relationship found between Total
  Fidelity Scores and outcomes at discharge
• Correlations were consistently low even when
  significant

26
Implications

• Implications for teams to support a youths
  school attendance and involvement in work,
  training, and other community activities (e.g.,
  church, sports, art, etc.) as well as support
  community-based living situations for youth (as
  an alternative to residential or institutional
  care) in order to improve outcomes.

27
Challenges of the WFI

• Direct feedback to teams was not provided
• The utility of the WFI in its relationship to
  outcomes was limited based on EMQs analysis of
  the data
• Time intensive and limited resources available to
  conduct 3 structured interviews per youth

28

• EMQ Practice Standards Initiative

29
Initiative to Improve Practice

• Quality Improvement Project launched in January
  2004 to implement practice standards and feedback
  tools across positions
• Family Facilitator
• Family Specialist
• Family Partner
• Goal was to standardize job positions, provide
  role clarity, and improve consistency in job
  performance and supervision

30
Example of EMQ Feedback ToolFamily Facilitator

• Scoring
• Opportunity for Improvement (1)
• Acceptable (2)
• Exemplary (3)
• Not Observed ()
• 3 Dimensions
• A. Families receive services that are consistent
  with wraparound values
• B. Families receive services that are
  collaborative, integrated and adhere to best
  practices
• C. Children and Families served are safe and
  stable at home and in the community

31
Facilitator Field Feedback Tool

• A. Families receive services that are consistent
  with wraparound values
• 1. Arrived on time and was prepared for the CFT
• 2. Started the CFT with whats working to
  maintain strength based focus and future
  orientation
• 3. Facilitated the CFT to validate and
  incorporate ideas, requests, and concerns of all
  members
• 4. Evidence that needs and strategies were
  reviewed within the CFT to assure the family
  voice and preference guide all activities
• 5. Evidence that plans were reviewed for progress
  on current measurable, observable goals and
  updated as needed to address new and changing
  needs
• 6. Evidence that the next CFT and staff visits
  were scheduled according to family preference

32
Facilitator Field Feedback Tool

• B. Families receive services that are
  collaborative, integrated and adhere to best
  practices
• 1. For children open six months or longer,
  evidence that the CFT was comprised of at least
  50 non-paid informal members
• 2. Facilitated the CFT to allow pertinent agenda
  items to be covered according to family
  preference and time availability
• 3. Demonstrated engagement and rapport with all
  team members including system partners
• 4. Therapy, medications and adjunctive services
  were evaluated relative to plan goals as needed
• Specific tasks were assigned
• C. Children and Families served are safe and
  stable at home and in the community
• 1. Evidence that team has a plan B to back up
  plan A
• 2. If the CFT was for a child in out of home
  placement, this was addressed as a priority with
  specific plan, tasks, and target date
• 3. Evidence that strategies and interventions
  developed emphasize the use of typical resources
  in the childs community

33
EMQ Feedback Tools

• Data analysis showed that the tools were not
  measuring what was intended therefore, tools
  were discarded
• Led to an examination of the national standards
  for Wraparound
• EMQ Sacramento decided to collect data on the
  Child and Family Team process using the
  Wraparound Observation Form Version 2.0
  (WOF-2.0)

34
Wraparound Observation Form Second Version
(WOF-2)

• Measure to assess the implementation of the
  Wraparound approach by direct observation of
  youth and family team meetings
• WOF has demonstrated good inter-rater reliability
  (kappa.886 Nordess Epstein, 2003)
• 48 items that measures 8 characteristics
• Community-Based ? Unconditional Care
• Individualized ? Measurable Outcomes
• Family-Driven ? Management of Team
  Meeting
• Interagency Collaboration ? Care Coordinator

35
Implementation of the WOF

• WOF data collection began July 2005
• Feedback provided in individual supervision if
  completed by direct supervisor
• Verbal feedback provided to facilitators after
  the child and family team meeting if completed by
  neutral observer. Copy of tools also provided to
  supervisors

36
Wraparound Observation Form Second Version
(WOF-2)

• Community-based resources (5 items)
• 1. Information about resources / interventions
  in the area is offered to the team.
• 2. Plan of care includes at least one public
  and/or private community service/resource.
• 3. Plan of care includes at least one informal
  resource.
• 4. When residential placement is discussed,
  team chooses community placements for child(ren)
  rather than out-of-community placements, whenever
  possible.
• 5. Individuals (non-professionals) important
  to the family are present at the meeting.

37

• Individualized services for the family (9 items)
• 6. If an initial plan of care meeting, the
  parent is asked what treatments or interventions
  he/she felt worked/didnt work prior to WRAP.
• 7. Facilitator advocates for services and
  resources for the family (e.g., identifies and
  argues for necessary services).
• 8. All services needed by family are
  included in plan (e.g., no needed services were
  not offered).
• 9. Barriers to services or resources /
  interventions are identified and solutions
  discussed.
• 10. The steps needed to implement the plan of
  care are clearly specified by the team.
• 11. Strengths of family members are identified
  and discusses at the meeting.
• 12. Plan of care that includes life domain(s)
  goals, objectives, and resources/interventions is
  discussed (or written).
• 13. Plan of care goals, objectives, or
  interventions are based on family/child
  strengths.
• 14. Safety plan/Crisis plan developed/reviewed.

38

• Family-driven services (10 items)
• 15. Convenient arrangements for familys
  presence at meeting are made (e.g., location,
  time, transportation, day care arrangements).
• 16. The parent/child is seated or invited to
  sit where he/she can be included in the
  discussion.
• 17. Family members are treated in a courteous
  fashion at all times.
• 18. The familys perspective is presented to
  professionals from other agencies.
• 19. The family is asked what goals they would
  like to work on.
• 20. The parent is asked about the types of
  services or resources/interventions he/she would
  prefer for his/her family.
• 21. Family members are involved in designing
  the plan of care.
• 22. In the plan of care, the family and team
  members are assigned (or asked) tasks and
  responsibilities that promote the familys
  independence (e.g., accessing resources on own,
  budgeting, maintaining housing).
• 23. The team plans to keep the family intact
  or to reunite the family.
• 24. Family Members voice agreement/disagreement
  with plan of care.

39

• Interagency collaboration (7 items)
• 25. Staff from other agencies who care about
  or provide resources/interventions to the family
  are present at the meeting.
• 26. Staff from other facilities or agencies
  (if present) have an opportunity to provide input
  
• 27. Informal supports (if present) have an
  opportunity to provide input.
• 28. Problems that can develop in an
  interagency team (e.g., turf problems, challenges
  to authority) are not evident or are resolved.
• 29. Staff from other agencies describe support
  resources interventions available in the
  community.
• 30. Statement(s) made by a staff member or an
  informal support indicate that contact/
  communication with another team member occurred
  between meetings.
• 31. Availability of alternative funding
  sources is discussed before flexible funds are
  committed.

40

• Unconditional Care (3 items)
• 32. Termination of Wrap services is discussed
  because of the multiplicity or severity of the
  childs/familys behaviors /problems.
• 33. Termination of other services (not Wrap)
  is discussed because of the multiplicity or
  severity of the childs/familys behaviors
  /problems.
• 34. For severe behavior challenges (e.g.,
  gangs, drugs), discussion focuses on safety
  plans/crisis plans (e.g., service and staff to be
  provided) rather than termination.
• Measurable Outcomes (3 items)
• 35. The plan of care goals are discussed in
  objective, measurable terms.
• 36. The criteria for ending Wrap involvement
  are discussed.
• 37. Objective or verifiable information on
  child and parent functioning is used as outcome
  data.

41

• Management of team meeting (5 items)
• 38. Key participants are invited to the meeting
  (e.g., family members, CPS worker, teacher,
  therapist, others identified by family).
• 39. Current information about the family (e.g.,
  social history, behavioral and emotional status)
  is gathered prior to the meeting and shared at
  meeting (or beforehand).
• 40. All meeting participants introduce
  themselves (if applicable) or are introduced.
• 41. The family is informed that they may be
  observed during the meeting.
• 42. Plan of care is agreed on by all present at
  the meeting.

42

• Facilitator (6 items)
• 43. Facilitator makes the agenda of meeting
  clear to participants.
• 44. Facilitator reviews goals, objectives,
  interventions and/or progress of plan of care.
• 45. Facilitator directs (or reflects) team to
  discuss family/child strengths.
• 46. Facilitator directs (or reflects) team to
  revise/update plan of care.
• 47. Facilitator summarizes content of the
  meeting at the conclusion of the meeting.
• 48. Facilitator sets next meeting date/time.

43
EMQ WOF-2 ResultsTotal Element Scores (N30)
44
Results of WOF

• EMQ Sacramento shows higher scores on elements of
  Family-driven, Collaboration, Unconditional Care
  Management of CFT Meetings
• EMQ Sacramento continues to be challenged by
  lower scores on Community-based Resources,
  inclusive of natural supports
• In contrast to the results of the WFI for
  Outcomes (high scores), the WOF showed low scores
  on this element
• Slight increase in Total WFI scores since the
  implementation of the WOF

45
Back to the WFI

• Could the use of the WOF impact WFI scores?
• A look at WFI scores across time

46
WFI Total Scores Across Time
WOF Implementation Begins 7/05
47
WFI Respondent Scores Across Time
High Scores on Voice and Choice, Cultural
Competence, Individualized Services, Strengths,
Outcome-Based Services Low
Scores on Natural Supports, Community-Based
Services, and CFT Element
48
Challenges of the WOF

• Limited resources in staff to become neutral
  observers
• Managers wanted to use the WOF as a supervision
  tool
• Willingness and comfort level of staff to
  participate in direct observation
• Staff perception of fear of families being judged
  by observers
• Sponsorship by management
• Competing priorities in data collection

49
Next Steps

• Supervisors will continue to utilize the WOF to
  provide feedback
• Operational realities limit the use of neutral
  observers
• EMQ will no longer collect the WFI due to
  limitations in resources, the utility of the tool
  for feedback, and its relationship to outcomes
• Continue to develop practice standards

50
References

• Achenbach, T. M., Rescorla, L.A. (2001). Manual
  for the ASEBA school-age forms profiles.
  Burlington, VT University of Vermont, Research
  Center for Children, Youth, Families.
• Epstein, M.H., Nordess, P.D., Hertzog, M.
  (2002). Wraparound Observation Form-Second
  Version.
• Epstein, M.H., Nordess, P.D., et al. (2003).
  Assessing the Wraparound Process During Family
  Planning Meetings.
• Hodges, K. (2000). Child and Adolescent
  Functional Assessment Scale (CAFAS). CAFAS
  Self-Training Manual and Blank Scoring Forms
  (2003). Ann Arbor, MI
• Nordness, P.D., Epstein, M.H.(2003).
  Reliability of the Wraparound Observation Form
  Second Version.
• Suter, J., Force, M., Bruns, E., Leverentz-Brady,
  K., Burchard, J. (2002). Manual for training,
  administration, and scoring of the WFI 3.0.
  Burlington University of Vermont.

51
Contact Information

• Enrica Bertoldo
• ebertoldo_at_emq.org
• Quality Support Manager, Sacramento
• Mary Ann Wong
• mwong_at_emq.org
• Research Specialist, Sacramento

52

• Wraparound Supervisor Adherence Measure (W-SAM)
  A Pilot
• Eleanor Castillo, Ph.D.
• Veronica Padilla

53
Rationale

• Supervision identified as a critical aspect of
  Wraparound
• No tool to date to measure this aspect of
  Wraparound
• Concept adapted from Multisystemic Treatments
  (MST) Supervisor Adherence Measure (SAM)

54
Rationale

• WFI-3.0 measures adherence to Wraparound
  principles from the Youth, Caregiver, and
  Resource Facilitator perspective
• WOF-2.0 measures the Child and Family Team aspect
  of Wraparound

55
Description

• Facilitator rates the Clinical Program Manager
  (CPM)
• 40 Items
• Likert Scale (1 Never 5 Almost Always)
• Administered quarterly
• Highly reliable (alpha .84-.98)

56
Process Implementation

• Sponsorship from the Directors
• Program quality improvement versus mandate
• Immediate feedback

57
Quality Management

• Training
• Evaluation
• Overall Program Results
• Individual CPM
• Performance Appraisal
• 360
• Annual

58
Quality Management

• 3-Tiered Agency-wide, Program, and Individual
• Training and Coaching
• Evaluation
• Performance Appraisal
• Quarterly 360 feedback
• Annual
• Correlation between supervisor focus on the
  Wraparound process and youth outcomes

59
Quality Management

• Trend analyses
• Fidelity consistency over time
• What factors affect fidelity
• Learning curve with new manager or facilitators

60
Next Steps

• Continue pilot
• Conduct further analyses on the psychometric
  properties of the scale
• Factor analyses
• Items reduction

61
Contact Information Eleanor Castillo,
Ph.D. Director of Outcomes Quality
Assurance Email ecastillo_at_emq.org