Ohio Outcomes: Helping Recovery and Resiliency Grow for Over Eleven Years

1
Ohio Outcomes Helping Recovery and Resiliency
Grow for Over Eleven Years

• James Healy
• Ohio Department of
• Mental Health (ODMH)
• healyj_at_mh.state.oh.us
• October 4, 2007
• Wellington, New Zealand

2
Learning Objectives

• How Ohio implemented Outcomes
• What efforts have worked well
• What has not worked as well
• What challenges now face the Outcomes system

3
Why Are We in Business?

• System should enable consumers and families to
  build resiliency and achieve their life goals
  unimpeded by mental illness or emotional
  disturbances (recovery)
• Measurement of consumer outcomes at treatment
  start, end, and periodically throughout, lets us
  know if these all-important objectives are met
  AND supports consumer-driven treatment planning

4
Founded 1803About OhioState Motto With God
All Things are Possible

• Splintered State level human services system
• 88 Counties, 50 Boards, 400 service providers
• Local (for some), State, Federal Funding
  Streams have many complicating requirements
• Local Control State--Supposed primacy of local
  control, Actual primacy of federal funding
  requirements
• Previously divisive nature ofboard-provider
  relations
• Best public Mental Health System in the United
  States? (Highest Grade of B from NAMI)

5
Then Outcomes Launch Historical Setting-1990?1996

• Tail-end of de-institutionalization efforts in
  state
• Birth pangs of recovery movement in Ohio
• Managed care efforts
• Lawsuits regarding authority of boards to control
  services

6
Then Driving Factors for Launching Ohio Outcomes
Initiative

• Need for measure of effectiveness
• Push for accountability at federal, state and
  local levels
• A multi-county (but not statewide) effort to
  establish standards would have splintered data
  collection efforts
• Void at federal level for standard mental health
  outcomes measures

7
Then and Now Potential Benefits of Outcomes

• Improved consumer input into treatment
• Improved clinical outcome of using Outcomes
  feedback
• Standardized evidence of performance
• Ongoing self-evaluation and improvement
• Cost-benefit analyses
• Addresses external requirements/demands
• Better long-term relationships with external
  stakeholders
• Method to differentiate organization from
  competitors

8
Now 11 Years and Counting

• The tree planted in September, 1996 By Mike Hogan
  begins to bear fruit. We never thought it would
  take this long!

9
Now Outcomes Success!

• A Number of Indicators
• Consumer Outcomes Rule (5122-28-04)
• Outcomes planning, collection, data flow, data
  use
• Widespread Implementation
• 340 providers
• All 50 boards
• 2,000,000 administrations
• 500,000 unique consumers (vs. Ohios 11,000,000
  citizens)
• Consumer Outcomes Data Mart
• Unique in the United States
• Adoption in Other Places
• Texas, Connecticut, Washington, California,
  Nebraska, Pennsylvania, Ireland, England, Canada,
  

10
Now Integral to System Activities

• Consumer Outcomes Used In
• Planning Analysis at Agency, Board and State
  levels
• Tied into new Service Standards (ACT adults,
  IHBT youth)
• Used in ODMH Certification Process
• Use in special ODMH funded programs, used for
  federal funding accountability requirements
• Provide agencies with acceptable solution for
  other funders (e.g. United Way) need for Consumer
  Outcomes data
• Setting a common data standard for service
  research data

11
We Can See the Summit from Here!
12
Outcomes Task Force 1996?1998

• Mission Define Consumer-Level Mental Health
  Outcomes Design Content
• Three criteria for consumer Outcomes
• Indicators of health or well-being for individual
  or family
• Measured by statements or observed
  characteristics of consumer/family
• Not characteristics of the service system
• Narrow focusnot drug abuse or satisfaction
• 42 members agencies, boards, consumers, ODMH,
  advocates, academics
• Strong leadership, expert facilitation

13
Outcomes Task Force Guiding Principles

• Multiple Constituencies
• Representative of all involved constituencies
• Routine reporting back to the field
• Deliberate Process
• In-depth review
• Documentation of meetings and decisions
• Consensus Model
• Work toward group consensus
• Formal decision process, if necessary
• These all worked, and were retainedfor every
  subsequent work group

14
Outcomes Task ForceValues

• Recovery philosophy drives service provision
• Providers and consumers share responsibility for
  environment of hope and for service planning
• Services driven by consumer-identified needs and
  preferences

15
Outcomes Task ForceValues

• Accurate information needed for continuous
  improvement of outcomes and for accountability
• Methodologically sound and cost effective
  outcomes measurement
• Balance between improved information and
  reasonable implementation

16
OTF Assumptions

• Integration with Other Data
• Outcomes data should be used with other data for
  continuous quality improvement
• Outcomes findings are indicators requiring
  further exploration and planning
• Outcomes are system-wide, data will be shared,
  i.e. data follows the consumer
• Availability
• All stakeholders should be able to use the
  outcomes findings

17
OTF Assumptions

• Consumer Perspective
• Outcomes should be measured primarily from
  consumer perspective
• Measures should complement the clinical judgment
  of practitioners
• Values-Based
• Incremental and innovative addition to Ohios
  mental health system improvement
• Should be evaluated to ensure that itfulfills
  the OTF values

18
Outcomes Clinical Mantra

• The single most important use of Outcomes data is
  to inform the clinical process!
• Utility of Outcomes data diminishes with distance
  from use in clinical process?Clinical
  Process?Program Evaluation/Quality
  Improvement?System Evaluation?State Evaluation

19
Outcomes Task Force What Worked, What Did Not

• What Worked
• Almost everything
• What Did Not
• Choosing pay-per-use instruments for kids
• Setting multiple instruments for adults

20
A Whole Mountain Range Was Ahead
21
Outcomes Implementation Pilot Coordinating Group
1999?2001

• Implementation and Evaluation
• Testing instrument reliability, validity
  acceptability
• Creating first tools for clinical use (Strengths
  Red Flags reports)
• Developing proof-of-concept data collection and
  reporting system for agency use
• Revised youth instruments (public domain)
• Procedural Manual, other documentation
• Key learning was that agencies who understood
  recovery accepted outcomes, and vs. versa

22
OIPCG What Worked, What Did Not

• What worked
• Proof-of-concept agency database still used by
  gt50 of agencies
• Replacing youth instruments with public domain
• The procedural manual, the website
• What Did Not
• Not collecting information that would collect
  program information with Outcomes data
• Not publishing collected RV work
• Did it work or not??
• Not defining use of Outcomes in clinical practice
• Multitude of information system
• Reliance on local board as intermediary

23
Outcomes Incentive Grants 2000-2001

• Implementation
• US3,000,000 in federal Center for Mental Health
  Services (CMHS) Block Grant funds were awarded to
  forty-four local systems to support Outcomes
  implementation in 192 agencies (US15,625 per
  participating agency)
• Training
• US420,000 in additional CMHS funds were used to
  develop a toolkit of training materials

24
Outcomes Incentive GrantsWhat Worked, What Did
Not

• What Worked
• Giving money to the agencies and board addressed
  real budgetary needs
• Grant followed by dial-in User Groups meetings
  that answered a lot of how-to questions
• What Did Not
• The majority of the technology purchased!? Little
  expertise at agencies in software acquisition,
  little control possible from ODMH
• The training materials were expensive with little
  impact? shorter videos manuals wouldhave been
  had more acceptance

25
The ODMH Quality Agenda 2001

• Outcomes are a key piece of the ODMH Quality
  Agenda, and are most useful when used in context
  of other pieces

You Are Here
26
Statewide Data Reports Workgroup 2002

• Outcomes Data Use and Reporting
• Developed standard reports to track Outcomes data
  reporting levels at agencies, boards and state
• Developed separate youth and adult diagnostic
  categories for reporting
• Developed standard report format
• Conceived the Outcomes Data Mart

27
Statewide Data Reports Workgroup What Worked,
What Did Not

• What Worked
• The missing data report stimulated collection and
  reporting
• What Did Not
• Only producing compliance-oriented agency-level
  missing reports? left agency boards to say
  thats all that mattered

28
New Administrative Rules in Place(enacted
September 2003)

• Outcomes Rule Requirements
• March 4, 2004 Collecting Outcomes data
• September 4, 2004 Flowing Outcomes production
  data to ODMH
• September 4, 2005 Evidence of use of Outcomes
  data in both treatment planning and agency
  performance improvement
• Also, Performance Improvement Rule Requires Use
  of Outcomes Data

29
Administrative Rule What Worked, What Didnt

• What Worked
• Requirement brought along the unwilling
• Larger scheme to trade government regulation for
  accreditation
• Required uses of Outcomes Data
• What Did Not
• Requirement made a good thing smell bad
• Narrow focus on outcomes use in treatment
  planning, rather than treatment, missed
  opportunity to require use later in treatment

30
Outcomes Data Mart Committee 2002?2004

• Available to Non-Technical Users
• Mental health consumers and family members
• Community mental health boards and agencies
• ODMH
• General public
• The actual construction of the Outcomes Data
  Mart was conducted over the next 15 months

31
Outcomes Data MartWhat Worked, What Did Not

• What Worked
• Easy to use
• Widely available
• What Did Not
• Not longitudinal
• Not a true data cube (no cross-tabulation)
• No Program analysis means no evidence for best
  practices

32
Outcomes System Quality Improvement Group
2005?2006

• Formalize ODMH Commitment to Review the Outcomes
  System Implementation
• Examine local experiences in areas of
• Leadership, training, automation, forms
  administration, data use, and constituency
  relationships
• Focus on improving Outcomes data use in
• Individual treatment planning, agency program
  evaluation and quality improvement, Board and
  ODMH oversight
• Advise ODMH on how to improve Outcomes
• Implementation, Operations, Administration

33
OSQIG Major Findings

• 14 Procedural changes
• Instrument-related issues deferred
• OSQIG found that many contextual (Bigger than
  Outcomes) issues affect what happens with
  Consumer Outcomes
• Technology
• Workforce
• Funding
• Recovery and Resiliency
• Corporate Culture

34
OSQIGWhat Worked, What Didnt

• What Worked
• Long term collection of issues identified by
  system constituents
• Simplification clarification of procedures in
  place
• Eliminating multiple adult instruments
  (Empowerment for all!)
• Agency toolkit to guide changes
• What Did Not
• No historical perspective yet to see what went
  wrong

35
Outcomes System Quality Improvement
Group-Instrument 2007???

• OSQIG-I focused on improving the Outcomes
  instruments to better support the integrated use
  of Outcomes data in treatment planning, agency
  program evaluation and quality improvement, and
  board and ODMH oversight.
• Probably 2 years to finish, then 2 years 15
  million dollars to implement changes
• Over 250 Identified Issues to sort through

36
Are There any Mountains Left to Climb? Any Seeds
Yet to Plant?
37
Technology Barriers Inherent in Outcomes

• Poorly performing initial implementations were
  widely adopted
• Complexity of Outcomes data relative to other
  clinical data
• A variety of people interact with Outcomes data
  systems
• Variability of Outcomes data use requirements

38
Non-Outcomes Technology Barriers

• Agencies do not perceive themselves as software
  managers
• information technology is integral to the
  business Thus, business models and software need
  to be co-developed
• Lack of awareness of steps in information
  system/software development lifecycle inhibits
  useful participation
• A typical software development lifecycle might
  look like Planning? Requirement Definition?
  Functional Design? System Design? Programming
  ?Testing? Implementation? Maintenance

39
Non-Outcomes Technology Barriers

• Agencies often do not have the economy of scale
  to participate in meaningful software development
  
• Agency culture may drive good technology practice
  out dont blame Information Technology!

40
Environmental Technology Factors

• Existing systems may be isolated in silos, making
  integration of data very difficult
• Existing systems may be monolithic and not
  readily changed
• Vendors may not have the capacity to deliver on
  promises
• Rapid changes in technology make adoption more
  problematic
• Rapid changes in requirements make development
  more difficult

41
Corporate Cultural Barriers

• Agency Leadership is key to change lacking clear
  direction from the top will inhibit needed
  changes
• Learning culture where mistakes are part of
  learning (instead of leading to termination)
  allows for uptake of new requirements

42
Cultural Barriers

• Multiple language versions of instruments
• Empowerment not universally recognized as a valid
  Outcome for all
• Multiple factors of culture Religion, race,
  gender, lifestyle, ethnicity
• Cultural evolution
• Given all this, what do Outcomes mean?? Need for
  individualized treatment

43
Work Force/Training Barriers

• High turnover in staff force constant training
• Often staff have no training on outcomes
• Training is often antithetical to using Outcomes
• Low pay structure makes MH a training ground for
  other human service fields

44
Funding Barriers

• How do we pay for the funding of Outcomes when
  its not built into the cost of doing business and
  we cant pay for what services actually cost
  because of federal funding requirements?
• Generically, how do you fund Outcomes work?

45
Even for those near the top, significant
challenges lie ahead
Significant Challenges
46
Key Top-Performer Agencies (N26) Survey Results
47
Contact Us!
http//www.mh.state.oh.us/oper/outcomes/outcomes.i
ndex.html
Any Question can be sent to Outcome_at_mh.state.oh.u
s OR Contact Policy, Outcomes History Dee
Roth rothd_at_mh.state.oh.us 614-466-8651 Policy,
General, Certification, Reports Generator Jim
Healy healyj_at_mh.state.oh.us 614-752-9311
Technical, Template, Testing Geoff
Grove groveg_at_mh.state.oh.us 614-644-7840 Certif
ication, Plans of Correction, Q/I Marsha
Zabecki zabeckim_at_mh.state.oh.us 614-466-9933 Poli
cy, Outcomes History Leslie Brower browerl_at_mh.st
ate.oh.us 614-752-9704