2005 Assessment of Performance for Standards for Public Health

1
2005 Assessment of Performance for Standards for
Public Health

• Standards Committee October 19, 2005
• Marni Mason and Barbara Mauer,
• MCPP Healthcare Consulting

2
Purpose of Todays Review

• Approval of Overall System Report and
  Recommendations
• Get Committee input on
• Draft Proposed Administrative Standards Report
  approval on 12/8
• Draft Programs Report approval on 12/8

3
Joint Conference on Health

• Two concurrent sessions
• Exemplary Practices at 400, Monday, 10/10
• Regina Delahunt
• Nancy Goodloe
• Sherri Bartlett
• Assessment Results at 1030, Wednesday, 10/12

4
Important Articles

• Berwick Controlling Variation In Health Care
• Berwick Continuous Improvement as an Ideal in
  Health Care
• New York Times Its the Simple Things, but Some
  Hospitals Dont Do Them

5
Overall Systems Report - 2005 Assessment of
Performance
6
Executive Summary

• Major Themes
• Combined LHJ and DOH Topic Area charts
• Recommendation Statements
• System Improvements
• Revisions to Standards and Measures
• Ongoing, 3-year improvement measurement cycle

7
Report Narrative - Redline version

• Thank you for all your comments and revisions
• A redline version sent to all Committee members
  reflecting the substantive changes
• Comments? Questions?

8

• Recommendations for Action

9
Reports Recommendations

• Consultants have recommendations to Standards
  Committee regarding
• Improvements to standards and measures, as well
  as some Administrative Standards
• Potential areas for system-wide, DOH, LHJ
  improvements
• Recommendations on ongoing 3-year cycle to
  sustain assessment and improvement efforts

10
Recommendations for Improvement

• Establish and monitor program goals, objectives,
  and performance measures
• Identify specific goals, objectives, and
  performance measures for every LHJ and DOH
  program and establish regular monitoring,
  reporting and use of results
• Evaluate Program Effectiveness
• DOH and LHJs should conduct monitoring of
  performance measures, including self-audits
  compare to targets, identify needed improvements,
  take action

11
Recommendations for Improvement

• Standardize DOH Program Requirements
• Collect/evaluate formats and use those
  consistent with Standards, Develop model
  templates, adopt in all programs, integrate into
  regional structures (HIV)
• Standard State Databases Documentation
• Implement standardized systems for data
  collection and analysis, including EH
  surveillance system Develop and distribute data
  dictionary, and assess capacity to support
  evaluation of program performance measures

12
Recommendations for Improvement

• Standard Statewide Results Measures to Track
• Develop core set of statewide results
  measures, pilot in LHJs, develop DOH capacity to
  report to each LHJ, monitor and report
• Establish a Quality Improvement Plan
• Use results measures monitoring and program
  evaluations to develop and implement QI plan
• DOH Technical Assistance and Consultation
• Information to LHJs on services can provide
  and how to obtain the services

13
Recommendations for Improvement

• Model Templates, Policies and Procedures
• DOH and LHJ jointly develop needed policies for
  EH investigation, CD protocols, evaluation and
  self-audits
• Training
• Establish systematic training processes
  (including non-classroom technology) and tracking
  methods for Emergency Response Plan and role,
  Program evaluation methods, Confidentiality, Risk
  communication, EH enforcement, Community
  involvement, Health promotion, QI methods and
  tools,, multi-program contract requirements

14
Recommendations for Standards

• Eliminate two standards
• Access standard 4 QI training and QI plan, move
  measures to Administrative Standards
• Combine PP standards 4 and 5 and eliminate
  standard 5 prevention and health promotion
  services
• Move Assessment standard 5 to Administrative
  Standards
• Confidentiality of shared data and computer
  security

15
Recommendations for Measures

• Combine all training to HR in Administrative,
  except Emergency Response Plan and roles
• Program evaluation methods
• Confidentiality and HIPAA
• Risk communication
• EH enforcement procedures
• Community involvement (mobilization)
• Health promotion methods
• QI tools and methods

16
Recommendations for Measures

• Establish new measure to evaluate websites
• 24hour contacts for reporting
• Notifiable conditions
• Assessment information
• Educational materials for EH and PP programs
• Information on how to obtain consultation and TA
• Links to legislation, regulations, codes and
  ordinances

17
Recommendations for Measures

• Quality Improvement Plan measure in
  Administrative
• Reflects strategic plan, annual goals and
  community assessment data
• Results measures performance
• Outbreak response evaluation results
• After-action debrief results
• Program evaluations results

18
Recommendations for Establishing an Ongoing Cycle
for Performance Management and Measurement
19
Implementation Recommendations

• DOH LHJ Leadership Take Action
• Develop Implementation Work Plan for both DOH and
  LHJs to address the results adopt by year end
  2005
• Jointly identify system-wide improvements with
  shared state and local focus
• Communication Key Messages
• Develop and implement a communication plan for
  performance improvement and assessment

20
Revise and Adopt New Standards

• Revise current measures for mid-2006 release,
  including new Administrative Topic Area, with
  crosswalk from current to new
• Revised materials, self-assessment guides and
  DOH/LHJ Matrices
• Glossary of Terms
• Technical assistance, QI training, community
  involvement training, health promotion training

21
Stages of Organizational Performance
22
Adding Statewide Results Measures

• Identify process to identify core results
  measures to be included in performance scoring
  (See Results Measures recommendation)
• Establish DOH capacity to create an annual report
  of LHJ performance on results measures

23
Use of Internal Reviewers

• Establish formal system to select internal
  reviewers
• Job description and clear roles and expectations
• Evaluation/feedback process
• Use teams of internal reviewers for Standards
  training and site visits, with training and
  involvement of consultants

24
Orientation, Training, Mentoring

• Assure orientation to Standards is part of all
  DOH and LHJ orientations
• Provide regularly scheduled training in Standards
  content and interpretation
• Use web-based and DVD/video technology
• Basic Standards (101)
• Preparing for Site Visit training (102)
• Series by topic areas and PDCA cycle (200)
• Mentor Program
• Fall 2007 - Site visit preparation training, 6
  months before site visits (spring 2008)

25
Site Visit Schedule

• All materials and schedules done by mid-2007
• Takes 12 months to conduct training to prepare
  for site visits, conduct site visits, and
  generate and approve reports
• Fall 2007- winter 2008 Site visit preparation
  training, 3-5 months before site visits
• Spring - early summer 2008 Site visit, allow
  more reviewer days per site, and schedule only
  2-3 days of site visit per week
• Use teams of internal reviewers with consultants
  to conduct site visits.

26
Measurement Reporting Database

• Create database to collect and report Standards
  performance data, including regular performance
  and core indicator reports
• Provides for data-driven decisions and
  improvements
• Provides ad-hoc and inquiry capacity
• Specific, actionable data and reports

27
Proposed Administrative Standards Draft
Evaluation Report
28
Assessment sites - LHJs

• All 35 LHJs reviewed, 31 data points
• Whitman and Columbia reviewed as one site
• Skamania no admin review
• Two other LHJs Not Able to Rate

29
Assessment Sites- DOH

• 25 DOH programs reviewed
• 13 in CFH, CWP, and IDRH
• 5 in EH
• 3 in EHSPHL
• 2 in HSQA
• Health Promotion
• Office of Secretary
• State Board of Health
• DOH programs reviewed just for applicable
  standards using DOH matrix

30
Administrative Standards Report

• The report for the Field Test of the Proposed
  Administrative Standards is at the system level
  for aggregate LHJ, DOH and overall results
• The results for the Administrative topic area
  will not be reported at the site specific level
• Theses are minimum or required to keep the
  doors open, not stretch standards

31
Administrative Standards

• Fiscal Capacities (9 measures)
• Human Resource Capacities (6 measures)
• Information System Capacities
• (8 measures)
• Leadership and Governance Capacities (11
  measures-DOH, 12 measures-LHJ)

32
Summary of Findings-DOH/SBOH

• DOH higher performance- all 4 standards (66-84,
  gt 80 on 3 standards) than LHJs (45-65)
• DOH demonstrated performance in 26 of 35 measures
  - only 1 or 2 sites
• DOH - 3 measures with no program demonstrated
  performance
• AD 1.1State audits and internal controls
• AD 1.7Contract have requirements are monitored
  for deliverables and compliance
• AD 4.5QI plan is implemented

33
Summary of Findings-LHJ

• Highest performance in HR65 demonstrated (31
  sites)
• Lowest performance in Leadership/Governance with
  45 demonstrated (31 sites)
• LHJ had 80 or higher demonstrated performance on
  8 of 36 measures
• AD 4.5 QI plans had 3 demonstrated 1 site

34
LHJ Results Fiscal Capacities

• Fiscal 2/3 demonstrated, need improvement in
• Assuring budgets are aligned with agency
  strategic plans
• Budgets are monitored regularly
• Systematic monitoring of contractor deliverables

35
LHJ Results HR and IS Capacities

• Human Resources - all measures more than 50
  demonstrated
• Information Systems 2/3 more than 50
  demonstrated
• Documentation of security evidence of
  monitoring policies
• IT staff trained and available
• Strategies for future technologies

36
LHJ Results Leadership/Governance

• Leadership Governance ¼ had 50 or more
  demonstrate performance
• Written guidelines risk management
• Procedures for communication with BOH inc.
  orientation, operating rules, voting
• QI plans are implemented
• Strategic/operations plan with mission, goals
• Guidelines for media
• Customer service standards

37
DOH/SBOH Results - Fiscal

• Fiscal all 9 measures evaluated in 1 or 2
  sites, 7 measures demonstrated
• Need improvement in
• Audit reports and internal controls
• Systematic monitoring of contractor deliverables

38
DOH/SBOH Results HR, IS, LG

• Human Resources - all measures more than 50
  demonstrated
• Information Systems all measures more than 50
  demonstrated
• Leadership Governance all but 1 measure had
  50 or more demonstrated
• Quality improvement plans implemented

39
Recommendations - Improvement

• Fiscal Capacities
• Assure budgets are aligned with agency strategic
  plans
• Assure budgets are monitored regularly
• Systematic monitoring of contractor deliverables

40
Recommendations - Improvement

• Human Resources
• Conduct regular staff performance evaluations
• Conduct training in required topics
• Information Systems
• Documentation of security evidence of
  monitoring
• Assure IT staff trained and available
• Assure IS plans include strategies for future
  technologies

41
Recommendations - Improvement

• Leadership Governance
• Implement written guidelines for risk management
• Develop and implement procedures for
  communication with BOH inc. orientation and
  operating rules
• Implement QI plans
• Develop Strategic/operations plan
• Implement customer service standards

42
Recommendations - Standards

• Administrative Standards be revised and
  integrated into current standards as 6th topic
  area
• Determine the extent of Administrative measures
  for DOH level
• Revise measures to 18 (from current 36)

43
What questions do you have?
44
Draft Programs Report
45
Program Reports

• Programs Report for the 10 programs included in
  the EH and PP menus
• Aggregates all LHJ results for each program by
  measure
• DOH program results for each of the 10 programs
  for the same measures as evaluated in the LHJs

46
Programs - Number Selected
47
Summary Findings-EH Programs

• EH 1.1L Information is available about
  environmental health, including newsletters,
  websites, and other had 92 or higher
  demonstrated performance in all five programs at
  the local level.
• EH 1.2L The community and stakeholders are
  involved in appropriate ways in addressing
  environmental health issues and EH 4.2L There
  are written procedures to follow for enforcement
  actions. had more than 65 demonstrated
  performance in all five programs at the local
  level.

48
Summary Findings-EH Programs

• Second lowest performing measure in LHJs - AS
  3.3L Program performance measures are monitored,
  the data is analyzed and regular reports document
  progress toward goals. with 29-Drinking Water,
  27-Food Safety, 24-Waste Water, 0- Water
  Recreation, and 0-Zoonotics.
• Lowest percent demonstrated in LHJs - EH 4.3L A
  selected number of enforcement actions are
  evaluated each year to determine compliance with
  0-Drinking Water, 15-Food Safety, 4-Waste
  Water, 0-Water Recreation, and 0-Zoonotics at
  the local level.

49
Summary Findings-EH Programs

• Two DOH programs, Drinking Water and Zoonotics,
  demonstrated performance in all applicable
  measures related to the LHJ program review.
• Three DOH programs demonstrated performance in
  46 of applicable measures (Wastewater), 75 of
  applicable measures (Water Recreation) and 80 of
  applicable measures (Food Safety).

50
Drinking Water Program

• 7 LHJ sites-averaged 58 demonstrated
• Program with highest percent demonstrated
• For 2/3s of the measures (8 of 13 measures or
  62) at least 50 or more of the LHJs
  demonstrated performance
• Need improvement in
• Evaluate program effectiveness, educational
  programs, educational materials, and
  effectiveness of enforcement actions
• Monitor, analyze and report performance measures
  data

51
DOH Drinking Water Program

• DOH program demonstrated performance for all 13
  of the applicable measures related to LHJ program
  review

52
Food Safety Program

• 27 LHJs averaged 56 demonstrated
• For just over half of the measures (7 of 13
  measures or 54) 50 or more LHJs demonstrated
  performance
• Need improvement in
• Evaluate program effectiveness, educational
  programs, educational materials, and
  effectiveness of enforcement actions
• Monitor, analyze and report performance measures
  data

53
DOH Food Safety Program

• Demonstrated 8 of the 10 (75) applicable
  measures related to LHJ program review
• Two measures partially demonstrated indicates
  improvement needed in
• Assuring a planned, systematic process that
  describes how appropriate data are used to
  evaluate program effectiveness and that programs
  have written goals, objectives and performance
  measures
• Assuring that program performance measures are
  monitored, the data is analyzed and regular
  reports document progress toward goals

54
Wastewater Management

• 26 LHJ sites - averaged 53 demonstrated
• For almost half of the measures (6 of 1346) at
  least 50 or more of the LHJs demonstrated
  performance
• Need improvement in
• Evaluate program effectiveness, educational
  programs, educational materials, and
  effectiveness of enforcement actions
• Monitor, analyze and report performance measures
  data
• Identify key EH illness indicators, track and
  report

55
Wastewater Management

• Demonstrated 8 of the 13 (62) applicable
  measures
• Did not demonstrate performance for two measures
  EH 4.4S documented process for periodic review
  of enforcement actions and a selected number of
  enforcement actions are evaluated to determine
  compliance with and effectiveness of enforcement
  procedure. and EH 4.5S Enforcement actions are
  logged (tracked) and are reported to other
  agencies as required.

56
Wastewater Management

• Partially demonstrated 4 measures indicating
  improvement needed in
• Providing coordination to develop EH indicators
  and data standards
• Identifying key indicators of EH risks and
  illnesses and reporting and tracking of trends in
  suspected EH illnesses
• Gathering/disseminating EH information about best
  practices
• Assuring written enforcement procedures that
  specify the type of documentation needed

57
Water Recreational Safety

• 3 LHJ sites - averaged 48 demonstrated
• For almost half of the measures (6 of 13 46)
  at least 50 or more of the LHJs demonstrated
  performance
• Need improvement in
• Evaluate program effectiveness, educational
  programs, educational materials, and
  effectiveness of enforcement actions
• Monitor, analyze and report performance measures
  data

58
DOH Water Recreational Safety

• Demonstrated 9 of 12 (75) measures
• Three measures partially demonstrated indicate
  areas of needed improvement in
• Assuring program performance measures are
  monitored, analyzed and regular reports document
  progress
• Identifying key indicators of EH risks and
  illnesses and reporting and tracking of trends in
  suspected EH illnesses
• Assuring that enforcement actions are logged
  (tracked) . and are reported to other agencies
  as required.

59
Zoonotics Program

• 3 LHJ sites-averaged 41 demonstrated
• For just over one-third of the measures (5 of 13
  39) at least 50 or more of the LHJs
  demonstrated performance
• Need improvement in
• Evaluate program effectiveness, educational
  programs, educational materials, and
  effectiveness of enforcement actions
• Monitor, analyze and report performance measures
  data

60
DOH Zoonotics Program

• Demonstrated performance for all 10 applicable
  measures related to LHJ program review

61
Summary Findings-PP Programs

• PP 4.4 L Staff providing prevention, early
  intervention, and outreach services have
  appropriate skills and training had 75 or
  higher demonstrated performance in LHJs.
• PP 5.3 L Health promotion efforts have goals,
  objectives, and performance measures .number and
  type of activities are tracked, reported, There
  is an evaluation process for health promotion
  efforts that is used to improve programs or
  revise curricula. had lowest percent
  demonstrated performance with 20 in Child Care,
  15 in First Steps, 17 in Immunizations, 0 in
  Nutrition and Physical Activity, and 15 in
  Tuberculosis.

62
Summary Findings-PP Programs

• None of the five selected DOH PP programs fully
  demonstrated performance in all eight applicable
  measures related to LHJ program review.
• The range of percent demonstrated in the five DOH
  programs is 88 in Child Care to 29 in Nutrition
  and Physical Activity in the applicable measures
  related to LHJ program review.

63
Child Care Program

• 10 LHJ sites - averaged 41 demonstrated
• For one-quarter of the measures (2 of 825) at
  least 50 or more of the LHJs demonstrated
  performance
• Need improvement in
• Evaluate program effectiveness, educational
  materials
• Establish, monitor, analyze and report
  performance measures data
• Staff training on health promotion methods

64
Child Care Program

• Demonstrated 7 of 8 (88) measures
• AS 3.2S not demonstrated, appropriate data are
  used to evaluate program effectiveness. Programs
  have written goals, objectives and performance
  measures... due to no documentation said NA

65
First Steps Program

• 13 LHJ sites - averaged 37 demonstrated
• For one-quarter of the measures (2 of 8 measures
  or 25) at least 50 or more of the LHJs
  demonstrated performance
• Need improvement in
• Evaluate program effectiveness, educational
  materials
• Establish, monitor, analyze and report
  performance measures data
• Staff training on health promotion methods

66
DOH First Steps Program

• Demonstrated performance for 2 of the 8 (25)
  measures
• PP5.4S not demonstrated, Health promotion
  efforts have goals, objectives, and performance
  measures. The number and type of health promotion
  activities are tracked, and reported, including
  information on content, target audience, number
  of attendees. There is an evaluation process for
  health promotion efforts that is used to improve
  programs or revise curricula.

67
DOH First Steps Program

• 5 measures partially demonstrated indicate
  improvement is needed in
• Use data to evaluate program effectiveness, and
  programs have written goals, objectives and
  performance measures
• Performance measures are monitored, analyzed and
  regular reports document progress and used for
  program improvements
• Providing consultation and TA on implementation
  and evaluation

68
Immunization/CHILD Profile

• 24 LHJ sites - averaged 41 demonstrated
• For just over one- third of the measures (3 of 8
  measures or 38) at least 50 or more of the LHJs
  demonstrated performance
• Need improvement in
• Evaluate program effectiveness, educational
  materials
• Establish, monitor, analyze and report
  performance measures data
• Staff training on health promotion methods

69
Immunization/CHILD Profile

• Demonstrated performance for 6 of the 8 (75)
  measures
• 2 measures partially demonstrated indicate areas
  of needed improvement in
• Assuring prevention services performance measures
  are monitored, tracked, analyzed and used for
  program improvements
• Health promotion efforts have goals, objectives,
  and performance measures, including tracking and
  reporting health promotion activities

70
Nutrition Physical Activity

• 3 LHJ sites - averaged 50 demonstrated
• For almost two-thirds of the measures (5 of 8
  63) at least 50 or more of the LHJs
  demonstrated performance
• Need improvement in
• Evaluate program effectiveness, educational
  materials for improvement
• Establish, monitor, analyze and report
  performance measures data

71
DOH Nutrition Physical Activity

• Demonstrated 2 of the 7 (29) of applicable
  measures
• 5 measures partially demonstrated indicate areas
  of needed improvement in
• Providing consultation and TA on implementation
  and evaluation
• Reviewing interventions for compliance with
  standards and state requirements
• Assuring health promotion efforts have
  performance measures that are monitored, tracked,
  analyzed and used for program improvements
• Evaluation process is used to improve programs

72
Tuberculosis Program

• 13 LHJ sites - averaged 32 demonstrated
• Only one measure had 50 or more of the LHJs
  demonstrated performance PP 4.4
• Need improvement in
• Evaluate program effectiveness, educational
  materials for improvement
• Establish, monitor, analyze and report
  performance measures data
• Staff training on health promotion methods

73
DOH Tuberculosis Program

• Demonstrated 5 of the 6 (83) applicable measures
  
• 1 measure partially demonstrated indicates
  improvement is needed in
• Providing consultation and TA on program
  implementation and evaluation

74
Recommendations

• Identify group of DOH and LHJ staff from each
  program to discuss how to improve performance
  against the Standards at both the state and local
  levels, as needed.
• Build on system wide improvement activities where
  possible to minimize resources needed for
  improvement.
• Integrate appropriate program review methods in
  the next round of measurement.

75
What questions do you have?
76
Reports Next Steps

• Final Administrative Standards Report to
  Committee for approval 12/8
• Programs Report to Standards Committee for
  approval - 12/8
• Exemplary Practices Report to Committee 12/8
• Review of 2006 work plan to implement ongoing
  cycle of measurement

77
Strategies for Using Results

• Increase system capacity
• External - resources
• Internal - system efficiency
• Improve system performance
• External - accountability
• Internal - target improvements, exemplary
  practices
• Improve measurement system
• External - results measures
• Internal - fewer measures, electronic process

78
Communication Next Steps

• Tailor communications for different audiences and
  specific topic areas
• Findings can be communicated on web site, in fact
  sheets, presentations, etc.
• PHIP communicates messages and information
• Media press kits prepares information for release
  to media and the public