davidr:

1
davidr
C-8 Moving Toward Excellence Integrating
Performance and Outcome Measurements
A Data-driven Organizational Quality and
Performance System
Jill Pfitzenmayer, Ph.D. Child Family Services,
Newport, RI David Robinson, Ed.D. Center for
Evaluation and Research with Children and
Adolescents, Massachusetts Society for the
Prevention of Cruelty to Children, Boston, MA
2
Presentation Objectives
1.   Participants will learn how to identify
program goals, objectives and specific program
activities toward the accomplishment of program
mission, vision and values. 2.   Participants
will obtain information about establishing an
organizational structure that supports critical
analysis of program performance and quality
improvement. 3.   Participants will identify
specific outcome indicators, measurement tools
and data collection methods relevant to their
program plans.
3
Presentation Objectives
4.   Participants will learn how to link staff
training, employee retention, program performance
and outcome measurement efforts. 5.   Participant
s will be able to understand when internal
outcome assessment can be enhanced by the
introduction of an outside evaluator as
consultant or primary investigator.
4
Agenda

• Conceptual framework
• Description of the identified needs and problems
• Discussion of the methodology and approaches
  used to address the systems issues,
• Review of specific agency initiatives and
  reporting structures developed to enhance
  organizational quality, and
• Discussion by an outside evaluator familiar with
  the agency who was brought in for consultation

5
Definitions

• Performance Measurement Ongoing monitoring and
  reporting of program accomplishments,
  particularly progress towards preestablished
  goals.
• type or level of program activities (process)
• direct products and services delivered (outputs)
• results of products and services (outcomes)
• Program Evaluation Individual systematic
  studies conducted periodically or on an ad hoc
  basis to assess how well a program is working.
• internal or external experts and program
  managers
• learn benefits of program or how to improve it

GAO, April 1998
6
Integrating Performance Measurement and Program
Evaluation A Conceptual Framework

• Linked to institutional mission, goals,
  objectives, strategic plan, process and outcome
  evaluation
• SMART - Simple, Minimum paper, Actively used,
  Revised, and Transferable to new programs
• Best Practices continuously updated
  incorporating new findings, technology and
  resources
• Comprehensive - Quantity, Quality, Effort and
  Effect

7
Program Evaluation
8
Performance Measurement System
1. Gilbert Performance Engineering Approach

• Develop a MODEL of process, department, program
• Define INDICATORS of valued results and
  expectations
• Take MEASURES to see how well process is working
• Implement METHODS of improvement and evaluate
  results

9
Performance Measurement System
2. Friedman Four Quadrant Approach
QUANTITY
QUALITY
INPUT
OUTPUT
10
Effective Organizational Performance Measurement
Systems

• Commitment Leadership team committed to
  measurement
• Clarity Degree of clear strategies and metrics
• Metrics Indicators measure valuable processes
  and expectations
• Alignment Performance measures aligned with
  key people processes and structures
• Involvement Key stakeholders involved in
  defining, tracking, and ongoing review of quality
  performance improvement system

11
Information about CFS

• Multi-service organization over 136 years old.
  CEO been in office for 22 years powerful Board
  of Directors
• Primary services
• Nine child adolescent residential sites
• Two child care centers
• Family counseling program
• Home-based program
• School-based programs
• Transitional housing program

12
Implementation of Continuous Quality Improvement
(CQI) Program

• Minimal COA requirements
• Chart review
• Customer satisfaction
• Utilization review in select areas
• Individual outcome data
• Indication of systematic review of agency

13
Implementation Model Challenges
Problem 1 disconnect between accrediting body
and agency leadership perceptions of
need Problem 2 global discomfort with/
suspicion of science and data Problem 3
Continuous Quality Improvement is supposed to
involve all levels of staff, not be a top-down
process
14
Steps Required to Develop an Outcome System Year
1

• 1. Assemble committee representing key
  constituents of the agency
• Agency leadership
• Information systems
• Human Resources
• Business Office
• Program staff
• Parent
• Consumer

15
Steps Required to Develop an Outcome System Year
1

• Develop a conceptual model to help staff review
  program activities and outcomes (e.g., logic
  models)
• Map out organizational structure and evaluation
  activities (e.g., department teams)
• Identify strategy that makes sense to staff to
  collect, analyze and use information

16
Steps Required to Develop an Outcome System Year
1

• Develop protocol and process for all levels of
  staff participation
• Develop tools for reporting program activities at
  least quarterly and identify survey instruments
• Train staff on recommended model and process

17
Identify a Solution-Focused Process

• PLAN
• Decide what is the problem
• Brainstorm solutions

• DO
• Enact solutions
• Collect data on how solutions are working

• ACT
• Revise action plan if needed
• Discontinue plan
• Continue to monitor

• CHECK
• Analyze data
• Discuss findings

18
CQI Activities Related to COA Standards
CFS develops long-term strategic plan (4
yrs) (G2.3)
Identified stakeholders participate in all
appropriate activities (G2.2)
Each dept. develops annual plan of goals and
objectives (G2.4)
Internal external audits reviews (G2.10,
G2.5)
Chart Reviews (G2.6, G2.1)
Customer satisfaction surveys (G2.8, G2.10)
Outcomes Measures (G2.7, G2.10)
Quarterly review of each dept.achievements by
agency CQI Team (G2.9)
Reassessment, as necessary (G2.11)
Annual reports of year-end progress(G2.9)
19
Flow of CQI Activities

• Annual plan
• Quarterly Monitoring summary
• Summaries of client/referral satisfaction forms
• Summaries of incidents that occurred over the
  quarter
• Summary of external review activities
• Review department-specific and aggregated
  findings

20
Year 2 Challenges to CQI System

• Many staff did not understand logic models
• Program Directors didnt understand basic
  concepts (e.g., difference between benchmark and
  baseline data)
• Challenges existed around grasping conceptual
  material and managing task of putting information
  on computer!

21
Year 2 Challenges to CQI System

• Some programs never tracked outcome data at all
• State-funded programs required their own
  outcome tools that were either cumbersome or
  lacked validity
• Staff lacked buy-in, experienced turnover or
  couldnt make use of findings
• Several programs had a hard time identifying
  individual outcome indicators

22
Year 2 Challenges to an Outcome System

• Mechanically tracking data and outcomes became
  a challengeno standardized data base in the
  agency
• Changes implemented as a result of outcome data
  were often not documented
• Staff were busy but couldnt document real
  client change (i.e., conflating outputs and
  outcomes)

23
Year 2 Small Successes for an Outcome System

• Some programs used the process to study and
  implement real change
• Residential Network system for getting paperwork
  accomplished
• Family Counseling saw change in patient
  population via use of SCL-90-R
• School-based outcomes supported pilot program
  around prevention education

24
Creating Organizational Change

• Year 2 Agency/Board leadership in strategic
  planning meetings, which required a SWOT
  (Strengths, Weaknesses, Opportunities, Threats)
  analysis of programs
• Lack of careful investigation of program outcomes
  forced leadership to rely more on fiscal and
  output information than on outcome data
• Frustration Innovation

25
Creating Organizational Change

• Continuous Quality Improvement
• Reactive
• Minimal Board involvement
• Driven by COA requirements
• Voluntary management participation

• Organizational Quality Performance
• Proactive
• Board subcommittee
• Driven by strategic planning process
• Sr VP assigned to job

26
Coordination of OQP Activities
Agency Strategic Plan
Organizational Quality Performance Assess
performance and outcomes to align agency goals
and program activities
Learning Institute Link staff needs with training
and learning activities
Employer of Choice Initiatives Maintain focus on
performance to attract and retain highest
quality staff
27
Moving Toward Excellence Changes in Culture and
Systems

• Staff are now required to identify steps in
  developing programs reflecting excellence and
  best practices
• Management are held accountable for process and
  program outcomes
• Systematic data collection and analysis requires
  new MIS system
• Agency-wide committees and initiatives brought
  into oversight process

28
Moving Toward Excellence Changes in Culture and
Systems

• New OQP committee members include Corporate
  Communications and Development in order to ensure
  inter-departmental coordination
• Methodology for tracking process and outcomes in
  place
• OQP SVP meets provides technical assistance in
  program plan development

29
Moving Toward Excellence Goals of OQP Process

• Department-specific process and outcome data will
  be collected, analyzed and acted upon
• Inter-departmental coordination will improve
• Corporate Com, MIS Development included in
  program planning
• Programs work together on mutual or complimentary
  projects
• Inter-program conflicts will be quickly
  identified and resolved

30
Moving Toward Excellence Goals of OQP Process

• Communication across departments and up, down
  chain of command will improve
• Agency leadership will have improved and speedier
  methods of determining where there are problems
  and strengths in programs
• Agency and program leaders have tools to
  continuously plan for the future

31
CFS Program Performance Plan
32
CFS Program Outcome Plan
33
CFS Quarterly Program Performance Report
34
CFS Quarterly Program Outcome Report
35
Moving Toward Excellence How Do Program Staff
Use OQP System?

• Program Directors use process to bring their
  vision into program planning
• Staff are oriented to OQP performance and outcome
  plans upon hire as a learning tool
• Program leaders use plans to maintain an eye on
  program improvement activities

36
Moving Toward Excellence Lessons Learned

• Appreciate what you bring to the process from
  your own discipline and theoretical orientation
  and how they inform your outcome assessment work
• Ask for help. Use volunteers, experts, etc.
• Get organized and dont panic
• Dont underestimate the value of communication
• Maintain a stance of being inquisitive, not
  judgmental
• Remind yourself and your staff that the process
  is developmental
• Aim for direction, not perfection

37
Genuine Unsolicited Testimony from Staff
I wanted to share with you my excitement of
having completed the baseline ECERS and ITERS for
child care today!With this data we can now
create our baseline and use it to establish our
movement forward to improved overall quality in
our Centers for the children, families and our
staff! This is so exciting. Good things are in
our future, along with much hard workI am
confident that I will lead the way to our
success! Patty Shelley, Child Care Program
Director
38
ASSESSMENT Case Study of CQI Performance
Measurement System

• Commitment Leadership team committed to
  measurement
• Clarity Degree of clear strategies and metrics
• Metrics Indicators measure valuable processes
  and expectations
• Alignment Performance measures aligned with
  key people processes and structures
• Involvement Key stakeholders involved in
  defining, tracking, and ongoing review of quality
  performance improvement system

Rating Scale 1 A Little 2 Just Enough 3
Success
SCORE
39
QUESTIONS?
ADVICE?
FEEDBACK?
COMMENTS?
40
CONTACT INFORMATION
Primary Presenter Jill Pfitzenmayer,
Ph.D. Title Senior Vice President,
Organizational Quality and Performance Agency
Child Family Services Address 24 School
Street, Newport, RI 02840 Phone
401-848-4186 Fax [phone] Email
jpfitzenmayer_at_cfsnewport.org   Co-Presenter
David Robinson, Ed.D. Title Director for the
Center for Evaluation and Research with Children
and Adolescents (CERCA) Agency Massachusetts
Society for the Prevention of Cruelty to
Children Address 399 Boylston Street, Boston,
MA 02116 Phone [phone] Fax
617-587-1582 Email drobinson_at_mspcc.org or
CERCA_at_mspcc.org Web www.mspcc.org