A Program Performance Measurement System to Enhance Accountability

1
A Program Performance Measurement System to
Enhance Accountability

• A Presentation to CIRPA/ACPRI 2004
• Alan Vladicka
• Executive Director, Strategic Planning
• Grant MacEwan College

MacEwan Strategic Planning
2
Outline

• Background
• Guiding Principles
• Performance Measures Calculations
• Performance Reports
• Program, Dean College levels
• Impacts
• Further Development

MacEwan Strategic Planning
3
About Grant MacEwan College

• 10,600 FTE students (97 increase in last 10 yrs,
  43 in last 5)
• 81 programs (certificate, diploma, applied degree
  and university transfer)
• Programming in the arts sciences,
  business/commerce, communications, education
  phys. ed., engineering, health human services,
  performing visual arts
• Operates 4 campuses in Edmonton ( numerous
  outreach locations throughout Alberta)
• Budget crunch with end of Access funding (past
  levels of enrolment growth no longer sustainable)

MacEwan Strategic Planning
4
Project Background

• Established process of regular program evaluation
  and collection of performance data (grad.
  employment, transfer rates, satisfaction, etc.)
• Provincial KPIs (institutional level)
• Decentralized academic administration
• Little accountability at program level (programs
  could get away with years of low performance,
  poor evaluations)

MacEwan Strategic Planning
5
Project Background (cont.)

• Fiscal constraints increased the need for a
  rational, data-driven system of performance
  measurement to support decisions on program
  change
• Boards strategic direction to enhance access
  requires increased focus on program success
• Need to increase accountability for program
  performance

MacEwan Strategic Planning
6
Guiding Principles

• Use established measures that deans, chairs
  faculty understand and accept as valid
• Use only objective, quantitative data
• Use Provincial KPI measures if appropriate
  (College is assessed on these measures overall)
• Develop a model that balances key elements of
  program operation (inputs, processes outcomes)
  and reflects a range of perspectives (student,
  institutional, govt)
• Minimize the impact of single-year anomalies

MacEwan Strategic Planning
7
Guiding Principles (cont.)

• Avoid lengthy disputes on weighting of measures
  weight all performance measures equally
• Treat program costs as separate from program
  performance measures
• Provide program chairs and deans with clear,
  easily understandable ratings of performance on
  each key measure, and an overall rating
• Keep it simple! (maximum 5-6 measures)

MacEwan Strategic Planning
8
The Measures

• Student Demand (ratio of applicants to available
  spaces)
• Enrolment ( of FLE target achieved)
• Program Completion ( of FT cohort who complete
  within program length 2 yrs.) KPI
• Student Satisfaction ( fully satisfied with
  overall quality of educational experience) KPI

MacEwan Strategic Planning
9
The Measures (cont.)

• Graduate Employment Rate (in related positions)
  KPI
  or
  University Transfer Rate
• Program Cost/FLE student. KPI costing
  methodology

MacEwan Strategic Planning
10
Putting It Together

• 3-year averages are used for all measures to
  minimize impact of any anomalous blips
• Performance on each measure converted to a
  score out of 10 (/10, scales developed for
  demand ratio, enrolment, and cost)
• College means and standard deviations form the
  basis for a letter rating system
• A for 1 STD above mean
• B between mean 1 STD
• C between mean -1STD
• D for 1- STD below mean

MacEwan Strategic Planning
11
Putting It Together (cont.)

• Aggregate score is calculated for the 5
  performance measures, with cost treated as a
  separate dimension
• VPA and deans receive a report that ranks
  programs numerically, with performance measures
  receiving 2/3 of the weighting, and costs 1/3

MacEwan Strategic Planning
12
Program Reports

• Program chairs receive the following
• A data summary providing 5 years of data on each
  measure (and several others not used in the
  performance ratings), including comparison with
  college faculty/school averages
• A report card with numeric and letter grades
  on each measure
• A graph that plots the programs rating on 2
  axes aggregate performance and program cost.

MacEwan Strategic Planning
13
The Report Card
MacEwan Strategic Planning
14
A Visual Perspective
MacEwan Strategic Planning
15
Faculty/School Reports

• Deans receive the following
• The complete package of reports for all programs
  in their faculty/school
• A spreadsheet providing a comparative view of
  their programs performance on each measure and
  in aggregate
• A graph that plots all programs in the
  faculty/school on the aggregate performance and
  program cost axes

MacEwan Strategic Planning
16
Faculty Performance Chart
MacEwan Strategic Planning
17
Program Rankings

• Increasing program accountability and performance
  are the primary goals of this system
• To guide programming decisions in an era of
  fiscal constraint, academic administration also
  requested a top-to-bottom ranking of overall
  program performance
• Numeric scores based on a 2/3 weighting of
  aggregate performance score, and 1/3 weighting of
  program cost

MacEwan Strategic Planning
18
Program Performance Ranking
MacEwan Strategic Planning
19
So Whats Happened?

• All performance data was already available to
  chairs, deans and VPA
• The difference information is presented as a
  single package, and from a variety of
  perspectives (longitudinal, comparative, visual)
• The report card format and charting performance
  appear to have made the performance data more
  user-friendly

MacEwan Strategic Planning
20
Impacts

• Increased accountability awareness of and
  concern with program performance issues has
  increased
• The penny has dropped that decisions on program
  sustainability are being made on the basis of
  performance data
• Deans have more information available to them
  regarding the particular strengths and weaknesses
  of programs in their faculty/ school

MacEwan Strategic Planning
21
Impacts (cont.)

• Facilitated decisions on closure of continually
  weak programs, provided a clear rationale for
  these decisions for use with the Board, faculty
  and other stakeholders
• Provides guidance for chairs seeking to
  strengthen their programs (identifies problem
  areas comparators)
• Basis for decisions on programs to expand as well
  as to contract, review or discontinue

MacEwan Strategic Planning
22
Further Development

• Utilize financial contribution model vs. program
  costs alone (high cost programs may also be high
  revenue generators)
• Refine costing model to include costs of space
  utilized by individual programs
• Improve methods of footnoting special
  considerations (e.g.. programs that close
  applications early demand underestimated)

MacEwan Strategic Planning
23
Issues for Consideration

• Comparison with College averages is the basis of
  the model. Would comparison with similar
  programs at other institutions be more
  appropriate? Pros and cons?
• Influence of measures on program behaviours, e.g.
• Disincentive to grant advanced standing due to
  impact on FLE counts
• Accept applications even when full to boost
  demand stats
• Tighter screening of applicants to boost
  completion rates
• Are measures influencing behaviour in the
  direction(s) we desire?

MacEwan Strategic Planning
24
Questions?

• Contact
• Alan Vladicka
• Exec. Director, Strategic Planning
• Grant MacEwan College
• vladickaa_at_macewan.ca

MacEwan Strategic Planning