Recognize Progress: Measuring Outcomes Not Outputs

1
Recognize Progress Measuring Outcomes Not
Outputs

• Yvonne Attard, Director, Customer Development,
  Oakville Public Library
• Rebecca Jones, Partner, Dysart Jones Associates

2
Agenda

• Basis for discussion
• Measurement System On Overview
• Critical success factors
• Options
• Can you really measure impact?
• Measurement Framework Theory and Reality
  Oakville example
• Gaps and hurdles
• Libraries change lives

3
Basis for discussion

• There is no one magic measure
• There IS a strong correlation between an
  organizations success and
• Its clarity of purpose
• Its ability to understand its culture and its
  stakeholders
• A well-designed performance measurement system
  that fits that culture.
• Value is client or stakeholder defined
• Definition is dynamic
• It is two-dimensional economic psychological
• It is relative to alternatives

4
Basis for discussion

• Measuring for Results The Dimensions of Public
  Library Effectiveness by Joe Matthews, 2004
• Asserts that few public libraries have a culture
  of assessment
• Difficult and complex
• Most measures indicate past performance
• No cause-and-effect relationship between measures
• Performance measures measure quantitatively, but
  library outcomes are largely qualitative

5
Measurement System an overview
Impact
6
Measurement System - Input

• Inputs are basically resources or capabilities
• Financial resources
• Content
• Staff
• Technology
• Facilities
• Ultimately, who enables you to obtain these
  inputs?

7
Measurement System - Output

• The service or program created by using the
  Input.

8
Measurement System - Outcomes

• What the client is able to do with the service or
  product how the client uses the Output.

9
Measurement System - Impact

• The effect or influence of the service or
  program.
• What changed for the client and/or the client
  organization.
• How the library changed a life or a group of
  lives.

10
Measurement System - Example
11
Measurement System - Oakville
12
Systems View
Stakeholders
Inputs
Impact
Feedback
Library
Clients
Outputs
13
Critical Success Factors

• For defining, measure communicating progress
  impact?
• Alignment of stakeholder goals and your goals
• Balance of government demands vs. business plan
  vs. library demands
• Your definitions, measures communication style
  match those of your stakeholders, other service
  organizations, non-profits and others in the
  local community
• Doable capture and analysis of measures
• Measures communicated in a meaningful and
  understandable way for stakeholders
• Conversations outside the library

14
Some Options

• Balanced Scorecard
• LibQual
• Cost-benefit

15
Balanced Scorecard

• Aligns measures with strategies to track
  progress, reinforce accountability and prioritize
  improvement opportunities
• A system of measures based on 4 perspectives
• customer
• internal
• financial
• Innovation
• Limits measures to those most critical

16
Balanced Scorecard
Customer Perspective How do we look to our
clients? Goals Measures
Innovation Perspective How can we improve
create value? Goals Measures
Internal Perspective What must we excel
at? Goals Measures
Financial Perspective How do we look to our
funders or stakeholders? Goals Measures
Kaplan Norton
17
(No Transcript)
18
(No Transcript)
19
Balanced Scorecard Example

• METRIC U.1.a. OVERALL RATING IN STUDENT AND
  FACULTY SURVEYS.
• Target1 A score of at least 4.00 (out of 5.00)
  from each of the major constituencies
  undergraduate students, graduate students,
  humanities faculty, social science faculty,
  science faculty.
• Target2 A score of at least 3.90 from each of
  the major constituencies.
• Method The University Library conducts extensive
  surveys of our clientele approximately every
  other year. A final question in each survey asks
  the respondent to "rate your overall
  satisfaction" with the Library using a 1 to 5
  scale. This metric will consider the two latest
  surveys on record.

20
Benefits of Scorecard

• a clear understanding of what drives value
  within your area and what doesnt, greater
  insight into senior managements strategic plans,
  and a better knowledge not only of the strategic
  role you play within the organization but how you
  can enhance that role and sit at the
  decision-making table
• Joseph DeFeo, Measuring What Matters.
• Industrial Management, v.42,n.3, May 2000

21
LibQual(TM)

• Based on ServQual
• Collaboration of Association of Research
  Libraries (ARL) Texas AM University Libraries
• Diagnostic tool for measuring library users'
  perceptions of service quality
• Identifies gaps between desired, perceived,
  minimum service expectations
• Defined survey questions, dimensions, data
  gathering processes for academic libraries
• http//www.libqual.org/

22
Benefits of LibQual(TM)

• Provides better understanding of patrons
  perceptions desires, how services are meeting
  these
• Libraries using this have learned that they must
  keep narrowing categories questions

23
Cost-benefit

• Does the benefit exceed the cost?
• St. Louis Public Library
• Measures the value people place on the
  consumption of a service in excess of what they
  pay to receive that service
• Found benefits received gt10 for each dollar of
  tax support
• Other public libraries place a fair market value
  on their output measures
• San Diego Library and Miami-Date Public
  Libraries benefits exceed costs by 61

24
Benefits of cost-benefit

• Stakeholders may easily understand
• Difficult, however, to determine valid market
  prices

25
Understand the context
Align objectives
Define measures
Manage collection
Critical Success Factor Measurement Framework
Interpret data
Communicate results
Adapted from Measuring What Matters A
Library/LRC Outcomes Assessment Manual by Lindauer
26
Context for your framework

• Who are your key stakeholders?
• Who do you really need to prove your value to?
• How do they prove themselves?
• What measures/indicators do they use?
• Ask yourself, in our environment, what is it
  critical for us to we measure?

Page 1 of Framing Template
27
Framework Align

• This is where you invest the most time
• Requires conversations with councillors, or local
  government superiors
• Profile critical stakeholders
• What are their goals objectives?
• What makes them look good to their superiors or
  constituents?

Page 2 of Framing Template
28
Align Profile Stakeholders

• Stakeholders are those who can put a stake of
  support under your organization or a stake of
  destruction through your organization
• Superiors
• Funders
• Constituents or market
• Clients potential clients

29
Align Know Your Stakeholders

• Who are your stakeholders
• How do they determine value?
• What comprises value to them?
• How do they express it?
• What are the most important questions they have?
• What are the most important decisions they make?
• What are they communicating to their clients?
• Which of their goals objectives do you
  contribute towards?

30
Framework Define

• Clearly define this alignment by articulating
  documenting your goals objectives
• Clarify how these contribute towards your
  stakeholders desired outcomes
• Test these goals objectives with your
  stakeholders
• Test them with staff to ensure they are in
  keeping with your purpose will aid with
  planning decisions

Page 2 of Framing Template
31
Framework Identify

• How will you track your success/progress towards
  meeting these goals objectives?
• What indicators or measures will you use?
• Qualitative? Quantitative?
• What data needs to be collected? How?
• Keep it key (KISS principle)
• Dont get mired

Page 3 of Framing Template
32
Framework Collect

• Do it
• Determine a collection schedule
• Are you currently collecting data or indicators
  that are no longer relevant?
• How long do you need to keep data?
• Who is responsible?

Page 3 of Framing Template
33
Framework Analyze Interpret

• So..what?
• What does the data say? What doesnt it say?
• Examine it from various angles
• What is the progress towards the goals
  objectives?

Page 3 of Framing Template
34
Framework Communicate

• Actually begins back at the define stage
• If your goals objectives are meaningful for
  stakeholders, your measures will be too
• Your message to them
• If your goals objectives are meaningful for
  your planning decision-making, your measures
  will be indispensable
• Your message to you to staff we have to
  start, we have to stop we have to continue

Page 3 of Framing Template
35
Oakville Public Library

• Background
• Oakville Library Board operates on 3 year
  business planning model
• Arms-length relationship with Town of Oakville
  Community Services Commission, operates on a
  rolling 3 year Integrated Business Planning model
• Brand focused business plan goals and KPIs

36
Context

• Key stakeholders Municipality, board, donors,
  public, staff, other libraries
• Measures that are important
• efficiency measures (ROI) and
• effectiveness measures (usage, market penetration
  etc.)

37
(No Transcript)
38
IDENTIFY KPIs used at Oakville
39
COLLECT

• Tools used for collection include
• Citizens survey measures value and awareness
  every 3 years
• Cardholders per capita (Dynix/Horizon)
• Informal satisfaction surveys and feedback forms
  focus/advisory groups, web surveys, print
  surveys
• Collection satisfaction survey

40
ANALYZE AND INTERPRET

• Service transactions circulation people
  entering program attendance outreach
  information queries answered e-transactions
• Trends analysis
• Comparative analysis with other areas (libraries,
  town departments)
• Variance reporting

41
COMMUNICATION

• Look at frequency, depth and message and
  customize for specific stakeholder
• Yearly communication to municipality through
  budget process
• Yearly communication to Library board, with
  quarterly updates throughout the year
• Annual Report to the Community to donors and
  public
• Semi-annual updates to staff through semi-annual
  reports, intranet and staff newsletter

42
Gaps and Hurdles

• Consistency in measurement over the years
• Ability to measure
• Staff engagement and education
• Detail of measurement required for different
  stakeholders
• Meaningful communication
• Extrapolation errors its part of the equation
• Cause and effect direct or indirect?
• Advocacy plan
• Identify comparators across libraries, local
  community groups and other non profits etc.

43
Can you really measure impact?

• Are there times you cannot?

44
Can we measure impact?

• Sometimes what counts cant be counted, and what
  can be counted doesnt count.
• Albert Einstein
• We should be a bit wary of the little library
  For when it is good, it is very, very good and
  when it is bad, its a pretty good library for a
  town this size.
• - E. Rodger, New Zealand Libraries, March 1990

45
Measuring Impact - Libraries change lives

• Knowledge of alphabet letters at entry into
  kindergarten is a strong predictor of reading
  ability in 10th grade Every Child Ready to
  Read, West Bloomfield Township
• it helped be improve my ability to help
  students in their essay and civic topics all
  useful says a teacher-librarian of the Youth
  Online! program