Measuring Your Impact:

1

• Measuring Your Impact
• Using Evaluation for Library Advocacy

Mid-Atlantic Chapter of the Medical Library
Association 2007 Annual Meeting October 12,
2007 Baltimore, MD
2
Instructors

• Susan Barnes
• Assistant Director, NN/LM OERC
• sjbarnes_at_u.washington.edu
• Maryanne Blake
• Evaluation Coordinator, NN/LM PNR
• blakema_at_u.washington.edu

3
Todays plan

• Agenda
• Resources/Packet
• Exercises
• Housekeeping

?
4
Workshop objectives

• By the end of the workshop participants will
• Understand the librarys value in terms of the
  mission of the larger organization
• Describe some tools used to assess the library,
  its users and stakeholders

5
Workshop objectives

• Identify the tools and methods used for data
  collection and analysis
• Design an evaluation plan for a service in a
  library
• Use evaluation results to communicate library
  value

6
What this workshop is not

• Custom-tailored
• A quick fix
• One size fits all

7
How what to evaluate

• What you decide to evaluate depends on
• What you need or want to know
• What your users feel is important
• What certain stakeholders want to have evaluated
• Use smallball evaluation

?
8
Demonstrating the Librarys Value
9
Why demonstrate value?

• To show the impact of the library on the larger
  organizations mission and goals
• To show accountability for your resources the
  librarys contribution to the bottom line
• To demonstrate that the library provides
  value-added services to solve staff problems
• To use as an advocacy and marketing tool

10
Yesterdays Librarian
That was one of the delightful things about her
job. It brought her in touch with so many people,
allor almost allof whom made her not only feel
welcome but that she was doing something really
worthwhile. She gave a little sigh of pleasure
as she unlocked the library door. She loved
working here and she loved the work
itself. From Jan Marlowe, Hospital Librarian.
Margaret Malcolm. Toronto New York Harlequin,
1976, (c)1960. Harlequin's collection ed.
11
Think like a capitalist

• If we had attended business school, we would
  have learned the importance of selling our own
  image, as well as our product.  

Kalan A. Are we sabotaging ourselves with our
professional image? Amer Lib 200233(5)42.
12
Means, not ends

• Libraries are not ends in themselves, and they
  should not be supported because they have
  intrinsic value.

Plutchak TS. Means, not ends. J Med Libr Assoc
2004 Jul92(3)294.
13
And remember

• There are no sacred departments ... Any
  function can be contracted to other professional
  agencies that want the business

14
So

• What would we do differently if we had to compete
  with outside agencies to keep the internal client
  we have come to take for granted?

Lee F. If Disney Ran Your Hospital 9 ½ Things
You Would Do Differently. Bozeman MT Second
River Healthcare Press, 2004.
15
The bottom line

• You are either generating revenue, or supporting
  those who do
• OR
• You are helping to control operating expenses or
  supporting those who do
• OR
• You are creating expenses that add recognized
  value
• OR
• You are creating expenses that must be controlled
  or eliminated to reduce overhead

Will Welton, DrPH, Program Director, Master of
Health Administration, University of Washington
16
The big question!

• What is the library doing to support the bottom
  line?

17
Step Oneget out of the library!

• Talk to people
• Find out what information problems they are
  trying to solve
• Show them how the library can help
• Remember, solving problems is a value-added
  service

18
Library service is value-added

• Medical Staff Support
• Evidence-based clinical decision-making
• Marketing
• Resource for patient/family education
• Community support
• Legal/Risk Management
• Human Resources Support
• Staff education and development
• Planning Program Development
• Competitive intelligence

19
Library service is value-added

• Educational Support
• Resources for students and faculty
• Liaisons to academic programs
• Evidence-based clinical decision-making
• Accreditation
• Outreach
• Patients/families
• Community at large
• Research Support
• Services for research faculty

20
A Plan to Show Your Librarys Value
21
Metaphor building a house

• Assess
• The vision
• The environment
• Create
• The blueprint
• Assess and assemble
• The tools

22
And then

• Build it
• Have a housewarming

23
The Vision
24
Know what you want to build and why

• Align the librarys mission goals with the
  organizations mission goals
• The librarys value librarys contribution to
  achieving the organizations mission goals

?
Vision
25
Mission Concept Organizational Goal

• Provide excellent clinical care
• Provide services, resources needed for teaching
  and learning
• Reduce corporate risk
• Increase profitability
• Improve the lives of patients and their families

• Clinical Care
• Education
• Management of operations
• Service

Abels EG, Cogdill KW, Zach L. Identifying and
communicating the contributions of library and
information services in hospitals and academic
health sciences centers J Med Libr Assoc. 2004
Jan92(1)46-55.
?
Vision
26
Time for energy!
27
The Environment
28
Why look at the environment?

• Understand needs, desires and problems in
  context
• Validate assumptions about your contributions and
  services
• Provide a baseline for future evaluation
• Help to develop the blueprint to plan and
  evaluate your contributions and services

Environment
29
Components of environment

• The organization
• Your library
• Clients/users
• Stakeholders
• The community

Environment
30
Assessing the environment

• User/stakeholder input
• Observation
• Library statistics and records
• SWOT analysis

?
Environment
31
SWOT analysis

• Strengths
• Internal
• Positive statements about your library

• Weaknesses
• Internal
• Statements about what is lacking in your library

• Opportunities
• External
• What do clients want you to do that you are not
  doing?

• Threats
• External
• Factors that can adversely impact your librarys
  goals

?
Environment
32
Creating a blueprint
33
First, analyze

• What you need (Goal)
• Comfort?
• Convenience?
• What meets your need (Outcome)
• A renovation of existing space?
• An addition?
• A new house?

Blueprint
34
Then, decide

• What actions must be taken? (Activities)
• How much of what must be produced? (Outputs)
• What resources will be required? (Resources)

Blueprint
35
Plan Backward, Implement Forward
36
Where to go how to get there

• Goals
• Outcomes
• Activities and outputs
• Resources

Blueprint
37
Goals

• Statement(s) of Purpose why we do what we do
  and for whom
• Examples
• Patients and their families will have improved
  health information literacy
• Acquisition of library materials will be cost
  effective
• Nurses will have access to health information for
  clinical decision making

Blueprint
38
Goals based on

• The librarys contribution to the organizations
  mission and goals
• Environment
• SWOT analysis threats and weaknesses as well as
  the strengths and opportunities
• Other assessments

39
Outcomes

• Outcomes are changes in behavior, skills,
  knowledge and/or attitude.
• Examples
• Short-term outcome 50 more nurses know there is
  a library in the hospital 3 months into the
  project.
• Intermediate outcome Web pages with clinical
  resources for nurses are available on the library
  website 1 year into the project
• Long-term outcome 40 more nurses access
  clinical health information using the library
  website by the end of the project than did at the
  start

40
Outcomes

• Goal-related outcomes should be SMART
• Specific
• Measurable
• Action-oriented
• Realistic
• Timed
• Short, medium and/or long term
• Intended or unintended
• Positive as well as negative

Blueprint
41
Exercise
Goal The library will acquire and manage
information resources in a cost effective
manner. Outcome What outcomes do you expect?
Blueprint
42
Activities and outputs

• Activities
• What will you do?
• Outputs
• How many did you do?
• How many attended?
• How many were distributed?
• How many times was it used?

Blueprint
43
Resources

• What you have
• Income
• Equipment
• Collection
• Etc.
• What you need
• Operating expenses (e.g., personnel,
  acquisitions, maintenance, etc.)
• Funds for new initiatives or services
• Etc.

Blueprint
44
A POP QUIZ!
45
POP QUIZ What is a logic model?

• A beautiful person who thinks deep thoughts

46
POP QUIZ What is a logic model?

• A set of plastic pieces to assemble step by step

47
POP QUIZ What is a logic model?

• A planning tool that links activities to results

48
The logic model

• Helps organize your thoughts both before and
  during a project/program
• Provides a framework for planning and evaluating
  programs
• Clarifies intended outcomes
• Acts as a communication tool
• Good source for more information
• W.K. Kellogg Foundation Logic Model Development
  Guide (see class bibliography for link)

Blueprint
49
Parts of the logic model

• Resources
• What will you need to carry out your activities?
• Activities
• What will you do?
• Outputs
• How many of what will your activities produce?
• Outcomes
• The so what the difference your program
  makes, the benefits that accrue because of your
  program

Blueprint
50
Sample logic model

• Goal
• Acquire and manage information resources in a
  cost-effective manner

51
Logic model towards this goal
52
Logic model worksheet
Goal_____________________________________________
____
53
Logic model group activity

• Choose a goal
• Identify outcome(s)
• List
• activities
• outputs
• resources

Blueprint
54
Sample goals

• Patients and their families will have improved
  health information literacy
• Staff will have information for timely clinical
  decision-making
• Acquisition of library materials will be cost
  effective

55
The model will change
Blueprint
56
Its lunch time.
57
The Evaluation Plan
58
The evaluation plan

• Builds on the logic model
• Evaluates success
• Assesses value

Blueprint
59
Consider the purpose

• Who is your audience?
• Your users
• Administrators
• Fund raisers
• How will the information be used?
• Financial savings or justification
• Intangible or non-monetary value of library
  benefits to community
• Advocacy for and marketing the library

60
What do you want to know?

• Indicators
• How will you know you have achieved the outcomes?
  
• Data
• Sources
• Methods
• Frequency
• Resources
• Expertise or tools needed to collect and analyze
  data

Blueprint
61
Evaluation plan example

• Outcome Reduce institutional costs by buying
  shared resources.

?
62
Example part 1
Outcome ________
Blueprint II
63
Example part 1 (filled in)
Outcome Reduce institutional costs by buying
shared resources
Blueprint
64
Example part 2
Outcome ________
Blueprint
65
Example part 2 (filled in)
Outcome Reduce institutional costs by buying
shared resources
Blueprint
66
(No Transcript)
67
Evaluation plan
Outcome (from logic model)_______________________
_________________
68
Design Your Own Evaluation Plan
Blueprint
69
From your logic model

• Select an outcome and describe
• An activity that will help achieve the outcome
• Audience(s) affected by the activities
• Indicators of outcome results
• Targets for indicators
• Data source
• Data collection frequency
• Data analysis methods

Blueprint
70
The Tools To Build It
71
What to measure and how

• Not everything that can be counted counts and
  not everything that counts can be counted.
• Albert Einstein

Build It
72
Considerations before collecting data

• Indicators and outputs to be measured
• Types of data to be gathered
• Qualitative vs. quantitative
• When and from whom
• Focused on the audience perspective

Build It
73
Measuring outcomes

• Savings statistics (output, use, financial)
• Attitudes and beliefs surveys, interviews, focus
  groups
• Awareness proxy measures, surveys, interviews,
  focus groups
• Behavior proxy measures, interviews, focus
  groups, observation
• Knowledge written or oral tests
• Skills observations, hands-on tests

Build It
74
Sources of data

• Existing records and statistics
• Observation
• User and stakeholder input
• Survey results
• Formal
• Informal
• The literature
• Specialist or expert input

Build It
75
Make sense of the data

• Think about data analysis before collecting the
  data
• Identify experts that can help with data
  analysis
• Test your collection and analysis tools
• Leave enough time
• Go smallball if needed

Build It
76
Examples of analysis tools

• Benchmarking
• How you compare to others of similar size and
  circumstance
• A common measuring stick to evaluate process
  performance
• Cost/Benefit Analysis
• Use to evaluate the benefits of a program or
  service
• Return on Investment
• How much your investment in the program or
  project earned for the company

Build It
77
Benchmarking

• Can improve your library's performance
• Can help you gain upper management support
• Can help prove the value of your library

http//www.mlahq.org/members/benchmark/ (from the
MLA Benchmarking Network web page)
Build It
78
Benchmarking resources

• Your own library
• The competition
• Other hospitals/libraries in your system
• Other hospitals/libraries in your area
• MLA Benchmarking
• AAHSL Survey Statistics

Build It
79
CBA cost/benefit analysis

• The ratio showing dollar value of benefits gained
  for dollar value of costs
• Use to evaluate the benefits of a program or
  service
• Benefits divided by costs gives the value
  realized by transaction

Build It
80
CBA

• BENEFITS COSTS
• Or
• BENEFITS
• COSTS

81
ROI return on investment

• Percentage showing the return (increase in value)
  on dollars spent to achieve a benefit
• Amount the companys investment in a program
  earned for the company
• Earnings on money spent

Build It
82
ROI

• ((BENEFITS COSTS) COSTS) 100
• Or

BENEFITS COSTS COSTS
X 100
83
Preparing for CBA and/or ROI

• Select the program or service to value
• Electronic journal subscriptions
• Identify benefits derived from program/service
• Electronic journal articles on staff desktops
• Identify who gets the benefit
• Library users
• Convert the benefits to quantifiable terms (based
  on surveys and/or records)
• of journals used of articles accessed

Build It
84
Convert benefits to quantifiable terms

• Identify basis for valuing the benefits
• Calculate value of benefits cost of a single
  personal subscription X of subscriptions
• Identify and calculate costs related to producing
  the benefits
• Cost of total number of subscriptions
• Staff salaries, space costs, etc. to support the
  online subscriptions (overhead)

Matthews JR Internet Outsourcing Using an
Application Service Provider A How-To-Do-It
Manual for Librarians. How-To-Do-It Manuals for
Librarians, Number 110. New York Neal-Schuman,
2002 71-72.
Build It
85
Costing tips

• Consider tangible vs. intangible benefits
• Be conservative in estimating benefits and
  liberal in assessing costs
• Obtain administrative approval of the value of
  the benefits for each alternative involve
  financial staff
• Express benefits and costs in a value system
  shared by all parties
• Think about inflation and depreciation if study
  extends over several years

Build It
86
A CBA/ROI Example

• Electronic journal subscription to Fictional
  Medical Journal at Anytown Institution

87
Fictional Medical Journal Online

• The library spends 1600 for an institutional
  subscription to Fictional Medical Journal (FMJ)
  online
• 7800 articles are viewed per year using the
  librarys subscription

Build It
88
Assumption (for teaching only)

• Each person reads on average 2 articles per week
• 2x52104 articles per user
• 7800 uses represents 75 users
• 7800/10475
• Without a library, each person who needed FMJ
  would subscribe at a cost of 99 for online only

89
Benefit to the institution

• 7,425 (cost to buy 75 personal subscriptions
  75 x 99)
• -1,600 (cost for library to buy institutional
  subscription)
• 5,825 savings for online institutional
  subscription

90
Value of the benefit

• 7425/16004.64
• (cost of personal subscriptions /institutional
  subscription)
• 4.64 of benefit for each dollar spent by library
  on FMJ online

91
Summary benefit/cost ratio

• Cost 1,600 spent on FMJ
• Benefit 7,425 (value of 75 subscriptions at
  99/subscription)
• The ratio of benefits to costs is 7425K1600K,
  or 4.641, or 4.64 in benefits for every 1
  spent by the library

The example continued
92
Calculating ROI

• ROI reflects the money realized by making an
  investment
• ROI should be larger than the likely return on a
  bank account or certificate of deposit or some
  other investment instrument
• Superior stock market return is 10-12

The example continued
93
ROI

• Benefits
• Less costs of carrying out the service or program
  (subscription cost plus staff cost)
• Divided by costs
• Multiplied by 100

Benefits Costs
X 100
Costs
The example continued
94
Simple benefits and costs

• Benefits 7,425
• (cost to buy 75 subscriptions)
• Net cost 1,800
• (price of an institutional subscription plus
  200 overhead)

The example continued
95
Calculating ROI

• 7,425 1,800

X 100
1,800
312.5
The example continued
96
The Housewarming
97
Use the evaluation results

• To communicate your value
• To promote and advocate for the library
• To maintain or get additional funding
• As baseline information to show improvement over
  time
• To improve library services and programs

Housewarming
98
Communicate the results

• Identify the purpose of the communication
• Decide who needs to know the results
• Administrators, staff and other partners, library
  users, community groups (stakeholders)
• Develop communication strategy

Housewarming
99
Some communication methods

• Written and/or oral reports
• Formal
• Informal
• Executive summary
• Press release for newsletter or local paper
• Promotional materials for the library
• Staff meetings
• Informal hallway or elevator talk

Housewarming
100
Communication tips

• Have others review the communications
• Frame message in terms of added value to the
  mission and goals of the organization
• Use figures, tables, graphs to make your point
• Should be able to stand alone
• Have a narrative to give more details

Housewarming
101
More communication tips

• Know and rehearse your message
• Talk to administrators one-on-one when possible
• Focus on benefits
• Dont use library jargon
• Educate, but dont bore

Housewarming
102
What sticks?

• SUCCESs
• Simple
• Unexpected
• Concrete
• Credible
• Emotional
• Stories

(From Made to Stick. Heath, Chip. New York
Random House, 2007)
Housewarming
103
Elevator talk

• Simple, memorable mini-speech
• Ready for any brief encounter
• What your library contributes and why that
  matters
• Sadly and too often, the people who have the
  most influence over our future are really not all
  that interested in libraries.

Gardner Reed S. Making the Case for Your Library
A How-To-Do-It Manual. How-To-Do-It Manuals for
Librarians, Number 104. New York Neal-Schuman,
2001 6
Housewarming
104
Elevator talk exercise

• We had three outcomes resulting from our
  activities to meet this goal the library will
  acquire and manage information resources in a
  cost effective manner.
• Choose one of the three outcomes and
• Invent evaluation results
• Determine the audience
• Create a soundbite that highlights the outcome
  you have chosen

105
The outcomes

• Reduce costs by buying shared resources.
• Increase value of organization's expenditures by
  ensuring that ordered materials are available
• Users know that library owns title

106
Go forth evaluate

• Do smallball evaluation
• Relate your value to the mission and goals of
  your organization
• Plan your evaluation before you begin
• Share your value with many audiences
• Dont wait for a crisis start now!

107
Want more information?

• Resources list for this class
• The OERC Web site
• http//nnlm.gov/evaluation/
• Your regions Evaluation Liaison
• J. Dale Prince dprince_at_hshsl.umaryland.edu

108
Enjoy Baltimore!