Measuring Your Impact: Using Evaluation to Demonstrate Value

1

• Measuring Your Impact Using Evaluation to
  Demonstrate Value
• Colorado Council of Medical Libraries
• Denver, Colorado
• April 2 2004

2
Instructors

• Betsy Kelly
• Assessment Evaluation Liaison, NN/LM MCR
• kellyb_at_msnotes.wustl.edu
• --314-362-2783
• Maryanne Blake
• Communications Education Coordinator, NN/LM PNR
• blakema_at_u.washington.edu
• --206-221-3405

3
Workshop Objectives

• Workshop participants will be able to
• Describe the process in an evaluation plan for a
  library service
• Describe tools used to assess communities of
  users and stakeholders
• Identify elements of a logic model

4
Workshop Objectives

• Workshop participants will also be able to
• Identify tools and methods used for data
  collection
• Design an evaluation plan for a service in a
  library
• Report evaluation results

5
Challenges Facing Hospitals and Health Care

• Hospitals are closing
• In 1975 there were 7,156 hospitals
• In 2002 there were 5,794 hospitals
• Health care costs are rising
• of GNP in 1960 was 5.3
• of GNP in 2001 was 14.1
• Medicare/Medicaid payment shortfalls
• In 2001 29 of hospitals had negative margins
• Source Centers for Medicare Medicaid Services

6
Challenges Facing Hospital Libraries

• Cost of library resources are rising
• Library budgets are being cut
• Library staffing is cut as workload rises
• The everything is free on the Internet syndrome
  is rampant
• Closure of some hospital libraries

7
Why Demonstrate the Librarys Value?

• To show the impact of the library on the
  organizations mission and goals
• To show accountability
• As an advocacy and marketing tool
• To be proactive

8
From MLA

• Are you jeopardized by hospital administrators
  only concerned with the bottom line or merely by
  those uneducated about what a real librarian
  does? No matter what the reason, address the
  priorities of the organization as a whole,
  instead of attending only to the needs of the
  library. Do not use your jargon use their
  jargon. . Show that you meet the needs and
  further the goals of the organization.
• http//mlanet.org/resources/survive/survive3.html

9
So, To Show Your Value

• To show value to its parent organization the
  library must link its services and contributions
  to the organizations mission and goals and show
  how it furthers those.
• Look at the King, Marshall and other studies

10
What About Other Types of Libraries?

• How are they doing?
• How do they show their value?
• Special MN/DOT Report
• Public St. Louis Public Library Project

11
Success Stories

• Two hospital librarians in Washington State saved
  their jobs.
• Do you have any success stories?

12
So, How Do I Demonstrate My Librarys Value?

• There is no single method
• The hospitals political and organizational
  culture are big variables
• When programs operate in real communities where
  influences and forces are beyond your control,
  evaluation is generally more about documenting a
  programs contribution than proving something.
  W.K. Kellogg Logic Model Guide. Pg. 17

13
A Plan for Showing Value

• Assess the library and its communities
• Choose what to evaluate
• Design the evaluation plan
• Make sense of the data
• Communicate the results

14
Assess the Library and Its Communities
15
The State of the Library

• A state of the library report
• What is your librarys mission?
• Does it fit with the institutions mission?
• What communities do you serve?
• What are the services you provide?
• What are the important resources you have?
• What are your assumptions about the library and
  its services and your users?

16
Types of Communities

• Clinical staff in the hospital
• Other hospital workers
• Hospital administration
• Patients and their families
• The public in your community
• Clinicians in the community
• Public health workers

17
Topics of Interest

• Current information needs and access
• Current information resources
• Barriers to accessing health information
• Characteristics of the group
• Demographics, economic, social and salient health
  topics

18
Tools for Assessment

• Use secondary data from national/local sources
• Use the literature
• Existing records and statistics
• Observation
• Obtain user input focus groups, interviews

19
Practical Approach

• To gain a general understanding
• Focus groups
• Open-ended survey questions
• Interviews with key informants
• Collecting statistics

20
Case Scenario

• The Medical Library at
• Rockanda Regional Medical Center

21
Choose What To Evaluate
22
Utilize the Assessment Results

• Understand needs, desires and problems
• Validate assumptions about your services
• Provide data for later evaluation
• Use the results to develop a plan to evaluate
  your services

23
Choose What To Evaluate

• What you decide to evaluate will depend on
• What you need or want to know
• What your users feel is important
• What certain stakeholders want to have evaluated

24
BREAKOUT GROUPS
25
Introducing the Logic Model

• Framework for program and evaluation
• Fluid
• Changes over time
• Makes evaluation easier

26
Logic Models May Change Over Time
27
Plan BackwardImplement Forward

• Outcomes
• Activities
• Resources

28
Parts of the Logic Model

• Inputs or Resources
• What will you need to carry out your activities?
• Activities
• What will you do?
• Outputs
• How many of what will your activities produce?
• Outcomes
• So what the difference your program makes, the
  benefits that accrue because of your program

29
Outcomes

• Outcomes should be
• specific
• measurable
• action oriented
• realistic
• timed

30
Sample Logic Model
31
The Evaluation Plan

• Based on the logic model
• Consider the purpose
• Who is your audience?
• Your users
• Administrators
• Fund raisers
• How will the information be used?
• Financial savings or justification
• Intangible or non-monetary value of program
  benefits to community

32
The Evaluation Plan, contd.

• Questions
• What do you want to know about your program?
• Indicators
• How will you know you have achieved the outcomes
• Data
• Sources
• Methods
• Frequency
• Resources
• Expertise or tools needed to collect and analyze
  data

33
Resources needed for evaluation

• What resources are required
• to prepare for the evaluation?
• to collect data for the evaluation?
• to analyze the data for the evaluation?
• to 'build a case' based on the findings and
  outcomes of your service?
• How can each of these resources be obtained?

34
Methods Tools for Collecting Data

• What to measure and how
• What indicators and outputs need to be measured
• What type of data needs to be gathered
• When and from whom will it be gathered
• Focus on the users perspective

35
What Do You Want to Measure?

• Attitudes written instruments
• Awareness proxy measures,
• Behavior self-report, observation, proxy
  measures
• Knowledge written or oral tests
• Skills observations, skill tests

36
(No Transcript)
37
Data Collection Techniques

• Written questionnaires
• Surveys
• Interviews
• Focus groups
• Observation
• Library statistics and records

38
Design Your Own Evaluation Plan
39
Make Sense of the Data
40
Make Sense of the Data

• Data analysis should be thought about before
  collecting data
• Test your collection and analysis tools
• Be sure to leave enough time
• If needed, limit the evaluation goals and reduce
  the number of data collected

41
Hot Analysis Tools

• CBA is Cost/Benefit Analysis
• Use to evaluate the benefits of a program or
  service
• ROI is Return on Investment
• How much your investment in the program or
  project earned for the company
• BSA is Balanced Scorecard Approach
• A four pronged look at performance
• Benchmarking
• How you compare to others of similar size and
  circumstance
• A common measuring stick to evaluate process
  performance

42
CBA

• Direct vs indirect benefits
• Contingent valuation
• Willingness-to-pay (WTP)
• Willingness-to-accept (WTA)
• Valuation of time
• Surplus value

43
ROI

• Business tool to determine earnings on money
  spent a percentage value
• CFOs are charged with getting the most return
  from every dollar invested or spent
• ROI of a library must show that the expense
  provides the highest return of any potential use
  of the budget dollars

44
ROI Process

• Collect data
• Convert data to monetary values
• Isolate the effects of the program
• Costs of the program must be included
• (Sum of value of benefits costs)/costs
• Difficult to use with qualitative data

45
BSA

• Balanced Scorecard Approach
• Adds dimensions to business evaluation beyond
  pure profit
• Considers the financial perspectives
• but also
• Considers customer satisfaction
• Considers knowledge acquired/applied as a result
  of training opportunities
• Considers whether products and services are
  aligned with customer requirements

46
Perspectives of BSA
47
Benchmarking

• Benchmarking, conducted properly, can
• improve your library's performance
• help you to gain or improve upper management
  support and
• help to prove the value of your library.
• http//www.mlahq.org/members/benchmark/
• (from the MLA Benchmarking Network web page)

48
Communicate the Results
49
Communicate the Results

• Audience
• Information needs
• Report purpose or use
• Report format
• Dissemination strategies
• Timeline

50
Reporting Results, contd

• Active voice
• Brief, understandable, descriptive
• Use headings, figures, tables, graphs
• Imbed in text
• Should be able to stand alone
• Explain figures, etc in narrative

51
Reporting Results, contd

• Clear link between conclusions, recommendations,
  results and the original evaluation question
• Communication and persuasion
• Have partners review the draft report
• (hint worksheet 2, question 3)

52
Reporting Results, contd

• Oral or written
• Summary of results Executive Summary
• Provide purpose, background, methodology of the
  evaluation
• Conclusions and recommendations
• Attachments
• Cover letter

53
Disseminating Results

• Identify audiences (administration, targeted
  community, professional colleagues)
• Look for other publishing, presentation, or
  promotion opportunities
• Professional meetings and activities, websites,
  listservs, print and e-journals

54
Other Report Format Options

• Photo essays
• Posters
• Electronic
• Power Point
• Scenarios
• QA

55
Questions?

• Resources List for this class
• The OERC Web Site
• http//nnlm.gov/evaluation/
• Web Resources for Evaluation
• http//nnlm.gov/evaluation/webresources.htm