Folie 1

1
Impact measures for libraries and information
services
Roswitha Poll Münster
2
The importance of libraries
What libraries ensure (IFLA)
Democratic values
Cultural diversity
Libraries make a difference But who will believe
it?
The open information society
Intellectual freedom
Economic progress
Information equality
Information literacy
Reduction of poverty
3
input
a contribution of work, information, or material
activities
activities change input into output
output
the amount produced, the results supplied
the effect or influence of one person, thing,
oraction, on another
the consequence, visible or practical result
oreffect of an event or activity
outcome
the importance or preciousness of something,the
perception of actual or potential benefit
4
Don Revill
ACRL
5
Different outcomes
positive negative direct/immediate long-term
actual potential intended unexpected
Don Revill
ACRL
Peter Brophy
6
Impact/outcome of cultural institutions
changes in skills, competences, attitudes,
behaviour

• knowledge
• information literacy
• academic or professional success
• social inclusion
• individual well-being

7
Outcomes of universities

• recruitment and retention of students and
  excellent academic staff
• effective teaching - high graduation rates -
  high grades in examinations - high employment
  rates after examination
• effective research- high renown and use of
  research results and publications - high
  amount of special grants - awards, honours

Evidence needed The library supports these goals
8
Problems of assessing outcome

• Value and outcome of a service can differ between
  user groups
• Data may not be available because of data
  protection rules
• Results of projects are often not comparable
  because of different data collection methods
• Long-term effects cannot be assessed as persons
  are no more available
• All tested methods are time-consuming

Influences on individuals are manifold
9
Methods for assessing impact
Quantitative measures
"Surrogate measures" have to be used

• tests assessing user skills before and after
  training on a service
• performance monitoring / data mining
• observation (unobtrusive)
• analysis of documents and bibliographies compiled
  by users
• correlation between library use and success

10
Methods for assessing impact
Qualitative measures

• Surveys (print, telephone, online)
• Interviews
• Focus groups, discussion groups
• Self-assessment of users
• Facts and stories must be organized to show
  patterns

The "anecdotal evidence" supports data
11
User satisfaction as outcome measure?
ACRL
A change of behaviour? Or a basis for such change?
Jennifer Cram
12
Assessing impact
Correlation of library use and academic success
Library use, expressed by

• Frequency (loans, visits)
• Range of services used (reference service, user
  training, ILL)
• Range of collections used (undergraduate
  collection, E-journals, special material)
• Attendance at training sessions

Success, expressed by

• Duration of studies
• Grades in exams
• Student retention
• Employment rate after exam
• Quality of publications (impact
  factor,peer-review journals)

13
Assessing impact
The library's impact on information literacy
Outcome of a specific training or series of
trainings

• surveys after instruction
• pretest / posttest
• self-assessment of users
• behavioural observation
• transaction logs
• longitudinal studies
• assessing changes in students' bibliographies

14
Assessing impact
Importance of the local library for research
3

• Percentage of material cited in academic
  publications
• that was (could have been) retrieved via the
  local library.
• Citations in
• dissertations
• student papers
• publications of a faculty

• 2 Estimated percentage of the information needed
• for a publication / paper / report found via the
  local library
• questionnaire/interview after publication

15
Assessing impact
Impact on social inclusion and community life
4

• Does the existence of a specific library / of
  libraries further
• individual well-being
• the quality of life in a society
• - democracy
• - social inclusion
• - cultural life
• - local identity
• - life-long learning

• Method Assessing public opinion (users and / or
  non-users) as to the benefits of libraries and
  special library services
• to the population (indirect benefit)
• to the individual user (direct benefit)

16
Assessing impact
The financial value of libraries
5

• For every 1 of public funding
• the British Library receives each year,
• 4.4 is generated for the economy

Assessing time costs (replacement value of a
client's time)
Users invest time and effort in order to use
library services. The value that they - or their
institution - place on that use must be at least
as high as their sacrifice of time. (Costs are
calculated by time and average salary)
Using the contingent valuation method

• Willingness-to-pay

What would users pay for the maintenance of a
service?

• Willingness-to-accept

Which sum would users accept as equivalent for
the deletion of a service?
17
Using the results
Libraries on the agenda (Claudia Lux)

• accountability
• decision making
• resource management
• improvement of services
• promotion of the library's role

18
Outcome projects
eVALUEd in "evidence base", University of
Central England Toolkit for evaluating
electronic information serviceshttp//www.evalued
.uce.uk/index.htm
IMLS (Institute of Museum and Library Services)
outcome-based evaluation of projectshttp//www.im
ls.gov/index.htm
ARL New Measures Initiative several projects -
Learning outcomes - Higher education outcomes
research review - MINES (Measuring the impact of
networked electronic services)http//www.arl.o
rg/stats/newmeas/index.html
19
Outcome projects
IBEC Information School of the University of
Washington and University of Michigan School of
Information Toolkit for assessing the impact of
information in communitieshttp//ibec.ischool.was
hington.edu/default1024.aspx
SCONUL and LIRG (Library and Information Research
Group) impact initiative and mailing
listhttp//www.sconul.ac.uk/activities/performanc
e/impact.html
IFLA Section Statistics and Evaluation working
group on outcome/impact bibliography of
literature and projects worldwide
http//www.ulb.uni-muenster.de/outcome.html
20
(Goethe Faust)