Lies,

1
Lies, Damned Lies, and Statistics
2

• Hennens American Public Library Ratings
• are put together by a librarian to promote
  excellence in librarianship
• are based on nationally-gathered figures for the
  United States
• are published every year in American Libraries

3
2006 ratings for two largest population
groups Rank Library State Score
Population over 500,000 1 Cuyahoga County
Public Library OH 861 2 Multnomah County
Library OR 855 3 Columbus Metropolitan
Library OH 848 4 Denver Public Library CO
842 5 Baltimore County Public Library MD 807
Population 250,000 to 499,999 1 Santa Clara
County Library CA 915 2 Howard County
Library MD 897 3 St. Charles City-County
Library District MO 868 4 Johnson County
Library KS 839 5 Madison Public Library
WI 811
4
Populations of high-ranking libraries in
largest-population group Cuyahoga
1,393,978 Multnomah 681,454 Columbus
711,470 Denver 554,636 Baltimore
631,366 Populations of low-ranking libraries in
largest-population group Los Angeles
3,844,829 Chicago 2,842,518 New York 8,143,197
(divided three ways between NYPL, QBPL and BPL
2,714,399 served by each library)
Is it meaningful to compare these systems?
5
Hennen notes Perhaps most prominently absent
from the data are any measures of electronic use
or Internet service. While such measures would
have been desirable, the FSCS data simply are
simply not sufficient for such comparisons at
this time. Internet, electronic services and
audiovisual services are excluded because there
is simply not enough data reported by enough
libraries to make comparisons meaningful.
Are we obsessed with circ?
Does circulation good service?
6
How do we get over our addiction to circ
stats? 1. Collect many more kinds of
statistics 2. Compare them across many kinds of
peer institutions (not just population
size) 3. Collect statistics that are related to
user benefit 4. Collect statistics that are
related to economic measures 5. Collect and
analyse non-statistical information such as
anecdotal reporting, open-ended questionnaires,
and feedback
7

• 1. Collect many more kinds of statistics
• Hours open per week
• Materials processing time and cost
• Space usage ( of floorspace devoted to study
  areas, stack space, work areas, storage, meeting
  rooms)
• Collection attributes ( reference vs. lending
  print vs. non-print electronic vs. actual
  periodical vs. monograph)
• New construction, renovation, and major capital
  improvements
• Internet availability ( of stations) and use
  (hours of user time clocked per week)
• Reasons for checking out materials (pleasure,
  hobby/interest, school, learning a skill,
  legal/health/personal/financial decision-making)
• Cardholders as of population non-resident
  cardholders as of users
• Circulation peak hours visit peak hours
  database use peak hours email/chat reference
  peak hours web site use peak hours
• (Same for peak days of the week and peak months
  of the year)
• Percentage of cardholders who never borrow
  materials

8
2. Compare them across many kinds of peer
institutions (e.g. geographic)
From B.C. Public Library Statistics http//www.b
ced.gov.bc.ca/pls/reports.htm
9
3. Collect statistics that are related to user
benefit

• benefit is not the same as satisfaction
• user surveys are usually designed as user
  satisfaction surveys
• whats wrong with user satisfaction surveys?

10

• General problems with user satisfaction
  surveys
• satisfaction is a subjective experience the
  distinction between satisfied and very
  satisfied is not the same for any two people
• it makes an effort to quantify something that is
  inherently qualitative (a feeling)
• self-reporting is known to be unreliable across
  the social sciences
• there is no evidence that user satisfaction
  correlates strongly with actual effectiveness
  some studies show the link to be extremely weak
• D. F. Galetta and A. L. Lederer. Some cautions on
  the measurement of user
• information satisfaction. Decision Science,
  202534, 1989.
• W. G. Chismar, C. H. Kriebel, and N. P. Melone.
  Criticism of information
• system research employing user satisfaction.
  Graduate School of Industrial
• Administration, Carnegie-Mellon University,WP
  24-85-86, October 1985.
• N. P. Melone. A theoretical assessment of the
  user-satisfaction construct in
• information systems research. Management Science,
  367691, 1990.

11

• Library-specific problems with user
  satisfaction surveys
• homogenous jurisdictions score better than
  diverse ones, because their needs are also
  homogenous and can be met at a higher level with
  the same funding level (in general, suburbs score
  higher than cities).
• user satisfaction surveys are usually
  undertaken when a system is having trouble
  serving its population. This guarantees youll do
  poorly on the survey, further demoralizing your
  staff.
• praise and positive reinforcement are
  culturally-specific values. If you serve a
  community that believes in withholding praise to
  improve performance, you will do less well than a
  library that serves a group that believes their
  positive comments will serve to reinforce
  libraries.
• we ask the wrong things. How satisfied are you
  with the cleanliness of the library? is a silly
  question because cleanliness can be better
  assessed using other measures (e.g. take a
  picture).

12
The Alternative User Benefit Surveys User
satisfaction surveys ask On a scale of 1 to 5,
how satisfied are you with the service you
received today? User benefit surveys ask Did
someone help you in the library today? If
answer is yes Did you find more information
than you would have by yourself? Did you find
better/more accurate information than you would
have by yourself? Did you find what you needed
faster than you would have by yourself?
13

• User benefit surveys for many kinds of services
• For many measures, you can use staff observation
  rather than a questionnaire, which reduces user
  survey burnout.
• classes
• 9 of 10 students became proficient in using the
  mouse today
• at the end of 10 weeks of ABE classes, 4 of 7
  students were able to open a bank account
• programs
• 18 children out of 26 attending the program
  learned to say hello in Spanish
• 22 of 28 attendees reported learning a new
  knitting technique from todays slide
  show/demonstration
• computers
• 10 of 22 computer users requested assistance in
  searching databases, the catalogue or the
  internet between 600 and 700
• Lots more ideas for these kinds of questions in
• How Libraries and Librarians Help by Joan C.
  Durrance and Karen E. Fisher (Chicago ALA,
  2005)
• For services that simply cant be quantified,
  see section on anecdotal reporting

14
4. Collect statistics that are related to
economic measures Direct benefits to
users Cost-benefit analysis of library use
Costs are compared to bookstores, internet
cafes, NetFlix, community centre classes, private
classes and tutoring services located in the same
jurisdiction. Willing to pay price is
calculated from mean response to questionnaire
given to attendees at programs and reference
users.
adapted from Miami-Dade 1998-99 figures, in
Measuring for Results by Joseph R. Matthews
(Westport, CN Libraries Unlimited, 2004)
15
Direct benefits to local business Provision of
business-related information use of
business-related lending materials, database use,
reference transactions, and services (e.g. small
business mentoring programs) Direct benefits to
locality Library as employer of taxpayer
investment directly poured back into local
economy via salaries Purchase of goods spent
locally for books, supplies, building
maintenance, capital expenditures heres an
argument for buying locally Increased local
consumer traffic bringing non-local residents
into community via non-resident cards (parking
fees, buying lunch, other consumer activity
increases locally) and heres your argument
for not having user fees for non-residents see
The Effects of Non-Resident User Fees on Central
Library Use by Ken Roberts (Hamilton, ON HPL,
1996)
16

• Indirect benefits to the public
• increased property values
• decreased crime (after-school library hours
  correlate to lower crime during those hours)
• note that these benefits accrue to everyone,
  not just library users
• Indirect benefits to local business
• literate, tech-ready work-force
• Indirect benefits to locality
• general support for economic initiatives,
  entrepreneurship, and local improvement through
  collections and services
• Indirect benefits are much harder to quantify
  than direct ones
• Truly proving indirect benefits would require a
  large-scale study across multiple municipalities
  with comparable demographics
• Collect what you can, when you can, and jump on
  opportunities to collaborate in more complete
  data collection initiatives

17
5. Collect and analyse non-statistical
information such as anecdotal reporting,
open-ended questionnaires, and feedback The
social audit Use of library space rather than
collections or services social interactions in
the library decreased social isolation the
library as commons Users able to make informed
personal decisions Increased literacy/English
language proficiency Lifelong learning
support Local history and genealogical research
not available elsewhere Increased social
awareness support for democratic civic
life Staff involvement in professional
leadership, e.g. committee work, professional
associations, publications Staff involvement in
unrelated artistic or social endeavors, e.g. how
many published poets do you have on
staff Open-ended questionnaires What would make
it easier for you to find what youre looking
for? Do you use NetFlix Amazon, Google? If
so, what features do you like and dislike?

18

• Feedback
• user letters, suggestion box items,
  complaints/kudos
• indirect feedback (monitor blogs for mention of
  your library)
• these should be collated and regularly analysed
  for patterns
• How to collect stories and testimonials
• web form for public to share stories (make sure
  you ask for permission to publish)
• intranet form for librarians/staff to share
  anecdotes (must be cleansed of identifying
  information before being made public)
• collect systematically from funded projects, e.g.
  ESL classes
• make sure you collect benefits that accrue to
  the individual as well as to a wider group of
  people
• individual benefit Because I took the ESL class
  I can now talk to my doctor
• wider impact I no longer have to take my
  daughter out of school to translate for me

19
What do we do with all this assessment? 1.
Advocacy 2. PR/communications 3. Write required
legal documents such as the Annual Report 4.
Make operational decisions 5. Make strategic
decisions
20

• The difference between operational and strategic
  decisions
• Operational decisions
• involve ongoing tasks
• attempt to improve efficiency or outcomes of
  existing operations
• resolve problems that have occurred
• should be made by staff on the ground who have
  strong knowledge of operations
• Examples of operational decisions
• Lets take 30,000 out of periodicals, which are
  being used less because of databases, and put it
  into downloadables, for which there is increased
  demand
• Lets stop putting coloured stickers on DVDs to
  identify genresit takes a lot of staff time and
  the users dont know what they mean
• Lets move the circulating DVDs to the back of
  the floor so that patrons walk by the book
  display to get there (milk placement)
• Lets put up signage and make library card
  information brochures in Russian because theres
  a lot of new immigration to town and no
  Russian-speaking staff

21

• Strategic decisions
• relate to meeting the librarys mandate and
  vision statement
• typically take time and careful planning to
  achieve
• should be made at the highest levels (board and
  library director, senior managers) but should
  involve staff and public input
• Examples of strategic decisions
• Lets increase library card holders to 25 of our
  population
• Lets decrease the time from publication to
  shelf-availability by 5 working days
• Lets increase teen use of the library by 10

22
Final thoughts on assessment All kinds of
assessment tools (statistical, economic,
anecdotal) can be used to make either operational
or strategic decisions Most libraries do not
collect enough kinds of information Many
libraries only do operational planning and never
do strategic planning Assessment is as much
intuitive and creative as it is data-driven, but
data is an essential component that propels the
intuitive and creative process