One Librarys Take on New Measures

1
Culture, Counting and Plumbing
One Librarys Take on New Measures
ALCTS Collection Management and Development
Section Atlanta, June 2002 Joe Zucca Assessment,
Planning and Publications Librarian University of
Pennsylvania Library
2
Topics For This Presentation
1. Placing the New Measures into Penns
Context 2. Building the Plumbing for Measurement
The Penn Library Data Farm 3. Mining Data for
Information 4. Can Librarians Count?

New MeasuresOld Ways, and
a High Altitude View From the Field
3
1. The Penn Context
The Proliferation of Electronic Resources
Article indexes, e-journals and other full-text
resources
7000
6875
6000
4932
5000
4000
Number of Titles
3000
2438
2000
1394
1000
86
9
3
0
1991
1993
1996
1999
2000
2001
2002
4
1. The Penn Context
The Growth of Expenditures for Electronic
Resources
Annual Growth of Expenditures for Electronic
Information Based on 1991
E-Resources as a percent of acquisitions budget
1991 1993 1996
1999 2000 2001 3.7
3.2 5.5 13.2
13.9 15.7
5
1. The Penn Context
Strategy-Formation 1996-2000 Strategic Plan
Develop new ways of conceptualizing and measuring
library effectiveness to promote within the
Library growing accountability and responsiveness
to changing information environments and to track
progress toward our goals.
6
1. The Penn Context
Administrative Imperative 1999/2000 External
Review Report The Committee did not address the
difficult questions about quantitative indices
and performance measuresbecause no satisfactory
answers exist. we recommend that the Penn
Library become an active participant in the ARL
new measures initiative.
7
1. The Penn Context
Accountability

• From The Council of Deans, UPenn
• To Vice Provost and Director of Libraries, Paul
  Mosher
• Who uses the library(ies)breakdowns by school,
  status (student/standing faculty/research
  faculty/adjuncts/staff/ full vs part time)
• What types of usesbooks, journals, reference
  data, study space, etc.
• Intensity of usetime and volume measures for
  both onsite and remote uses
• Relationships among the abovetype of user by
  type of use by intensity etc.
• Patterns in the above over time
• Comparisons with peer institutions re the above
• Patterns/trends in intellectual areas of interest
  to users
• Patterns/trends in information resource use
  beyond the librarywhere do Penn people go for
  what types of information and how?

8
1. The Penn Context
Utilitarian Attitude/Willingness to Experiment

Leverage the data you have
usually theyre rigorous enough to validate and
inform organizational experience and judgment
9
User transaction
Building the Plumbing for Measurement
Tools That Deliver Service Also Enable Measurement
Vendor Reports
Web Other Servers
Server Logs
Library Data Farm
Log archive raw log output parsed stored
Perl
SAS/Excel/MS Access
Voyager/DLXS
Data harvest processing systems
Processed data files Reports and Analysis
Knowledge base statistical repository, data
files, assembled for access
Web host
Staff query interface
10
The Library Data Farm Mining Data for Management
Information
11
Circulation Statistics
12
Penn Library Franklin (OPAC) Activity Statistics
13
Image Collection Use Analysis
14
Ebook Use
15
(No Transcript)
16
(No Transcript)
17
Managing the Collection of Public Web Sites
18
Information Base, Data, E-Journal Usage
Statistics

19
Proxy Service Statistics
20
Dynamic Report Generation
21
Use of Licensed Resources
What Are the High Use E-Journals, Data for FY2001
Title

Log-ins Pct Total Log-ins
Log-ins


On Campus Off Campus
22
Use of Licensed Resources
How Much Bang Do We Get on the Dollar For
E-Journals?
E-Journal Subscription Costs Per Log-In, FY2002
(July-April)
Publisher Log-ins Pct
of Total Cost Per Login
ScienceDirect 139,727 27.1 0.63 ECO
70,730 13.7 0.09 JSTOR 48,668 9.4 0.35 Wil
ey 38,255 7.4 0.09 ACS
31,865 6.2 0.12 Ideal 30,568 5.9 5.51 Blac
kwell/Munksgaard 28,940 5.6 0.27 Journals_at_Ovid
26,982 5.2 n/a Oxford 14,819 2.9 0.20 Sprin
gerLINK 13,507 2.6 n/a ABI/Inform
12,785 2.5 3.08 Project Muse
11,438 2.2 1.22 AIP 7,873 1.5 5.01 Cambrid
ge 7,835 1.5 n/a Annual Reviews
7,215 1.4 0.08 IEEE 7,132 1.4 6.73 RSC
5,661 1.1 n/a Others 11,451 2.2 Total 515
,451 100 11 publishers
23
Use of Licensed Resources
What Databases Do Our Clients Use at What Cost?
15 Most Frequently Used Index/Abstract/Full-text
Databases in FY 2001
Database
Log-ins Pct Total Cost Per Login
24
Use of Licensed Resources
How Does Use Scatter Across Databases
Use Measured in Log-ins for FY 2001
25
Use of Licensed Resources
ScienceDirect Articles Viewed, FY 2001
vendor-supplied data
26
Use of Licensed Resources
Academic Press (Ideal) Articles Viewed, FY 2001
vendor supplied data
27
Use of Licensed Resources
Where Do Our Clients Access Information?
Database Log-ins by Domain, FY2001
Campus Residences 10
Off-Campus 15
In-Library 25
On-Campus Depts 50
28
Database Use by Penns Schools Centers
Use of Licensed Resources
School Pct of Log-ins
How Does Database Use Distribute By Communities?
Per Capita Use of Databases by Penns Schools and
Centers, FY 2001
55
50
45
40
35
30
Log-ins Per Capita
25
20
15
10
5
0
LAW
VET
ASC
MED
NUR
SAS
GSE
SSW
SEAS
GSFA
WHRT
ADM
DENTAL
School and Center Domains
Does not include resources licensed by the Law
Library for Law school affiliates
29
Use of Licensed Resources
Where Do Communities of Clients Work?
Database Log-ins from Off Campus as a Percent of
Total Log-ins, FY2001
Pct. of Log-ins
School or Center
30
Use of Licensed Resources
When Are They Working?
Database Use by Time of Day, FY2001
31
Use of Licensed Resources
How Does Audience Composition Change Through the
Day?
Database Use by hour, FY2001
90.0
Schools
80.0
Library
Student Residences
70.0
Dialin
Other
60.0
50.0
40.0
30.0
20.0
10.0
0.0
1-2 AM
2-3 AM
3-4 AM
4-5 AM
5-6 AM
6-7 AM
7-8 AM
8-9 AM
1-2 PM
2-3 PM
3-4 PM
4-5 PM
5-6 PM
6-7 PM
7-8 PM
8-9 PM
12-1AM
9-10 AM
12-1 PM
9-10 PM
10-11 AM
11-12 AM
10-11 PM
11-12 PM
32
Intelligence for Bibliographers
Return-Path ltolson_at_pobox.upenn.edugt Subject
Again, testing general databases To
sblack_at_asc.upenn.edu Date Wed, 10 Apr 2002
165411 -0400 (EDT) From olson_at_pobox.upenn.edu
Dear Sharon -- Just a second quick note begging
you, please, keep trying to look at those three
databases! Data farm usage logs indicate that
one-quarter of all database logins from Annenberg
IP addresses in 2001 were pointing to Academic
Index (followed by Lexis-Nexis and PsycInfo, both
with about 10-percent of all Annenberg database
logins). Also, 15-percent of all Academic Index
school-based logins last year came from Annenberg
IP addresses, more than from all schools except
Arts and Sciences (at 30-percent). Considering
how much Annenberg people use the general
database -- and you must know best how they can
raise Holy Ned over the least change, I hope
33
Impact of Behavior on Facilities Planning
34
Studying Technology Transfer
High impact chemistry journals regression of
online use on in-library use
Journal of the American Chemical Society Journal
of Organic Chemistry Tetrahedron Letters
Reshelves
Log-ins
35
Can Librarians Count?
36
Plus Ga Change, Plus Cest La Meme Chose
37
New MeasuresOld Ways?

• The Metrics are Faulty
• The Assessment Rationale is Vague
• The Data are Inexact
• We Need a Formal Study Process to Address the
  Problem
• Managements Data Need is Imperative (and the
  managers are impatient)
• The Result is Entropy

38
Measuring the New Measures A High Altitude View
From the Field

• Impetus for Culture Change (help librarians to
  embrace counting)
• Creation and Testing of Models/Methods (e.g.
  LibQual)
• Channels for Learning and Collaboration
• Further Cultural Change

39
Culture, Counting and Plumbing
One Librarys Measure of the New Measures
Joe Zucca
University of
Pennsylvania Library
zucca_at_pobox.upenn.edu Penn Library Data Farm
http//metrics.library.upenn.edu/prototype/datafar
m