Demonstrating Value and Creating Value: Evidence-Based Library Management through MINES for Libraries

1
Demonstrating Value and Creating
ValueEvidence-Based Library Managementthrough
MINES for Libraries
Martha Kyrillidou Director of the ARL Statistics
and Measurement Program Association of Research
Libraries Brinley Franklin Vice Provost for
University Libraries University of
Connecticut Scholars Portal Forum, Ontario
Council of University Libraries February 1
2006 Toronto
www.arl.org/stats/
2

• Today
• is
• Tomorrow
• - Peter Melinchok expressing dismay at his
  parents distorted sense of time

3

• Bangor University considers removing librarians
  posted by Blake on Thursday January 27, _at_0730AM
  -753 hits       Ms Information writes
  "News from the University of Wales Bangor in the
  UK. senior management no longer feel that subject
  librarians / academic liaison librarians are
  needed in the modern academic library. They have
  made restructuring proposals which include
  removing all bar one of the subject librarians
  and a tier of the library management, including
  the Head of Bibliographic Services. The
  university management thinks that technology has
  'deskilled' literature searching. As far as I
  know, this proposal is unprecedented in the
  United Kingdom.In essence, there will remain 4
  professional librarians serving a 'research-led'
  university of 8,000 plus FTEs and with 8 library
  sites. These will be the university librarian,
  cataloguing librarian, acquisitions librarian and
  Law librarian.Has anything like this happened
  anywhere that you know of? If so, what have been
  the effects?

4
(No Transcript)
5
(No Transcript)
6
(No Transcript)
7
(No Transcript)
8
Rhodes University Anne Moon
9
Total Circulation
Note. M. Kyrillidou and M. Young. (2003). ARL
Statistics 2002-03. Washington, D.C. ARL, p.8.
10
Reference Transactions
Note. M. Kyrillidou and M. Young. (2003). ARL
Statistics 2002-03. Washington, D.C. ARL, p.8.
11
ARL Overall
12
Libraries Remain a Credible Resource in 21st
Century
98 agree with statement, My library contains
information from credible and known sources.
Note. Digital Library Federation and Council on
Library and Information Resources. (2002).
Dimensions and Use of the Scholarly Information
Environment.
13
Changing Behaviors
Recent Survey Only 15.7 agreed with the
statement The Internet has not changed the way I
use the library.
Note. Digital Library Federation and Council on
Library and Information Resources. (2002).
Dimensions and Use of the Scholarly Information
Environment.
14
The Internet Goes to College

• Early data from ethnographic interviews
• I use Google because I heard it searches for
  more things (than other sources).
• I believe I can find anything on the Internet.
  There hasnt been anything I havent been able to
  find.
• Because Im lazy.
• Books have so much information that no one can
  go through it all.
• I use the Internet first is because it is more
  convenient.
• I go to the library because thats what teachers
  like.
• Google has gotten me through college.
• Source Steve Jones, The Internet Goes to
  College, ARL Talk

15
a revolution in making

• Il est plus nécessaire d'étudier les hommes que
  les livres
• FRANÇOIS DE LA ROCHEFOUCAULD (16131680)

16
Into the future

• StatsQUAL
• ARL Statistics
• E-Metrics
• LibQUAL
• DigiQUAL
• MINES for Libraries

17
MINES for Libraries
The MINES Survey Methodology
Brinley Franklin Vice Provost for University
Libraries University of Connecticut
brinley.franklin_at_uconn.edu http//www.minesforlibr
aries.org
www.arl.org/stats/
18
What is MINES?

• Action research
• Set of recommendations for research design
• Set of recommendations for web survey
  presentation
• Set of recommendations for information
  architecture in libraries
• Plan for continual assessment of networked
  electronic resources

19
MINES for LibrariesTM

• MINES is a transaction-based research methodology
  consisting of a web-based survey form and a
  random moments sampling plan.
• MINES typically measures who is using electronic
  resources, where users are located at the time of
  use, and their purpose of use.
• MINES was adopted by the Association of Research
  Libraries (ARL) as part of the New Measures
  toolkit in May, 2003.
• MINES is different from other electronic resource
  usage measures that quantify total usage (e.g.,
  Project COUNTER, E-Metrics) or measure how well a
  library makes electronic resources accessible
  (LibQualTM).

20
MINES for LibrariesTM Survey Form Five
Questions and a Comment Box
21
Questions Addressed By MINES for Libraries for
the OCUL Scholars Portal

• How extensively do sponsored researchers use
  OCULs Scholars Portal? How much usage is for
  non-funded research, instruction/education,
  student research papers, and course work?
• Are researchers more likely to use the Scholars
  Portal from inside or outside the library? What
  about other classifications of users?

• Are there differences in Scholars Portal based on
  the users location (e.g., in the library
  on-campus, but not in the library or
  off-campus)?
• Could MINES, combined with usage counts, provide
  an infrastructure to make Scholars Portal usage
  studies routine, robust, and easily integrated
  into OCULs administrative decision-making
  process for assessing networked electronic
  resources?

22
OCUL/MINES Methodological Considerations

• The sampling plan was determined at the outset.
  Surveys were conducted once a month for two hours
  a month between May, 2004 and April, 2005
• The selection of the monthly survey periods were
  weighted based on usage counts by time of day and
  were chosen randomly.
• Participation was mandatory, negating
  on-respondent bias, was based on actual use in
  real-time, and was brief (to minimize user
  inconvenience).
• OCUL designed the local questions, mounted the
  survey, collected data and sent it to ARL for
  tabulation in aggregate and by individual
  institution.
• If more than one search was conducted by a user,
  the survey form was auto-populated with initial
  responses as the default.

23
OCUL/MINES Methodological Considerations
(continued)

• Each participating library explained the survey
  and its confidentiality provisions to their local
  constituency.
• Research ethics officers and/or Ethics Review
  Boards, where necessary, reviewed and approved
  the survey instrument and methodology.
• OCUL determined that individual institutions and
  their institution-specific data collected during
  the survey periods would not be disclosed.
  Individual data was anonymous.
• The mandatory nature of the survey required
  discussion on some campuses and caused one OCUL
  member library to withdraw from the study.
• Two institutions pre-tested the survey in
  January, 2004. Data collection programming and
  configurations/links had to be revised in
  February and March, 2004.
• After completing the survey, users were connected
  to their desired Scholars Portal networked
  electronic resource.

24
Issues with web surveys

• Research design
• Coverage error
• Unequal access to the Internet
• Internet users are different than non-users
• Response rate
• Response representativeness
• Random sampling and inference
• Non-respondents
• Data security

25
MINES strategy

• A representative sampling plan, including sample
  size, is determined at the outset. Typically,
  there are 48 hours of surveying over 12 months at
  a medical library and 24 hours a year at a main
  library.
• Random moment/web-based surveys are employed at
  each site.
• Participation is usually mandatory, negating
  non-respondent bias, and is based on actual use
  in real-time.
• Libraries with database-to-web gateways or proxy
  re-writers offer the most comprehensive
  networking solution for surveying all networked
  services users during survey periods.

26
MINES strategy (continued)

• Placement
• Point of use
• Not remembered, predicted or critical incident
• Usage rather than user
• What about multiple usages
• Time out ?
• Cookie or other mechanism with auto-population
• Distinguish patron association with libraries.
• For example, medical library v. main library.
• But what if the resources are purchased across
  campus for all. Then how to get patron
  affiliation?

27
Web Survey Design Guidelines

• Web survey design guidelines that MINES followed
  
• Presentation
• Simple text for different browsers no graphics
• Different browsers render web pages differently
• Few questions per screen or simply few questions
• Easy to navigate
• Short and plain
• No scrolling
• Clear and encouraging error or warning messages
• Every question answered in a similar way -
  consistent
• Radio buttons, drop downs
• ADA compliant
• Introduction page or paragraph
• Easy to read
• Must see definitions of sponsored research.
• Can present questions in response to answers
  for example if sponsored research was chosen,
  could present another survey

28
Quality Checks

• Target population is the population frame
  surveyed the patrons who were supposed to be
  surveyed - except in libraries with outstanding
  open digital collections.
• Check usage against IP. In this case, big
  numbers may not be good. May be seeing the
  survey too often.
• Alter order of questions and answers,
  particularly sponsored and instruction.
• Spot check IP against self-identified location
• Spot check undergraduates choosing sponsored
  research measurement error
• Check self-identified grant information against
  actual grants
• Content validity discussed with librarians and
  pre-tested.
• Turn-aways number who elected not to fill out
  the survey
• Library information architecture -- Gateway v.
  HTML pages there is a substantial difference in
  results.

29
Mandatory UConn Libraries (3 months)
30
Optional UConn Libraries (3 months)
31
Issues with web surveysbrief bibliography

• Cook, Colleen Heath, Fred and Russell L.
  Thompson. 2000 (December). A Meta-Analysis of
  Response Rates in Web- or Internet-Based
  Surveys. Educational and Psychological
  Measurement 60(6) 821-836.
• Couper, Mick P. Traugott, Michael W. and
  Lamias, Mark J. 2001. "Web Survey Design and
  Administration," Public Opinion Quarterly, 65
  (2) 230-253.
• Covey, Denise Troll. . 2002. Usage and
  Usability Assessment Library Practices and
  Concerns. CLIR Publication 105. Washington DC
  Council on Library and Information Resources.
• http//www.clir.org/pubs/reports/pub105/contents.h
  tml
• Dillman, D.A. 2000 (December). Mail and
  Internet Surveys, The Tailored Design Method.
  2nd Ed. New York John Wiley Sons.
• Gunn, Holly. 2002. Web-based Surveys Changing
  the Survey Process. FirstMonday 7(12).
• http//firstmonday.org/issues/issue7_12/gunn/index
  .html
• LIBQUAL Spring 2004 Survey. 2004. Cook,
  Colleen, and others.
• http//www.libqual.org/documents/admin/ARL_Noteboo
  k2004.pdf
• Schonlau, Matthias Fricker Jr., Ronald D. and
  Elliott, Marc N. 2002. Conducting Research
  Surveys via E-Mail and the Web. Santa Monica,
  CA RAND.
• Tenopir, Carol, with the assistance of Brenda
  Hitchcock and Ashley Pillow. 2003 (August). Use
  and Users of Electronic Library Resources An
  Overview and Analysis of Recent Research Studies.
  Washington DC Council on Library and
  Information Resources.
• http//www.clir.org/pubs/reports/pub120/contents.h
  tmls
• Thomas, Susan J. 2004. Using Web and Paper
  Questionnaires for Data-Based Decision Making
  From Design to Interpretation of the Results.
  Thousand Oaks, Corwin Press.
• Thompson, Bruce. Cook, Colleen. Thompson,
  Russell L. 2002. Reliability and Structure of
  LibQUAL scores Measuring Perceived Library
  Service Quality. portal Libraries and the
  Academy.3-12.

32
MINES for Libraries
OCUL MINES What do the data tells us?
Martha Kyrillidou Director of the ARL Statistics
and Measurement Program Association of Research
Libraries
http//www.minesforlibraries.org
www.arl.org/stats/
33
Questions Addressed By MINES for Libraries for
the OCUL Scholars Portal

• How extensively do sponsored researchers use
  OCULs Scholars Portal? How much usage is for
  non-funded research, instruction/education,
  student research papers, and course work?
• Are researchers more likely to use the Scholars
  Portal from inside or outside the library? What
  about other classifications of users?
• Are there differences in Scholars Portal based on
  the users location (e.g., in the library
  on-campus, but not in the library or
  off-campus)?
• Could MINES, combined with usage counts, provide
  an infrastructure to make Scholars Portal usage
  studies routine, robust, and easily integrated
  into OCULs administrative decision-making
  process for assessing networked electronic
  resources?

34
MINES for LibrariesTM Survey Form Five
Questions and a Comment Box
35
Analysis

• Web deliverables
• Crosstabulations in html for all OCUL data
• Interactive crosstabs for all OCUL and
  institutions
• Print deliverables
• summary tables for OCUL
• summary tables for each institution
• Final report

36
Add them up and down
37
(No Transcript)
38
OCUL Scholars Portal UsageAffiliation
39
(No Transcript)
40
(No Transcript)
41
(No Transcript)
42
Affiliation by Purpose of Use
  Purpose of Use Purpose of Use Purpose of Use Purpose of Use Purpose of Use Purpose of Use  
Affiliation Coursework Other Activities Other Research Patient Care Sponsored Teaching Total
Applied Sciences 24.0 7.6 17.7 0.6 46.3 3.7 100.0
Business 34.8 7.6 30.0 0.9 10.8 16.0 100.0
Education 40.9 5.4 17.1 0.8 11.8 24.0 100.0
Environmental Studies 43.5 2.5 24.0 0.3 23.3 6.3 100.0
Fine Arts 56.3 6.9 20.6 1.3 5.6 9.4 100.0
Humanities 51.5 10.8 21.0 0.5 9.5 6.7 100.0
Law 67.5 6.8 12.8 0.9 2.6 9.4 100.0
Medical Health 29.7 5.5 18.4 8.6 32.0 5.7 100.0
Other 51.9 22.8 10.9 2.1 7.4 5.0 100.0
Sciences 44.6 9.7 11.1 0.4 31.8 2.4 100.0
Social Sciences 62.6 4.5 14.4 0.7 13.6 4.2 100.0
Total 42.0 7.5 16.2 2.4 26.2 5.6 100.0
43
User Status by Purpose of Use
  Purpose of Use Purpose of Use Purpose of Use Purpose of Use Purpose of Use Purpose of Use  
User Status Coursework Other Activities Other Research Patient Care Sponsored Teaching Total
Faculty 1.5 4.7 21.2 4.4 42.6 25.6 100.0
Graduate Professional 19.5 3.9 25.5 2.5 45.4 3.2 100.0
Library Staff 23.5 24.1 13.1 16.5 17.7 5.2 100.0
Other 6.0 35.2 20.8 8.7 26.8 2.5 100.0
Staff 3.5 9.5 20.6 2.1 51.6 12.7 100.0
Undergraduate 75.8 7.8 7.7 0.9 5.9 1.9 100.0
Total 42.0 7.5 16.2 2.4 26.2 5.6 100.0
44
Location by Purpose of Use
  Purpose of Use Purpose of Use Purpose of Use Purpose of Use Purpose of Use Purpose of Use  
Location Coursework Other Activities Other Research Patient Care Sponsored Teaching Total
Library 52.8 14.9 10.8 1.2 12.3 7.9 100.0
Off-campus 47.2 7.0 17.3 4.1 19.9 4.6 100.0
On-campus 29.2 4.0 17.9 0.9 42.2 5.7 100.0
Total 42.0 7.5 16.2 2.4 26.2 5.6 100.0
45
Reason for Use
Reason for Use (n20293) Frequency Reason for Use (n20293) Percent
Important Journal 10219 Important Journal 50.4
Recommended Colleague 2436 Recommended Colleague 12.0
Reference/Citation 6090 Reference/Citation 30.0
Recommended Librarian 620 Recommended Librarian 3.1
Course Reading 925 Course Reading 4.6
Other 4388 Other 21.6
46
(No Transcript)
47
(No Transcript)
48
(No Transcript)
49

• How extensively do sponsored researchers use
  OCULs Scholars Portal? How much usage is for
  non-funded research, instruction/education,
  student research papers, and course work?
• MINES for Librariesshows that the Scholars
  Portal resources are heavily used by faculty and
  students in all OCUL. The majority of the use
  is from the sciences and the medical field and
  particularly in those fields the majority of the
  use is for sponsored research purposes.

50

• Are researchers more likely to use the Scholars
  Portal from inside or outside the library? What
  about other classifications of users?
• Most faculty, graduate professionals and
  undergraduates uses of the Scholars Portal are
  from outside the library building.
  Undergraduates though do show many uses of the
  Scholars Portal from within the library as they
  are probably becoming more exposed to these
  resources by having more physical contact with
  the library.

51

• Are there differences in Scholars Portal based on
  the users location (e.g., in the library
  on-campus, but not in the library or
  off-campus)?
• Most of the faculty and graduate professionals
  use Scholars Portal either from on-campus
  locations outside the library or from other
  off-campus locations. Most of the uses from
  these locations outside the library are for
  sponsored research purposes.

52

• Could MINES, combined with usage counts, provide
  an infrastructure to make Scholars Portal usage
  studies routine, robust, and easily integrated
  into OCULs administrative decision-making
  process for assessing networked electronic
  resources?
• Yes

53
Discussion

• How do we allocate expenditures for electronic
  resources?
• How do we allocate indirect costs for electronic
  resources?
• What is the appropriate balance between
  electronic and print?
• What is the appropriate balance between
  centralized and distributed purchasing?
• How are electronic resources affecting learning
  and research outcomes?

54
MINES for Libraries
OCUL MINES Possible Future Directions
Brinley Franklin Vice Provost for University
Libraries University of Connecticut
http//www.minesforlibraries.org
www.arl.org/stats/
55
OCUL MINES Possible Future Directions

• Individual Institutional Analysis
• Longitudinal Data Collection
• Beyond the Scholars Portal

56
OCUL Scholars Portal Users by Purpose of
Use(n20,300)
57
MINES for LibrariesTMLocation of Electronic
Resources Users
U.S. Main Libraries Total Usersn 25,698
U.S. Medical Libraries Total Users n 31,883
OCUL (Canada) Libraries Total Usersn 20,300
58
MINES for LibrariesTM Demographics by Location
of UserOntario Council of University Libraries
Inside the Library n 4,047
Off-Campus n 9,163
On Campus, Not in the Library n 7,090
59
Beyond the Scholars Portal

• DD/ILL
• ILLiad enable at the ILLiad logon screen
• Ask Reference
• Enable at the Ask Reference page or icon
• Digital libraries
• Represent an enormous investment
• Primary clientele is outside the library.
• Introduces non-authenticated group

60
Beyond the Scholars Portal

• Online catalog
• 856 field
• Serials solutions
• List of ejournals
• Referrer server
• Create a passthrough gateway
• Mirrored web server
• Drop in mirrored HTML page with survey links at
  survey period
• Mirrored HTML pages enabled by scripts

61
(No Transcript)
62
Beyond the Scholars Portal

• Because of the point of use requirement,
  libraries that have either a virtual gateway in
  library web architecture (or mount their own
  files like OCUL or OhioLink) succeed the best.
• Rewriting proxy server
• Database-to-web solutions
• Serials Solutions
• OpenURL solutions are a gateway.

63
Beyond the Scholars Portal

• Networked electronic resources are accessible
  from many different web pages and web servers
• Patrons bookmark networked electronic resources
  locally on their own workstations.
• Academic departments, librarian liaisons, anyone
  with a web page copies and pastes library links
  into their own web sites
• The survey data must be collected and
  commensurable for all networked electronic
  resources, including e-journals, e-books, online
  databases or traditional library request services
  offered in the online environment, such as
  Interlibrary Loan.
• The results of the survey have to be uninfluenced
  by caching issues, both local, web browser
  caching and proxy server or Internet Service
  Provider caching.
• The survey has to be meaningful for networked
  electronic resources, no matter how they were
  implemented.
• Different authentication methods have to be
  accommodated, whether the institution used IP,
  password, referring URL, or an authentication and
  access gateway.
• Remote usage has to be measured, regardless of
  the channel of communication, whether locally
  implemented proxy server, modem pool, or other
  institutional service.

64
September 25-27, 2006 Charlottesville, VA, USA
65
OCUL MINES Possible Future Directions
brinley.franklin_at_uconn.edu martha_at_arl.org http//
www.minesforlibraries.org http//www.statsqual.org