ACRL: Outcome Assessment Tools for the Library of the Future: MINES at OCUL

1
MINES for Libraries
ACRL Outcome Assessment Tools for the Library of
the Future MINES at OCUL
Toni Olshen York University
Association of College and Research
Libraries ACRL Conference 2005 Minneapolis April
7, 2005
www.arl.org/stats/
2
Ontario Council of University Libraries

• OCUL is a consortium of twenty university
  libraries in the province of Ontario
• The member libraries cooperate to enhance
  information services through resource sharing,
  collective purchasing, document delivery and many
  other similar activities.

3
OCUL Members Those in Green are ARL Libraries

• Queen's University
• Royal Military College of Canada
• Ryerson Polytechnic University
• University of Toronto
• Trent University
• University of Waterloo
• University of Western Ontario
• Wilfrid Laurier University
• University of Windsor
• York University

• Brock University
• Carleton University
• University of Guelph
• Lakehead University
• Laurentian University
• McMaster University
• Nipissing University
• Ontario College of Art Design
• University of Ontario Institute of Technology
• University of Ottawa

4
Member Institution Enrolments (2003)- Total
Undergraduate and graduate students

• University of Ontario Institute of Technology
  936
• Royal Military College of Canada
  1,941
• Ontario College of Art and Design
  3,062
• Nipissing University 5,478
• Lakehead University 7,304
• Trent University 7,388
• Laurentian University 8,751
• Wilfrid Laurier University
• 12,426
• Brock University 15,527

• University of Windsor 16,266
• University of Guelph 19,096
• Queen's University 20,034
• McMaster University 22,064
• Carleton University 22,535
• University of Waterloo 25,029
• Ryerson University 27,221
• University of Ottawa 30,948
• University of Western Ontario
• 
  32,784
• York University 46,794 
• University of Toronto 68,290
• Total 391,933
• 90 undergrads 10 graduate
• 12,500 faculty

5
Scholars Portal What is it?

• A unique set of shared information resources
• and services
• Resources acquired and managed through OCUL with
  funding support from a 5-year grant from the
  Ontario Innovation Trust (OII), a provincial
  funding body
• Resources are made available to researchers and
  students in Ontario through their own university
  libraries

6
Scholars PortalOntario Council of University
Libraries (OCUL)

• Ontario Information Infrastructure (OII) funded
  by the Ontario Innovation Trust in 2001 for five
  years
• Consortia-purchased electronic resources offered
  through the Ontario Scholars Portal
• March 2004, we began the evaluation phase of 7.6
  million dollar OII project

7
Scholarly Information Resources

• As of the end of March, contains 7,547,904 full
  text articles from 6,783 full text journals
  published by 12 academic publishers
• Coverage of most disciplines but concentration in
  sciences
• Current and historic coverage
• One of the largest collections of electronic
• journals available to researchers anywhere

8
Scholars Portal Resources

• Academic Press,
• American Psychological Association,
• American Chemical Society,
• Berkeley Electronic Press,
• Cambridge University Press,
• Emerald Publishing,
• Elsevier Science (Elsevier Science, Harcourt
  Health Sciences),
• Kluwer (Kluwer Academic Publishers, Kluwer Law
  International and Kluwer/Plenum),
• Oxford University Press,
• Project MUSE,
• Springer-Verlag, and
• John Wiley Sons.

9
Scholars Portal Project Goals

• Centrally mount and deliver information resources
  acquired through OCUL consortia purchases to
  ensure rapid and reliable access
• Provide for the long term, secure archiving of
  resources to ensure continued availability

10
Scholars Portal Project Goals

• Ensure that the resources and services provided
  meet the needs of faculty, students and staff.
• Ensure that resources and services can be
  seamlessly integrated to the local library and
  information systems

11
Measuring Success

• OCUL provides a sophisticated statistical report
  mechanism. (see next slide). Download statistics
  are a rough measure of value but we need more to
  properly assess impacts.
• Need to measure also the significance for
• research of access to e-journals
• Employing ARL MINES Survey methodology to
  capture information on how resources are being
  used (from where, by whom, and for what purposes)

12
SP Statistics and Report Generator
13
Why Evaluation?

• Feedback to OII and University funders
• Understand who, where, and why the digital
  resources are used
• Supplement usage numbers to answer the key
  question
• What is the impact of Portal content on research
  at Ontario academic libraries?

14
Evaluating Success

• Evaluating Scholars Portal from user and staff
  points of view
• Use a mix of quantitative and qualitative tools
  for a richer assessment MINES, focus groups,
  staff survey
• Are OII projects improving research services?
• Does Scholars Portal meet OCUL user and staff
  expectations?

15
MINES (Measuring the Impact of Networked
Electronic Services)

• MINES survey is one of a new breed of assessment
  tools that did not exist before because services
  were not digital. 

16
MINES (Measuring the Impact of Networked
Electronic Services)-Desired Outcomes

• To capture in-library and remote web usage of the
  Scholars Portal in a sound representative sample
  using MINES methodology
• To identify the demographic differences between
  in-house library users as compared to remote
  users by status of user

17
MINES (Measuring the Impact of Networked
Electronic Services)-Desired Outcomes

• To identify users purposes for accessing
  Scholars Portal electronic services (funded
  research, non-funded research, instruction/educati
  on use, student research papers and course work)
• To assist with the evaluation of the project as
  well as to capture information for OCUL about
  indirect research costs and
• To develop an infrastructure to make studies of
  patron usage of networked electronic resources
  routine, robust and integrated into the
  decision-making process.

18
MINES Methodology

• What user groups use SP?
• What specific resources are used?
• From where?
• How do users learn about SP?
• Are there differences in the use of digital
  resources based on the user's location?
• Why use SP? (sponsored research? Instruction?
  patient care?)
• Does use differ by discipline? user group?
  location?

19
MINES Methodology

• Web-based surveys conducted over the course of a
  year for each institution
• Activated during randomly selected 2-hour survey
  periods each month as users access one of SPs
  journals
• Mandatory, short, and anonymous

20
ARL/MINES Jan. 04-Dec. 05

• ARL developed random schedule of two-hour
  sessions per month
• OCUL designed local questions, mounted survey,
  collects and sends data to ARL
• ARL compiles survey results for all sites
• ARL reports findings on a semi-annual basis
• ARL presents findings and final report to project
  participants on an aggregated and individual
  institution basis

21
Development of survey form

• Finding balance between simplicity, ease and
  richness of data elements
• Bilingual University of Ottawa, Laurentian
  University, Glendon College at York University
• Ultimately a change in focus to the creation of a
  unique data set

22
MINES Survey Form Five Questions and a Comment
Box
23
Survey Form

• Survey form determined
• users status
• Discipline (affiliation)
• location or where accessed from
• purpose of use (sponsored research, instruction,
  patient care, course work)
• how the resource was identified (bibliography,
  colleague, librarian, important journal in field
  etc.)

24
OCUL Definition of Usage for MINES

• A successful search connecting the user to an
  article of interest for viewing, printing or
  downloading
• Unique to Scholars Portal because of consortia
  server setup and archiving of all journals

25
MINES Methodology

• Random sampling plan and the mandatory nature of
  the questions are both required to create a
  statistically sound study
• If the survey is not mandatory, the group of
  non-respondents is likely to be different from
  the group of respondents, and we will not know
  what that difference is
• One of the strengths and innovations of this
  survey technique is that it is based upon actual
  use, not on predicted, intended, or remembered
  use

26
OCUL Implementation of MINES

• Once the survey is completed, the respondent's
  browser is forwarded to the desired networked
  electronic resource
• If more than one search is carried out, the
  survey form is auto-populated with users
  responses as defaults which only have to change
  if response is different

27
Informed Consent

• Because this is a Web-based survey, the
  respondents consent to participate by electing to
  fill out the survey questionnaire
• It is the participating librarys responsibility
  to provide an explanation of the survey and
  information pertaining to its confidentiality

28
Confidentiality of Data

• Institutional data are confidential. Individual
  institutions and/or their specific data will not
  be identified.
• Individual data are anonymous. The respondents
  privacy is protected because only very indirect
  information is captured, which would be difficult
  to trace back to an individual.

29
Ethics Review

• A major step was contacting research ethics
  officers and/or Ethics Review Boards to get
  approval, where necessary, to run the survey
• Purpose of ethics reviews for human subjects is
  to prevent putting subjects at risk
• Officers/Boards on 16 OCUL campuses accepted that
  no physical or psychological harm would come to
  library users who are asked to fill out a brief
  mandatory anonymous survey before they are
  connected to the title of their choice.

30
Ethics Review

• Reference to interesting opinion piece by J. Paul
  Grayson. How Ethics Committees are Killing
  Survey Research on Canadian Students. University
  Affairs, January 2004.
• http//www.universityaffairs.ca/issues/2004/jan/pr
  int/opinion.html

31
Mandatory Survey

• If individuals chose to avoid filling out the
  brief anonymous survey, they might be
  inconvenienced for a maximum of a two-hour
  period, but they would not be harmed
• We needed to balance good data for making
  decisions and the inconvenience caused to the
  user.

32
Ethics Review Issues and Problems

• Mandatory nature of the survey required
  discussion on some campuses
• Several campuses did not require approval because
  the survey fell into quality assurance guidelines
  and was seen as a library management tool (8)
• Several schools received approval after an
  application process (8)
• One Library and Review Board did not support the
  mandatory nature of the methodology so that
  school dropped out of the project.

33
Pre-testing and False start January March 2004

• ARL prepared a schedule for the random two-hour
  monthly runs.
• A test run was planned at York and Wilfrid
  Laurier in January with the real survey
  commencing at the end of February.
• The pilot in January failed at York and
  highlighted the need for all institutions to be
  using a link resolver URL when connecting to SP
  journals from their catalogues or eResources
  databases.
• Each site reviewed their configuration and
  necessary changes were made.

34
Pre-testing and False start January March 2004

• Survey form and the explanatory material were
  translated into French for bilingual Ottawa,
  Laurentian, and Glendon College at York.
• February run highlighted concerns about the data
  collection.
• The technical infrastructure was capturing only
  access through library catalogues or eresource
  databases, but not from the use of the SP
  directly.
• There were some technical problems with the
  February and March runs and the validity of the
  data was under question. The data-collection
  programming was revisited.

35
Lessons Learned

• Early runs taught us a great deal about the
  different ways OCUL libraries access the SP
• We needed to reflect that in the data gathering

36
Lessons Learned

• As originally planned, we now capture as much
  usage as possible that comes from
• local eresource databases
• library catalogues
• Scholars Portal browse and search functions.

37
New Definition of Usage for MINES

• A successful search is now defined as connecting
  the user to an article of interest for viewing,
  downloading or printing
• Definition is unique to Scholars Portal because
  of consortial server setup and archiving of
  content
• We cancelled the April 20 run and reset the dates
  of the survey from May 2004 through April 2005,
  considering the February and March runs as tests.
  

38
New Definition of Usage for MINES - Innovation

• We continue to build on the unique opportunity we
  have to gather useful data that is not open to
  other types of library groups. By the end of
  March about 22,500 surveys have been completed.
  One more month to go!
• By implementing the MINES survey, OCUL is ahead
  of other projects in that we are not held
  "hostage" to the limitations and inconsistencies
  of vendor statistics
• We have opportunities to disseminate research on
  measurement of networked resources through
  conferences and publications

39
MINES Very Preliminary Output MAY AUGUST 2004
5223 respondents
40
Very Preliminary Findings 4 months of data
-Subject Affiliation

• Applied Sciences
• Business
• Education
• Environmental Studies
• Fine Arts
• Humanities
• Law
• Medical Health
• Sciences
• Social Sciences
• Other

• 804 17.5
• 146 3.2
• 176 3.8
• 160 3.5
• 22 .5
• 93 2.0
• 21 .5
• 1341 29.2
• 1031 22.4
• 673 14.6
• 129 2.8

41
Very Preliminary Findings 4 months of dataUser
Status

• Faculty
• Graduate/Professional
• Undergraduate
• Library Staff
• Staff
• Other

• 764 16.6
• 2068 45.0
• 1039 22.6
• 47 1.0
• 427 9.3
• 251 5.5

42
Very Preliminary Findings 4 months of data -
Location

• Library
• Off-Campus
• On-Campus ( but not in the library)

• 578 12.6
• 1978 43.6
• 2040 44.4

43
Very Preliminary Findings-4 months of data -
Purpose of Use

• Sponsored research
• Other non-sponsored research
• Teaching
• Course work
• Patient care
• Other activities

• 2189 47.6
• 919 20.0
• 278 6.0
• 686 14.9
• 143 3.1
• 381 8.3

44
Cross Tabulations

• Purpose of use by affiliation, user status,
  location, why
• Location by affiliation, user status, purpose of
  use, why
• Why by affiliation, user status, location,
  purpose of use
• Which titles used by which users for which
  purposes

45
Location and Purpose of Use
46
Additional Qualitative Data

• MINES Survey respondent comments
• Staff Survey What does the range of
  institutional experiences reveal?
• Focus Groups What anecdotal data can faculty and
  students add to the development of the Scholars
  Portal?

47
Thank you for your attention!

• Questions?
• Toni Olshen tolshen_at_yorku.ca