Ottawa, Canada

1
Quantitative and Qualitative Methods for
Assessingand Improving Outcomes and Service
Quality

• Ottawa, Canada
• June 14, 2006
• Presented by
• Dr. Colleen Cook, Dean
• Texas AM University

2
Why Assess?

• In an age of accountability, there is a pressing
  need for an effectiveprocess to evaluate and
  compare research libraries.
• 700 participants in LibQUAL
• 123 Association of Research Libraries (ARL)
  alone, over 3.4 billion dollars were expended in
  2003/2004

Note. M. Kyrillidou and M. Young. (2005). ARL
Statistics 2003-04. Washington, D.C. ARL, p.5.
3
Libraries Remain a CredibleResource in 21st
Century

98 agree with statement, My library contains
information from credible and known sources.
Note. Digital Library Federation and Council on
Library and Information Resources. (2002).
Dimensions and Use of the Scholarly Information
Environment.
4
Changing Behaviors

Recent Survey Only 15.7 agreed with the
statement The Internet has not changed the way I
use the library.
Note. Digital Library Federation and Council on
Library and Information Resources. (2002).
Dimensions and Use of the Scholarly Information
Environment.
5
Faculty Dependence onElectronic Resources Will
Increase
I will become increasingly dependent on
electronic research resources in the future.
http//www.arl.org/arl/proceedings/144/guthrie_fil
es/guthrie.ppt
6
Research BehaviorPersonal Control

When searching for print journals for research

• Only 13.9 ask a librarian for assistance
• Only 3.2 consider consulting a librarian a
  preferred way of identifying information

Note. Digital Library Federation and Council on
Library and Information Resources. (2002).
Dimensions and Use of the Scholarly Information
Environment.
7
Total Circulation
Note. M. Kyrillidou and M. Young. (2005). ARL
Statistics 2003-04. Washington, D.C. ARL, p.6.
8
Reference Transactions
Note. M. Kyrillidou and M. Young. (2005). ARL
Statistics 2003-04. Washington, D.C. ARL, p.6.
9
Web UsageTotal File Requests - UT Austin
Libraries 2000-2003
10
Enter LibQUAL

• The necessity of assessment
• Rapid shifts in information-seeking behavior
• The reallocation of resources from traditional
  services and functions

11
The Challenge ofAssessment in Libraries

• Traditional statistics emphasize inputs,
  expenditures, acquisitions, holdings, etc.
• Helping funding agencies understand success of
  investment
• No demonstrable relationship between expenditures
  and service quality
• Lack of metrics describing outcomes success
  from the users point of view
• Need to redesign library services to better meet
  changing patterns of use
• Building the climate, tools, and skill set for
  library assessment

12
ARL New MeasuresInitiative

• Collaboration among member leaders with strong
  interest in this area
• Specific projects developed with different models
  for exploration
• Intent to make resulting tools and methodologies
  available to full membership and wider community

13
LibQUAL Goals

• Improve mechanisms and protocols for evaluating
  libraries
• Develop web-based tools for assessing library
  service quality
• Identify best practices in providing library
  service
• Support libraries seeking to understand changes
  in user behavior
• Assist libraries seeking to re-position library
  services in the new environment

14
LibQUAL Outcomes

• Securing information that contributes
  meaningfully to planning and improvement efforts
  at a local level
• Providing analytical frameworks that
  institutional staff can apply without extensive
  training or assistance
• Helping decision-makers understand success of
  investments
• Finding useful inter-institutional comparisons

15
The LibQUAL Premise
PERCEPTIONS SERVICE

• .only customers judge quality
• all other judgments are essentially
• irrelevant

Note. Zeithaml, Parasuraman, Berry. (1999).
Delivering quality service. NY The Free Press.
16
(No Transcript)
17
76 Interviews Conducted

• York University
• University of Arizona
• Arizona State
• University of Connecticut
• University of Houston
• University of Kansas

• University of Minnesota
• University of Pennsylvania
• University of Washington
• Smithsonian
• Northwestern Medical

18
(No Transcript)
19
(No Transcript)
20
Dimensions ofLibrary Service Quality
21
Affect of Service

• I want to be treated with respect. I want you
  to be courteous, to look like you know what you
  are doing and enjoy what you are doing. Dont
  get into personal conversations when I am at the
  desk.
• Faculty
  member

22
Library as Place

• One of the cherished rituals is going up the
  steps and through the gorgeous doors of the
  library and heading up to the fifth floor to my
  study. I have my books and I have six million
  volumes downstairs that are readily available to
  me in an open stack library.
• Faculty
  member

23
Library as Place

• I guess youd call them satisfiers. As long as
  they are not negatives, they wont be much of a
  factor. If they are negatives, they are a big
  factor.
• Faculty
  member

24
Information Control

• first of all, I would turn to the best search
  engines that are out there. Thats not a person
  so much as an entity. In this sense, librarians
  are search engines just with a different
  interface.
• Faculty
  member

25
Information Control

• By habit, I usually try to be self-sufficient.
  And Ive found that I am actually fairly
  proficient. I usually find what Im looking for
  eventually. So I personally tend to ask a
  librarian only as a last resort.
• Graduate student

26
Multiple Methodsof Listening to Customers

• Transactional surveys
• Mystery shopping
• New, declining, and lost-customer surveys
• Focus group interviews
• Customer advisory panels
• Service reviews
• Customer complaint, comment, and inquiry capture
• Total market surveys
• Employee field reporting
• Employee surveys
• Service operating data capture
• A SERVQUAL-type instrument is most suitable for
  these methods

Note. A. Parasuraman. The SERVQUAL Model Its
Evolution And Current Status. (2000). Paper
presented at ARL Symposium on Measuring Service
Quality, Washington, D.C.
27
LibQUAL Resources

• An ARL/Texas AM University joint developmental
  effort based on SERVQUAL.
• LibQUAL initially supported by a 3-year grant
  from the U.S. Department of Educations Fund for
  the Improvement of Post-Secondary Education
  (FIPSE)
• Initial project established a expert team,
  re-grounded SERVQUAL concepts, and designed
  survey methodology
• Survey conducted at over 500 libraries resulting
  in a data base of over half a million user
  responses
• NSF funded project to refocus LibQUAL on the
  National Science Digital Library (NSDL)

28
World LibQUAL Survey 2005
29
Rapid Growth in Other Areas

• Languages
• American English
• British English
• French
• Dutch
• Swedish
• In development
• Chinese
• Greek
• Spanish
• German
• Consortia
• Each may create 5 local questions to add to their
  survey

• Types of Institutions
• Academic Health Sciences
• Academic Law
• Academic Military
• College or University
• Community College
• European Business
• Hospital
• Public
• State
• Countries
• U.S., U.K., Canada, the Netherlands, South
  Africa, Sweden, France, Australia, New Zealand,
  Malaysia

30
22 items
31
Survey Instrument
32
And a Box

• Why the Box is so Important
• About 40 of participants provide open-ended
  comments, and these are linked to demographics
  and quantitative data.
• Users elaborate the details of their concerns.
• Users feel the need to be constructive in their
  criticisms, and offer specific suggestions for
  action.

33
Reliabilityalpha By Language
By Language Service Info. Lib as Group n
Affect Control Place TOTAL American
(all) 59,318 .95 .91 .88 .96 British (all)
6,773 .93 .87 .81 .94 French (all)
172 .95 .90 .89 .95
34
Reliability alpha by University Type
By University Type Service Info. Lib as Group
n Affect Control Place TOTAL Comm Colleges
4,189 .96 .92 .89 .97 4 yr Not ARL 36,430 .95 .91
.88 .96 4 yr, ARL 14,080 .95 .90 .87 .96 Acad
Health 3,263 .95 .92 .90 .96  
35
Validity Correlations
Validity Correlations
Serv_Aff Info_Con LibPlace TOTALper Serv_Aff 1.
0000 .7113 .5913 .9061 Info_Con .7113 1.0000
.6495 .9029 LibPlace .5913 .6495 1.0000 .8
053 TOTALper .9061 .9029 .8053 1.0000 ESAT_TO
T .7286 .6761 .5521 .7587 EOUT_TOT .5315 .6
155 .4917 .6250  
36
Understanding LibQUAL Results

• Measures the distance between minimally
  acceptable and desired service quality ratings
• Perception ratings ideally fall within the Zone
  of Tolerance

37
Key to Bar Charts
38
LibQUAL 2004 Summary Collegesor Universities
American English
(n 69,449)
39
Score Norms

• Norm Conversion Tables facilitate the
  interpretation of observed scores using norms
  created for a large and representative sample.
• LibQUAL norms have been created at both the
  individual and institutional level

40
Institutional Norms for PerceivedMeans on 25
Core Questions
Note Thompson, B. LibQUAL? Spring 2002 Selected
Norms, (2002).
41
LibQUAL InteractiveInstitution Statistics
42
Adequacy GapThe difference between the minimum
and perceived score
LibQUAL
43
In Closing LibQUAL

• Focuses on success from the users point of view
  (outcomes)
• Demonstrates that a web-based survey can handle
  large numbers users are willing to fill it out
  and survey can be executed quickly with minimal
  expense
• Requires limited local survey expertise and
  resources
• Analysis available at local and
  inter-institutional levels
• Offers many opportunities for using demographics
  to discern user behaviors

44
LibQUAL Resources

• LibQUAL Website http//www.libqual.org
• Publications http//www.libqual.org/publication
  s
• Events and Training http//www.libqual.org/even
  ts
• LibQUAL Bibliography http//www.coe.tamu.edu/
  bthompson/servqbib
• LibQUAL Procedures Manual http//www.libqual.o
  rg/Information/Manual/index.cfm

45
LibQUAL Contact Information

• MaShana Davis
• Technical Communications Liaison
• mashana_at_arl.org
• Richard Groves
• Customer Relations Coordinator
• richard_at_arl.org
• Mary Jackson
• LibQUAL Services Manager
• richard_at_arl.org
• Martha Kyrillidou
• Director, ARL Statistics and Service Quality
  Programs
• martha_at_arl.org

woof
46
(No Transcript)