Beyond LibQUAL

1

• Beyond LibQUAL Effective, Sustainable, and
  Practical Assessment
• Other tools ClimateQUAL, MINES for Libraries,
  annual statistics
• Other approaches Return on Investment, Value
  Impact
• Martha Kyrillidou, Glasgow, May 24 2010

2
ARL Mission

• Non-profit organization of the libraries of
  research institutions in North America
• Forum for exchange of ideas
• Agent for collective action

3
What makes a research Library?

• Breadth and quality of collections and services
• Sustained institutional commitment to the library
• Distinctive resources in a variety of media
• Services to the scholarly community
• Preservation of research resources
• Contributions of staff to the profession
• Effective and innovative use of technology
• Engagement of the library in academic planning
• --from ARL Principles of
  Membership

4

• What makes a quality research library?
• Quality much like beauty is in the eye of the
  beholder

5
Whats in a Library

• A word is not crystal, transparent and unchanged
  it is the skin of a living thought, and may vary
  greatly in color and content according to the
  circumstances and time in which it is used.
• --Justice Oliver Wendell Holmes

6
Thinking Strategically About Library Futures

• What is the central work of the library and how
  can we do more, differently, and at less cost?
• What important set of services does the library
  provide that others cant? What new roles are
  needed?
• What advantages does the research library
  possess?
• What will be the most needed by our community of
  users in the next decade? How is user behavior
  changing?
• What should our libraries aspire to be ten years
  from now? What are the implications of technology
  driven change?
• What are the essential factors responsible for
  the success of the library?

www.arl.org
7
Assessment at ARL

• A gateway to assessment tools ARL StatsQUAL
• ARL Statistics -- E-Metrics
• LibQUAL
• ClimateQUAL
• DigiQUAL
• MINES for Libraries
• Library Assessment Conferences
• Service Quality Evaluation Academy
• Library Assessment blog
• Effective, Sustainable, Practical Assessment
• Profiles Scenarios . Balanced Scorecard .
  LibValue

www.arl.org
8
ARL Tools for Library Assessment

• As we look to the future

Users
StatsQUAL

• Common User Interface
• Unified Data Structure
• Enhanced Data Mining/Warehousing Opportunities
• Faster Development Cycle for New Tools
• Common Workflow

9
ARL Tools for Library Assessment

• As a result of the work of the New Measures and
  Assessment Initiative (1999)

MINES for Libraries Since 2003
LibQUAL Since 2000
DigiQUAL Since 2003
ClimateQUAL Since 2007
ARL Statistics Since 1907-08
10
(No Transcript)
11
Partners

• 2008
• Arizona State University
• Cornell University
• Duke University
• Emory University
• Kansas State University
• New York University
• Northwestern University
• University of Houston
• University of Maryland
• University of Massachusetts, Amherst

• 2007
• Texas AM University
• University of Arizona
• University of Connecticut
• University of Iowa
• University of Kansas

• 2009
• George Mason University
• Illinois State University
• Johns Hopkins University
• Oberlin College
• University of California, Berkeley
• University of Hawaii at Manoa
• University of Nebraska-Lincoln
• University of Wyoming

12
Goals Research Question for Project

• Develop a tool that assesses the health of your
  library.
• Whether the policies, procedures and practices of
  your library are supporting your mission and
  facilitating meeting current and future customer
  needs.
• Develop large database of norms to help libraries
  interpret their results.
• Develop an active community of libraries that
  share experiences and interventions to improve
  the effectiveness of libraries participating in
  community
• Track changes in libraries over time to permit
  objective feedback regarding effectiveness of
  attempted practices/interventions.
• Empirical validation of the healthy organization
  theory.

13
Core Concepts

• Climate for Diversity
• Climate for Fairness
• Distributive Justice
• Procedural Justice
• Informational Justice
• Interpersonal Justice
• Climate for Innovation
• Climate for Continual Learning
• Climate for Teamwork
• Climate for Psychological Safety
• Climate for Customer Service
• Immediate Supervisor Scales
• Authentic Leadership Leader-Member Relationship
  Quality
• Work Group Conflict
• Worker Attitudes

14
2000 U. Maryland Survey

• Transformative Changes
• Holding All-Staff Meetings
• Emphasis on learning
• Creating training for supervisors
• Created new position - Coordinator of Personnel
  Programs
• Focus on team building

• Results
• Issues identified
• Managerial training
• Mentoring
• Standardization of procedures
• Decision-making practices, etc.
• Recruitment/selection of ethnic
• minorities
• Performance review
• Rewards and recognition

15
2004 U. Maryland Survey

• Transformative Changes
• Conducted focus groups within two divisions
• Groups and teams identifying strategies
• Developing supervisor core competencies
• Incorporating Organizational Citizenship
  Expectations into work plans
• Beginning workforce planning

• Results
• Support of diversity and communication practices
• Employees feel they are treated fairly
• Teamwork is valued
• Increase in Organizational Withdrawal from 2000
  to 2004
• Indication of consistent ethnic and divisional
  differences

16
Community Building

• No one-size-fits-all method of interpreting
  results or implementing changes
• Creation of intimate community of participants
• In-person events, an online shared workspace, and
  conversations
• Allows for shared experiences concerning
  improvement strategies

17
Assessing the Value of Networked Electronic
Services
The MINES survey

Measuring the Impact of Networked Electronic
Services (MINES) - MINES for Libraries
www.arl.org/stats/newmeas/mines.html
18
What is MINES?

• A research methodology consisting of a web-based
  survey form and a sampling plan.
• Measures who is using electronic resources, where
  users are located at the time of use, and their
  purpose of use.
• Adopted by the Association of Research Libraries
  (ARL) as a part of the New Measures toolkit
  May, 2003.
• Different from other electronic resource usage
  measures that quantify total usage (e.g.,
  COUNTER, EQUINOX, E-Metrics, ICOLC guidelines,
  ISO and NISO standards) or measure how well a
  library makes electronic resources available
  (LibQUAL, DigiQUAL).

19
Questions Addressed

• How extensively do sponsored researchers use the
  new digital information environment?
• Are researchers more likely to use networked
  electronic resources from inside or outside the
  library?
• Are there differences in usage of electronic
  information based on the users location (e.g.,
  in the library on-campus, but not in the
  library or off-campus)?
• What is a statistically valid methodology for
  capturing electronic services usage both in the
  library and remotely through web surveys?
• Are particular network configurations more
  conducive to studies of digital libraries patron
  use?

20
Effective, Sustainable and Practical Assessment

• Association of Research Libraries (ARL) project
  Making Library Assessment Work began in 2005
  to Assess the state of assessment efforts in
  individual research libraries, identify barriers
  and facilitators of assessment, and devise
  pragmatic approaches to assessment that can
  flourish in different local environments
• Funded by participating libraries
• Conducted by Steve Hiller and Jim Self
• 1.5 day site visit
• Presentation and best practices
• Interviews and meetings
• Report to the library with recommendations

21
Library Assessment Process

• Focuses on customer needs, defining measurable
  outputs and offering services that meet those
  needs
• Collects, analyzes and uses data for management,
  program development, and decision-making
• Emphasizes ongoing communication with customers,
  opportunities for collaboration, qualitative
  measures and circular process of continuous
  improvement

22
MLAW/ESPData Collection Methods

• Pre-Visit
• Survey on assessment activities, needs etc.
• Telephone follow-up
• Mining library and institutional web pages
• Visit (1.5 days)
• Presentation on effective assessment
• Group meetings
• Follow-up and report
• Pursue leads and additional information

23
Commonly Identified Assessment Needs (30
Libraries)
24
Organizational Indicators of Effective
Assessment

• Library leadership/management truly supportive
• Customer focus is a shared library value
• Organizational culture receptive to change
  improvement
• Assessment responsibility recognized and
  supported
• Library has strategic planning process and
  prioritizes
• Evidence/Data used to improve services/programs
• Web sites (usability)
• Facilities (qualitative methods)
• Serial subscriptions (emetrics)
• LibQUAL results are followed-up

25
Building a Community of Practice

• Biennial Library Assessment Conference
• 220 registrants for 2006 conference in
  Charlottesville, VA
• 380 registrants for August 2008 in Seattle, WA
• Workshops
• Biennial Service Quality Evaluation Academy
• Full day and half day workshops
• Library Assessment SPEC Kit (December 2007)
• SCONUL/ARL collaboration for UK and Irish
  libraries
• Assessment tools
• LibQUAL (Millions served)
• MINES for Libraries
• ClimateQUAL, DigiQUAL and more

26
Library Assessment Conference
27
Library Strategy Maps as of April 14, 2009
28
The intellectual crossroads where research and
teaching, tradition and innovation, faculty and
students all intersect.
Facilitate world-class research, teaching and
learning through leadership in the creation,
access, use, and preservation of information and
knowledge for scholars at the University of
Virginia
Customer Needs
Internal Processes
Marketing Communications
Library Services
Operational Efficiency
Develop effective digital services
Prioritize strategic programs and adjust budget
and staffing accordingly
Promote Library as intellectual crossroads
Align administrative, technical, and public
service priorities
Strategic Skills / Workforce
Recruit, develop, and retain productive, highly
qualified staff
Learning And Growth
29
UW Strategy Map Based on Strategic Plan
Mission
Enriching the Quality of Life and Advancing
Intellectual Discovery by Connecting People with
Knowledge
Customer Perspective
Financial Perspective
Strengthen digital and physical delivery services
Reshape library spaces
Optimize organizational structure and efficiency
Enhance undergraduate services
Demonstrate library value to research
productivity
Internal Perspective
Build a management information structure
Communicate and market library services
resources
Provide access to information resources for
world-class TLR
Develop and promote scholarly communication plan
Integrate resources services into user
environments
Strengthen library liaison program
Assess effectiveness of services programs
Marketing Communications
Library Services
Operational Excellence
Create a workplace of choice
Develop staff expertise that supports current
future TLR
Provide technological infrastructure to support
staff and user needs
Learning and Growth Perspective
Culture
Infrastructure
Strategic Skills
30
Strategy Map
Mission advance teaching, learning and research
at McMaster by teaching students to be
successful, ethical information seekers,
facilitating access to information resources,
providing welcoming spaces for intellectual
discovery and promoting the innovative adoption
of emerging learning technologies
Customer
Improve discovery of and access to scholarly
resources
Create world-class teaching learning
environments
Strive for exemplary service that is responsive
to user needs
Internal Processes
Marketing and Communications
Operational Effectiveness
Services
Enhance the Librarys commitment to strategic
planning
Integrate the Library into the Universitys
teaching, learning, and research mission
Promote the Librarys role in the discovery,
dissemination, and preservation of knowledge
Track efficiency and effectiveness of Library
programs and services
Learning Growth
Develop highly-trained, technologically-fluent
superlative staff
Nurture a healthy, collaborative, and dynamic
organization
Grow an evidence-based culture
Encourage innovation and risk taking
Finances
Increase alternate sources of revenue e.g.
fundraising, grants and revenue generation
Align the Librarys budget with the Universitys
mission
31
JHU Library Strategy Map
Mission To advance research, teaching, learning
at Johns Hopkins University by providing relevant
and significant collections, services, and
instruction for faculty, students, and staff.
The Sheridan Libraries through the University
Libraries Council partners with other Johns
Hopkins Institutions libraries to maximize
support for our constituents while minimizing
costs.
Customer Needs
Financial Perspective
Ensure fast, easy access to resources and
services needed for research and teaching
Provide productive, user-centered workspaces
(virtual and physical)
Develop strong support base
Minimize costs
Internal Processes
Communications
Programs
Operational Efficiency
Integrate and strengthen instructional and
research support services and programs
Identify and improve processes and infrastructure
with greatest impact
Promote resources and services
Expand and preserve access to information
including digital library development
Create productive research, learning, study, and
social space
Learning And Growth
Recruit, develop, and retain productive highly
qualified staff
Promote an adaptive, collaborative, diverse,
engaged, innovative workplace
Create a culture of assessment and accountability
32
ARL Profiles

• Describes the here and now and strategies
• 86 qualitative descriptions of research libraries
• Five pages each
• Address issues related to collections, services
  and collaborative relations
• Preliminary report available directors.arl.org
• Working on identifying key areas and defining an
  ARL Statistics 2.0 survey

33
ARL Scenarios

• Process led by Karla Hahn, Associate Executive
  Director for Transforming Research Libraries,
  with external consultant
• Interview and focus groups on out of the box
  thinking attempting to describe different
  scenarios of the future of research libraries in
  20 years or so
• A visioning tool
• FAQ on ARL website

34
IMLS Lib-Value Grant

• Partner Institutions U of Tennessee, U of
  Illinois at Urbana-Champaign and ARL
• Primary Goals
• Establish a fact-based articulation of the value
  and ROI of the university library resources and
  services within the wider mission of university
  administration.
• Develop a model for ROI and tools that implement
  this model which can be used by other academic
  libraries.

35
Impact

• More informed librarians who understand how to
  conduct and present ROI studies for their
  libraries and who can measure all aspects of
  library services and collections
• Improved understanding by university library
  funders of the ROI and value of university
  libraries to the overall mission of the
  university and what products and services are the
  best investments

36
Introductions - Management Team

• Carol Tenopir UT (ctenopir_at_utk.edu)
• Paula Kaufman UIUC (ptk_at_uiuc.edu)
• Martha Kyrillidou ARL (martha_at_arl.org)
• Donald King - UNC (donaldwking_at_gmail.com)
• Bruce Kingma -Syracuse (brkingma_at_syr.edu)
• Tina Chrzastowski UIUC (chrz_at_illinois.edu)

• Gayle Baker UT (gsbaker_at_utk.edu)
• Ken Wise UT (kwise_at_utk.edu)
• Rachel Fleming-May UT (rfmay_at_utk.edu)
• Regina Mays UT (rmays_at_utk.edu)
• Crystal Sherline UT (csherli1_at_utk.edu)
• Andrea Baer UT (abaer1_at_utk.edu)

37
Overview of Functional Areas
38

All Stakeholders considered -- emphasis on
faculty and students)
39
Overview of Sub-Areas
40
June Workshops

• In Washington DC .
• Return on Investment Lib-Value with Paula
  Kaufman, Don King and Bruce Kingma
• Value and Impact with Stephen Town

41
Thank You