Evaluation of Digital Libraries

1
Evaluation of Digital Libraries

• Iris Xie

2
Evaluation Approaches

• Information Retrieval approach
• Library evaluation approach
• User-centered approach
• Interface evaluation

3
INFORMATION RETRIEVAL APPROACH

• PERFORMANCE MEASURES
• DEFINITIONS OF RELEVANCE USER-CENTERED VS SYTEMS
  ORIENTED DEFINITIONS
• Recall
• Precision

4
LIBRARY EVALUATION APPROACH

• COST-BENEFIT ANALYSIS
• BUDGET ANALYSIS
• PERFORMANCE EVALUATION
• COLLECTION EVALUATION

5
USER-CENTERED EVALUATION

• SUCCESS
• SATISFACTION
• USES AND USAGE
• INDIVIDUAL, SOCIAL, ECONOMIC IMPACT
• READING HABITS
• SCHOLARLY COMMUNICATION

6
Mental Models of USERS

• MENTAL MODELS OF SEARCHING (DANIELS ARTICLE,
  EXPANDING UNIVERSE TOOL TO SIMULATE
  CONVENTIONAL LIBRARY STRUCTURE)
• STUDIES OF ADAPTATION OF PRE-EXISTING MENTAL
  MODELS TO NEW SEARCHING ENVIRONMENT

7
HUMAN FACTORS STUDIES

• STUDY PEOPLES RESPONSES TO PROTOTYPE DIGITAL
  LIBRARIES
• PRIMARILY USABILITY ANALYSIS
• ITERATIVE DESIGN
• DESIGN, DEVELOP, TEST, EVALUATE, REDESIGN,
  RETEST

8
Interface Evaluation

• Database selection/classification of all the
  materials
• navigation
• Query formulations and reformulations
• Search features
• Browsing features
• Help system

9
Evaluation of a Statewide Digital Library

• State-wide initiatives to facilitate access to
  database vendor resources via Internet
• Available since Fall 1998 to residents of
  Wisconsin
• Provides web-based access to full text and
  bibliographic databases from EBSCOhost and
  ProQuest
• Accessible with dedicated access (known users)
  and dial-in access (unknown users)

10
Methodology

• EBSCO and Bell Howell databases use data
  provided by the vendors (January June 1999)
• 239 library responses towards the BadgerLink
  service collected by questionnaire
• 81 individual user responses towards the
  BadgerLink service collected by Web survey

11
Vendor Data

• Type of user usage
• Database usage
• Title usage
• Search and browsing behavior

12
Research Questions for Libraries

• How is the statewide digital library service
  used?
• Who are the users?
• Which factors impact how the service is used?
• What is the impact of the availability of the
  service on the usage of physical libraries within
  the state?

13
Model for Successful Implementation of State
Digital Library Service    
14
Research Questions for Individual Users

• Who are the most frequent users? And what are the
  primary BadgerLink access sites?
• How do users learn how to use BadgerLink? And
  what do they use Badgerlink for?
• Which database and title resources are most
  frequently accessed?
• What are users searching and browsing behaviors?

15
Further Research

• Statewide digital library design should satisfy
  diverse user needs
• target nonusers and infrequent users and
  determine their needs and attitudes
• Develop model for the successful implementation
  of general purpose digital library
• In-depth study of how individual users seek
  information in general purpose digital library
  for system design

16
EVALUATION ISSUES I

• WHAT ARE APPROPRIATE CRITERIA FOR EVALUATING DLS
  HOW CAN WE COMBINE IR LIBRARY MODELS
• SHOULD WE STUDY REAL USERS WITH REAL INFORMATION
  NEEDS?
• WHAT COMPONENT OF THE DL WILL BE EVALUATED?
  (INTERFACE COLLECTION ETC.)

17
EVALUATION ISSUES II

• HOW CAN WE EVALUATE INTERACTION IN DIGITAL
  LIBRARIES?
• INCORPORATING MULTIPLE METHODS OF EVALUATION
• HOW GENERALIZABLE ARE RESULTS?
• COORDINATING DEVELOPMENT AND DESIGN CYCLES