Evaluation Chat Reference Service from User

1
Evaluation of a Chat Reference Service from the
Users Perspective
Lili Luo Jeffrey Pomerantz School of Information
Library Science UNC Chapel Hill ltluo,
pomerantzgt_at_unc.edu
Tapping the vast reservoir of human knowledge
--Louis Round Wilson, founder, 1931
2
Brief Outline
Research Context NCknows Research Questions
Theoretic Framework Study Design
Methodology Results Discussion Future Work
3
NCknows

• Collaborative state-wide chat service in North
  Carolina (www.ncknows.org)
• 18-month pilot phase launched in February 2004,
  with 19 libraries participating.
• Librarians in all 19 libraries staffed the
  service during the pilot.
• Participants public, research university,
  community college, government libraries urban
  and rural, large and small.
• Pilot phase is completed. Service had its final
  launch in July 2005.

4
Evaluation

• 4 Components to the Evaluation
• Baseline Statistics from the 24/7 reference chat
  application
• Quality control Peer reviews of chat transcripts
• Administration Interviews with librarians
• Use user satisfaction Exit surveys follow-up
  interviews

5
Research Questions
Motivation What motivates users to use a chat
reference service? Use How do users use the
information provided by the chat reference
service? Satisfaction What is the users level
of satisfaction with chat reference service?
6
Dervins Sense-Making Theory
7
Study Design
Exit Survey
an exit survey popped up for the user at the
conclusion of a chat session
Email Telephone Interviews
semi-structured interviews were conducted two to
three weeks after users chat sessions
8
Results
Prior to Use of the Service

• Use of Other Reference Services
• Discovery of NCknows
• Motivation for Using the Service
• Motivation for Asking the Question

The Point of Service

• Satisfaction Levels
• Role of User

Use of the Information Provided by the Service
9
How did you find out about the NCknows chat
service? (Follow-up interview - 68
responses) Four routes by which users learned
about the service emerged
1. The local library, or library-produced
publicity materials (69). 2. Online searching
(12). 3. School system (12). 4. Friends and
colleagues (7).
10
Have you ever used any of your librarys other
reference services? Which? (Follow-up interview -
67 responses)
Reference Services Respondents
desk reference service 48
e-mail reference service 19
telephone reference service 33
never used any other reference services 19
did not answer the question directly 7
librarians or library staff 7
11
Why did you use the chat service to get this
question answered rather than any of these other
services? (Follow-up interview - 68 responses)

• Convenience (47).
• Other means of seeking information were not
  helpful (15).
• Curiosity (13).
• Serendipity (12).
• Recommended by others (7).
• Personal characteristics/habits (7).
• Other reference services were not available
  (1.5).

12
Can you tell me about what motivated the question
that you asked the NCknows chat
service? (Follow-up interview - 72 responses)
To satisfy an inquiry about the library itself, 3
To help others look for information, 3
Others, 3
To locate a known item, 8
To answer a question arising from the users
personal life, 32
To answer a work/school related question, 51
13
Business-related Im a small business owner and
I need the information to negotiate with other
type of business people to do a joint
venture. School-related I was doing a
research paper on homeschooling. Want some hard
stats numbers from the website about how
homeschool students do academically. Genealogy
Wanted to find passenger ships of 1800 that had
been lost at sea, trying to find one that may
have been carrying wife and children of my great
grandfather. Vocational or academic
change doing research on school that meets the
criteria that Im looking for. I want to go back
to school. Known-item search I was looking
for an audio book and couldnt understand the new
system. On behalf of others It was my sons
schoolwork questions in Geography class. He had
found the answers to many of his geography
questions but we had searched and searched using
Google and also used Ask Jeeves but were not
successful in finding the answers to a couple of
questions.
14
How satisfied were you with the ? (Follow-up
interview - 72 responses)
Completeness of the answer Speed that the librarian answered your question Helpfulness of the librarian Ease of use of the software
Very satisfied 68.5 67.7 81.2 82.7
Satisfied 23.55 24.9 13.1 14.2
Dissatisfied 5.4 5.6 3.4 1.8
Very dissatisfied 2.6 2.1 2.4 1.3
Total n 387 381 382 387
15
Would you recommend this chat service to a
friend? (Exit survey - 385 responses)
Recommend Percentage
Very likely 87.5
Maybe 10.1
Unlikely 1.6
Never 0.8
Total 100
16
You asked this question in your role as
a(n) (Exit survey - 379 responses)
Role Percentage
Other 29.7
Student Undergraduate 23.2
Student Graduate 13.5
Student K-12 10.3
Parent 7.8
Librarian 7.7
Medical Professional 6.3
Teacher K-12 3.6
Higher Ed Faculty 3.5
Adult Educator 3.1
Administrator 2.7
Teacher Pre-school 0.8
Policymaker 0.0
Politician 0.0
17
Have you had a chance to use the information that
NCknows provided to you? (Follow-up interview -
67 responses)
18
Use It was utilized as background information
for my lecture. I used the information in the
negotiation with other business owners. Through
the knowledge gained, we have (hopefully) brought
another business to Fayetteville. At least we
are talking now. Partial Use NCknows told me
the person to call at my local library who said
they can't provide that information. I was going
to use it in my graduate school thesis.
Alas I am still working on this matter but
was provided good leads from your service. No
use The librarian did give me a website for on
loan tapes, but they took to long to
receive. I plan to call the number I got from
the service in a couple of time.
19
Discussion
Target User Groups Campaigns to market chat
reference services should be targeted to user
groups that are likely to have a need for the
service for example, in grade schools, colleges,
and universities businesses and other corporate
settings and social organizations.
Expand Service Hours The availability of the
service at all hours of the day and night is one
important factor in the convenience of the
service.
Identify Appropriate Reference Medium for
Users Encourage users to use a reference service
in the most appropriate medium. Research
question How to identify the most appropriate
medium for a user / for a question?
20
Limitations

• Non-response bias
• Self-selection bias

Future Work

• Methodological enhancements alternative subject
  recruiting methods
• Further the exploration of service value what
  users gain from using the information provided by
  the service

21
Questions Comments?

• ltluo, pomerantzgt_at_unc.edu