Needs Assessment and Program Evaluation: Keys to the UserCentered Library

1
Needs Assessment and Program EvaluationKeys to
the User-Centered Library

• Steve Hiller
• University of Washington Libraries
• April 6, 2000

2
SESSION GOALS

• Importance of assessment
• Choosing the right assessment method
• Analysis and presentation of assessment data
• Turning results into action

3
WHY DO ASSESSMENT?

• Make better informed decisions
• Improve programs and services
• Accountability to funding agencies
• Institutional/Program Accreditation
• Political benefits of user involvement

4
Developing a Culture of Assessment

• A Culture of Assessment is an environment in
  which decisions are based on facts, research and
  analysis, and where services are planned and
  delivered in ways which maximize positive
  outcomes and impacts for library clients. A
  culture of assessment is an integral part of the
  process of change and the creation of a
  user-centered library. (Betsy Wilson,
  University of Washington)

5
The User-Centered Library

• Focus explicitly on the information resources and
  services needed by users
• User is at the center of service and resource
  decisions
• Quality services and user satisfaction are goals
  shared by all library staff
• Personalized service which recognizes diverse
  user needs
• Operations organized to attain user-centered
  objectives
• Evaluation and assessment focused on user impacts
• User-centered not user driven

6
LISTENING TO OUR USERS
7
DIFFERENT USERS AND NEEDS

• Faculty
• Students
• Graduate/Professional
• Student
• Distance education
• Staff
• Visitors

• Research
• Teaching/learning
• Subjects/disciplines
• Format
• Onsite/Remote

8
STARTING ASSESSMENT

• Identify and prioritize assessment needs (you
  cant do everything)
• Importance
• Timeliness
• Feasibility
• Cost
• Select Appropriate Assessment Method
• Expertise available (local or external)
• Degree of user involvement wanted
• How will you use assessment information

9
SO MANY METHODSSO LITTLE . . . Time, money, staff

• Counts (manual and automated)
• Observations
• Interviews
• Logged activities
• Critical path analysis
• Usability
• Focus groups
• Surveys

10
CHOOSE THE RIGHT METHOD

• Appropriate
• Timely
• Cost effective
• Amount of staff support/training
• Degree of user involvement
• Representative population
• Results can lead to change

11
USE SEVERAL METHODS FOR MORE POWERFUL ASSESSMENT

• Methods may complement one another
• Follow up issues identified by another method
• Large projects may need different methods
• Provide three dimensional view of issues or users

12
UW LIBRARIES3 Assessment Methods Used Recently

• OBSERVATION
• Fast, Cheap, and in Control!
• FOCUS GROUPS
• Watch out below!
• SURVEYS
• Size Matters!

13
OBSERVATIONAL STUDIES

• Describe user activities in terms of
• what they do
• how they do it
• how much time they spend
• problems they encounter
• Can be obtrusive or unobtrusive
• Can be tied in with interviews or usability
• Well-developed data collection method is
  essential

14
OBSERVATIONAL STUDY

• USE FOR
• Time sensitive
• Low-cost support
• Reality check
• Help identify/define issues (including usability)

• BE AWARE OF
• Not representative
• Limited focus
• Data collection and analysis issues

15
OBSERVATIONAL STUDY

• Define issues
• Develop methodology
• Identify participants
• Record information
• Summarize results
• Report and recommend

16
OBSERVATIONAL STUDIES UW

• How do faculty find and use electronic journals?
  (1997)
• Observations/guided interviews at faculty office
• Six faculty members observed/interviewed
• Web Gateway Design (1998)
• Usability study
• Conducted in Technical Communications Usability
  Lab
• Public Workstation Use (1999)
• Observational sampling of more than 200 public
  workstations
• 3 time periods in one week period
• Workstation in use/not in use
• Category of use noted (library catalog, library
  licensed database, other library resource, other
  Web site, email, games)

17
OBSERVATIONAL STUDIESBottom Line

• Fast 2 weeks start to finish
• Cheap No direct costs
• Low staff involvement
• In Control One-on-one observation
• Scripted

18
Sometimes You Just Cant Tell By Looking
19
FOCUS GROUPS

• Planned discussion to obtain user perceptions and
  observations on a topic
• Usually composed of 6-10 participants and may be
  repeated several times
• Facilitator or moderator guides discussion
• Participants encouraged to share perspectives
• Participants learn from each other

20
FOCUS GROUPS

• USE FOR
• High user involvement
• Clarify issues
• User defined perspective
• Focus group bounce
• Intermediate time/cost
• Results can lead to use of other assessment
  methods

• BE AWARE OF
• External facilitator
• Not representative
• Complex logistics
• Wandering discussion
• Costly transcription
• Complicated analysis

21
UW FOCUS GROUPS 1998-2000

• Bioscience information needs 1998 (3 groups)
• Fine Arts service and space needs 1998 (2 groups)
• Interlibrary loan user needs 1999 (2 groups)
• Electronic reserves 1999 (3-4 groups)
• Faculty- where and how do they find information
  2000 (3 groups)
• Students - where and how do they find information
  2000 (5 groups planned)

22
FOCUS GROUP SESSIONS PROCESS

• Select topic
• Define issues covered
• Develop budget and timeline
• Choose facilitator
• Select participants

• Record session
• Transcribe discussion
• Analyze session
• Report results and recommendations

23
FOCUS GROUPSLogistics

• SESSIONS
• Offer more than 1 session if possible
• PARTICIPANT SELECTION
• 6-10 participants per session
• OK to ask for volunteers
• LOCATION/TIME
• Neutral site, convenient for participants
• Lunchtime or afternoon
• INCENTIVES Feed them and they will come
• Lunch
• Copy cards

24
FOCUS GROUPSRecording/Transcription

• AUDIO TAPE RECORDERS
• Unobtrusive (video can be obtrusive)
• Microphone placement
• May need two tape recorders
• OBSERVER
• Take notes (include names, emphasis etc.)
• TRANSCRIPTION
• Transcription device may help
• Lengthy and time consuming

25
FOCUS GROUPS Bottom Line

• TIME
• 2 months from start to finish
• COSTS (direct)
• 300 for 3 sessions
• lunch, recording equipment rental
• LOOK OUT BELOW!
• STAFF TIME
• 75-100 hours (including analysis)

26
USER SURVEYS

• Ask directly a series of questions dealing with
  use, needs, satisfaction etc.
• Large enough (and randomly selected) population
  or sample size can provide generalizeable results
  
• Results are quantifiable and can be analyzed
  using statistical techniques
• Qualitative information can also be obtained

27
USER SURVEYS

• DIFFERENT TYPES OF USER SURVEYS
• Mail
• Electronic/Web
• Telephone
• Site specific
• Spot

28
USER SURVEYSAdvantages

• Acquire both quantitative and qualitative
  information
• Generalize from sample population
• High degree of statistical correlation
• Analyze data for entire group, within group,
  between groups and over time
• Customize questions and length
• Can use as marketing tool
• Data gathered directly is more politically
  powerful
• ANTIDOTE FOR THE ANECDOTE

29
USER SURVEYSBe Aware Of

• Potentially long lead time needed
• Expense (direct and indirect costs)
• Coordination with external units
• Sample size (especially for subgroups)
• Statistical expertise needed
• Return rate - users may feel oversurveyed
• Data saturation and interpretation issues

30
USER SURVEYSQuestions Before You Begin

• What information do you need?
• Who do you need it from?
• How will you use it?
• Is a survey the best way to get it?
• How soon do you need the information?
• Who will do the work?
• What resources are available and/or needed?

31
USER SURVEYSGetting Ready

• Approvals (library, institution)
• Budget
• Timeline
• Staffing
• Logistics
• Communication

32
SIZE MATTERS!
33
USER SURVEYS Survey Design

• What do you want to know?
• How can you elicit that information in this
  survey?
• Survey length SIZE MATTERS!
• How will you use the results?
• Who are other interested parties or stakeholders?
• Reality check - PRETEST
• Can users understand survey questions and
  directions?
• Will responses provide useful information?

34
USER SURVEYSSurvey Methodology

• Define your survey population
• Determine your survey size SIZE MATTERS!
• Select your survey group
• Choose your survey time
• Produce and distribute survey
• Encourage survey completion and return
• Arrange for data entry and analysis
• Communicate results

35
USER SURVEYSIncreasing the Response Rate

• Make survey as short as possible
• Keep it simple
• Use pre-survey publicity
• Explain why its important
• Offer incentives
• Do follow-up reminders

36
USER SURVEYSUW Libraries

• Survey UW faculty and students on 3 year cycle
• Surveys conducted in 1992, 1995, 1998
• Surveys mailed to
• all faculty (to encourage return and analyze
  subgroups)
• random sample of graduate and undergrad students
• Surveys obtained information on
• Satisfaction
• Use
• Needs/Priorities

37
SURVEY98Timeline

• Survey design, pretest, revision 2 months
• 1st survey printing/mailing 3 weeks
• 2nd survey printing/mailing 2 weeks
• Surveys out and returned 9 weeks
• Survey data entry 3 months
• Survey results/analysis 6 weeks
• Survey results distribution Ongoing
• STARTED 2/15 RESULTS COMPLETE 9/30
• THIS WAS OUR THIRD SURVEY EFFORT!

38
SURVEY98
39
SURVEY 98Direct Costs

• Printing 6000
• Mail (outgoing) 3000
• Survey return 2400
• Data entry 7000
• Return incentives 600
• TOTAL 19000
• (7 per returned survey)

40
USER SURVEYS Collaborate With Other Campus Units

• Reduce costs
• Take advantage of expertise
• Avoid oversurveying
• Build relationships
• Marketing

41
EDUCATIONAL OUTCOMES IMPORTANCE
42
GRADUATING SENIORS (98,99) Satisfaction with
Undergraduate Education
43
MAKING SENSE OF THE DATA

• DATA INTERPRETATION AND PRESENTATION

44
STATISTICAL LITERACY?

• Everyone knows that you can use statistics to
  prove anything.
• 14 of all people know that.
• Homer Simpson

45
IDENTIFY WHAT IS IMPORTANTSurvey Results

• Work initially from basic frequency results
• Satisfaction levels are an important indicator
• Are there consistent themes that emerge
• Is there variation within and between groups
• What types of statistical analysis make sense
• Pay attention to written comments
• Be statistically literate
• Seek internal validation
• Take care in making longitudinal comparisons

46
IDENTIFY WHAT IS IMPORTANTFocus Group Results

• Use debriefing session and transcript analysis
• Identify key themes
• Categorize them
• Review comments for
• Frequency
• Extensiveness
• Intensity
• Specificity
• Consistency
• Level of group participation

47
COMPARING GROUPS AND SUBGROUPS

• Analyze for consistency or variation
• between your respondents and population you
  surveyed
• within group as a whole
• between academic areas
• by year or class or rank
• by gender
• over time
• in depth

48
COMMUNICATIONPost-Data Analysis

• Present key results to
• Library administration/Institutional
  administration
• Library staff
• Community you surveyed
• Other libraries/interested parties/stakeholders
• Make sure results are
• Timely
• Understandable
• Usable
• Good enough statistics

49
COMMUNICATION TIPS

• Provide executive summary
• Present brief context on scope and methodology
• Identify key findings (not all results!)
• Mix text, data, and graphics
• Provide user comments that buttress findings
• Avoid jargon (statistical and otherwise)
• Know your audience(s)! Make it understandable!
• Identify potential action items and/or follow-up
• Maintain assessment momentum

50
USE THE RESULTS
51
TURNING RESULTS INTO ACTION

• Identify WHO or WHAT GROUP is responsible
• Develop implementation process
• Provide responsible party with authority/resources
  
• Communicate action to stakeholders
• Evaluate results of action

52
FOCUS GROUP ACTION Interlibrary Loan

• Remove 5 item per week limit
• Mail items (including books) to on-campus address
• Keep requesters informed about request status
• Improve customer service skills
• Update reference staff on interlibrary loan
  policies and procedures

53
PUBLIC WORKSTATION USE
54
PUBLIC WORKSTATION ACTION

• Reallocate public workstations based on use
• Block known email sites
• Establish catalog only terminals
• Time limits on some workstations

55
BIOSCIENCE SURVEYFUNDING ELECTRONIC JOURNALS
56
EJOURNAL ACTION

• Cancel duplicate subscriptions first
• New subscriptions are electronic only, if
  available
• Switch to electronic only, when feasible
• Avoid publisher packages which lock in higher
  costs

57
FACULTY TOP PRIORITIES
58
UNDERGRAD TOP PRIORITIES
59
FACULTY AND STUDENT PRIORITIES

• Faculty Priorities
• Collections and InformationResources
• Grad Student Priorities
• Collection and Information Resources
• Access (hours)
• Undergraduate Priorities
• Services!!!

60
FACULTY PRIORITIES BY AREA DELIVER FULL TEXT
61
UNDERGRADUATE PRIORITIES BY GENDER
62
REASONS FOR VISITING LIBRARY (AMONG THOSE WHO
VISIT WEEKLY OR MORE OFTEN)
63
SERVICES NEEDED WEEKENDS AND EVENINGS
64
FACULTY AND UNDERGRADS WHO VISIT LIBRARY AT LEAST
WEEKLY
65
FACILITIES ACTION

• Libraries are student places
• Provide diversified study and work areas
• Add computers (both library and those with
  application software)
• Extend hours of opening (but not reference)

66
ASSESSMENT What Happens If You Dont?