Evaluating and Assessing Library Instruction

1
Evaluating and Assessing Library Instruction

• Examples Best Practices

2
Presenters

• Martin Garnar Regis University
• Rhonda Gonzales Eleni Adrian CSU-Pueblo
• Lorrie Evans Auraria Library
• Patrick McCarthy - CSU

3
Evaluating and Assessing Library Instruction

• Regis University
• Martin Garnar

4
ACRL Guidelines for Evaluation and Assessment

• program evaluation plan
• criteria for evaluation
• learning outcomes and assessment methods
• coordination of assessment with teaching faculty
• data gathering and analysis

5
Evaluating and Assessing Library Instruction

• CSU-Pueblo
• Eleni Adrian Rhonda Gonzales

6
CSU-Pueblo Library Instruction Evaluation

• Overview
• Librarians at CSU-Pueblo administered a brief
  evaluation form after each instruction session.
• Collected these evaluations from 1999-2004.
• Entered results into an Access Database.
• Data Clean-up
• Statistical Analysis
• Conclusions

7
CSU-Pueblo Library Instruction Evaluation

• Evaluation Form
• Background questions
• Satisfaction questions
• Ranking question

8
CSU-Pueblo Library Instruction Evaluation

• Evaluation Forms
• Collected Fall 1999 through Summer 2004.
• 181 classes
• 2441 evaluation forms

9
CSU-Pueblo Library Instruction Evaluation

• Access Database
• Entered several semesters of data
• Revealed errors in our database design
• New database structure
• New input forms

10
CSU-Pueblo Library Instruction Evaluation

• Access Database

11
CSU-Pueblo Library Instruction Evaluation

• Access Database

12
CSU-Pueblo Library Instruction Evaluation

• Data Fixes New Database Design
• Relationship between the classes and the
  evaluations tables.
• No answers for background questions.

13
CSU-Pueblo Library Instruction Evaluation

• Data Fixes Form Changes
• Two questions added, two questions removed.
• Re-entered all old evaluation form data to match
  the final version.
• Double-checked that the form changes were
  accurately re-entered.

14
CSU-Pueblo Library Instruction Evaluation

• Data Fixes Data Integrity
• Duplicate Class records
• Duplicate and blank Evaluations records
• Typos in Status (year in school)
• General data entry errors

15
CSU-Pueblo Library Instruction Evaluation

• Results Background Questions

Year in School
16
CSU-Pueblo Library Instruction Evaluation

• Results Background Questions

Attended Previous Session
17
CSU-Pueblo Library Instruction Evaluation

• Results Background Questions

Specific Assignment
18
CSU-Pueblo Library Instruction Evaluation

• Results Background Questions

Professor Participated
19
CSU-Pueblo Library Instruction Evaluation

• Results Satisfaction Questions

Start Research
20
CSU-Pueblo Library Instruction Evaluation

• Results Satisfaction Questions

Clear Presentation
21
CSU-Pueblo Library Instruction Evaluation

• Results Satisfaction Questions

Hands-on Experience
22
CSU-Pueblo Library Instruction Evaluation

• Results Satisfaction Questions

Respect
23
CSU-Pueblo Library Instruction Evaluation

• Results Ranking Question

Most Important Thing Learned
24
CSU-Pueblo Library Instruction Evaluation

• Comparisons

25
CSU-Pueblo Library Instruction Evaluation

• Comparisons

26
CSU-Pueblo Library Instruction Evaluation

• Conclusions
• Distribution of responses skewed towards high
  end.
• Student evaluations of faculty teaching across
  campus are not normally skewed towards the
  positive to this extent.
• These results show that a large majority of
  students felt that our instruction sessions were
  helpful!

27
CSU-Pueblo Library Instruction Evaluation

• Conclusions - Meeting Our Goals?
• Since most of our sessions are introductory in
  nature, one of our primary goals is to show
  students how to start their research using our
  library.
• 88.17 of students that took the survey indicated
  that they moderately or strongly agreed that the
  sessions helped them learn how to start their
  research.
• 36.12 of students who took the survey responded
  that learning how to locate a journal article or
  book was the most important thing they learned.

28
CSU-Pueblo Library Instruction Evaluation

• Conclusions - Factors Affecting Results
• What factors may have affected student responses?
• Chi-square tests indicate that the following
  factors proved to be statistically significant
• Having a specific assignment in conjunction with
  this presentation was a significant factor on the
  number of students that strongly agreed with A
  C
• Attending a prior session was a slightly
  significant factor in determining how many
  students strongly disagreed with D
• Survey Design
• Even number of response choices forced students
  to choose positive or negative
• Unclear wording My Professor participated in
  this session.
• Lack of N/A forced students to choose a response
  even if it did not apply
• Survey Administration
• Administered at the end of each session, when
  students were in a hurry to leave.
• Tendency of librarians may be to skip the survey
  if the session hasnt gone well.

29
CSU-Pueblo Library Instruction Evaluation

• Indications for Best Practices
• Role of Evaluation
• combine learning assessment with program
  evaluation
• Survey Design
• concise, clearly worded, avoid mild statements,
  allow neutral and N/A responses, define
  hypotheses prior to creating the form

30
Evaluating and Assessing Library Instruction

• Auraria Library
• Lorrie Evans

31
Evaluation of instruction at Auraria Library

• Survey student learning in single session library
  classes (random selection).
• Survey students in full semester credit bearing
  classes (assignments, final project)
• Survey the faculty toward the end of the semester
  after papers and assignment are turned in.

32

Challenges in the single session library class

• Do the students understand the most important
  concepts?
• No other chance for a follow up or review.
• When students leave the room we will not see them
  again

33
Why?
Look in the mirror are we teaching what we
think we are teaching. Collaborate with campus
faculty on a pro-active level. Make library
instruction a more integrated part of the
academic culture. Accreditation assessment
mandates driven externally.
34
The tool short online survey

• In class three to five questions mapped to
  learning goals.
• Customized for each class

Broadcast results, review answers main
concepts. Students share concerns.

• Students see their answer along with classmate
  responses.
• Librarian can respond to student input.
• Review main points.

35
Sample results

• English 1020
• Student comments view
• Psychology freshman seminar data
• Graduate Political Science - single session post
  test.
• Graduate English Literature - semester course

Jump past results
36
Comments
return
37
Psych
Freshman Seminar sample items
return
38
Graduate English Literature survey sample items
return
39
Political Science
post test
return
40
Right tool for the job question
41
Faculty survey -- end of the semester

• Program effectiveness as reflected in quality of
  student projects and assignments.
• Satisfaction with content, scheduling and
  teaching.

42
Faculty Evaluationsurvey results
Zoomerang link
43
Faculty evaluationssample items
44
How can we improve?
45
Feedback loop

• In class survey instant feedback for both
  students and librarian. Librarian can go over
  confusing concepts.
• Results discussed at departmental meetings used
  as source of information to make adjustments in
  teaching (more depth with less quantity).
• Results made available to instructors and
  students.
• End of the year report submitted to the campus
  Outcomes Assessment Advisory Committee. The
  committee provides a review of the report,
  recommends improvements.

46
Evaluating and Assessing Library Instruction

• CSU Fort Collins
• Patrick McCarthy

47
Evaluating Learning Outcomes at the Reference
Desk - CSU

• Why?
• Determine if learning takes place as a result of
  interactions at the reference desk
• Identify those skills most suitable to reference
  desk instruction
• Establish reasonable reference desk learning
  outcome benchmarks
• Recommend behaviors and methods that lead to
  student learning

48
Categories of Reference Assessment

• Accuracy response rate
• User satisfaction
• Course/instructor opinion survey
• Learning outcomes

49
Methodology?

• Quantitative approach
• Powerful, not practical
• Qualitative approach
• Powerful, versatile

50
Procedures

1. Patron completes preliminary survey at completion
   of reference interaction
2. Researcher analyzes preliminary surveys and
   selects relevant patrons for follow-up assessment
3. Selected patrons are interviewed on specifics of
   reference transaction
4. Interview recordings are transcribed and coded
   for assessment

51
Analysis

• Transfer assessment
• Accuracy assessment
• Patron must provide evidence
• Authentic assessment

52
Group Discussion Guiding Questions

• 1. How have you determined the learning needs of
  your community?
• 2. What's your best method for evaluating library
  instruction?
• 3. What's your best method for assessing progress
  on learning outcomes?
• 4. Do you have examples of successful
  collaboration with classroom faculty for
  evaluation and/or assessment?
• 5. How have you used data from your instruction
  program evaluations and/or assessments?

53
Assessment of Instruction Examples Best
PracticesPresenter Contact Information

• Martin Garnar, Reference Librarian
• Dayton Memorial Library, Regis University
• 3333 Regis Blvd., Mail Stop D-20
• Denver, CO 80221-1099
• 303.964.5459
• 303.964.5497 (fax)
• mgarnar_at_regis.edu

Rhonda Gonzales, Interim Dean Eleni Adrian,
Instruction Coordinator University
Library Colorado State University Pueblo 2200
Bonforte Blvd. Pueblo, CO 81001 719.549.2361 rhond
a.gonzales_at_colostate-pueblo.edu eleni.adrian_at_colos
tate-pueblo.edu

• Lorrie Evans,
• Library Instruction Services
• Auraria Library
• 1100 Lawrence Street
• Denver, CO 80204
• 303.556.6776
• Lorrie.Evans_at_cudenver.edu

• Patrick McCarthy
• Coordinator of Instruction
• Morgan Library
• Colorado State Univ.
• 501 University Avenue
• Fort Collins, CO 80523
• 720.491.1881
• pmccarthy_at_manta.colostate.edu