Assessing Information Literacy Skills: A Look at Project SAILS

1
Assessing Information Literacy Skills A Look
at Project SAILS

• Joseph A. Salem, Jr.Kent State University
• ARL New Measures Initiatives
• CREPUQ
• February 11, 2005

2
Context

• Explosion of interest in information literacy
• Accountability
• Assessment
• Formative for planning/improvement
• Summative evidence/documenting

3
What Is an Information Literate Person?

• Information Power - American Association of
  School Librarians
• 9 standards
• Big6
• Information Competency Standards for Higher
  Education ACRL
• 5 standards, 22 performance indicators, 87
  outcomes, 138 objectives

4
Our Questions

• Does information literacy make a difference to
  student success?
• Does the library contribute to information
  literacy?
• How do we know if a student is information
  literate?

5
The Idea of SAILS

• Perceived need No tool available
• Project goal Make a tool
• Programmatic evaluation
• Valid
• Reliable
• Cross-institutional comparison
• Easy to administer for wide delivery
• Acceptable to university administrators

6
The Project Structure

• Kent State team
• Librarians, programmer, measurement expert
• Association of Research Libraries partnership
• Ohio Board of Regents collaborative grant with
  Bowling Green State University (part for SAILS)
• IMLS National Leadership Grant
• Working with many institutions
• www.projectsails.org

7
Project Parameters

• Test based on ACRL document
• Test development model Systems design approach
• Measurement model Item Response Theory
• Tests cohorts of students (not individuals)
  programmatic assessment
• A name
• Standardized Assessment of Information Literacy
  Skills

8
Test Development

• Systems design approach
• 1. Determine instructional goal
• 2. Analyze instructional goal
• 3. Analyze learners and contexts
• 4. Write performance objectives
• 5. Develop assessment instrument
• The Systematic Design of Instruction. 6th ed. By
  Walter Dick, Lou Carey, James O. Carey. Boston
  Pearson/Allyn and Bacon, c2005.

9
ACRL Standards Significant Challenges

• Breadth and depth
• Objectives
• Multi-part
• Multi-level
• Habits/behaviors versus knowledge

10
Consider Skill Sets

• Regrouping the ACRL objectives (and some
  outcomes)
• 12 sets of skills organized around
  activities/concepts
• More closely mirrors instructional efforts?

11
Item Development Process

• Review competencies and draft some items
• "How can I know that a student has achieved this
  competency?"
• Formulate a question and answers. This may take
  several iterations.
• Develop additional responses that are incorrect,
  yet plausible. Aim for five answers total.

12
Testing the Test Items

• Conduct one-on-one trials
• Meet with individual students, talk through test
  items
• Conduct small group trials
• Administer set of items to group, engage in
  discussion after
• Conduct field trials
• Administer set of items to 500 students, analyze
  data

13
Measurement Model

• Item Response Theory
• Also called Latent Trait Theory
• Measures ability levels
• Looks at patterns of responses
• For test-takers
• For items
• Rasch measurement using software program
  Winsteps (www.rasch.org)

14
Response Pattern Example
  C gave correct answer
15
Data Reports

• Based on standards and skill sets
• Looking at cohorts, not individuals
• Show areas of strength and areas of weakness

16
The Person-Item Map

• Plots items according to difficulty level
• Plots test-takers according to their patterns of
  responses
• Can mark average score for cohorts
• Cross-institutional average
• Specific institution average

17
The Bar Chart

• Another representation of the information
• Group averages
• Major, class standing, etc.
• Which groups are important to measure?
• How do you know which differences in means are
  important?

18
Current Instrument Status

• 158 items developed, tested, and in use
• Most ACRL learning outcomes covered
• Not Standard 4 Uses information effectively to
  accomplish a specific purpose.
• 12 skill sets developed based on ACRL document

19
IMLS Grant Status

• Phase I complete
• 6 institutions participated
• Feedback from institutions
• Phase II underway
• 36 institutions participated
• Phase III started June 2004
• About 70 institutions participating
• Wrap-up summer 2005

20
Project Highlights

• Discipline-specific modules
• Canadian version of the instrument
• Automated survey generation
• Automated report generation

21
Next Steps for SAILS

• IMLS grant period ends on September 30, 2005
• Stop administering SAILS to allow analysis of the
  instrument
• Does the instrument measure what we want it to?
• Are institutions getting what they need?

22
Next Steps for SAILS

• Analyze data and input from institutions
• Validate the instrument
• Factor analysis and skill sets
• Outside criterion testing through performance
  testing
• Test-taker characteristics
• Sex, ethnicity, class standing, GPA
• Test administration methods

23
Next Steps for SAILS

• Re-think how results can be presented or used
• Scoring for the individual
• Pre and post-testing
• Cut scores
• Administrative challenges
• Automate data analysis
• Re-engineer administrative tools
• Create customer interface
• Test development
• Develop new items

24
Summary

• Vision
• Standardized, cross-institutional instrument that
  measures what we think it does
• To answer the question
• Does information literacy make a difference to
  student success?

25
For More Information

• www.projectsails.org
• sails_at_kent.edu
• Joseph Salem, jsalem_at_lms.kent.edu
• Mary Thompson, project coordinator
• mthomps1_at_kent.edu 330-672-1658

26
Questions?