Out of the Blue: Electronic Assessment Replaces Paper Bluebooks

1
Out of the Blue Electronic Assessment Replaces
Paper Bluebooks

• Greg Robinson, Ph.D.
• Academic Technology Services
• College of Arts and Sciences
• University of NC at Chapel Hill
• Melissa Graham Meeks
• Doctoral Candidate
• Department of English

2
Agenda

• Placement Exam Context of Securexam Pilot
• Outline of Placement Exam Objectives
• Brief Demonstration of Software
• Logistics, Outcomes, Feedback
• Student Interviews
• UNC-Chapel Hill Status
• Comments and Questions

3
The Problem Time

• The placement exam generates hundreds of pages of
  student writing, which must be duly processed
  within the first week of class so that students
  can be placed in the appropriate courses.
• Grading handwritten exams is slower than reading
  typed exams.

4
Context The Writing Program

• The WP consists of three courses ENGL 10, 11,
  and 12.
• Undergraduate requirements mandate that students
  receive credit for or be continuously enrolled in
  11 and 12.
• Placement is initially determined by the
  verbal/English scores for the SAT, ACT, SAT II,
  or AP exams.

5
The Placement Exam

• Students have the option to take the Placement
  Exama timed, two hour exam requiring students to
  read a passage and analyze its rhetorical
  features and argumentative effectiveness.
• Summer Bridge participants, junior transfers,
  nontraditionally aged students, and students who
  are dissatisfied with their placement usually
  take this exam. About 10 of the FY class, or 300
  students, take the exam in any given year.

6
ETS-based Process of the Placement Exam

• Exam Session
• Anchoring Selects exams that match the 6 rubric
  descriptions.
• Calibration Trains scorers to match anchor exams
  to the rubric.
• Scoring Each exam receives two scores.
• Rescoring Exams with scores that differ by more
  than 1 point are scored by a 3rd grader.
• Distribution analysis
• Placement decisions posted, and students
  reschedule their classes as necessary.

7
Convenience is bad pedagogy, so why ask students
to type?

• Global revision, which the WP emphasizes, is
  easier.
• Editing is also easier.
• For most students, typing is also easier than
  handwriting.
• The WP does not accept handwritten essays in any
  other context.

8
How might Type Affect Scoring and Scores?

• Because our goal is to place students in the
  course that will most benefit them, we are
  delighted that typed exams give students more
  opportunity to revise and edit A stronger
  writing sample means a stronger placement
  decision.
• Typed exams eliminate the noise of legibility
  in the scoring process.
• Potentially, it reduces the time necessary to
  read each exam.

9
How Can We Get Type?

• We needed a software that
• Did not significantly alter the writing space
  with which most students could be expected to be
  familiar
• Could be integrated into the exam process without
  compromising the timeline
• Could deliver typed exams with minimal technical
  fuss

10
Securexam Software

• Purpose Secure Electronic Assessment
• Setting Exams administered in a secure
  environment using Microsoft Word, locking out
  access to all other files, programs, and the
  Internet.
• http//www.softwaresecure.com

11
Three Security Methods

• Lockdown Effects
• Alarm Security System (detects exit attempts)
• Post-exam Encryption prevents access

12
Software Demonstration
13
Implementation Logistics

• Students download and install software from the
  Website before exam.
• Provide IT help before and during exam.
• Provide extra outlets per room
• Collect exam files on USB drives (secure,
  reliable, network-free, and cheap) at multiple
  exiting stations.
• Batch Decrypt, Print, and Archive files.

14
Results of Implementation

• 172 exams were typed.
• 14 were handwritten, by election or technical
  failure.
• The reduced number of total exams is related to
  the upward shift of SAT scores, not technophobia.
• Every stage of the scoring process was timely.

15
Feedback 1 Laptop Preference

• If you had the option to complete this exam with
  pencil and paper or using your laptop, which
  option would you choose and why? (n 134
  students)
• Laptop 76.3. Easy edits, speed, comfort,
  efficiency.

16
Feedback 2 Laptop Disadvantages

• Q2 Disadvantages to using a laptop?
• Apprehension about a new environment
• Impetus?
• Needless layer of complexity

17
Feedback 3 Essay Quality

• Do you feel that using your laptop allowed you
  to write a more cohesive and coherent essay?
• Yes 80.2

18
Feedback 4 Implementation

• Do you feel that you would be comfortable using
  your laptop for other in class assessments such
  as short answer essays or exams?
• Yes 86.6

19
Summary Reported Benefits

• Benefits for students
• Ease of Editing and Revision
• Rapid Composition
• Improved Organization and Cohesiveness
• Review, Depth, Content-Rich
• Benefits to scorers
• 100 Easier to grade
• Instant conversion to legible essays
• Better student performance

20
Observed PE Challenges

• Student Responsibilities
• Uninformed and Unprepared
• Skipped Exiting Station (!)
• User Apprehension
• Technical Challenges

21
Technical Challenges

• Spyware, P2P apps, Office Add-ins
• Macs and Some PC Hardware
• Win 98 and Office 97 trouble.
• Power Dead batteries, outlets, etc.

22
Lessons Learned

• Students follow instructions poorly outside of
  the exam setting.
• Post removal tools (Spybot SD, AdAware)
• Post Office Add-ins Disable Tool
• Technical Restart required before transferring
  the file to USB drive.

23
Securexam Pros

• Students already know software (Word)
• Integrates with CCI (Laptop Initiative)
• Hardware Infrastructure already in place
• Office Software already licensed
• Blackboard Infrastructure
• Potential on campus load?
• Recovery Capability
• Accommodates batch processing

24
Student DVD Feedback
25
Present Status at UNC-CH

• Purchased and Used 1000 licenses across the
  College of Arts and Sciences
• English, Music, Communication Studies, EXSS,
  Political Science, AFAM Studies, American
  Studies, Public Policy, German.
• Implementations
• English Composition Placement Exam
• Midterm and final exams
• Graduate Qualifying exams

26
Standard Course Implementation

• Post Application to Course Blackboard Site
• Collection Blackboard or email (not USB)
• Student Information http//securexam.unc.edu

27
Closing Comments

• Positive Student Feedback
• Positive Instructor Feedback
• 100 (n10) who used in Fall 2004 exams requested
  for Spring 2005
• 100 (n10) would recommend to peers, chair, and
  IT Administration.
• Securexam Failsafe Feature Word Backup Files
• IT Support Goal Instructor Autonomy, Campus
  Support, Facilities

28
Contact Information

• UNC-Chapel Hill Greg Robinson, Ph.D. Senior
  Academic Technology Liaison College of Arts and
  SciencesUNC-Chapel HillGreg_Robinson_at_unc.edu
• http//securexam.unc.edu
• Vendor Software Secure, Inc.http//www.softwares
  ecure.com