Summative Assessments Online: High Stakes Low Risks

1
a
Summative Assessments Online High Stakes Low
Risks
Dr Richard Parsons Centre for Learning and
Teaching University of Dundee r.parsons_at_dundee.ac
.uk
2
University of Dundee

• Considerable experience of online assessments
• Blackboard VLE users
• Delivering up to 2000 formative assessment
  sessions a day with QM Perception
• Delivering 50 summative exams a year with QM
  Perception
• Further expansion expected

3
Online Assessment System
Assessment hardware design deployment
Assessment software design deployment
Clear policy and procedures
Central assessment support
Learning systems integration
Staff and student education
4
The Hardest Task

• Remains writing quality questions
• Staff time is precious
• Students are more critical than ever before
• Assessments to be aligned
• Questions need to be prepared ahead of time
• Quality runs to feedback, distracters, variety,
  depth, currency
• Authoring Q within the software is secondary

5
Staff and Student Education

• At Dundee we have an Online Assessment course
• Covers technical and pedagogical aspects
• 5 weeks long, 3 to 6 hours a week.
• Formally accredited by our Education Faculty (10
  SC)
• Building a bank of staff skills
• Most students need little guidance

6
Examples of Assessment Strategies

• 15 Q from pool of 50, max mark taken (Languages)
• Take assessment first, then repeat in an open
  fashion. Mark taken from first run. (Medicine)
• Revision questions with feedback relating to
  lectures (Art History)
• 10 numerical questions, each from a pool of 50
  randomly generated Q (Life Sciences)
• Student revision students invited to compose 2
  Q each with FB, all compiled automatically,
  checked made available for open assessment (Env
  Sci)
• Open assessment, one submission only (Law)
• Summative (range of departments)

7
Quality Hardware and Systems

• Central servers built maintained by ITS
• LEU involved in system design
• Dual servers, one formative, one summative
• Compaq ML370 Dual Pentium 1400 MHz 2GB RAM,
  single NICs
• Databases
• Formative on Oracle Unix server
• Summative on local Access

8

Quality Hardware
Development CAA system
Development VLE environment
QMP Test CAA server CAA-03
Blackboard VLE ELS 6.0 Test
Blackboard VLE server replica of production
system Sun Blade 1000, 2 750MHz processors, 2 GB
RAM A1000 300GB RAID5 array
QMP examination CAA server CAA-01
Existing operational
CAA-01 Username Matric no.
QuestionMark Perception application and database
server Windows 2000 Server, IIS, Access database
Compaq ML370 Dual PIII 1400MHz 2 GB RAM 200 GB
RAID5 array
LSExam1
Assessment publication
LEU (DJW RP PCs) linked by Netdrive and Access
publish tool
9
Clear Policy and Procedures

• Central online assessment system required clear
  guidance
• Prepared draft Policy and Procedures
• Generally follows BS79882002
• Developed for about a year, inc. consultation
• Now passed at University Senate
• Excellent for ensuring quality
• Runs to 16 pages
• Additional documentation for Online Assessment
  system administrators

10
Summative Procedures

• Assessment must be tested
• Assessment loaded to both servers, scheduled on
  one
• Central staff attend examinations
• Use special exam accounts
• Use Perception Secure Browser
• Candidates complete Examination Attendance form
• Immediately after the exam a backup of results is
  taken
• Use two way radios or mobiles
• No system is fullproof, 99 success target ?

11
(No Transcript)
12
(No Transcript)
13
(No Transcript)
14
(No Transcript)
15
(No Transcript)
16
(No Transcript)
17
Final Points of Interest

• Once had a problem of results displayed
  immediately to students finishing a summative
• Industrial action on assessments left the
  machines running unsupported
• The best people to assess a subject are experts
  in the subject, not experts in online assessment
• OA is appropriate for usage outside of the
  sciences
• Staff are becoming managers of teaching
• Online assessment is important, but not life
  threatening

18
The Future for Online Assessment

• Secure, stable, performing system
• Increased range and depth of questions
• Innovation within and beyond QM
• Work with the staff on questions
• Increased sharing/sales of question banks
• Foresee increased integration with online
  learning systems

19
Summary

• Dundee has an institutional online assessment
  system
• Software, hardware, PP, staffing, integration
  and education
• Secure and reliable, but straightforward to use
  for students and staff
• Let the academics create the Qs to stimulate
  learning
• Costs are neither insignificant or astronomical
• Expect the internet developments to continue

20
January 2004
University of Dundee Managed Learning Environment
GroupWise Web Email
Student Management System
P3
QM Formative server CAA-02
User exists 3 weeks before appointment Completed
daily
Module enrolment Bb client Mark return to Bb
Student exists Staff details also stored in
SMS Module enrolments of students Module
assessment mark storage LDAP client?
Blackboard VLE ELS 6.0
QM Exam server CAA-01
Hardware 2 480s Version 6.0.10 Users created
from SITS SMS system Module enrolment Staff
assignments from web system LDAP client Module
assessment mark storage Training teaching staff
Special exam enrolment Mark return
Library
Catalogue access Resource management system LDAP
client?
e-Directory
LDAP
Create user name and email process LDAP
authentication Remove user process
Dual Novell LDAP servers
Staff Assignment Record
File store
Novell Fileservers Student Desktop
Secure web access
Richard Parsons, CLT LEU