Experiences Conducting Examinations Online

1
Experiences Conducting Examinations On-line

• H.C. Davis, L.A. Carr, and Su White
• Learning Technology Group
• Electronics and Computer Science
• University of Southampton
• hcd_at_ecs.soton.ac.uk

2
Warning This is NOT a paper about on-line
objective testing
3
Problems with Traditional Exams

• Not an authentic test of ability/learning
• Done under rushed conditions
• with unfamiliar tools (pen and paper!)
• Sometimes pen and paper are not appropriate tools
• Particularly challenging for those with any
  special needs

• Closed book exams tend to test the wrong things
• or persuade the students to revise
  inappropriately
• Attempting to read student handwriting is a
  punishment academics do not deserve!

4
Our Situation

• Team teaching
• first year Programming Principles (180
  students)
• final year Advanced Web Technologies and
  Hypertext Research (100 students)

• Both courses
• have a significant hands on programming element
• have an important non-programming reflective
  element
• It is necessary have examinations of some kind
• Butstudents complained about writing progams
  with pen and paper

5
On-line examinations

• Booked enough computer rooms
• plus 10 extra computers per room
• plus an extra room
• Used our standard hand-in machine per part
  question (had floopy disks on standby)
• Allowed all normal network resources on line
  notes and Google
• Re-wrote Standard Instructions for invigilators

• Employed all PG lab TAs (1 TA per 12 students) to
  help invigilate
• Liaised with ISS to ensure all machines and
  services working
• Phone contact between main invigilator in
  different rooms
• Students with disabilities allowed to take exam
  at home
• ..Spent hours explaining what we were doing
  to examination authorities!

6
Programming Principles first attempt on-line

• examination pretty similar to the regular two
  hour written exam
• Gave them 3 hours
• Used objective testing for non programming
  components

7
Web Tech/Hypertext -first attempt on-line

• Three hours Compared to previously a one and a
  half hour paper
• 3 Qs out of 4
• 2 questions involved coding
• 1 question involved a reflective essay
• 1 question involved a design

8
Initial Mistakes

• Approach
• Coding at computers takes a long time and is
  stressful
• It is not appropriate to set reflective questions
  in the same slot as activity/coding questions

• Mechanics
• Needed some method for students to confirm what
  they believed they had handed-in
• Cannot invigilate final year students by watching
  their screens too fast!
• Consequence
• Students wrote much to much, mostly badly
  constructed, text for essay questions

9
Changes

• Split examinations into two separate time bound
  parts
• one for coding and other computer use
• one for reflective question answering/ essay
  writing We aim to give too much time
• Significantly decrease the amount of coding

• Coding questions much more tightly specified
• particularly what they dont have to do
• Strict (and quite small) word limit for essay
  questions
• Along the way we have changed the atmosphere
  from examination to time bound, unaided
  assessed coursework

10
Advantages

• Equity
• Students believe style of assessment is fairer
• Students (and staff) believe marking is fairer
• Authenticity
• Students seriously challenged by having open
  web makes them think about What is learning?
• Efficiency
• Staff time on marking significantly reduced
  (including essay questions)

11
Thoughts on QA

• Much easier to give individual students feedback
  on their examination performance

• Much more difficult to associate comments when
  marking with questions
• Had trouble producing necessary evidence for
  external examiners
• Need some improvements to our hand-in system

12
From our student Wicki..

• What attracted me to this course was the fact
  that it has a computer-based exam. You sit in the
  lab at a computer with full access to the
  internet and your files as usual. This means that
  revision does not require cramming, but rather
  making notes of where you can find information
  you might need. And you don't even have to
  remember those notes - just stick them on the web
  for yourself to read later! That's not cheating,
  it's preparation.

13

• This course gives you a run up of history on the
  web and hypertext (which you'll find out is a
  hell of a lot more than just blue-underlining)
  up to the latest stuff.
• All very interesting me thinks..

14
And later
Oh dear. How wrong was I? Just done the exam and
feel awful for it. OK, the first part (actually
doing practical XML and WebML stuff) was good
when you know what you're doing, but the second
half (the exam is in two separate sessions) was
hell. .
15
OK, everything you need is there for you (papers,
google, your own notes) but that's not much good
when you can't remember which paper uses the
phrase "temporal continuous media" to include in
your referencing. It's a nice course, but
don't be fooled into thinking you need less
revision because it's all online - you don't have
time to read the stuff in the exam, so take the
time to do it before.
16
Thank You ?

• Questions and comments please.