Blind double marking

1
Blind double marking

• Aim to raise some issues contingent on the
  introduction of blind double marking of essay
  exams

2
BChD (Bachelor of Dental Surgery)

• 5 year course
• non-modular (curriculum largely determined by
  General Dental Council)
• degree exams in 1st, 2nd, 3rd and 5th year
  plus Progress to Finals

3
The written exam in Finals

• takes place in June of 5th year
• 40 of marks for Finals
• 2 three-hour papers
• 4 compulsory essay questions in each paper
• each essay marked out of 25
• traditionally double-marked
• first double-marked blind in 1999

4
Double-marking vs.blind double-marking

• traditionally 2nd marker agreed with 1st mark
  70 - 80 of time and disagreements nearly always
  only 1 - 2 marks (out of 25)
• marking blind, absolute agreement dropped to
  between 48 and 17 with, in one question, 35 of
  marks being gt2 marks apart

5
Extent of agreement between markers in 1999 final
written exam
6
Analysis of markers consensus process (all
questions, 48 students)
7
Inappropriate use of correlation statistics

• Correlation statistics show association, not
  agreement
• e.g. correlation would be perfect if all marks
  given by 2nd marker are exactly 10 points higher
  than those given by 1st marker
• therefore use kappa, which measures absolute
  agreement with reference to the frequency with
  which marks would be expected to agree by chance
  Note
  agreement can be evaluated as
  Poor (lt0.20), Fair (0.21 - 0.40), Moderate
  (0.41 - 0.60), Good (0.61 - 0.80) or Very Good
  (0.81 - 1.00)

8
Standard ways of improving inter-marker
reliability

• re-visit assessment criteria/marking scheme
• have standardisation meetings with markers
• downgrade importance of essays in assessment and
  use more objective methods
• continue to monitor marking

9
Action taken between June and November 1999

• Issues raised and discussed at Undergraduate
  Dental Education Committee and in Dental School
  newsletter Teaching Quality Matters
• Assessment criteria reviewed,
  re-formatted and published in Nathan Bodington
• Markers for Operative Dentistry prize exam
  (December 1999) given verbal briefing about
  assessment criteria
• Result a marked improvement in reliability !!
  (but.)