CertaintyBased Marking CBM

1
Breckenridge April 2008
Certainty-Based Marking (CBM) aka
Confidence-Based Marking rewarding good
judgment of what is or is not reliable Tony
Gardner-Medwin Physiology, University College
London www.ucl.ac.uk/LAPT
2
Certainty-Based Marking
How should we reward students' knowledge?
Teacher leadership encouragement
Praise criticism
3
Certainty-Based Marking
How should we reward students' knowledge?
Teacher leadership encouragement
Praise criticism
4
How should we reward students' knowledge? Mark
Grade Score for a single question

• How is knowledge related to probability?
• What is Certainty-Based Marking?
• How easy is it to use CBM?
• What are the learning benefits?
• What are the assessment benefits?
• Why doesn't everybody use it?

5
Ordinary words we use to describe Knowledge

• Knowledge is a function of certainty
  (confidence, degree of belief)
• There are states a lot worse than acknowledged
  ignorance

6
How can you measure and reward this knowledge?
- the origin of CBM gt100 years ago.
The key is to have a "proper" or
"motivating" reward scheme, which ensures that
the person does best by expressing their actual
level of uncertainty
7
What is CBM ?
Each answer is marked according to the students
certainty that their answer is correct.
If you're sure, obviously you're best with C3,
but you must convince yourself there is a low
risk of a penalty. If unsure, you gain by
acknowledging this (with C1) and avoiding the
risk of a penalty.
8
(No Transcript)
9
How well do students discriminate reliability ?
10
Thinking about justification and
uncertainty stimulates understanding
Confidence (Degree of Belief)
Choice
To understand to link correctly the facts
that bear on an issue.
11

• Using CBM
• With software from UCL www.ucl.ac.uk/LAPT
• Offline with a CD - ditto
• 3. With Moodle - full integration work in
  progresss
• 4. With other VLEs - the IDEAL !!
• 5. Secure exams, with Optical Mark Reader (OMR)
  Cards

12
Student Learning Principles they readily
understand

• You need to know the reliability of your
  knowledge to use it
• Confident errors are serious, requiring attention
  to explanations
• Expressing uncertainty when you are uncertain is
  a good thing
• Confidence is about understanding why things
  cannot be otherwise, not about personality
• if over- or under-confident, you must calibrate
  through practice
• reflection and justification are essential study
  habits

In evaluation surveys, a majority of students
have always said they like CBM, finding it useful
and fair. Early on they asked to include it in
exams, and recently at UCL they voted 52 30
to retain it.
13
Perhaps we don't need to test knowledge, now we
have Google?
It's so easy to find stuff out, why test
knowledge?
But cheap knowledge puts an absolute premium on
- 1) Identifying misconceptions - "unknown
unknowns" .... the things you will get
wrong and not Google! and distinguishing them
from "don't knows"
Cheap knowledge puts an absolute premium on - 1)
Identifying misconceptions - "unknown unknowns"
.... the things you will get wrong and
not Google! and distinguishing them from "don't
knows" 2) Judging reliability and uncertainty
correctly .... setting a threshold for seeking
help. If checking is expensive, you can
only pick your best choice and "go for it"
These lessons are core things that CBM teaches
14
Student Assessment
CBM quite closely follows the ideal ignorance
measure
The student loses about 3 marks per 'bit' of
ignorance- up to a maximum of 3 bits
15
CBM increases the reliability of exam data
'Reliability' indicates to what extent a score
measures something about the student's ability,
as opposed to 'luck' or chance.
16
CBM increases the effective test length With
increased 'Reliability' you don't need so many
exam questions to get data of equal quality.
17
CBM data is a more valid measure of
ability 'Validity' means it measures what you
want, rather than just something easily measured.
18
Why doesn't everybody already use CBM ? - a
puzzle

• Enthusiasm was exhausted before the age of
  'online'
• Some CBM methods were complex, opaque or
  non-motivating
• Reluctance to treat certainty as integral to
  knowledge
• Mistaken worries about 'personality bias'
• Under-rating of self-assessment practice as
  learning tools
• Worry that CBM would need new questions
• Adaptation of procedures for standard-setting
• Inertia and vested interests

"THE IDEAL" needs CBM !
19
SUMMARY

• Why CBM?
• Get students to think more carefully
• Reward recognition of uncertainty, either
  personal or in a group
• Highlight misconceptions
• Engage students more - the game element of CBM
• Encourage criticism of Qs (intolerance of
  ambiguity or looseness)
• In general enhance self-assessment as a learning
  experience
• NB All of the above arise with little or no
  practice with CBM. The following do require
  practice
• More searching diagnostic data
• More valid and reliable assessment data
• (But NB with CBM you have conventional assessment
  data too.)

20
A few of the names associated with confidence
testing in education

• Kate Hevner
• Darwin Hunt
• Dieudonné Leclercq
• Emir Shuford

• Andrew Ahlgren
• Jim Bruno
• Robert Ebel
• Jack Good

• London Colleagues
• Mike Gahan
• David Bender
• Nancy Curtin

21
We fail if we mark a lucky guess as if it were
knowledge. We fail if we mark misconceptions as
no worse than ignorance.
www.ucl.ac.uk/lapt
22
Lessons from experience with CBM

• Practice is needed before use in exams
• Exams should re-use questions from an open
  database only very sparingly
• Over-confidence and diffidence are both
  unhealthy traits that can be moderated by
  practice to achieve good calibration
• With multi-option questions, students tend (at
  least initially) to over-estimate reliability
• Standard setting - it is easy (but important!)
  to scale CBM marks to match familiar scales based
  on number correct.

23
Some Questions about CBM !

• Are there problems using it?
• Why doesn't my VLE support CBM?
• Do students need practice?
• Isn't computer marked assessment just factual?
• Does CBM increase retention?
• Do I need new questions?
• What are the best Q types?
• What about school education?
• Is it relevant to my subject, where opinions
  differ?
• Isn't it bad to encourage guessing?
• What if my only assessments are exams?
• How do I convince an exam board?
• Isn't it right/wrong that really matters?

24
CBM increases the reliability of exam data with
True/False Questions 'Reliability' indicates to
what extent a score measures something about the
student's ability, as opposed to 'luck' or chance.
To achieve these increases using only correct
would have required on average 58 more
questions.
25
When you know a thing, to hold that you know
it. And when you do not know a thing, to allow
that you do not know it. This is knowledge.

Confucius