Theres no confidence in multiplechoice testing

1
Theres no confidence in multiple-choice testing

• Phil Davies
• School of Computing
• University of Glamorgan
• South Wales

2
Who has NO confidence?

• STAFF
• They can just guess the answer
• Its not the way it is in the real world
• there are varying degree of knowledge
  (Echternacht, 1972)
• dont worry its easy, its multiple choice
• Possibly more importantly -- STUDENTS

3
The First Test

• 40 Questions / Formative
• Time Consuming / Difficult to create sensible
  distracters / Rocket Science
• I thought Average 1 minute per question
• Actual all finished in ten minutes apart from
  one who took 50 minutes
• Why? No reason to think

4
On-Line Assessment Learning OLAL

• Level Two Degree .. Computer Communications
  Networks
• Failures High Programming to Demonstrate ..
  Technical Subject .. Essay-type answers in
  examinations
• Assess .. Knowledge Understanding Learning
  Outcomes . Directed

5
(No Transcript)
6
On Line Assessment LEARNING

• Four Tests
• Weeks 3,6,9,12 5, 10, 15 20
• Cumulative
• Two Passes of SAME test
• Learning .. Short term?
• Slowed Down .. Thought about WHEN WRONG .

7
What decides the QUALITY of a question?

• In the recent World cup qualifier between England
  and Greece, who scored the all important
  equalizing goal?
• David Beckham
• Victoria Beckham
• Brooklyn Beckham

8
Rewards?

• Know it
• On the tip of my tongue
• If I see the answers Ill know it
• Deduction
• Guess
• Know it oh no Ive got it wrong ..
  misinformation

9
(No Transcript)
10
Allocation of Marks

• Logarithmic 3, 2, 1, -3, -2, -1
• This is not the same!! - not seeing answer
• Students were involved / Provisional
• You can try this out but it must not have a
  negative affect upon our results compared with
  last year
• Reward knowledge 4, 2, 1, -2, -1, 0
• If we dont know it, then why take off 1

11
2002 OLAL 1/3 .. Better by approx. 6 5, 3, 1,
-2, -1, 0
12
(No Transcript)
13
(No Transcript)
14
Percentile Frequency Distribution
15
(No Transcript)
16
(No Transcript)
17
Proportion of Questions Correct

• High Confidence 84 correct
• Fairly Confident 58 correct
• Not confident 47 correct
• 72 of all questions answered with high confidence

18
Based on Test 3, who were the most confident (60
Questions)

• 75
• 52 7 1
• 50
• 44 13 3
• 25
• 42 15 3
• 0
• 33 19 8

19
Student Feedback Thoughts on Multiple Choice

• useful learning tool
• second test is a really useful way of learning
• immediate feedback helps me to learn
• doesnt reflect my lack of knowledge
• not fair, I found it difficult to tell the
  difference between the answers
• negative marking made me panic

20
Student thoughts on Confidence

• Eliminates guesswork
• I was waiting for a leggy blond to bring on my
  prize at the end
• separated the lucky students from me
• at least I didnt lose marks for selecting no
  confidence
• are you testing my confidence or knowledge

21
Quality of a multiple choice question?

• In the recent World cup qualifier between England
  and Greece, who scored the all important
  equalizing goal?
• Teddy Sheringham
• Michael Owen
• Robbie Fowler
• QUALITY OF QUESTION NOT DISTRACTERS

22
Question Format

• One or more questions were impossible to judge
• I wish youd put more detail in the question
• Perhaps it would have been better to show the
  answers, ask for confidence, and then show the
  question

23
Conclusions?

• Assess subject area not individuals confidence
• Quality of questions question NOT distracters
• Allocation of Marks (based on past?)
• Student Accept / Staff Accept
• Time for tests increases really had to think

24
(No Transcript)