The Continual Assessment of Confidence or Knowledge with Hidden MCQ? Short Paper W.I.P.

1
The Continual Assessment of Confidence or
Knowledge with Hidden MCQ?Short Paper W.I.P.

• Phil Davies
• School of Computing
• University of Glamorgan

2
Conference 1999Learning Through Assessment

• Phil Davies
• School of Computing
• University of Glamorgan

3
Implementation of OLAL

• Tests Weeks 3,6,8,12 (60 Subject Mark)
• Percentage Marks 5,15,15,25
• Beep Test
• 2 Correct -1 Incorrect(Really -3)

4
(No Transcript)
5
Student Concentration

• Guessing
• Why should the student care if incorrect?
• Allow TWO PASSES of the test
• By concentrating on first pass
• will get reward on second pass
• formative

6
Results of The Tests
Improvement of Test Results Pass One to Two
7
Who benefited from the test?
8
Student Feedback

• Excellent way of gaining knowledge
• 2nd Test Allowed us to learn
• Good assistant to learning
• Encourages Revision
• Very good, promotes learning of a subject early
  on
• Better than sex!!!!

9
Theres no confidence in multiple-choice testing

• CAA Conference 2002
• Phil Davies

10
What decides the QUALITY of a question?

• In the 2001 World cup qualifier between England
  and Greece, who scored the all important
  equalizing goal?
• David Beckham
• Victoria Beckham
• Brooklyn Beckham
• Rebecca Loos

11
Rewards?

• Know it
• On the tip of my tongue
• If I see the answers Ill know it
• Deduction
• Guess
• Know it oh no Ive got it wrong ..
  misinformation

12
(No Transcript)
13
Proportion of Questions Correct

• High Confidence gt 84 correct
• Fairly Confident gt 58 correct
• Not confident gt 47 correct
• 72 of all questions answered with high confidence

14
Student thoughts on Confidence

• Eliminates guesswork
• I was waiting for a leggy blond to bring on my
  prize at the end
• separated the lucky students from me
• at least I didnt lose marks for selecting no
  confidence
• are you testing my confidence or knowledge

15
Quality of a multiple choice question?

• In the 2001 World cup qualifier between England
  and Greece, who scored the all important
  equalizing goal?
• Teddy Sheringham
• Michael Owen
• Robbie Fowler
• QUALITY OF QUESTION NOT DISTRACTERS

16
Bite The Bullet

• Use Confidence Testing for all four tests (10,
  15, 20 25)
• Weighting of Passes (50/50, 60/40, 70/30 80/20)
• How to judge if the students had really
  learned?
• Identify improvement gt
• Select five questions that were answered
  poorest in test one pass through to test two
• Select five questions that were answered
  poorest in test two pass through to test
  three ( original five) . Etc.

17
How to identify Poorest

• Using confidence, degree of correctness of
  answering
• Using confidence scores to provide a degree of
  poorness
• i.e. 4, 2, 1, -6, -5, -4
• High degree of poorness could just be a BAD
  question
• Swing between 1st and 2nd pass of degree of
  correctness should be Reasonable

18
(No Transcript)
19
(No Transcript)
20
Identifying Poor Questions Test One
21
Identifying the Poorest Questions Test two to
three
22
Identifying Poorest Questions from Three to Four
23
Passed through to Test Four

• 13 of the poorest question
• 6 new questions based upon additional work
  between tests 3-4
• 21 questions (cross section of questions and
  topics from previous tests)

24
Improvement of Results Through Tests
25
Improvement Pass One of Questions
26
Improvement for Question 5

• Stronger Students improved quickly through the
  tests
• Became more confident in answering the questions
• Weaker Students improved less quickly than
  stronger through the test progress

27
Is it just Confidence or Knowledge i.e. questions
right or wrong?
28
Some Concerns
29
Overall Comments

• Work in Progress
• Results look very positive
• Correctness Factor is dependant upon weighting of
  scores
• Guessing not so much of a problem
• Changed students mode of revising / study
• Fine granularity in the QUALITY OF A QUESTION
• Possibility of Computerized Adaptive Testing
• However note the quality of the question change
  throughout the course of the cumulative tests
• Does gender come into it??

30
Contact Information

• Email pdavies_at_glam.ac.uk
• Phone [phone]
• School of Computing
• University of Glamorgan
• Pontypridd
• Mid Glamorgan
• South Wales