Heriot Watt University Breakout Session MCQ with Confidence

1
Heriot Watt UniversityBreakout SessionMCQ with
Confidence

• Dr Phil Davies
• Division of Computing Mathematical Sciences
• Department of Computing
• FAT
• University of Glamorgan

2
E-Assessment can be viewed from different
perspectives

• Commercial / Large Scale Implementations
• Question Marks Perception
• MOODLE
• Triads (25 different types of Multiple Choice
  Type Questions!)
• TOIA (FREE!) Tool for On-line Interoperable
  Assessment
• Blackboard Assessment (version!)
• Standards (IMS QTI Specification)
• Try it and see

3
How easy to do MCQ?Cost?

• Question
• Key
• Distracter One, Distracter Two, ..
• Note
• Key(s) must always be correct
• How many students would you expect to get it
  right for it to be a GOOD question?
• Distracter One / Two / Three equal of
  selections
• Throw away BAD questions?

4
TOO HARD .. ?

• Too time consuming
• Often needs to be an iterative process
• Does it matter if a question is NOT perfect?
• Seeking perfection not feasible
• I suggest that assessment MUST be included within
  a module, not only at end make it embedded
  within TEACHING LEARNING experience
• Formative / Summative / Both

5
My Entry into CAA

• 1st Attempt TOTAL FAILURE 1998 OLAL
• 5 minutes all finished apart from one student
• Guessing
• Why should the student care if incorrect?
• How to make it worthwhile slow students down
  make them think
• Negative marking used 2 and -1 (actually becomes
  -3 double whammy!)
• Allow TWO PASSES of the test
• By concentrating on first pass
• will get reward on second pass
• Formative
• IMPORTANT WORTH MARKS

6
(No Transcript)
7
Own Background Objective Testing via MCQ

• Four Tests MCQ .. Two passes 50/50 (became
  graduated)
• Cumulative Tests 5, 10, 15, 20
• Amateur .. No Research Pedigree (CAA Conference)
• Improved Class results not easier more
  directed
• Weaker students improved dramatically
• Assessed the learning outcomes of subject
• END JUSTIFIES THE MEANS

8
What did the students say about it?

• Excellent way of gaining knowledge
• 2nd Test Allowed us to learn
• Good assistant to learning
• Encourages Revision
• Very good, promotes learning of a subject early
  on
• Better than sex!!!!

9
What decides the QUALITY of a question?

• In the 2004 World cup qualifier between England
  and Greece, who scored the all important
  equalizing goal?
• David Beckham
• Victoria Beckham
• Brooklyn Beckham
• Romeo Beckham

10
Rewards?

• Know it
• On the tip of my tongue
• If I see the answers Ill know it
• Deduction
• Guess
• Know it oh no Ive got it wrong ..
  misinformation

11
TRIED ONE CONFIDENCE TEST ONLY SHOWS ANSWERS ONCE
CONFIDENCE SELECTED RELATES TO MARKS 4, 2, 1,
-2, -1, 0
12
Proportion of Questions Correct

• High Confidence 84 correct
• Fairly Confident 58 correct
• Not confident 47 correct
• 72 of all questions answered with high confidence

13
Student thoughts on Confidence

• Eliminates guesswork
• I was waiting for a leggy blond to bring on my
  prize at the end
• separated the lucky students from me
• at least I didnt lose marks for selecting no
  confidence
• are you testing my confidence or knowledge

14
Bite The Bullet

• Use Confidence Testing for all four tests (10,
  15, 20 25)
• Real SHOCK to students
• Weighting of Passes (50/50, 60/40, 70/30 80/20)
• How to judge if the students had really
  learned?
• Select five questions that were answered
  poorest in test one pass through to test two
• Select five questions that were answered
  poorest in test two pass through to test
  three ( original five) . Etc.

15
(No Transcript)
16
Identifying Quality of a Question

• Negative on first pass e.g. -180
• Difference between first and second pass
• Too big an improvement may just signify a very
  poor question e.g. 500?
• Create heuristic that identifies degree of
  difficulty of a question

17
Passed through to Test Four

• 13 of the hardest question
• 6 new questions based upon additional work
  between tests 3-4
• 21 questions (cross section of questions and
  topics from previous tests)

18
Improvement of Results Through Tests
19
Improvement Pass One of Questions
20
Improvement for Question 5

• Stronger Students improved quickly through the
  tests
• Became more confident in answering the questions
• Weaker Students improved less quickly than
  stronger through the test progress, but DID
  improve

21
Is it just Confidence or Knowledge i.e. questions
right or wrong?
22
Quality of a multiple choice question?

• In the 2004 World cup qualifier between England
  and Greece, who scored the all important
  equalizing goal?
• Teddy Sheringham
• Michael Owen
• Robbie Fowler
• QUALITY OF QUESTION NOT DISTRACTERS

23
Developmental v Assessment

• Need not be at alternative ends of the scale
• Student motivation / attendance increased
  considerably
• Testing knowledge (not done at moment for higher
  order skills)
• Good students have best results i.e. number of
  questions correct

24
COMMENTS?

• HMCQ
• Confidence
• Two Pass
• Formative Summative
• Graduated Assessment
• Continual Assessment

25
(No Transcript)