Assessment: Challenge for learners

1
Assessment Challenge for learners markers

• Professor Colin Mason

2
Why assess?

• "Assessment is the most powerful tool teachers
  possess in moulding student effort and learning."
• Graham Gibbs and Trevor Habeshaw, 1988
• "If we wish to discover the truth about an
  educational system, we must look into its
  assessment procedures .. the spirit and style
  of student assessment defines the de facto
  curriculum."
• Derek Rowntree, 1987

3
Pick the best answerWhy are lectures still used
so frequently in higher education?

• So that lecturers can demonstrate how expert they
  are in their subject.
• Because they are a very efficient means of
  transmitting information to large numbers of
  students.
• To provide a common space where students can be
  enthused by a lecturer exploring key concepts of
  their subject.
• Students expect them as part of university
  (rather than school) education and prefer
  listening to having to contribute themselves.
• Lectures are easier to prepare than more
  interactive resource-based techniques such as
  case study discussions or problem-based tutorials.

4
Tensions in Assessment(George Brown, 1996)

• EFFECTIVENESS
• Fits aims and learning outcomes
• EFFICIENCY
• Saves time (and effort - eventually)
• ENABLEMENT
• For understanding and expertise

5
Some examples of innovation in methods of
assessment
Exams
Coursework
Unseen exams
Presentations
Annotated Bibliography
Seen exams
Reports
Design and create VLE course
Posts to discussion boards
Essays
PBL scenarios
MCQs
OSCEs
Workbooks, diaries, logs
Reflective Report
Practical tests
Authentic Work-based observations by professionals
Higher Cognitive skills, Transferable skills
Knowledge, Understanding
See also Integrative Assessment Guide no 3
6
Complex outcomes of learning

• Complex outcomes including higher order
  academic abilities (analysis, critical reasoning)
  and soft skills (teamwork, leadership) are
  rarely and inconsistently defined
• Advanced skilful practice is acquired slowly
  (years?)
• Precision reliability of such assessment is
  often only attained at the expense of validity

7
Forms (Modes) of Assessment
Reality
Theory
8
So, formative assessment .

• Helps learning
• Provides diagnostic information for students and
  staff (even if only scores)
• Provides opportunities for Feedback (corrections)
  and Feedforward (suggestions for improvement for
  next time)
• Ensures wider coverage of all learning outcomes

9
Speedier Feedback

• Speeding up feedback accelerates subsequent
  learning
• Solutions
• objective assessments with instant feedback
• exemplar answers
• Objective assessment (eg MCQ, matching etc),
  especially utilising online approaches, form a
  significant element of an overall assessment
  strategy EVS also.
• Problem-based learning with ideal or exemplar
  solutions made available (often online)

10
Assessment Efficiency

• Synoptic summative assessment
• Multiple Choice Questions (MCQ)
• More sophisticated Objective Tests
• Computer-based (online) administration of
  assessments

11
Blended assessment between units Synoptic
approach?
LATERAL
LONGITUDINAL
TRIMESTER 2
TRIMESTER 1
12
Assessment Efficiency MCQs?http//www.ukcle.ac.
uk/resources/trns/mcqs/index.html

• What attracts me about using MCQ assessment?

What are my concerns about using MCQ assessment?
13
Assessment Efficiency MCQs?http//www.ukcle.ac.
uk/resources/trns/mcqs/index.html

• What is an MCQ?
• What could I use MCQs for?
• What are the features of MCQs?
• Does adopting MCQ assessment mean 'dumbing down'?
  
• Are MCQ scores unrealistically high?
• Can students guess their way to success in an
  MCQ?

14
Assessment Efficiency MCQs? http//www.ukcle.ac.
uk/resources/trns/mcqs/index.html

• Does using MCQ assessment encourage rote or
  surface learning?
• Can MCQs test oral and written skills?
• What should I think about before I design an MCQ
  test?
• How can I write effective MCQs?
• How can I provide effective feedback for my MCQs?
  
• What can I learn from the student responses to
  the MCQs?

15
What is an MCQ Design issueshttp//www.le.ac.uk
/castle/resources/

• A Multiple Choice Question (MCQ) is one that
  students are asked to select one answer from a
  given list of options in response to a question
  stem
• Which of the following is the currency unit used
  in Albania?
• Dinar
• Rouble
• Kwacha
• Lek
• Rupee

16
What are the features of MCQs?

• Objective marking
• a right or a best answer
• but scope for crediting more than one right
  answer or 'near miss' answers, depending on the
  marking scheme
• Testing is efficient
• greater part of syllabus assessed by compulsory
  questions.
• Answers to Questions are easy to score or mark
• Especially online or computer-assisted eg DSO,
  WebCT, Blackboard, QuestionMark.
• Scoring uses the entire marking range (of 100
  scale)
• raises issues where the MCQs are used in
  summative assessment.

17
What are the features of MCQs?

• Results are quantifiable
• possible to analyse the level of student
  achievement in each question and thus identify
  areas of student difficulty or problematic
  questions
• Feedback can be targeted more effectively
• Setting questions is time consuming and
  challenging
• Initial investment is high, but questions can be
  re-cycled in databanks
• Recycling of questions raises quality issues
  for summative assessments

18
MCQs Checking Knowledge?Understanding and
Analysis?
19
Rank (R) the options (1 - 5)There is an
increasing interest in the use of formative
assessment in higher education because

• Fewer students submit assignments, reducing staff
  marking loads.
• Students can choose whether or not to complete an
  assessment assignment.
• It allows students to choose when, where and how
  they complete an assessment assignment.
• It provides non-threatening, low stakes,
  opportunities for students to learn from their
  mistakes and feedback provided on their assessed
  work.
• Academic staff need to worry less about the
  accuracy of their marking because it doesnt
  count.
• (If correct, R15 marks R23 marks R32 marks
  R41 R50)

20
Evaluate the sentence explaining why MCQ tests
are used in higher education these days

• MCQ tests are increasingly used in assessment in
  higher education courses BECAUSE they can easily
  be marked by computer systems and are thus more
  objective
• The assertion and the reason are both correct,
  and the reason is appropriate or valid.
• The assertion and the reason are both correct,
  but the reason is inappropriate or invalid.
• The assertion is correct but the reason is
  incorrect.
• The assertion is incorrect but the reason is
  correct
• Both the assertion and the reason are incorrect

21
High Marks and Guessing What to do about it?

• An MCQ with 5 options presents a 20 chance of
  'guessing' the correct answer
• negative scoring?
• adopting mathematical strategies to 'normalise'
  the mark (see Bush, 1999)?
• raising the pass mark for the MCQ element of
  assessment?
• making MCQs one component (low weighting) of the
  assessment strategy?
• concentrating on using MCQs in formative
  situations where the 'result' is less important
  than the process?

22
True/False -a special case MCQ

• 100 T/F
• (Design1 mark correct 0 mark incorrect)
• Monkey Mark 50/100 50 (Pass?)
• 100 T/F
• (Design 1 mark correct -1 mark incorrect)
• Excellent student 100
• Monkey Mark 0

23
MCQ Negative mark Designs

• 3 option MCQ
• Unfair 1 mark correct -1 mark each incorrect
  distractor
• So .
• 1 mark correct -0.5 mark each incorrect
• But, easier .
• 2 mark correct -1 mark each incorrect

24
I C/(n-1)MCQ scheme for negative marking
25
Negative scoring for the whole test

• Score Deserved score undeserved score (for
  guessing)
• 100 MCQ with 5 options correct option 4 marks
• 25 guesses 1 in 5 chance (20)
• undeserved score 20 of 25 x 4 marks
• undeserved score 5 x 4 20 marks
• Deduction for undeserved score No. Qs (25-5) x
  I
• Deduction for undeserved score 20 x 1 20
• So, adjust Score according to
• Score Deserved undeserved deduction
  undeserved
• Score Deserved 20 20
• Score Deserved

26
What about educated guessing?

• Situation 100 MCQs (5 options)
• Student knows 50 guesses 25 but educated
  guesses 25 (8 correctly 17 incorrectly)
• Scoring
• Deserved 50 x 4 200
• Undeserved 5 (20 of 25) x4 20xI 0
• Partial 8 x4 -17 x1 15
• Total 200 0 15 215 (out of 400)
• If no negative marking ie 0 for incorrect
  answer)
• Total 200 (5x4 20x0) (8x4 17x0)
• Total 200 20 32 252

27
Other Adjustments

• Order of preference (or ranking) of options
• Confidence
• Choose option and assign confidence level (1,
  2, 3)
• If correct marks 1, 2 or 3
• If incorrect marks 0, -2, -6
• Normalisation
• X ? (x-20) 5/4 (x unadjusted mark)
• Liberal (more attempts) MCQ
• Eg 5 option MCQ
• 1 attempt score 4/4 100 on Question
• 2 attempts score (4-1)/4 75 on Question
• 3 attempts score (4-2)/4 50 on Question etc

28
Formative online assessment

• On-line approaches
• Deakin Studies Online (DSO)
• e.g. Blackboard, Moodle
• Assessment Management Systems
• e.g. QuestionMark, TRIADS
• Tools
• e.g. Respondus, StudyMate
• Techniques for Formative Assessment
• Simple
• Medium
• Advanced
• Fun

29
On-line Assessment Simple

• Self Assessment Exercises
• DSO Self Assessment Tool
• MCQs within content using
• (simple) javascript

30
On-line Assessment Medium

• DSO tests/quizzes
• Multiple Choice
• True/False
• Multiple Answer
• Matching Pair
• Fill in the blanks
• Short Answer
• Paragraph/Essay

31
Example

• What are the Christian names for the Islamic
  prophets Ibrahim, Daviyd, Musa and Isa?

32
Multiple Choice
33
Multiple Answer
34
Fill in the blanks
Answered Correctly
Alternative Answers
35
Matching Pair
36
On-line Assessment Advanced

• Same as Medium Technique
• PLUS
• Randomisation of a questions possible answers
• Randomisation of questions
• All randomised
• Randomised from bank
• e.g. any 30 questions from 100
• Pre-defined and randomised
• e.g. first 10 defined then 30 random
  questions

37
On-line Assessment Advanced

• Same as Medium Technique
• PLUS
• Timed assessments (beat the clock)
• Timed released
• Incentive driven e.g.
• If score gt80 release other learning materials
• Extra support
• If score lt40 release more learning materials
• extra test

38
On-line Assessment Fun

• Flash based - can be used within DSO or on
  standard web pages e.g.
• Crosswords
• Pick a Letter
• Challenge
• And more

39
Useful URLs

• Approaches to assessment and feedback that foster
  independent learning
• http//www.deakin.edu.au/itl/pd/tl-modules/assessm
  ent/index.php
• Online assessment Strategies
• http//www.deakin.edu.au/itl/dso/strategies-teachi
  ng/assessment.php
• UK resources to assist MCQ designers
• http//www.le.ac.uk/castle/index.html
• Studymate
• http//www.deakin.edu.au/itl/dso/tools/studymate.p
  hp
• Wimba Create (CourseGenie)
• http//www.wimba.com/products/wimba_create/
• Votapedia
• http//www.votapedia.com/index.php?titleMain_Page
  

40
Contact us

• Institute of Teaching and Learning
• http//www.deakin.edu.au/itl/
• ITL Support Service
• http//www.deakin.edu.au/itl/contact/contact-us.ph
  p
• And,
• Phone [phone] (ext. 78127)
• Email itl-support_at_deakin.edu.au