COMPUTERADAPTIVE TESTING: A CASE STUDY

1
COMPUTER-ADAPTIVE TESTING A CASE STUDY

• Mariana Lilley and Trevor Barker

2
Objectives

• Overview of ongoing research into the use of
  computer-adaptive tests (CATs)
• Background to study
• Findings
• Summary
• Future work

3
Research overview

• University of Hertfordshire
• Research started in 2001
• Computer-Adaptive Test
• Based on Item Response Theory (IRT)
• If a student answers a question correctly, the
  estimate of his/her ability is raised and a more
  difficult question is presented
• If a student answers a question incorrectly, the
  estimate of his/her ability is lowered and an
  easier question follows
• Final score (i.e. proficiency level) based on
  number of correct/incorrect responses and level
  of difficulty of questions

4
CAT-UH

• The CAT prototype comprised
• Graphical user interface
• Calibrated database of questions
• Adaptive algorithm

5
Graphical user interface
6
Adaptive Algorithm

• Dichotomously scored questions, such as
  multiple-choice and multiple-response
• Three-Parameter Logistic Model
• , test-takers proficiency level (or ability)
• b, questions difficulty
• a, questions discrimination
• c, pseudo-chance

7
The Study

• 205 Computer Science undergraduate students
  participated in three assessment sessions within
  the Visual Basic.NET subject domain
• All assessments sessions were summative and took
  place in computer laboratories, under supervised
  conditions

8
The Study

• Assessment 1
• A traditional Computer-Based Test (CBT) session,
  where all students were provided with the same
  set of predefined questions
• Scores ranged from 0 (lowest) to 100 (highest)
• Assessment 2
• A Computer-Adaptive Test (CAT) session using our
  CAT prototype
• Proficiency level scores (CAT-PL) ranged from -3
  (lowest) to 3 (highest)
• Assessment 3
• A Practical Programming Exam (PPE), where each
  participant had to implement a program based on a
  set of specifications
• Scores ranged from 0 (lowest) to 100 (highest)

9
Performance summary (N205)
10

• To investigate the fairness of the CAT approach,
  student results were subjected to a Pearson's
  Product Moment correlation

11
Pearson's Product Moment correlation (N205)
12
Summary

• The findings from this study were taken to
  indicate that students were not disadvantaged by
  the CAT approach adopted in this study
• Those learners performing well on the CAT
  component also performed well on the CBT and PPE
  components of the assessment and vice-versa
• Earlier work by the authors has shown benefits of
  the CAT approach, such as increased learner
  motivation
• In this study, we were able to show that CATs
  offer comparable measurements of student
  performance to traditional assessment methods

13
Future work

• Research into how CAT proficiency level estimates
  could be employed to support learners individual
  development

14
Contact Details

• Mariana Lilley
• M.Lilley_at_herts.ac.uk
• Dr Trevor Barker
• T.1.Barker_at_herts.ac.uk