Innovative Assessment Technologies for eLearning

1
Innovative Assessment Technologies for eLearning
Stewart Long
2
Presentation Overview

• Background
• Traditional CBA vs. Authentic Assessment
• OCR case study
• Applications to eLearning
• Demonstration

3
Background

• 1993-6 TLTP funding at UEA
• 1996-2003 OCR (RSA) funding to automate
  assessment of IT skills
• 2003 LARGO Technologies formed to carry forward
  world leading assessment technologies developed
  at UEA.

4
Traditional CBA

• Multiple Choice Questions
• Atomic Functions Tests
• Simulations/limited functionality
• Over the shoulder model tracing

5
Authentic Assessment

• Practical skills
• Multiple methods, backtracking, etc.
• Complex, compound tasks
• Realistic contexts
• Industry standard tools
• Complete output assessed
• Complex assessment criteria

6
Basis of Automated Assessment

• Exercise output files gathered from real
  applications
• File comparison of model and candidate solutions
  identifies differences (potential errors)
• Differences counted and categorised under
  assessment objectives using context sensitive
  rules
• Overall result classification determined
• Output

7
OCR Case Study

• Automated assessment of New Clait scheme
• Word processing, spreadsheets, databases, graphs
  and charts, etc.
• Many exam centres, thousands of candidates
• Internet submission
• Central assessment control via automated workflow

8
OCR Benefits

• Work reduction for centres
• Speed - assessor and system
• Accuracy
• Capacity
• Cost reductions
• Improvements in QA
• OCR profile

9
Applications to eLearning

• Usage scenarios
• Workplace, home, training centre, schools, etc.
• Practice, formal assessment, tuition
• Users
• Tutors
• Learners
• Benefits
• Access, speed, adaptability, authentic skills

10
Mechanisms for adapting to eLearning

• Assessment objectives as high level learning
  goals
• Link advice to individual assessment (error
  counting) rules
• Delivery and feedback issues
• Web/CD/network, In-application/external, user or
  tutor controlled, feedback grain size, immediate
  or delayed feedback

11
High Level Objective Output
12
Context-Sensitive Error Level Feedback
Advice Text

• IF difference type is INSERT FULL STOP and
  context is MOVED TEXT
• THEN AO is MOVE TEXT, count type is PER
  EXAMINATION with priority 2
• ADVICE Be careful you dont miss the final full
  stop when moving text.
• IF difference type is INSERT FULL STOP and
  context is INSERTED TEXT
• THEN AO is DATA ENTRY ACCURACY, count type is PER
  WORD with priority 2
• ADVICE Always check your data entry accuracy,
  including punctuation.

13
Innovative Assessment Technologies for eLearning
Stewart Long
14
Multiple Choice Questions
15
Atomic Functions Tests
16
Simulations/limited functionality
17
Over-the-shoulder model tracing