ICT

1
ICT Examinations in Dutch secondary education

• Mark Martinot
• Projectmanager ICT and examinations
• Cito / unit secondary education
• Arnhem, Netherlands

2
National Institute for Educational Measurement
Cito

• 500 employees 2,000 freelancers
• Develop instruments for objective evaluation of
  students achievement
• Growing focus on use of ICT (itembanking, CBT,
  CAT)
• Introducing CBT in national examinations

3
(No Transcript)
4
National examination in figures

• Cito composes together with 1000 teachers the
  examinations
• 600 different examinations
• 200,000 examinees
• more than 700 schools
• 49 million A4 pages

5
Digital examination

• Start changing from paper and pencil to CBT
• Working at a countrywide CBT-infrastructure
• Existing computerexamsturning to one system
  (CitoTester)

6
Project Computers Examinations

• Start in 2002
• Working together with all partipicants-
  Ministry, examination board (Cevo)- Information
  Managenent Group (IBG)- Test-construction
  groups- Schools officials, ICT-staff, teachers,
  students
• Selection of exams where ICT gives 'added value'
  - use of audio/video, interactions (arts,
  biology) - flexibel organization (prevocational
  education)
• Gradual implementation- software simple to more
  complex- items simple (multiple choice) to rich
  (multtmedia, interaction)- interest low stake
  to high stake- schools small numbers to large
  numbers

7
Software-development (CitoTester)
2002 version 1 (simple, only multiple choice,
pictures, 'stand-alone') 2004 version
2 (multi-media, open answers, marking-module,
manager-module, webserver for data) 2006
version 3 (more security, more itemtypes,
adaptive testing, independant second marking
) 2008 version 4 (more generic modules, new
technologies / smart clients)
8
Implementation-stages
A Proof of concept try-outs B Experimental
use with schools (small scale - large scale) C
Available for all schools (choice for paper or
CBT) D Regular use (only CBT, no paper)

• - go / nogo -
• - go / nogo -
• - go / nogo -

9
Implementation-stages
A Proof of concept try-outs B Experimental
use with schools (small scale - large scale) C
Available for all schools (choice for paper or
CBT) D Regular use (only CBT, no paper)

• - Version 4
• - pre-university and senior general secondary
  exams (20-50 schools)
• - pre-vocational exams (500 schools)

10
Results Reactions

• Students
• Generally positive(less language, more
  structure)
• Interaction problems e.g. scrolling
• Teachers
• Positive when ICT really makes a changee.g.
  questioning in realistic context
• Positive when ICT saves time

11
Quality criteria / requirements

• Item-types- multiple choice, open answer,
  interaction/simulation- using audio/video, hot
  spot, simulation, plug-ins (Flash)- different
  scoring possibilities (automated, marking by
  teachers)
• Test-formats- lineair, random, adaptive-
  exercise / informal - scheduled / formal-
  reporting and feedback
• Itembank-system- support for new formats,
  scoring rules, test-definitions- publishing to
  different standards (IMS/QTI compliant)

12
Quality criteria / requirements

• Data-collection/processing - testdata and
  metadata - interoperability with analysing-tools
• Scalability- support for large number of tests
  taking place simultaneously- no dependencies of
  internet-connections during test- smooth
  performance of multimedia (video) under all
  conditions
• Security - fall back scenario's in case of PC-,
  server- or networkfailures- encryption of
  (distributed) items / answers / scores, save
  storage - blocking other applications (email,
  chat) during test - logging activities during
  test ('walkthrough')- authorization of users-
  time-lock on content, prohibit printing

13
Architecture / design

• Flexibility / maintainability - modules for
  different components (item- testcreation,
  scheduling, testdistribution, testlaunching,
  scoring marking, data-collection,
  data-analysis, reporting) - documentation, use
  of standards, metadata
• User-friendly (schools)- using local
  ICT-infrastructure, low system requirements-
  efficient for local managers- simple menu's /
  intuitive interface (navigation, lay-out issues,
  etc.)

14
CitoTester Modules

• TestCenter

• Student
• exercise- making exams

• TestManager

• School-secretary
• auhorize users
• scheduling exams
• sending results to Cito

• MarkingManager

• Teachers/Censors
• marking answers- at school or at home

15
(No Transcript)
16
(No Transcript)
17

• CitoTester / TestCenter

18
(No Transcript)
19
Voorbeeld CBT examen
20
(No Transcript)
21
Voorbeeld CBT examen
22
(No Transcript)
23
New developments

• Support for disabled pupils (text to speech)
• New itemtypes(drawings, simulation, interaction)
• More sofisticated scoring strategies(multiple
  response-items, interaction)