Design and Evaluation of an XMLBased PlatformIndependent Computerized Adaptive Testing System - PowerPoint PPT Presentation
1 / 20
Title:
Design and Evaluation of an XMLBased PlatformIndependent Computerized Adaptive Testing System
Description:
... factor: 0.526. Reporter: Tsung-Yao Lee (???) Date: 2005/9/10 ... Precision and Efficiency. CAT platform (PC or PDA) does not cause significant difference on ... – PowerPoint PPT presentation
Number of Views:37
Avg rating:3.0/5.0
Title: Design and Evaluation of an XMLBased PlatformIndependent Computerized Adaptive Testing System
1
Design and Evaluation of an XML-Based
Platform-Independent Computerized Adaptive
Testing System
- Author Rong-Guey Ho and Yung-Chin Yen
- Source IEEE Transaction on Education, VOL. 48,
NO. 2, May 2005 - SCI 10/20 50 Impact factor 0.526
- Reporter Tsung-Yao Lee (???)
- Date 2005/9/10
2
Outline
- Introduction
- IRT-based CAT
- XML-based CAT system
- Experiment
- Result
- Conclusion
- Comment
3
Introduction
- CAT can be tailored to the ability of each
examinee - Based on item response theory (IRT)
- Benefit
- Improve the accuracy of testing
- Save testing time
- Require fewer items per examinee
- Using XML to describe the item bank, one is able
to solve the problem while implementing a CAT
system on a different platform (PC, PDA, Handheld
device)
4
Item Response Theory (IRT)
ai item discrimination bi item difficulty ci
pseuso-guessing
ai 2.0, bi 0.0, ci 0.25 ? -3.0 to 3.0
5
Adaptive strategy
- 1) An initial ability is estimated in some way
- 2) The initial ability estimate is used to select
an appropriate item from the item bank. - 3) The item is scored as either correct or
incorrect, and the examinees ability level is
reestimated - 4) If the estimate is sufficiently precise or a
prespecified criterion has been reached, the
procedure is terminated. Otherwise, the estimate
is further refined by the return to Step 2
6
XML-Based CAT System
7
(No Transcript)
8
Workflow 1/3
- Step 1) The item parameters of each item in the
item bank will be sent from the item retrieval
subsystem to the CAT subsystem as soon as the
system is activated. - Step 2) The system first detects the type of
platform - Step 3) Assuming the examinee is of median
ability, the system will pick an initial item of
medium difficulty and send its item number to the
item retrieval subsystem.
9
Workflow 2/3
- Step 4) CAT subsystem extracts the XML-format
item from the bank and sends it to the XSL
implementation procedure. - Step 5) XSL implementation procedure generates
the HTML-format item by applying the
corresponding XSL template - Step 6) displays HTML-format item on the screen
10
Workflow 3/3
- Step 7) After the answer has sent, the system
will check the answer and reevaluate current
ability - Step 8) The system will then check whether the
stopping condition is met. If the stopping
condition is not met, the next step will return
to Step 3). - Step 9) If the stopping criterion is met, the
system shows his or her final estimation of
ability on the screen and sends the response data
to response database for further analysis.
11
Experiment design
- Participants
- Fifty senior high school students, 95.92 had at
least one year of computer experience. - Only 28 had ever taken tests on the computer,
and only 22 of them had experience with a PDA. - Item bank
- 84 items from the vocabulary sections of CEEC
- The value of IRT parameters of each item was
calibrated using the BILOG program following a
3-p logistic model based on 5000 high school
students - Properties in item bank
12
Procedure
- Group A (n25) PC-version CAT taken first
PDA-version CAT taken second. - Group B (n25) PDA-version CAT taken first
PC-version CAT CAT taken second. - PC-CAT was administered on Intel Pentium III PC
with 128M RAM - PDA-CAT was on HP Jornada 565 and COMPAQ
iPAQH3850.
13
Precision and Efficiency
Ability
Time
Item no.
- CAT platform (PC or PDA) does not cause
significant difference on - accuracy (ability)
- efficiency (time, item no.)
14
Participants Response Toward Testing Environment
15
Conclusion
- CAT on PDA is practical
- The results may differ when experiment is of a
different age or other domains - Probably is only a result of the examinees
curiosity regarding PDA
16
Future work
- Revising this system to a platform-independent,
on-line practice system - Implementing this system on other domains
- Expanding the diversity of the CAT administration
platform
17
Comment
- ??????????,??XML????????????
- ???????????,??????SCORM????
- ???XML???????,??????SCORM????
18
IRT
- IRT?Load?1980???????????(item response
theory,??IRT),????????????????,????????,??????????
????????? - 1.???(Difficulty index)???????????????????????,?
??????? - 2.???(Pseudoguessing parameter)?????????????????
,?????????????? - 3.???(item-descriminating power)?????????,??????
?????????????
19
(No Transcript)
20
(No Transcript)
