A TestSheetGenerating Algorithm for Multiple Assessment Requirements

1
A Test-Sheet-Generating Algorithm for
MultipleAssessment Requirements

• Authors Gwo-Jen Hwang
• Source IEEE Transactions on Education,
• Vol. 46, No. 3, pp. 329-337,
• August 2003 (SCI and EI)
• Speaker Yi-Ching Lee(???)
• Date 2004/09/04

2
Outline

• Introduction
• Test-Sheet-Generating Approach in ITED
• Implementation of the System
• Performance Evaluation
• Conclusion
• Comment

3
Introduction

• The accuracy of a test depends on
• the quality of the item bank
• the accurate measure of psychometric features for
  each test item
• Manage?maintain?integrity
• the way of the test sheet is constructed.
• It is important how to select appropriate test
  items.
• Assessment requirements of constructing a test
  sheet
• average difficulty degree
• average discrimination degree
• length of the test time
• number of the test items
• the specified distribution of concept weights.

4
Introduction

• Previous methods
• random selection
• Evaluation Balance Table (EBT)-1993
• for balanced concept weights
• Single -criteria greedy method
• In this paper, the multiple-criteria,
  test-sheet-generating problem is formulated, and
  a dynamic-programming approach is proposed for
  the specified requirements.

5
Fig. 1. PTTD project and its subtasks.
6
Test-Sheet-Generating Approach in ITED

• Analyzing Difficulty Level for Each Student
• Constructing the Test-Item-Clustering Table
• Formulating the Test-Sheet-Generating Problem
  With Dynamic Programming

7
Analyzing Difficulty Level for Each Student

• The students request for drills or
  self-assessments.
• This step is skipped for group tests.
• The instructors should consider IQ and test
  records (scores) of the students when determining
  the difficulty degree for each test item.

8
Analyzing Difficulty Level for Each Student

• learners level UIQ, test score evaluation
  results VHigh, Average, Low

9
Analyzing Difficulty Level for Each Student

• the relative weights for u1 and u2W(w1,w2)
• Calculate learners levelBW?R(b1,b2,b3),where
  bj Max(Min(w1,r1j),Min(w2,r2j)) for
  j1 to 3

10
Analyzing Difficulty Level for Each Student
11
Test-Sheet-Generating Approach in ITED

• Analyzing Difficulty Level for Each Student
• Constructing the Test-Item-Clustering Table
• Formulating the Test-Sheet-Generating Problem
  With Dynamic Programming

12
Constructing the Test-Item-Clustering Table

• clustering mechanism
• Fuzzy ART(Adaptive Resonance Theory)
• learning methods of neural networks
• A similarity threshold must be defined, ranging
  from 0 to 1.
• If a new event passes the test the parameters
• of the group are revised.
• If the new event fails to pass the test, a new
  group is created.

13
Constructing the Test-Item-Clustering Table
W11
Wn2
14
Constructing the Test-Item-Clustering Table
VGi(ki1,ki2,,kin) average weight of each
concept
15
Constructing the Test-Item-Clustering Table

• This table can also take the difficulty degree,
  the discrimination degree, and other parameters.
• The system is able to search for a test item with
  certain specified properties.
• least frequently used first
• increase the search performance
• additional costs

16
Test-Sheet-Generating Approach in ITED

• Analyzing Difficulty Level for Each Student
• Constructing the Test-Item-Clustering Table
• Formulating the Test-Sheet-Generating Problem
  With Dynamic Programming

17
Formulating the Test-Sheet-Generating Problem
With Dynamic Programming

• The number of stages is the number of test items
  to be included in the test sheet.
• The test item selected in the i th stage is
  represented as Qi .
• Q(Q1, Q2,,QN) the test items selected during
  the N stages.
• Si and Ei the degrees of difficulty and
  discrimination of the test sheet if a test item
  Qi is selected in i th stage
• Sg and Eg the expected degrees of difficulty and
  discrimination
• SQi and EQithe degrees of difficulty and
  discrimination of the test item Qi
• the distribution for the weights of
  concepts in i th stage.
• the
  accumulated weights of the concepts in the n th
  stage. Initially,

18
Formulating the Test-Sheet-Generating Problem
With Dynamic Programming

• Five test items are to be selected.
• difficulty degree Sg60 discrimination degree
  Eg70
• Initially, S00
  E00
• which group ? which test item

19

• 1st Stage

3 1 1 2 0 0 0
2
1
11
20
Formulating the Test-Sheet-Generating Problem
With Dynamic Programming

• Test item Q1 is selected from with G1 with
• and

21

• 2nd Stage

6 2 2 4 0 0 0
20
22
Formulating the Test-Sheet-Generating Problem
With Dynamic Programming

• Test item Q3 is selected from with G3 with
• and

23
Implementation of the System

• implement in the Windows NT environment
• access the systemvia WWW browsers
• set of parameters

24
Implementation of the System

• avoid generating the same test sheet
• After performing group test, ITED will update the
  difficulty degree and discrimination degree of
  test item.
• allows the user to mark unwanted test items
• An online discussion window is provided and then
  answer the same test items jointly.

25
Implementation of the System

• five text files question-id file, users answers
  file, correct solutions file, the related
  concepts file, and the weight of each concept in
  the questions.
• A well-learned concepts file and a poorly learned
  concepts file are generated.
• present the learning guidance to the users

26
Fig. 1. PTTD project and its subtasks.
27
(No Transcript)
28
Performance Evaluation
Distance(difficulty)
Distance(discrimination)
Sig and Eig represent objective degrees of
difficulty and discrimination for i th case, and
Si and Ei are the degrees of difficulty and
discrimination of the test sheet generated for i
th case. N is the number of the cases.
29
Performance Evaluation
30
Performance Evaluation
31
Performance Evaluation
32
Conclusion

• The approach of this paper achieves desirable
  performance and produce optimal solutions.
• An investigation of the study of finding better
  approaches to generate test sheets from very
  large item banks may be needed.
• A well-constructed test sheet not only helps
  evaluation of the learning status of the
  students, but also facilitates improved diagnosis
  of any problems within the learning process.

33
Comment

• The parameters of select item will associate IRT
  with subject concepts.
• Improve learning guidance

34
Appendix
35
(No Transcript)
36

37
Evaluation Balance Table (EBT)
Sj
1.2-42.8
38
Evaluation Balance Table (EBT)