Neural test theory model for graded response data

1
Neural test theory model for graded response data

• SHOJIMA Kojiro
• The National Center for University Entrance
  Examinations, Japan
• shojima_at_rd.dnc.ac.jp

2
Accuracy of tests

• Weighing machine
• A1 weighs 73 kg
• fW(A1)73
• fW (A1)?74
• fW (A1)?72

• Academic test
• B1 scores 73 points
• fT(B1)73
• fT(B1)?74 ?
• fT(B1)?72 ?

3
Discriminating ability of tests

• Weighing machine
• A1 weighs 73 kg
• A2 weighs 75 kg
• fW(A1)ltfW (A2)

• Academic test
• B1 scores 73 points
• B2 scores 75 points
• fT(B1)ltfT (B2) ?

4
Resolving ability of tests

• Weighing machine
• A1 weighs 73 kg
• A2 weighs 75 kg
• A3 weighs ...

• Academic test
• B1 scores 73 points
• B2 scores 75 points
• B3 scores ...

5
Neural Test Theory (NTT)

• Academic tests are an important public tool
• Precise measurements are difficult
• 10 measurement error
• Tests are at best capable of classifying academic
  ability into 520 levels
• Neural test theory (NTT)
• Shojima, K. (2009) Neural test theory. K.
  Shigemasu et al. (Eds.) New Trends in
  Psychometrics, Universal Academy Press, Inc., pp.
  417-426.
• Test theory that uses the mechanism of a
  self-organizing map (SOM Kohonen, 1995)
• Latent scale is ordinal

5
6
Graded evaluation ? Accountability ? Qualification
test
7
Statistical Learning Procedure in NTT

• For (t1 t T t t 1)
• U(t)?Randomly sort row vectors of U
• For (h1 h N h h 1)
• Obtain zh(t) from uh(t)
• Select winner rank for uh(t)
• Obtain V(t,h) by updating V(t,h-1)
• V(t,N)?V(t1,0)

Point 1
Point 2
7
8
NTT Mechanism
Response
Point 1
Point 2
Point 1
Point 2
Latent rank scale
8
9
Point 1 Winner Rank Selection
Likelihood
ML
Bayes

• The least squares method can also be used.

9
10
Point 2 Update the rank reference matrix

• The nodes of the ranks nearer to the winner are
  updated to become closer to the input data
• h tension
• a size of tension
• s region size of learning propagation

10
11
Analysis Example

• A geography test

N 5000
n 35
Median 17
Max 35
Min 2
Range 33
Mean 16.911
Sd 4.976
Skew 0.313
Kurt -0.074
Alpha 0.704
11
12
Fit Indices
ML, Q10
ML, Q5

• Useful for determining the number of latent ranks

12
13
Item Reference Profiles
13
Monotonic increasing constraint can be imposed
14
Test Reference Profile (TRP)

• Weighted sum of IRPs
• Expected value of each latent rank

• Weakly ordinal alignment condition
• TRP increases monotonically, but not all IRPs
  increase monotonically
• Strongly ordinal alignment condition
• All IRPs increase monotonically ? TRP also
  increases monotonically
• For the latent scale to be an ordinal scale, it
  must at least satisfy the weakly ordinal
  alignment condition (WOAC).

14
15
Rank Membership Profile (RMP)

• Posterior distribution of the latent rank to
  which each examinee belongs

RMP
15
16
Examples of RMP
16
17
Extended Models

• Graded Neural Test Model (RN07-03)
• NTT model for ordinal polytomous data
• Nominal Neural Test Model (RN07-21)
• NTT model for nominal polytomous data
• Continuous Neural Test Model
• Multidimensional Neural Test Model

17
18
Graded NTT ModelBoundary Category Reference
Profiles
19
Graded NTT ModelItem Category Reference Profile
20
Nominal NTT ModelItem Category Reference
Profile Correct selection, x Combined
categories selected less than 10 of the time
21

• Website
• http//www.rd.dnc.ac.jp/shojima/ntt/index.htm
• Software
• EasyNTT
• By Prof. Kumagai (Niigata Univ.)
• Neutet
• By Prof. Hashimoto (NCUEE)
• Exametrika
• By Shojima (NCUEE)

21
22
Demonstration of Exametrika
23
Can-Do Chart (Example)
23