Caracterization of the distribution of the index of person fit according to the estimated proficiency level

1
Caracterization of the distribution of the index
of person fit according to the estimated
proficiency level

• Gilles Raîche, Université du Québec à Montréal
• Jean-Guy Blais, Université de Montréal

http//www.er.uqam.ca/nobel/r17165/RECHERCHE/2005/
IMPS_05.pdf
International Meeting of the Psychometric Society
, July 2005
2
Contents

• Percentile values with person fit index
• The Lz index
• Objective
• Methodology
• Simulated data
• TCALS II Characteristics
• Analysis
• Results
• Diffrence between the empirical and the
  theoretical percentiles
• Residuals and coefficient of determination
• Predicted hit rate
• Conclusion

3
Percentile Values with Person Fit Index

• Not following a desirable probability
  distribution
• Proficiency level specific
• Estimated proficiency level specific
• Test specific
• Have to be simulated for each test at each
  estimated proficiency level

4
The Lz Index
5
Objective
Elaboration of easier strategies to obtain the
critical percentile values of person fit indices
here Lz
6
Methodology simulated data

• Simulations 2000 at each proficiency level
• Proficiency level -3.00 to 3.00, by steps of
  0.25
• Estimation methods WLE, MLE, MAP, EAP, EXP(EAP
  with uniform), AEAP (EAP with adaptive a priori)

7
Methodology TCALS II Characteristics

• 85 four-choices items
• Parameters average values (Bilog)

Mean Standard deviation
Difficulty (b) -1.11 0.86
Discrimination (a) 1.17 0.43
Pseudo-guessing (c) 0.20 0.06
8
TCALS II Test Characteristic Curve
9
Methodology analysis

• Predicted first and fifth percentiles according
  to
• Predicted first and fifth percentiles with
  multiple linear regression
• Predicted 0.01 et 0.05 hit rate by the multiple
  linear regression

10
Difference between the empirical and the
theoretical fifth percentiles
11
Difference between the empirical and the
theoretical first percentiles
12
Residuals and coefficients of determination for
the fifth percentile
13
Residuals and coefficients of determination for
the first percentile
14
Hit Rate at the 0.05 Predicted Error Level
15
Hit Rate at the 0.01 Predicted Error Level
16
WLE WLE MLE MLE MAP MAP EXP EXP EAP EAP AEAP AEAP
-3.00 0.098 0.046 0.050 0.012 0.051 0.011 0.060 0.012 0.022 0.000 0.031 0.006
-2.75 0.044 0.010 0.041 0.007 0.052 0.011 0.065 0.014 0.051 0.009 0.036 0.007
-2.50 0.038 0.011 0.047 0.011 0.042 0.008 0.047 0.007 0.067 0.015 0.051 0.008
-2.25 0.043 0.007 0.049 0.009 0.049 0.009 0.039 0.005 0.060 0.014 0.047 0.010
-2.00 0.056 0.011 0.060 0.014 0.053 0.010 0.039 0.007 0.064 0.011 0.056 0.013
-1.75 0.050 0.014 0.060 0.015 0.054 0.011 0.041 0.008 0.050 0.012 0.058 0.016
-1.50 0.047 0.009 0.051 0.010 0.057 0.015 0.031 0.007 0.050 0.013 0.052 0.010
-1.25 0.050 0.010 0.057 0.012 0.059 0.011 0.037 0.009 0.050 0.010 0.059 0.014
-1.00 0.048 0.009 0.057 0.011 0.053 0.013 0.030 0.004 0.034 0.005 0.055 0.009
-0.75 0.047 0.009 0.052 0.010 0.050 0.008 0.035 0.006 0.040 0.006 0.054 0.010
-0.50 0.050 0.010 0.056 0.012 0.050 0.012 0.031 0.005 0.029 0.005 0.058 0.012
-0.25 0.044 0.009 0.045 0.009 0.043 0.008 0.040 0.007 0.036 0.006 0.046 0.010
0.00 0.049 0.008 0.030 0.005 0.041 0.007 0.042 0.007 0.040 0.007 0.027 0.004
0.25 0.049 0.011 0.046 0.010 0.041 0.010 0.059 0.014 0.047 0.011 0.050 0.011
0.50 0.042 0.009 0.036 0.008 0.032 0.007 0.060 0.010 0.044 0.009 0.038 0.008
0.75 0.062 0.011 0.048 0.008 0.047 0.008 0.091 0.020 0.074 0.019 0.047 0.007
1.00 0.072 0.022 0.051 0.011 0.062 0.019 0.097 0.032 0.104 0.035 0.051 0.010
1.25 0.089 0.024 0.068 0.013 0.068 0.013 0.123 0.034 0.157 0.061 0.061 0.020
1.50 0.056 0.010 0.069 0.007 0.063 0.010 0.111 0.028 0.191 0.039 0.076 0.011
1.75 0.037 0.009 0.040 0.010 0.082 0.016 0.034 0.007 0.075 0.014
2.00 0.070 0.004 0.037 0.008 0.041 0.010
2.25 0.048 0.014 0.026 0.007 0.035 0.004
2.50 0.033 0.033
17
Conclusion

• This study considered only one specific fixed
  item test. The results are not very exportable
  for the moment. We have to apply this analysis
  for each specific fixed item test. To elaborate a
  more general strategy, later, percentiles
  prediction by multiple regression on any test
  conditional on the number of items and on the
  distribution of the item parameters would have to
  be analysed.