PREDICTING ATTRITION OF ARMY RECRUITS USING OPTIMAL APPROPRIATENESS MEASUREMENT

1
PREDICTING ATTRITION OF ARMY RECRUITS USING
OPTIMAL APPROPRIATENESS MEASUREMENT

• Oleksandr S. Chernyshenko
• University of Canterbury
• Stephen E. Stark
• University of South Florida
• Fritz Drasgow
• University of Illinois at Urbana-Champaign

2
Overview

• Purpose
• Examine optimal appropriateness measurement (OAM)
  methods to predict attrition using the AIM scales
• Analyses
• Data
• Item response theory (IRT) modeling of AIM items
• Optimal Appropriateness Measurement (OAM)
  classification of AIM respondents

3
Advantages of OAM Methods for Classification
Purposes

• OAM provides statistically most powerful methods
  for classifying examinees into two groups, such
  as stayers and leavers.
• If two groups differentially endorse AIM items,
  then OAM can classify any unknown response
  pattern as likely to come from one group or from
  another
• If item response model is correctly specified for
  each studied group, then the Neyman-Pearson lemma
  states that no other method can be used on the
  same data to provide more accurate classification
  (Levine Drasgow, 1988).

4
Data

• 22,666 Army enlisted personnel
• 22 individuals excluded prior to analyses because
  they were either National Guard, Reserve forces,
  or information on their Service was not available
• 6 AIM scales examined (e.g., Work Orientation)
• Scale names are omitted from our results

5
AIM Scoring

• Stems within each AIM tetrad were scored
  trichotomously
• Each statement received a score of 0, 1, or 2
  depending on whether it was positive/negative and
  selected/unselected
• As a result a unidimensional IRT model can be
  used to characterize the process of responding to
  AIM items

6
Classical Test Theory Analyses

• Item-total correlations
• Items with negative correlations identified and
  removed from final reliability analyses
• Reliability (alphas ranged from .6 to .7)

7
Principal Components Analysis

• Used to examine dimensionality of AIM scales

8
Predicting Attrition Using IRT

• Prediction of attrition involves three steps
• 1) Calibration of the AIM stems in samples of
  stayers and leavers
• 2) Examination of model-data fit
• 3) Classification of recruits via optimal
  appropriateness measurement

9
Step 1 Calibration

• Samejimas Graded Response (SGR) model was fit to
  trichotomously scored (0, 1, 2) response data for
  six AIM content scales parameters estimated
  using MULTILOG 6.3

10
Step 2 Model-Data Fit

• Model-data fit was examined using fit plots and
  chi-square statistics, provided by MODFIT.
• A cross validation approach was employed.
• Fit plots illustrate the correspondence between
  theoretical (ORF) and empirical (EMP) response
  functions
• Chi-squares for item singles, doublets, and
  triplets provide more sensitive tests of fit may
  also provide evidence concerning violations of
  local independence

11
Representative Fit Plot Results Stem 2,
Physical Conditioning
ORF
EMP
Good Fit
12
Representative Chi-Square Results Adjustment
Scale

• Adjusted chi-square to degrees of freedom ratios
  of 3 or less indicate very good fit.

• Similar results were found for other scales.

12
13
Step 3 Classification via Optimal
Appropriateness Measurement

• Respondents classified using likelihood ratio
  (LR) statistic, obtained by dividing marginal
  likelihood of being a leaver by marginal
  likelihood of being a stayer
• The marginal likelihood equation is shown below

14
Example of the OAM Procedure

• Compute the marginal probability of a
  respondents Physical Conditioning responses
  using the SGR item parameters for leavers.
• Compute the marginal probability of the
  responses using the parameters for stayers.
• Compute the ratio of these two probabilities.
• If the ratio is large (i.e., the responses are
  better described by the model for leavers),
  predict that the respondent will be a leaver

15
OAM Results

• Six LR statistics computed for each respondent
  (one per AIM content scale)
• Logistic regression used to determine best
  linearly weighted sum of LR values for predicting
  stayer/leaver outcome
• Receiver operating characteristic (ROC) curves
  computed for each content scale and the logistic
  regression composite

15
16
Representative ROC
OAM differentiated stayers and leavers to a
moderate degree
16
17
Representative ROC
17
18
Summary and Conclusions

• OAM composite provided the highest hit rate. It
  correctly identified
• 22 of stayers at a 10 false positive rate
• 35 of stayers at a 20 false positive rate
• 47 at 30
• 56 at 40
• 65 at 50
• OAM composite provided incremental validity in
  the prediction of attrition and, thus, AIM scales
  should be used collectively.
• OAM methodology provided an improvement over the
  current application of the Adaptability score in
  predicting attrition.
• At a 20 false positive rate, the current
  adaptability score yields 27 correct
  identification, while the OAM composite yields
  33.