Building Cognitive Ability Tests with Reduced Adverse Impact

1
Building Cognitive Ability Tests with Reduced
Adverse Impact
Harold Goldstein Baruch College
CUNYMAPACNovember 2008
2
Key Outcomes for Personnel Selection

• Validity
• Utility
• Adverse Impact

3
The History on Cognitive Ability Testing and
Personnel Selection

• High Validity
• High Utility
• High Adverse Impact
• Search for alternative tests and means of
  implementation

4
Based on the Psychometric Theory of Intelligence
(Spearman, 1904)

• We know
• The nature of intelligence (singular g)
• How to measure intelligence
• Intelligence is the most important predictor of
  job performance
• Whites possess more intelligence than minorities
  (e.g., African-Americans, Hispanics)
• Jensen (1998)

5
Psychometric Perspective

• Single latent variable (Spearman, 1904)
• Positive manifold (Spearman, 1927)
• Factor analysis (Carroll, 1993)
• Measures of g (Jensen, 1998 Ree Carretta,
  2002)
• Validation research (Schmidt Hunter, 1984)
• Spearman hypothesis of racial differences
  (Jensen, 1998)

6
Other Perspectives on Intelligence

• Other
• Definitions
• Structures
• Models/theories and Viewpoints
• Rationales for positive manifold and factor
  analytic outcomes
• Explanations for racial differences

7
Construct Validity and Measuring Intelligence

• Construct Relevance
• Construct Deficiency
• Construct Contamination

8
Designing New Measures

• Driving forces
• Education, Child Development, and Clinical
  Psychology
• Modern techniques
• Create theory driven measures
• Reduce use of language and prior knowledge
  (Fagan)
• Decrease impact of culture (Ortiz Ochoa)
• Un-entrenched approach (Sternberg)

9
Examples of New Measures

• Wechslers WAIS (Version 3)
• Fagan (Processing Approach)
• Sternberg (Rainbow Project)
• Naglieris Cognitive Assessment System (Nonverbal
  Focus)
• Siena Reasoning Test (SRT)

10
Siena Reasoning Test (SRT)

• 40 items 40 minutes
• Items tap
• processing and manipulating information
• drawing inferences
• reasoning
• making decisions
• integrating knowledge
• Adapt format
• Reduction of language
• Decrease level of prior knowledge required
• Non-entrenched

11
SRT Validity Studies
All uncorrected correlations
12
SRT Validity Comparison
All uncorrected correlations
13
Incremental Validity of SRT
All uncorrected correlations
14
Adverse Impact of the SRT

• Gender
• No difference
• Race
• Substantially mitigates adverse impact (relative
  to other written cognitive ability tests)
• Allows for a more selective cut score

15
Black-White Differences of SRT
16
Rethinking Cognitive Ability and Personnel
SelectionQuestions for Practical Application

• CATs show validity -- but what constitutes a CAT?
• Construct and theory focus
• Proper design
• Beyond psychometric perspective
• Can we achieve all three outcomes (validity,
  utility, adverse impact)?
• Incremental validity (as well as Incremental
  Adverse Impact and Incremental Loss of Utility)
• Systematic development (hurdles, cutoffs,
  combinations)
• Can we predict an expanded domain of performance
  (e.g., fit, OCB, engagement, turnover)?

17
Summary

• Personnel selection must go beyond the
  psychometric perspective on intelligence
• We have an opportunity to develop cognitive
  measures that have high validity, high utility,
  and low adverse impact
• QUESTIONS?