Occupational Analysis and Future Automated Testing

1
Occupational Analysis and Future Automated Testing

• Johnny J. Weissmuller
• Innovation Center,
• Occupational Data Applications Practices
  (ICODAP)
• 5 April 2005

2
Occupational Analysis and Future Automated Testing

• Opportunities and Challenges
• http//www.icodap.org/

3
Government Managers Executives

• CHINA - 1115 B.C. - The Imperial Court
• Problem Selection Retention of
• All Government Officials

4
Six Factor Model

• Writing
• Arithmetic
• Music
• Archery
• Horsemanship
• Ceremonies and Rites
• P.H. DuBois, 1970
• A History of Psychological Testing

5
Department of Defense

• JCS Memo, August 1949
• Initiate a study to determine the most
  appropriate methodology and techniques of
  military job analysis and job evaluation
  determine the extent to which the unilateral
  analyses accomplished to date are valid with
  relation to methodology provide for such further
  job analysis as may be required and initiate a
  study to relate all Army, Navy, and Air Force
  jobs to a common occupational structure.

6
What are we talking about?

• Not Training-Embedded Testing
• High-Stakes Testing
• Selection
• Promotion
• Pay-Affecting (Language Prof)
• Compromise is criminally actionable

7
Job Analysis in the Military

• Military Provides all employee training
• Military structures job families to balance
  training cost utility of force
• Many occupations undergo major technology shifts
  mission broadening
• Major changes lessens the value of senior
  personnel in targeting training
• Military Promotes from Within
• Military is a model for enlightened corporations

8
Opportunities AheadFrom Computerization

• Better Test Deployment Logistics
• Faster Test-to-Personnel Action
• Quicker Results Notification to Subject
• Smarter Tests (Branched/Adaptive)
• More Realistic (High Fidelity Scenarios)
• Tighter Controls against Compromise

9
Challenges Ahead

• Fragmentation of Occupational Families called
  Multi-skilling and Agile Workforce by
  Management
• Greater computer testing capabilities outstrips
  the classical and item response theory
  ability to rank individuals
• Assessment will need to move to the module level
  rather than job title

10
Laundry List of Concerns 1

• Test Compromise
• Prior Knowledge of Test Material
• Material Access during testing
• Multiple Forms versus Adaptive Item Pools

11
Laundry List of Concerns 2

• Changing Job Demands
• Test Proctors
• Test Developers
• New IT-Personnel (High Stakes Testing Demands)
• Test Psychologists analyzing Results

12
Laundry List of Concerns 2a

• Test Psychologists analyzing Results
• Test Compromise
• Alternate Forms/Bayesian Algorithms
• Test Equating Across Forms
• Personnel Programs which use these Metrics
• Comparability to Previous Scores / Master
  Database
• Use in Cross-Training Decisions / Differential
  Aptitudes

13
Laundry List of Concerns 3

• Cost
• Dollars/Budget
• Time-Quality Trade-off
• Infrastructure Creation Maintenance
• Logistics and Timelines for initial deployment
• Re-integration into operational personnel programs

14
Laundry List of Concerns 4

• Reliability of test scores across presentation
  formats
• Validity/Equivalence of test presentation formats
• Fidelity Scenario Approaches
• Data Points Per Minute
• Contingent Measures/Bayesian Algorithms
• Parallel Form Development
• Lost Opportunity for Quantum Leaps

15
A Modest Proposal

• Use Occupational Surveys to Identify Low Hanging
  Fruit
• Incorporate Standardized Rating Lists of What
  New Tests May Help to Select or Promote the right
  person (and how much)

16
A Modest Proposal - Examples

• Diagnosis problems by listening
• Diagnosis problems by visual inspection
• Understand instructions in a noisy environment
• Recognize good/bad practices within visual
  scenarios
• Detect problems by smell, taste, touch,
  kinesthetic sense, etc.

17
Questions?

• Obviously I have the questions!
• Do YOU have any of the answers?
• Email johnnyw_at_ICODAP.ORG