CompetencyBased Performance Measurement Technology Craig Haimson Webb Stacy 5 April 2006

1
Competency-Based Performance Measurement
TechnologyCraig Haimson Webb Stacy5 April
2006
Ó 2006, Aptima, Inc.
2
Background Pedagogy of experiential training

• Objectives
• Competencies to train
• Events
• Situations or actions
• Conditions
• Context in which events occur
• Measures
• Performance data
• Assessments
• Comparison w/ standard

3
Structure of measures
4
Sources of data

• Simulator output
• Game log files
• HLA/DIS
• Observer-based
• Self-report
• Eye tracker
• EEG
• ???

SPOTLITE is a product of Aptima, Inc.
5
Role of performance measurement systems

• Measurement may be the responsibility of separate
  system
• Collect data
• Extract measures
• Compute assessments
• Report results
• Adaptive experiential learning may require
  configuration of both simulation and performance
  measurement systems
• Sequencing can entail changes in measurements
  without any corresponding changes in simulation
  (especially when paired with new trainee
  instructions)
• Both individual- and team-level measures of
  previous scenario performance can drive
  sequencing for group training (although
  configuration must ultimately be synched across
  participants)
• SCORM/Simulation interoperability must take into
  account
• LMS configuring simulation/performance
  measurement
• Simulation communicating activity to LMS
• Performance measurement communicating assessment
  to LMS
• May even be multiple measurement systems (for
  different sources of data)
• Part of the solution common performance
  measurement language

6
Example performance measurement system A-Measure
Create Configure Collect Assess Review
7
A-Measures PMEngine
8
Team Performance Measurement Workbench

• Graphical measurement editor
• Outputs measures as XML measurement interchange
  language

9
Human Performance Measurement Language

• Measurement interchange language
• Separation of
• Definition
• Configuration
• Computation
• Reporting
• Store/reuse measures across scenarios
  simulations
• Context repr to be added
• Trigger measurement
• Modulate assessment

10
Measurement definition in HPML
11
Conclusions

• Measurement may be the responsibility of separate
  system
• Adaptive experiential learning may require
  configuration of both simulation and performance
  measurement systems
• Measures may be thought of a distinct element of
  learning content to be stored, discovered, and
  reused
• SCORM/Simulation interoperability must take into
  account LMS communication with one or more
  simulations one or more performance measurement
  systems
• A measurement interchange language may facilitate
  configuration of measures communication about
  assessments
• Issue To support AAR/DAR, language may need to
  capture richer set of concepts than is practical
  for SCORM RTE data model
• Additional mapping to competency model will
  enable roll-up (potentially via use of sequencing
  rules)

12
Craig Haimson

• Craig Haimson, Ph.D.
• Team Lead, Cognitive Training Technologies
• Aptima, Inc.
• 1726 M Street NW, Suite 900
• Washington, DC 20036
• 202-842-1548 x314 (voice)
• 202-842-2630 (fax)
• haimson_at_aptima.com
• www.aptima.com

Webb Stacy, Ph.D. Vice President,
Technology Aptima, Inc. 12 Gill Street, Suite
1400 Woburn, MA 01801 781-935-3966
x237(voice) 781-935-4385 (fax) wstacy_at_aptima.com w
ww.aptima.com