WP 3: Human Performance Metrics

1
WP 3 Human Performance Metrics

• 10 Oct 2002
• Brian Hilburn

2
Goals
Extract set of ASAS applicable HumPerf
Metrics Outline application methods
(guidance) Provide primary references Mapped
onto - Application (eg Airborne Sep.) -
Validation Type (eg Realtime, etc) - Agent
(ATC vs Flightdeck) - Human Performance Area
3
Human Performance Areas

• Workload
• Situation Awareness
• System Monitoring
• Teamwork
• Trust
• Usability and User Acceptance
• Human Error

4
Parallel WP2 (System Metrics)

• SYSTEM versus HUMAN performance?
• Distinction sometimes arbitrary
• Performance Areas differ
• Implicit links to Goals
• human performance influences SAFETY etc

5
WP 3 Deliverable
Fast t Real t
Wrkld SA etc
Obj Subj
Air Ground
ASAS f1
. .
ASAS fn
6
Methods
Literature review - ASAS sources (transatlantic
FreeFlight) - ATM validation efforts (INTEGRA,
RHEA,) - Aviation Human Factors - Human
Factors general Distillation of methods and
metrics Guidelines Including general Measurement
Issues
7
General Measurement Issues
Validation test subjects Experimental design in
validation Analytical techniques Balancing
validation realism data collection
8
Format of Metrics
9
Guidance Literature Consensus and Expert
Opinion...
EXAMPLE
10
100 Human Performance Metrics

• NASA Task Load Index
• Heart Rate Variability
• Pupil Diameter
• Human Computer Trust Scale
• Auditory Choice Secondary Task Response Time
• Situation Awareness Rating Technique
• Blink latency
• Choice Reaction Time secondary task
• etc...

11
Choosing a Metric

• Objectivity Objective / Subjective
• Intrusive High / Low
• Cost High / Low
• Reliability High / Low
• Validity High / Low
• Utility High / Low
• Expertise High / Low
• Resource High / Low

12
A Worked Example...

• Given Airborne Spacing application
• (in-descent spacing scenario)
• Controlled airspace
• Study Real time
• Question Transient workload peaks?
• Issues Time resolution, intrusiveness, cost

13
Applying the Framework
14
General Guidance
15
Conclusions

• Link between System and Human Performance metrics
• What are my goals?
• Relationship to higher validation goals
• Guidance, not a cookbook
• Ideally used in conjunction w SYSTEM metrics