A Glimpse at Archie: The LOSA Archive

1
A Glimpse at Archie The LOSA Archive

• Ashleigh Merritt, Ph.D.
• The University of Texas Human Factors Research
  Project (UT)
• First ICAO Global Symposium
• on TEM NOSS in ATC
• 9-10 November, 2005, Luxembourg

2
What is Archie?

• Archie is the LOSA Archive
• A database of de-identified airline data which is
  maintained and updated by UT
• Airlines that do a LOSA with The LOSA
  Collaborative contribute their data to Archie for
  research purposes
• Archie provides the airline perspective

3
How old is Archie?

• Five LOSAs in five years to develop the
  methodology
• Data in the Archive starts in 2000
• Had to wait for Archie to grow before doing any
  benchmarking or industry-wide analyses.
• If we think of Archie as a dog, he would be a
  young bloodhound

4
How big is Archie?

• As of September 2005, Archie has data from
• 23 airline LOSAs
• 4800 flights
• 4800 flight narratives written by trained
  observers
• 4800 ratings of TEM countermeasure performance
• 17,500 threats
• 12,500 errors
• 2,400 undesired aircraft states

5
Why is Archie here?

• To show what is possible when you build an
  archive of observational data based on TEM
• To show what is possible when two TEM archives
  talk to each other

6
Archies Attributes

• Archie is loyal and faithful
• Archie is a persistent and diverse tracker
• Archie plays well with others

7
After five years, Archie remains loyal to its
aviation master

• The Archive is housed at UT
• All data are de-identified as to airline and
  individual
• No airline has been publicly identified
• No individual pilots can be or ever have been
  identified from the Archive

8
Archie is a good tracker

• ATC threats
• Intentional Noncompliance
• Outstanding performance

9
Tracking ATC threat chains

• Data are based on 10 LOSAs 2426 Flights 9450
  threats
• 2350 ATC threats
• ¼ of all threats
• about 1 per flight
• 236 mismanaged ATC threats
• 30 of all mismanaged threats
• 10 of all ATC threats were mismanaged

10
Mismanaged ATC threats
2426 Flights 9450 threats
236 mismanaged ATC threats
11
ATC threats - Crew Errors
2426 Flights 9450 threats
280 errors
236 mismanaged ATC threats
Automation errors (20) wrong MCP/FCU altitude
setting dialed
12
ATC threats - Crew Errors - UAS
2426 Flights 9450 threats
2 UAS
280 errors
236 mismanaged ATC threats
58 UAS
56 Automation errors
9 UAS
21 UAS
5 UAS
13
ATC threats - UAS
2426 Flights 9450 threats
Lateral deviation 17 Speed too high
14 Vertical deviation
10 Unstable Approach 10 Continued Landing
9 Incorrect Automation
configuration 9 Taxiway/runway
incursion 6 Speed
too low 6 Incorrect Aircraft
configuration 5 Other
14
280 errors
236 mismanaged ATC threats
106 UAS
Bottom line 4 of flights had a UAS arising
from an ATC threat that was mismanaged. ¾
occurred during descent/approach/land
14
Archie Intentional Noncompliance

• Across 22 LOSAs, the average is 40 of flights
  with one or more noncompliance errors
• Range 23 - 90
• Most common noncompliance errors
• checklist performed from memory / nonstandard
  checklist use
• failure to cross-verify MCP/FCU altitude alerter
  changes
• PF makes own MCP/FCU changes
• Seems pretty trivial stuff, eh?

15
Noncompliance correlates with..

• Based on 22 LOSAs, airlines with higher rates of
  intentional noncompliance also have more flights
  with
• Mismanaged threats (r .7)
• Mismanaged handling errors (r .9)
• Undesired aircraft states (r .9)
• Mismanaged UASs (r .8)
• Noncompliance a measure of safety culture?

16
Archie is a diverse tracker

• The richness and complexity of the data allow
  literally thousands of paths to be pursued
• When did it happen, was it Captain or First
  Officer flying, how was it managed, what else was
  going on, what happened next, has it happened
  before, does it happen often, does it happen more
  on certain aircraft, does it happen at particular
  airports, etc. etc. etc.
• Archie can dig and dig

17
Test Your Understanding

• Q. What would you get if you asked Archie to find
  all the flights that have four or more threats,
  zero mismanaged threats, zero crew errors, zero
  UAS, and the observer rated the crew
  outstanding for overall effectiveness?

• A. Archie would find the flights that had above
  average operational complexity (4 or more
  threats), and that the crews managed very, very
  well.
• In the current Archive of 4800 flights, there are
  102 of these defined flights. We can read the
  flight narratives to explore best practices

• GOOD BOY, ARCHIE!

18
Archie plays well with others

• Queries from / data-sharing with
• ATC (NOSS group)
• IATA/ICAO (ITA)
• Boeing
• Airlines
• Incident Reporting systems
• Transport Safety Boards

19
Archie plays well with others

• LOSA/TEM/NOSS data can complement other safety
  management data
• Flight Data Recorder is the aircraft perspective
  
• Incident reporting is the actors perspective
• LOSA/NOSS is a neutral third party perspective
• Weve never really had that before.
• One Big Advantage Pilot-ATC see the others
  world!

20
Conclusions

• You have to be patient while the methodology
  matures and the Archive grows, but once in place
• You can benchmark within and across facilities to
  determine a facilitys safety strengths and
  vulnerabilities
• You can benchmark across the industry to
  determine systemic strengths, vulnerabilities,
  and best practices within the industry
• A TEM-based Archive would allow ATC to converse
  freely with the LOSA Archive on matters of mutual
  interest to pilots and controllers

21

• The University of Texas
• Human Factors Research Project
• www.psy.utexas.edu/HumanFactors