The LOSA Archive: The data and how it can be used

1
The LOSA Archive The data and how it can be used

• Ashleigh Merritt, Ph.D.
• The University of Texas Human Factors Research
  Project (UT)
• 2nd ICAO Global Symposium on Threat and Error
  Management (TEM) and Normal Operations Safety
  Survey (NOSS) in Air Traffic Control
• 7-8 February 2007

2
Purpose of Presentation

• Show how the LOSA Archive is used in the airline
  world
• Show how a similar NOSS Archive could benefit the
  ATC world

3
What is LOSA?

• Line Operations Safety Audits
• Forerunner/Prototype for NOSS
• Normal operations monitoring based on cockpit
  observations of regularly scheduled commercial
  flights
• Trained expert observers using TEM taxonomy to
  frame, describe, code the flight
• ICAO Industry best practice for normal
  operations monitoring (ICAO LOSA Manual, Doc
  9803)
• FAA (FAA Advisory Circular 120-90)

4
TEM in the Airline World

• Threats Adverse weather, traffic congestion,
  terrain, airport conditions, aircraft
  malfunctions, problems with ground, maintenance,
  dispatch, cabin, and of course.. ATC
• Errors Handling, automation, systems,
  procedures, briefings, communication
• Undesired Aircraft States Speed, lateral and
  vertical deviations, unstable approaches,
  incorrect engine/system configurations, taxiway
  incursions

5
The LOSA Archive (Archie)

• A database maintained and updated by UT
• Data from airlines who do a LOSA with TLC
• As of February 2007, data from
• 30 airline LOSAs (including 4 repeat LOSAs)
• 6000 regularly scheduled commercial flights
• 22,000 threats
• 15,000 errors
• 2,500 undesired aircraft states

6
LOSA Airlines
7
Archive Three Main Uses

• Benchmarking
• Industry Trends
• Interface with other safety information sources

8
Archive Use 1 Benchmarking

• Empirical TEM data
• New form of data - One hand clapping
• Example Airline X
• 36 of flights had a mismanaged threat
• 30 of flights had a mismanaged Handling error
• 26 of flights had some form of intentional
  noncompliance.
• Is that good, bad or average?

9
Airline X and the LOSA Archive

• 36 of flights had a mismanaged threat
• Average across 30 LOSAs is 34. Range 16-63
• 30 of flights had a mismanaged Handling error
• Average is 37. Range 13-88
• 26 of flights had some form of intentional
  noncompliance.
• Average is 46. Range 22-90

10
Archive Use 2 Industry Trends

• Normal Ops monitoring provides important base
  rate information
• Example Unstable Approaches
• We know how many incidents/accidents involve an
  unstable approach, but do we know how many
  unstable approaches occur without incident?
• Average is 5 of flights. Range 0-15 across 30
  LOSAs
• 10 of UASs are linked to a mismanaged ATC threat

11
ATC Threats in the LOSA Archive

• Data are based on 2400 Flights (9500 threats)
• ATC the most common threat
• ¼ of all threats were ATC threats
• 10 of all ATC threats were mismanaged by the
  crews
• In all, 30 of all mismanaged threats were ATC

12
ATC threats
2426 Flights
2349 ATC threats
13
Mismanaged ATC threats
2426 Flights
2349 ATC threats 236 mismanaged
14
ATC threats -gt Crew Errors
2426 Flights
280 errors
236 mismanaged ATC threats
Automation errors (20) wrong MCP/FCU altitude
setting dialed
15
ATC threats -gt Crew Errors -gt UAS
2426 Flights
2 UAS
280 errors
236 mismanaged ATC threats
58 UAS
56 Automation errors
9 UAS
21 UAS
5 UAS
16
ATC threats -gt UAS
2426 Flights
Lateral deviation 17 Speed too high
14 Vertical deviation
10 Unstable Approach 10 Continued Landing
9 Incorrect Automation
configuration 9 Taxiway/runway
incursion 6 Speed
too low 6 Incorrect Aircraft
configuration 5 Other
14
280 errors
236 mismanaged ATC threats
106 UAS
Bottom line 4 of flights had a UAS arising
from an ATC threat that was mismanaged. ¾
occurred during descent/approach/land
17
Archive Use 2 Industry Trends What does
Noncompliance signify?

• Has to meet one of three conditions to be coded
  as intentional noncompliance in LOSA
• Committed multiple times during one phase of
  flight (e.g., missing multiple altitude callouts
  during descent). (coded as one intentional
  noncompliance error)
• Crew openly discusses they are intentionally
  committing an action that is against published
  SOP
• Observer determines the crew is time-optimizing
  SOP when time is otherwise available (i.e.,
  performing a checklist from memory)
• Most errors are coded as unintentional/ honest
  mistakes. If observer is in doubt, error is coded
  as unintentional.

18
Is Noncompliance a Problem?

• The average (across 30 LOSAs) 46 of flights
  have one or more noncompliance errors
• Range 22 of flights at one airline to 90 at
  another
• Most common noncompliance errors
• checklist performed from memory / nonstandard
  checklist use
• failure to cross-verify MCP/FCU altitude changes
  
• PF makes own MCP/FCU changes when hand-flying
• Conclusion 1 Noncompliance is just experienced
  pilots taking optimizing shortcuts no big deal

19
Industry Trend What does Noncompliance signify?

• Across 30 LOSAs, airlines that have higher rates
  of intentional noncompliance also have higher
  rates of
• Mismanaged threats (r .60)
• Mismanaged errors - handling (r .81)
• Mismanaged errors - procedural (r .80)
• Undesired aircraft states UAS (r .81)
• Mismanaged UASs (r .73)
• Conclusion 2 Increased noncompliance decreases
  safety margins Noncompliance reflects the safety
  culture

20
Archive Use 3 Interface with other safety
information sources

• Queries from / data-sharing with
• Airlines
• Incident Reporting systems
• IATA/ICAO (ITA)
• Transport Safety Boards
• Boeing
• ATC (NOSS Archive?)

21
And so - a NOSS Archive?

• Have to be patient while the methodology matures
  and the Archive grows, but once in place
• You can benchmark within and across facilities to
  determine strengths and vulnerabilities
• You can trend to determine industry best
  practices problems
• A TEM-based NOSS Archive would allow ATC to
  converse freely with the LOSA Archive on
  matters of mutual interest to pilots and
  controllers

22

• The University of Texas
• Human Factors Research Project
• www.psy.utexas.edu/HumanFactors