Line Operations Safety Audits LOSA: Safety Data from the Trenches

1
Line Operations Safety Audits (LOSA) Safety Data
from the Trenches
11th Annual ALPA / US Airways Safety Forum

• James Klinect
• The University of Texas Human Factors Research
  Project
  

2
Presentation Overview

• Two part presentation hopefully something for
  everyone
  
• Part I LOSA Defined
• Part II LOSA Insights
• Summary and Future Directions

3
Part I LOSA Defined
4
Safety Data from the Trenches

• How do airlines monitor safety?
• Incident reports
• Data slanted to events resulting from system and
  flight crew failures
  
• Line checks
• Data slanted to crew proficiency and procedural
  knowledge
  
• Flight Data Recorders FOQA
• Data slanted to what happened in terms of
  flight parameters
  
• Need to go the trenches and monitor normal
  operations
  
• Provides a more realistic baseline of safety

5
Line Operations Safety Audit (LOSA)

• LOSA Jump seat observations of flight crew
  performance during regular scheduled flights
  
• Train observers to be a fly on the wall just
  there to collect data
  
• Team of 15 to 30 observers from different
  backgrounds
  
• Line pilots / Union representatives
• Check airmen
• Safety and Training pilots
• UT observers
• All data are DE-IDENTIFIED and CONFIDENTIAL

6
Purpose of LOSA

• Data provides a baseline of safety pertaining
  to
  
• Flight crew performance strengths and
  weaknesses
  
• Proficiency
• Decision-Making
• CRM skills
• Procedural compliance
• System performance strengths and weaknesses
• Culture
• National Airspace System Airports and
  Navigational Aids
  
• Aircraft design / automation
• Standards / Training / Safety / Maintenance
• Crew support ATC, Cabin, Ground, and Dispatch

7
The Main Challenge for LOSA

• What kind of database is going provide the
  desired information on flight crew and system
  performance?
  
• Concerns
• Observer overload cant capture everything
• Data analysts overload needed a data framework
  to provide direction
  
• Needed to establish a safety bottom line that
  would give the database structure (e.g., profit
  bottom line)

8
The Safety Bottom Line

• What do flight crews have to do to safely fly
  from A to B?

9
Error Avoidance
10
Error Avoidance

• Error avoidance is impossible Errors happen
• LOSA data across 429 flight segments at five
  airlines
  
• 64 of the flights observed had at least one
  error
  
• Conservative estimate because of observer
  limitations
  
• Must continue to examine how errors are induced
  to strengthen system defenses

11
Threat Management
12
Threat Management

• Threats - Originate outside the flight crews
  influence but require active management to
  prevent them from becoming consequential to safety

• Adverse weather
• Terrain
• Traffic
• Airport conditions
• A/C malfunctions
• Automation events
• Communication events

• Operational time pressures
• Non-normal operations
• ATC command events / errors
• Cabin events / errors
• MX evens / errors
• Dispatch events / errors
• Ground crew events / errors

• LOSA - 79 of the flights observed had at least
  one threat

13
Error Management
14
Flight Crew Error Types

• Intentional Noncompliance violations
• ex) Performing a checklist from memory
• Procedural Followed procedures but wrong
  execution
  
• ex) Wrong altitude setting dialed into the MCP
• Communication Missing information or
  misinterpretation
  
• ex) Miscommunication with ATC
• Proficiency Error due to a lack of knowledge
• ex) Lack of knowledge with automation
• Decision Discretionary decision that
  unnecessarily increased risk
  
• ex) Unnecessary navigation through adverse
  weather

15
Error Frequencies and Outcomes

• Intentional noncompliance errors Most frequent,
  least consequential
  
• Proficiency and decision errors Least frequent,
  most consequential

16
Intentional Noncompliance Errors

• Airlines cannot allow violations to normalize
• Why?
• It cultivates complacency and a disregard of
  rules
  
• Crews that commit at least one intentional
  noncompliance error are two times more likely
  to
  
• Commit unintentional errors (Procedural,
  Communication ..)
  
• Commit consequential errors that lead to
  additional errors or undesired aircraft states

17
More Error Management Results

• Error Responses Most errors are undetected
• 64 Undetected
• 31 Detected and effectively managed
• 5 Detected and mismanaged
• Error Outcomes Most errors are inconsequential
• Inconsequential 77
• Additional Errors 6 (Error Chains)
• Undesired Aircraft States 17

18
Undesired Aircraft State Management
19
Undesired Aircraft State Management

• Undesired Aircraft States Aircraft deviations
  at edge of the safety envelope

• Incorrect aircraft configurations
• Vertical deviations - altitude
• Lateral deviations - heading
• Speed too high
• Speed too low
• Abrupt aircraft handling

• Path / glide slope deviations
• Unstable approach
• Long landing
• Firm landing
• Wrong taxiway / runway
• Runway incursion

• LOSA 32 of the flights had at least one
  undesired aircraft state
  

20
LOSA Data Framework

• LOSA provides the data on how flight crew and
  system performance affects the bottom line

21
Part II Insights Gained from LOSA
22
The Riskiest Phase

• Descent / approach / land phase also contains the
  most variability in crew performance and
  consequential errors

23
The Blue Box
24
LOSA Blue Box Performance

• Error Avoidance - More automation and decision
  errors in this phase
  
• Threat management ATC threats are the most
  difficult to manage
  
• Airports have specific threat profiles that make
  them distinct from others
  
• Error Management Only 20 of the errors are
  effectively managed
  
• Undesired State Management 28 of the errors
  led to undesired states
  
• Most frequent states
• Incorrect A/C configurations
• Vertical deviations
• Speed too high

25
Threat and Error Countermeasures
26
CRM Refined

• LOSA has given new insight on the role CRM plays
  in the bottom line
  
• Error Avoidance
• Threat Management
• Error Management
• Undesired Aircraft State Management
• Framed CRM skills as threat and error
  countermeasures
  
• Four groupings of CRM markers of have emerged
• Team building
• Planning
• Execution
• Review / Modify

27
Team Climate Countermeasures

• Two markers
• LEADERSHIP In command, decisive, and encouraged
  participation
  
• COMMUNICATION ENVIRONMENT Communication
  channels established and maintained
  
• Critical in all levels of the bottom line
• Error avoidance
• Threat management
• Error management
• Undesired aircraft state management

28
Planning Countermeasures

• Four markers
• BRIEFING Quality of required briefings
• PLANS STATED Plans were communicated shared
  mental model
  
• WORKLOAD ASSIGNMENT Roles and responsibilities
  were defined
  
• CONTINGENCY MANAGEMENT Anticipated and
  developed strategies to manage threats
  
• Most critical in
• Error avoidance
• Threat management when the threat is expected
  (e.g., Adverse weather)

29
Execution Countermeasures

• Four markers
• MONITOR / CROSSCHECK Crosschecking of systems
  and actions
  
• WORKLOAD MANAGEMENT Staying ahead of the curve
• VIGILANCE Remained alert of the environment and
  aircraft position
  
• AUTOMATION MANAGEMENT Balance of automation and
  workload
  
• Most critical in
• Error management

30
Review / Modify Countermeasures

• Three markers
• EVALUATION OF PLANS Reviewed and modified plans
  when necessary
  
• INQUIRY Asked questions to clarify nothing
  taken for granted
  
• ASSERTIVENESS Stated critical information with
  persistence
  
• Most critical in
• Undesired aircraft state management
• Threat management when the threat is unexpected
  (e.g., A/C malfunction)

31
Summary

• LOSA monitors normal flight operations to provide
  baseline data on flight crew and system
  performance
  
• The bottom line data provides insight not
  answers
  
• How the system can provide flight crews with
  better support in
  
• Error Avoidance
• Threat Management
• Error Management
• Undesired Aircraft State Management
• LOSA is not a stand alone used as a supplement
  to other data sources (Incident reports, training
  evaluations, and line checks)
  

32
Future Directions

• Currently conducting LOSAs at at three major U.S
  airlines
  
• Will expand the current LOSA database by 800
  flight segments
  
• LOSA is still in development
• Will continue to bring LOSA from academic
  research to a safety tool used by the industry
  
• ICAO recently made LOSA a central focus for its
  Safety and Human Factors Program between 2000 and
  2004
  

33
The University of Texas Human Factors Project We
bsite
www.psy.utexas.edu/psy/helmreich/nasaut.htm