FAA Oversight of SafetyCritical Software Systems: A Case Study

1
FAA Oversight of Safety-Critical Software
Systems A Case Study

• William S. Greenwell
• Department of Computer Science
• University of Virginia

2
August 6, 1997 142 AM

• Korean Air flight 801 impacts terrain on final
  approach to Guam Intl runway 06L.
• En-route from Kimpo Intl, Seoul, Korea.
• Boeing 747-300 carrying 254 people
• 228 Fatalities, 26 Seriously Injured
• Instrument meteorological conditions prevailed at
  time of accident.

3
Runway 06L ILS Approach
560

• NOTAM Runway 06L glideslope unusable
• Without glideslope, pilots must execute
  nonprecision approach using step-down fixes.
• Runway threshold D3.3 from UNZ VOR.

Approach plate courtesy Jeppeson. NOT TO BE USED
FOR NAVIGATION.
4
NTSB
5
Wreckage of KA Flight 801
Runway 06L
VOR
Wreckage
NTSB
6
Proximity to UNZ VOR
7
NTSB Findings

• Captain lost awareness of aircrafts position on
  the landing approach. (p. 173)
• Captain possibly believed UNZ VOR was co-located
  with the runway, causing him to descend below
  intermediate approach altitudes. (p. 173)
• Classified as Controlled Flight Into Terrain.

Source NTSB. Controlled Flight Into Terrain,
Korean Air Flight 801, Boeing 747-300, HL7468,
Nimitz Hill, Guam, August 6, 1997. Aircraft
Accident Report NTSB/AAR-00/01. Washington, DC.
8
Barriers to CFIT

• On-board Barriers
• Instrument Landing System (ILS)
• Localizer/Glideslope Indicators
• Outer/Middle/Inner Markers
• Distance Measuring Equipment (DME)
• Published approach procedures
• Ground Proximity Warning System (GPWS)
• Pilot not flying, flight engineer
• Ground-based Barriers
• Minimum Safe Altitude Warning (MSAW) system

9
On-Board Barriers

• Instrument Landing System (ILS)
• Glideslope out of service since July
• Outer/Middle marker indicators suppressed
• Published approach procedures
• Pilot misunderstood/disregarded DME fixes.
• Ground Proximity Warning System (GPWS)
• Commonly ignored due to nuisance warnings
• Pilot not flying, flight engineer
• Didnt challenge approach soon enough

10
Ground-based Barriers

• Minimum Safe Altitude Warning System
• Inhibited by FAA due to nuisance warnings
• NTSB concluded
• Contributing to the accident was the Federal
  Aviation Administrations (FAA) intentional
  inhibition of the minimum safe altitude warning
  system (MSAW) at Guam and the agencys failure to
  adequately manage the system. (p. 175)

11
MSAW Overview

• Developed by FAA in response to NTSB Safety
  Recommendation A-73-46.
• Incorporated into ARTS IIA in 1990.
• Data Inputs
• ARTS track positions altitudes
• Terrain database elevation data
• Configuration file airport runway
  information, service area definitions
• General approach path monitoring

12
MSAW General Monitoring
Minimum Safe Altitude (MSA)
Terrain Clearance Altitude
13
MSAW Approach Path Monitoring
Glideslope Path
Alarm Trigger Area (100 below glideslope path)
Runway
1 nm
14
MSAW Deployment

• Installed at 193 ARTS IIA ARTS III sites.
• Each site employed customized terrain database
  and configuration file.
• Individual sites free to inhibit processing as
  needed to alleviate nuisance warnings.
• No guidance for defining inhibit zones
• No oversight of site adaptations
• Waiver required to turn off MSAW entirely.

15
Guam MSAW Chronology

• 1990 MSAW incorporated into ARTS IIA
• Originally configured with 55-nm service area.
• March 1993 Guam adapts MSAW parameters to
  include 54-nm inhibit zone.
• February 1995 New MSAW build becomes
  operational with inhibit zone.
• July 1995 Facility evaluation of Guam notes
  inhibition as informational item.

16
Guam Inhibit Zone
MSAW Service Area Boundary (55 nm)
Inhibit Zone (54 nm)
Guam
Pacific Ocean
17
Guam Chronology / Cont.

• February 1996 NOAA delivers new terrain
  database for Guam MSAW system.
• April 1996 New MSAW build becomes operational
  with updated terrain database 54-nm inhibit
  zone.
• May 1997 FAA reevaluates Guam ATC facility, but
  does not note MSAW inhibition.
• August 6, 1997 KA flight 801 accident

18
Effectiveness of MSAW

• NTSB/FAA simulation indicated that, without the
  inhibition, MSAW would have generated an alert 64
  seconds before impact.
• NTSB This would have been sufficient for the
  controller to advise KA 801. (p. 174)
• NTSB concluded that FAAs quality assurance of
  MSAW was inadequate.

19
MSAW Alert
NTSB
20
FAAs Post-accident Actions

• Recertified MSAW at all 193 equipped sites.
• Two other improperly configured systems
• Instituted policy for periodic recertification.
• MSAW inspection added to facility evaluation.
• Developed standards for site adaptation.
• Centralized configuration management.
• All configuration changes made by AOS.

21
Underlying Cause

• FAA changed a safety-critical system without
  reexamining the scenario the system addressed.
• Trial-and-error approach to site adaptation
• Sites permitted to make changes at their
  discretion without review or recertification.
• No instructions or guidance for making changes
• FAA allowed Guam to operate normally despite
  having two safety-critical systems out of
  service.

22
MSAW Safety-Critical?

• FAA Safe operation of aircraft is ultimately
  pilots responsibility.
• No certification criteria for ground systems
• MSAW merely an aid to AT controllers
• But
• MSAW only ground-based CFIT barrier
• FAA NFSD manager MSAW safety-critical item

23
Lessons for the FAA

• Systems that serve as barrierseven ground
  systemsprovide safety-critical functions to
  larger systems.
• When changing such a system, we must examine how
  that change will affect the safety of the overall
  system.
• How do we reduce nuisance warnings without
  compromising alerting capability?