Verification and Validation Challenges in Adaptive Flight Control Software

1
Verification and Validation Challenges in
Adaptive Flight Control Software
Stephen A. Jacklin National Aeronautics
and Space Administration Ames Research
Center Moffett Field, California Caltech
Workshop on Verification and Validation Pasadena,
California September 2324, 2009
2
Adaptive Flight Control Systems Proposed for Many
Applications
3
No Adaptive Flight Software Certified for Use in
Commercial Airspace, Except

• Gain-scheduled control methods
• Controller gains are function of the flight
  condition
• Controller stability is well-understood

Control Surface Commands
PID Controller (KP, KI, KD)
U
Error
-
e
Desired State, X
Sensor Feedback
Feedback Gains
Measured State, X
Y
4
Gain-Scheduled Adaptive Control

• Is it really adaptive?
• Yes, in that it can change controller behavior
  based on prescribed changes to the plant or
  environment
• No, simply a group of non-adaptive controller

• In principle, gain-scheduled control can be
  applied to any control problem

• In practice, cannot be used for applications that
  do not divide well into segments for scheduling
• Damage adaptive control
• Upset recovery
• Unknown combinations of control surface failures
• Slow degradation of control system components

5
Adaptive Controllers Use Learning Algorithms or
Systems Identification
Control Surface Commands
Desired State, X
U
LQG or PID Controller
Error
-
e
Measured State, X
Sensor Feedback
Y
Feedback Gains

• Adaptation makes software certification difficult

6
Why Is Adaptive Software a Problem?

• Why is it so difficult to verify the performance
  of adaptive flight control software?

• Why cant RTCA DO-178B guidelines be satisfied by
  adaptive controllers that use learning or system
  identification algorithms?

• Most guidelines can be satisfied

• There are 5 knowledge gaps or VV challenge areas
  for adaptive control software

7
Gap 1 Defining Adaptive Controller Requirements
and Test Plans
8
Gap 1 Defining Adaptive Controller Requirements
and Test Plans
9
Gap 1 Defining Adaptive Controller Requirements
and Test Plans

• Requirements define what the software is supposed
  to do

• Doesnt seem all that hard considering that the
  analysts know exactly what needs to be done and
  how to test it

• Or do they ?
• Derived requirements such as computer speed,
  bandwidth, I/O, memory, redundancy, fault
  detection
• Learning speed and controller stability
  requirements
• Noise rejection requirements
• Persistent excitation requirements
• Human-machine interaction requirements
• Adaptive controller performance requirements

10
Requirements Definition, cont

• Lack of metrics hampers specification of adaptive
  controller performance requirements
• Non-adaptive flight controllers have well
  established metrics to describe performance such
  as gain margin and phase margin
• DO-178B all requirements must be stated in a way
  that they can be tested
• And software tests written before coding

11
Expansion of Model-Based Design Methods Needed to
Validate Requirements

• Iterative path of software design, simulation,
  and testing against the requirements

• Software validation testing done before code is
  written for target flight computer
• Model-based Design will aid certification by
  showing early and complete validation

12
Gap 2 Lack of High-Fidelity Benchmark
Simulations and Simulation Tools

• DO-178B allows certification credit for
  high-fidelity simulation as well as flight testing

• Everyones got there own simulation
• Desktop simulation (Matlab/Simulink)
• Workstation with nonlinear aerodynamics
• Hardware-in-the-loop (target flight computer)
• Human-in-the-loop (motion based simulation)
• Sub-scale flight and wind tunnel testing
• Full-scale flight testing

• Will the FAA accept any and all of these?

13
Gap 3 Difficulties in Proving Adaptive
Controller Stability and Convergence

• Adaptive controller stability and performance
  depends on the stability and convergence of the
  learning method
• Instability in the learning or system
  identification process will lead to instability
  in control
• At present, most stability proofs based on
  Lyapunov stability theory

14
Lyapunov Stability Limitations for Certifying
Adaptive Controller Stability

• Mathematical complexity of Lyapunov proof not
  conducive to a certification argument
• Controller may be inadequate if learning does not
  happen quickly enough
• Polynomial form of plant required by the method
  (A and B matrices)
• Plant model may change with damage or flight
  condition
• Lyapunov theory does not guarantee the rate of
  convergence

15
Gap 4 On-line Monitoring Tools Needed to Assess
In-Flight Performance

• A requirement of DO-178B is that the fielded
  software be the same as that tested
• Although the equations of an adaptive controller
  do not change, the controller gains do
• On-line monitoring tools need to be developed to
  perform run time verification
• Determines when controller malfunctions occur
• A difficulty is finding appropriate indicators of
  bad performance

16
Conundrum for On-line Monitoring

• If you have a monitor thats smart enough to know
  when the adaptive controller is wrong, then why
  not use it instead ?
• Need inference tools that can monitor controller
  performance without knowing the right answer
• Preliminary efforts
• NASA Ames Neural network Confidence Tool
• NASA DFRC in-flight controller stability
  assessment for X-38

17
Gap 5 Certification Plan for Adaptive Controller
Needs to be Formed

• Many on-going research efforts
• AFRL VIVIACS study
• On-going program with SRI (Ashish Tiwari)
• NASA IRAC Program
• Safety Case approach needed to identify all
  software hazards and risks
• Describes how the risks are mitigated
• Provides evidence that the system is safe
• Presents a safety management plan
• A safety case argues for certification on the
  basis of evidence that says all the best
  practices for ensuring safety have been followed