High Integrity Software

1
High Integrity Software

• List typical high integrity applications
• Outline examples of failure
• Outline legal sources of responsibility
• Discuss the relevance of standards
• Discuss the role of formal methods, including
  static analysis

2
High Integrity Software

• Reliable Software
• Correct
• Robust
• High integrity software is needed when the cost
  of failure too large to bear

Correct Software meets specification Robust
Software behaves sensibly if out of
specification input is provided
What is the definition of Correct Software and
Robust Software?
3
Applications

• Safety Critical Software
• Vehicle systems plane, car, rocket
• Process control nuclear power station
• Security Critical Software
• Data communications
• Chip Pin
• Business Critical Software
• Share trading systems
• Handle millions per day

4
Therac 25

• A system for treating tumours
• 1 low energy electron beam treatment
• 2 high energy electron beam with a metal plate
  in front giving X-rays.
• A software interlock prevented a high energy beam
  without the plate
• Safety case claimed 10-11 probability for
  computer selects wrong energy.

5
Therac User Interface

• Operator not in room to avoid radiation
• Actions
• Types x (X-ray) rather than e (Electrons)
• Realises error,
• Types ? edit, e, enter within 8 seconds.
• System says Malfunction
• Clears error, sees beam ready, hits b
• Same error message, so tries again.

6
Therac The Root Cause

• The input exposed a race in the multi-tasking
  code, and stopped the plate.
• Gave lethal full-power beam overdose
• This input sequence hadnt been tested
• Intercom and video link out of service.
• The error messages usually meant treatment had
  not occurred

7
Ariane 5

• First launch of Ariane 5 exploded
• Inertial navigation system (INS) failed
• Overflow on converting 64-bit float to 16-bit
  integer exception not trapped
• This killed the primary and back-up INS
• Went off course was destroyed
• Software re-used from Ariane 4
• Less powerful rocket
• Value remained in 16-bit integer range

8
Relevant Legal Issues

• Contract (Specific agreement)
• Cant contract away liability for harm
• Negligence (General duty of care)
• Specific Laws
• Consumer protection
• Supply of Goods Services
• Covers specifically developed software
• Reasonable care in development
• Sale of Goods
• Covers packaged software
• Reasonably fit for purpose

9
A Legal Guideline

• UK, Health Safety at Work Act
• Obligation to keep risks As Low As Reasonably
  Practicable (ALARP)
• Definition of reasonably practicable the cost
  of undertaking the action is not grossly
  disproportionate to the benefit gained.

10
Failure Causes

• Hardware design / random error
• Software error
• Specification, design, implementation
• Compiler error
• Risky/ill-defined language features
• Dynamic memory
• Pointers
• Multiple inheritance
• User interface problems

Do formal methods help with any of these?
11
Techniques

• Safe programming languages
• Development methodologies
• Verification Validation
• Testing
• Inspections Reviews
• Static program analysis
• Formal Specification
• Program Proving

Define give examples of each of these
12
Standards

• What is a standard?
• How are standards defined?

Use the following terms de facto de jure
(committee-based) ISO, ANSI, BSI,
DIN Professional Body (IEEE)
13
Why Use Standards?

• Summarise good practice
• Better quality products
• Better processes
• Define minimum levels of quality
• Protect against lawsuits
• Demonstrate reasonable care
• Satisfy customers
• May be essential for a contract
• May allow a price premium

14
Standards ITSEC

• IT Security Evaluation Criteria
• ISO15408 International Standard
• Higher levels require formal methods
• EA 7 Formally Verified Design and Tested.
• Formal model and a formal presentation showing
  links between functional specification and high
  level design.
• Evidence of developer "white box" testing.
• Independent confirmation of test results.
• Complexity of the design must be minimised.

15
Standards IEC 61508

• Functional safety of E/E/PE safety systems
• Electical, Electronic, Programmable
• Generic standard
• Base for sector-specific standards
• Example applications
• Emergency shut-down system
• Fly-by-wire operation of aircraft
• Life-critical medical decision support tool

16
Standards IEC 61508

• Risk-based approach
• More care with high risk high hazard
• Life-cycle based
• Hazard risk analysis
• Development lifecycle
• Consider hardware software
• Failure mechanisms
• Random hardware systematic design
• Failure Prevention Management

17
Safety Integrity Levels

• Measure of risk of failure
• Guide selection of methods

18
Are SILs useful?

• Are levels logical?
• How do we demonstrate achievement?
• Rates too low to measure by testing
• Cant estimate probability of failure of software
  from code or design
• Do suggested methods reduce errors?
• Differ from zero defects approach
• Methods reducing defects save money
• Use at any SIL

19
Formal Methods

• Formal Specification
• Pre/post conditions (JML)
• Static Analysis
• Analyse code to show that
• Certain good things happen
• Certain bad things dont happen
• ESC/Java2
• Program Proving
• Mathematical proof of correctness
• Requires a formal specification

20
Static Analysis

• Analyse code without executing it
• e.g.
• Whatever happens, are all variables are
  initialised before use?
• Are all parameters used?
• Are array indices within bounds?
• Is there any unreachable code?
• Extend compiler checks

21
Summary

• High integrity software
• Significant consequences of failure
• Requires careful development
• Legal responsibility on developers
• Standards identify best practice
• But best practice changes
• Formal methods considered vital
• But dont cover all possible failure modes

22
(No Transcript)