Title: Certification Issues
1(No Transcript)
2Safety Critical SystemsandCertification
IssuesDO-178BAirborne Standard
3SC190 / WG-52 Application Guidelines For
RTCADO-178b/ED-12b
RTCA
EUROCAE
SC-167
WG-12
AvionicsIndustry
CNS/ATM Community
SC-190 / WG-52
Cast Position Papers
CAST (Certification Authority Software Team)
SC-190 Products
4DO-178B in a Nutshell
- Highly Process Oriented
- Requires Traceability
- Requirements
- High Level Design
- Detailed Design
- Source Code
- Test Procedures
- Test results
- Test Test Test Test and ..
5DO-178B Safety Levels
- Level A
- Catastrophic
- Failure Prevents Continued Safe Flight and
Landing - Level B
- Hazardous/Severe-Major
- Potential Fatal Injuries to a Small Number of
Occupants - Level C
- Major
- Discomfort to Occupants or Possible Injuries
- Level D
- Minor
- Increased Crew Workload
6Development Team
- What is COTS?
- How can objectives of DO-178B be satisfied, using
COTS? - There is much variation in applicants for COTS
certification credit - Is DO-178B clear on the interpretations?
Certification Evidence Available as COTS
Product is COTS
Together these satisfy safety objectives
7Testing without Source Code
Wrappers to Validate Parameters
Commercial O/S
Application
This cannot be trusted unless O/S is verified
8Special Considerations
Use of Service History for Certification
Air Traffic Control system
- Worked Correctly in US for years
- Transferred to U.K.
Actual Plane 2
Plane 1
Displayed Plane 2
Greenwich Meridian
9Use of Service History
Developed under a less stringent standard
(Military?)
Used for 4 years Problems tracked Quality Good!
Safety Critical System
Dead Code(Unintended Function)
Residual Error
Is this system safe for the next 4 years? At
Level A, B, C? We can bound inputs, but we
cannot check internal states without looking
inside
10Black Box Testing
- No single failure should prevent Continuous safe
flight and landing. - Statistical testing cannot show absence of a
single state that will cause a failure - Software has discontinuities
- Software does not follow Gaus/Normal Distribution
There is no foundation for statistical reasoning
about software faults or safety
11Verification Teamexamples of issues
- What are low level requirements? How can they be
tested - Data and Control flow coupling
- Use of higher level test results for lower level
requirements - What is the intent of structural coverage?
- Traceability of source to object code for
structural coverage - What is statement, decision, condition and MCDC
coverage testing (Modified Condition/Decision
Code) - Verification tool qualification
- etc..
12Coverage Analysisat Level B and C
- Statement Coverage
- Decision Coverage
- Entry Points
- Exit Points
- All Decisions
- All Outcomes
13Fixing anomaliesexample for level B/Library
Filter
B 3 A Filter (B) X X A
Compiler
Object Code
Source level coverage required
Even for Filter
14Boundary Level testingnot enough!
A B -- A and B are packed Boolean arrays
Run-time call to Bit_Block_Move (From, To,
Size) -- size in bits
- Min, Mid, Max in combination gives 27 (useless)
test cases - 5 Bits size
- 32 Bit size
- 67 Bit Size
- From overlaps To
- To overlaps From
Interesting test cases based on actual
code i.e. White Box Testing
15Coverage at Level A
- Coverage required at Machine Code level or
- Show source to object code traceability
- and test at source level or
- Use different language/compilers
- and use voting system
- MCDC testing required
- each condition must have effect on outcome
- Tools which modify source for traceability
- problem at level A
- Mitigation method use 3 different compilers
- (Now Look At Conditional Statements)
16Conditions/Decisions
Conditions
if AB and C or Dlt3 then
Decision
Boolean Variable
Boolean Operators
17What areConditions/Decisions
MCDC Coverage Requires all Branches AND all
Conditions Be Covered
18More BooleanConditions
Cannot hide fromTesting Obligations
and are Boolean Operators!
19Condition coverage
Coverage of Basic-Block may not capture
condition results
20Avoiding MCDC Testing
Modified Condition/Decision Code
Use Adas short-circuit conditions if A0 and
then Blt 2 and then Cgt5 then Or in C write if
A 0 B lt 2 C lt 5
21Why short-circuitconditions eliminateMCDC
if A0 and then Blt 2 and then Cgt5 then
if A0 then if Blt2 then if Cgt5 then
P end if end if end if
MCDC not required for this code
22Testing strategymust evaluate conditions
if A0 and then Blt 2 and then Cgt5 then
if A0 then if Blt2 then if Cgt5 then
P end if end if end if
BUT !!!
Testing must show thateach then part has been
tested True and False
MCDC not required for this code
23Inlining code
- If decisions/conditions introduced
- Decisions must be identified and verified (level
B) - Conditions must be identified and verified (level
A) - Verification may be done by analysis
- Traced to derived requirements
- ensure safety is not compromised
- Code may be deactivated
- As inlined code depends on local state it may be
very hard to test the conditions in accordance
with standards requirements - Intent - absence of unintended funtion
- Dead code not allowed
24Use of Tools
- Tool Qualification is required if tool replaces a
step of development process - Development tools
- Examples - Compiler, Design to code generator
- May introduce an error
- In general - NOT qualified, not trusted
- Verification tools
- Examples - Coverage analyser
- May conceal an error
- May be qualified - Trusted for verification
purposes - Additional verification process required if the
tool is not trusted
25CNS/ATM Process Integration
- Information matrix
- Regulators
- Committees
- Standards Bodies
- Standard
- Software Integrity Assurance Standards vs.
Software Development Standards - Relationships between DO-178B
- and IEC 61508
26Ground BasedCommunity
- Communications/ Navigation/Surveilance and Air
Traffic Management - in the loop - Ground Based Systems affect airborne software
- DO-178B addresses airborne only
- Guidance being prepared to encompass needs of
CNS/ATM community (SC-190 committee) - Standards tightening up
27Certification Standardsare Improving
- Holes in documents are being fixed
- Understanding of Certification Requirements is
spreading - Industry and Certification Authorities are
collaborating on Guidance materials - It will get more difficult to shop around for a
more lenient signature
28( Legal)Safety Systems
Case Law Precedence Interpretations Standards Guid
elines
Laws Regulations Standards Guidelines
PROCESS
Traceability
Visibility
EVIDENCE / RECORD
Confidence / Safety
29When Is Software Safe
30When Is Software Safe
We Dont Know !!
31What is our best guessabout the safety
- When applicable processes have been followed
- When we have verified the code from within
- When this has been checked
- and checked
- and checked
- and checked
- and
checked -
and checked
32The FAA/JAA Rulesare Strict
To Date no fatalities have been attributed to
Software Failure
Have we been lucky? Have we been safe?