The target technology for costeffective assessment of critical SW dependability and functional safet

1
The report is presented byUkrainian State
Nuclear Regulatory Committee IC System
Certification Center p/o 9871, Akademika
Prockury str., 1, Kharkiv, 61070, Ukraine
Tel/fax [phone]
Director Georgiy Chertkov web-site
http//scasu.com, e-mailadmin_at_scasu.com
The target technology for cost-effective
assessment of critical SW dependability and
functional safety independent verification and
qualification of critical SW for IC systems.
Project manager Professor, Doctor of Science,
Head scientific researcher Borys
KonorevTechnical Meeting on Increasing Power
Output and Performance of NPPs by Improved
Instrumentation and Control Systems 29-31 May
2007 Prague, Czech Republic
2
Content

• Introduction. Quality and reliability of critical
  software and IC systems safety.
• Normative regulation of IC software quality and
  safety.
• Life cycle model and quality assessment of
  critical software.
• Concept of target technology of critical software
  cost-effective assessment.
• Target technology of critical software
  cost-effective assessment Methodology.
• Target technology of critical software
  cost-effective assessment. Functional IDEF0 model
  .
• Unit tree of functional IDEF0 - model.
• Normalization of the software project Reference
  and assessment models. Normative base.
• Normalization of the software project
  requirements profiling. Screening technology.
• Diverse measurement of the software invariants.
• Model and scheme of software quality measurement.
  
• Inverse model of software residual defects for
  the composition of diverse verification
  technologies.
• Efficiency assessment of diversification of the
  verification technologies.
• Model of sensitivity and variety degree of the
  calibration of the diverse methods on the basis
  of the test defects drop injection.
• Cost-effectiveness of critical software
  assessment parametric control.
• Resulting software quality assessment. RMD.
  Convolution.
• Approaches to implementation, development stage.
• Target technology of critical software
  cost-effective assessment innovation and
  advantages.
• Summary

3
Introduction. Quality and reliability of critical
software and IC systems safety.
IC (Instrumentation and Control) systems are the
key factor providing the main dependability
characteristics (reliability, availability,
maintainability) as well as the safety of the
project for the critical areas such, as atomic
engineering, space activity, etc.
The level of system criticality is defined by the
consequences of abnormal operation in a range
ltmaterial losses - environmental damage - danger
to people health and lifegt taking into account
the probability of its occurrence.
The volumes of software-based critical functions
of IC systems tend to grow. The dependency of
safety use of IC systems from the quality of
critical software grows directly.
Residual software defects (not revealed on test
stages) are the risk factors of system abnormal
operation and emergency hazard. That is why the
status of critical software is the important
element of the normative regulation determining
dependability and safety of the IC system in the
frame-works of risk-informed approach to NPP
safety assurance.
The main procedures of normative regulation and
licensing activities are the independent
verification and qualification tests of IC
system. One of the basic objectives of
qualification consists in the cost effective
assessment of the risks of system abnormal
operation due to residual defects of critical
software.
4
Scope normative regulation of IC software
quality and safety
matrix of criticality mij (P,S)
Technical and science expertise of critical SW
conformity to the requirements
Probability
Application area
SW project
Severity
Expertise report
Regulative and licensing body
Expert group
??????? ?????????? ?????? ???????????? ??
1. A key part of safety control loop is the
scientific and technical expertise of conformity
critical software to the specified
requirements. 2. The procedure of expertise
represents the target technology of the quality
cost-effective assessment of critical software
(dependability and functional safety) . It
defines an urgency of this technology. 3.
Normative -methodical provision and the procedure
of a cost-effective assessment of critical
software define real opportunities to achieve
acceptable safety levels during operation of
critical IC systems.
Normative methodical base
Tools utilities of software expertise support
Organization performing software expertise and
independent verification
5
Life cycle model and quality assessment of
critical software
Acceptance certificate
Expert report
Contract
Supply
Validation
Qualification tests
System requirements
Acceptance test
verification
SW assessment
SW assessment
SW assessment
Requirement specification
Software requirements
verification
System test
Software logical model
Integration test
Architectural design
verification
Software physical model
verification
Detail project and production
Unit test
Arranged scope measurement of internal, external,
quality in use characteristics in real operation
conditions.
Measurement of internal, external, quality in use
characteristics on the results of pilot operation
Software module structure
t
Code
Maintenance. Life-time extension
Acceptance test
Preliminary test
Conformity expertise. Independent verification.

?????????????
Measurement of internal, external, quality in use
characteristics.
Cost-effective assessment of software quality and
functional safety is the basic procedure for all
right-hand branches of split life cycle V-model
including preliminary test, independent
verification, acceptance test, maintenance.
6
Concept of target technology of critical software
cost-effective assessment.

• The concept of target technology is the
  independent verification of critical software
  using the advanced technique of the IC software
  static analysis. Improvement consists in
  provision of objective tool measurements of
  semantic, interval-precision, logic and other
  software invariants.

• Invariants represent source software attributes
  (properties) permanent during software life
  cycle. These invariants represent the basis for
  measurement of the values of critical software
  dependability (reliability, availability,
  maintainability) and functional safety.

• Functional safety of critical IC systems
  regards as system being in conditions of
  acceptable risk of the abnormal operation
  specified by the design during life time. The
  metrics of cost-effective assessment and
  functional safety of critical software is the
  value (measure) mij (P, S) of predicted risks
  levels occurred due to residual software defects,
  at minimal resources input (terms, labor input,
  cost).

7
Target technology of critical software
cost-effective assessment Methodology. Includes
the following elements
1. Normalization of expertise object and the
development of assessment model (the control loop
of software attributes measurement). The purpose
consist in disclosing of specifications of the
specific software project invariants-attributes.
As the result of normalization the measurement
scheme determining the metrics (methods and
scales) for measurements of specific software
project and measures of a test coverage is
developed on the basis of assessment model. The
development of requirements normative profile
for the specific software project is a key
element of normalization. This development is
performed using the procedures of formalization,
screening and harmonization for normative profile
disjuncts (clauses) of technical documentation
for specific software project.
2. The diverse measurement of semantic,
interval-precision, logic, etc. software
invariants using the static analysis of source
software texts. Software invariants represent
software attributes fixed (permanent) during
software life cycle. Measurements results are
represented as the linear equations and
inequalities systems , describing invariants
trajectories for chains of operational display
of specific project at all levels of software
architecture hierarchy. Processing of the static
analysis results and the assessment of software
qualities characteristics consists in the
decision of the linear equations and inequalities
systems on the basis of logic conclusion and
restoring of the missed invariants values.
3. Cost-effective assessment on the basis of
parametric controlled procedure which consists in
calibration of sensitivity and a diversity
degree of composition methods of invariants
measurement. Calibration consists in
experimental definition of sensitivity and
diversity degree of composition methods of
software attributes measurement taking into
account the specificity of the concrete software
project . The method of a drop injection ("crop")
of software test defects is used corresponding to
the stated profiles. Specificity of the software
project is presented by its operational
mix. The final assessment is made on the
indicator, evaluating the reduction of residual
defects probability in the specific software
project in a range 0 100 (in a limit on 100
depending on software type) for a composition
of diverse assessment methods at acceptable
cost-effectiveness level.
8
Target technology of critical software
cost-effective assessment Functional IDEF0 model
.
Parametric control of calibration of the diverse
measurement methods for invariants.

• Control mechanism
• (restrictions)
• scenario
• plan

The scheme of software invariants measurement
Input
Software project normalization
Project documentation
Measurement of invariants
Output

• Summary
• cumulative assessment of dependability and
  safety
• profitability

Cumulative assessment. Risks
The results of equation and inequalities,
connecting the invariants at all hierarchy levels

• Realization mechanism
• techniques
• utilities
• regulations

The scenario of target technology presented by
three concept techniques. Realization mechanism
of these methods includes the techniques,
utilities integrated tool of the scenario
support on the analytical, information and
organization levels. Scenario including
procedures of experimental calibration of the
sensitivity and the diversity degree of
measurement methods provides the mechanism of
management of critical software cost-effective
assessment.
9
Unit tree of functional IDEF0 - model.
Defines the specification of performed activities
and tasks on the 0-1-2 levels of the model
including the following elements
0 -level
Scenario.
1 level concept- methods
Measurement of invariants in the static analysis
mode of source software
Quality assessment of software project
Software project normalization
2 level tasks

• development of the normative profile for
  software project

• instrumentation of the source software.
  Implementation of assessment model (measurement
  scheme)

• diversification, parametric control, calibration
  of sensitivity and diversity of invariants
  measurement methods

• disclosing if the specification of the software
  invariants and definition of the measurement
  scheme (assessment model)

• analysis of predicted risks of IC systems
  abnormal operation and cost-effectiveness of the
  assessment

• measurement of the invariants in the mode of
  interpretation of instrumented version

• compression of radial metrics and cummulative
  assessment of software project quality

• verification of the assessment model and
  evaluation of the test coverage completeness.
  Direct and inverse tracing of disjuncts
  ltmeasurement schemegt, ltspecificationgt, ltnormative
  profilegt, lttechnical documentationgt.

• processing of the static analysis results
  (solving of the systems of linear equations and
  inequalities )

10
Next slide provides an information on the models
which are the basis for target technology
methodology.
Normalization of the software project Reference
and assessment models. Normative base.
The purpose 1) Disclosing of the specifications
of software project invariants and development of
the assessment model for the software project.
2) Verification and an assessment of test
covering completeness on the basis of exhaustive
direct and inverse tracing of normative profile
disjuncts and assessment model of the software
project.
The normative base of the target technology is
harmonized with international standards IEC
61508, IEC 60815, IEC 61025, ISO/IEC 12207,
ISO/IEC 9126, ISO/IEC 14598, ISO/IEC 15504,
ISO/IEC 25 000TR and branch standards ?) for the
atomic engineering IEC 1226, IEC 61226, IEC
62138, IEC 60880 1,2 IAEA NS-G 1.1, IAEA TR No
384 ?) for the space industry ECSS ?40,
ECSS-Q-30, 40, 80 xx ESA PSS 05-xx
Reference model common normative profile
(CNP). Assessment model cartesian product for
the set of CNP disjuncts, project definition
files (PDF) and project justification files (PJF)

Reference model
Assessment model
Assessment techniques
The target technology of critical software
cost-effective assessment is the key element for
the set of fundamental practices of system risks
analysis, including PHA (preliminary hazard
analysis), FMECA (failure mode, effects and
criticality analysis), FTA (fault tree
analysis), HSIA (hardware software integration
analysis), HAZOP (hazard operational analysis),
CCFA (common cause failure analysis). RAMS
(reliability, availability, maintainability and
supportability)
11
Normalization of the software project
requirements profiling. Screening - technology
The availability of adequate normative profile of
software requirements is the necessary criterion
for quality assurance (dependability and
functional safety) of critical IC systems.
Standards
Basic standards of general industrial application

• Branch standards

The basis of normative profile making is the
taxonomy, including processes, methodology and
procedures.
The assessment model defines the attribute
measurement scheme, including the attribute
specification, the methods and metrics of the
measurement
Reference model Common normative profile
Skreening 2 Development of profile-formative
base
Processes (definitions, actions, tasks)
Screening technology is used for SCREENING 1
Development of the prepared profile-making
base. SCREENING 2 Development of basic
model. SCREENING 3 Development of assessment
model. Harmonization. (specialization and
generalization of profile-making base disjuncts)
Screening 1
Screening 3
Methodology (Methods and metrics, tasks,
operations)
Procedures (Techniques. Tools and environments)
Profile-making normative base
12
Model and scheme of software quality
measurement( is developed on the basis of
ISO/IEC 25000, 9126, 14598, 15504, 12207)
Defines a) Models, attributes and metrics of
quality assessment on different software life
cycle stages b) Reference model of software
quality assessment and the scheme measurement.
SW integration
Real platform
SW specification
influences
influences
influences
External quality
Quality in use
Internal quality
SW processes quality
depends on
depends on
depends on
is measured
is measured
defines
is measured
Attributes and metrics of quality in use ?3
Attributes and metrics of external quality ?2
Attributes and metrics of internal quality ?1
CMMI levels
influences
SW quality measurement scheme
Presents the sets ?1, ?2, ?3 superposition of
software attributes, defining internal, external
and quality in use.
Characteristic
?2

Sub-characteristic

Reference model for software quality analysis and
assessment
Attribute (source attributes SW invariants)
?1
?3

Metric (Method and scale)
13
Diverse measurement of the software invariants
The model of integrated checking capability of
diverse verification technologies composition
based on numeral and semantic SW invariants. It
represents the specification of the set of
software defects, detected by the diverse methods
of invariants measurement
U Ai reference model, base of software
parameters analysis and assessment for
composition of diverse technology
?1
?3
D1,D2 sets of detected faults (anomalies) of
invariant types 1 and 2.
?2
D initial set of defects
Comparison of diverse assessment results
D
D1
D2
D
D\D'1U D'2 - sets of faults, which are not
detected by 1st and 2nd methods (insensitivity of
the diverse methods composition)
D'1U D'2 superposition of faults detected by
1st or 2nd methods in SW address field. Defines
the real degree of diversity methods of software
invariants measurement.
D'1
D'2
Software address field
14
Inverse model of software residual defects for
the composition of diverse invariants measurement
methods
Software residual defects model is the modified
model of diversification of invariants
measurement. It represents the superposition of
virtual sub-sets of residual defects ?i for
composition of diverse methods verification with
different sensitivity (checking ability)
Method sensitivity is defined by software
residual defects probability
A software address field
A
? initial defects set
?
?1
?1 initial set of residual defects,
undetected at verification (the result of the
first method implementation)
M1 n M2
M1 n M2 sub-set of residual defects, undetected
for the composition of verification methods
?2
M\M1 U M2
n ?2 n Mi sub-set of residual
defects i1 undetected at the
verification by the diverse methods Mi (resulting
measure of diversity of methods composition for
invariants measurement)
Possible variants Common case M1 n M2
?Ø
n Pessimistic - M1 M2 n Mi
i1 Optimistic
- M1 n M2 Ø
M\M1 n M2 sub-set of detected defects for the
composition of verification methods
15
Efficiency assessment of diversification of the
verification technologies
The benefit (progress) in the result of the
composition of realization of diverse
verification methods is estimated using the
metrics indicator decrease of software
residual defects risk
Additive benefit
Local benefit
?
?
?1
?2
?i
Diversity measure
Software address field
Initial set of defects
??
_____ ??, ? 1,n virtual sub-set of
SW faults, undetected by each diverse method .
n n Mi diversity measure of methods
composition. i1
n Probability of
null defect for the composition of diverse
methods ?n (1-?(?1\ n Mi))
i1
Maximum benefit for the composition of diverse
methods can make ?(?1)
Indicator of software residual defects risks
decrease for verification and independent
verification composition
n ?1\ n Mi)

i1
n ? ?(?1) -
?(?1) ?(?1) (1-?(?1\ n Mi))

(?1)
i1
16
Model of sensitivity and variety degree of the
calibration of diverse methods on the basis of
the test defects drop injection.
Defines the parametrically controlled procedure
of accessible required cost-effectiveness of an
critical software assessment. Controll
parameters ltprofile of defectsgt ltinvariantgt ltnumb
er of test defectsgt
Steps (layers) of calibration (number of
dropped test defect)
N
For the specified calibration volume is
defined 1) total sensitivity of each method i U
?part i.
At every step defines 1. Disjunction of
undetected test defects of the for each ?i.
2. in pairs for all ?i (each taken together) the
diversity degree i j n (U ?part i) and the
development of diversification matrix
2. Conjunction of undetected test defects on each
defect type of the profile for each ?i..
?i
?n
?1
3. Partial sensitivity ?part i on each defect
type .
Methods
___ mij 1,0 mij ? mji
Defects profile (types and percentage composition)
17
Cost-effectiveness of critical software
assessment parametric control
The purpose to achieve allowable values of
predicted risks of IC system abnormal
functioning due to software residual defects at
the minimal (comprehensible) resources input
(time, labour input, cost).
Parameters of works management of the assessment
scenario 1. Test defects profile (specyfied
according to the operations spectrum of the
specific software project). 2. The specification
of diverse methods composition of invariants
measurement (specyfied according to the
operations spectrum of the specific software
project and statistics on methods
sensitivity). 3. Number of calibration steps
(layers) (specyfied according to the selected
reliability values, a confide?e interval and
accuracy of expectation probability assessment
and zero-defects (sensitivity) of calibrated
method.)
18
Resulting software quality assessment. RMD.
Convolution.
Basic procedure of the radial metrics diagrams
(KIVIAT) convolution area of RMD lower level ?-1
images into the value of radius-vector RMD ? of
the higher level taking into account relative
values and weight ratio.
weight ratio
corresponds completely
0,3
corresponds partly
Measured metrics value
0,3
Average time of nonoperability
doesnt corresponds
Usability
Recoverability
Operational safety
0,3
1
1
0,25
0,05
Availability
0,4
0,05
Recovery average time
Functionality
1
0,25
1
Reliability
0,5
0,5
0,5
0,5
1
1
1
1
0,5
0,5
0,5
0,5
0,5
0,5
0,5
0,5
0,5
1
1
0,5
1
1
0,5
0,5
Rationality
Rebootability
1
Rationality of recoverability
1
0,1
0,15
Reliability conformity
Resistance to deviations
1
1
0,1
Portability
0,15
Recoverability
Maintainability
0,2
0,1
0,15
0,15
2-d hierarchy level radial metric diagramm
3-d hierarchy level radial metric diagramm (RMD)
1-d hierarchy level radial metric diagramm
SW reliability "
SW recoverability"
"SW quality"
19
Approaches to implementation, development stage.
1. The parser (Gold Parser - free software) with
the mechanism of adjustment - adaptation to the
syntax of programming languages (C, C , C,
Java, VHDL) for source software static analysis
and development of the instrumented software
version providing invariants measurement is used .
2. Graphical user interface GUI of the
integrated tool environment utilities is
developed using the SDI (Single-Document-Interface
).
3. The utilities complexes are developed as
programmed Web - applications (services) on the
basis of actual Internet standards.
4. The theoretical basis and testable prototypes
of utilities complexes of the integrated tool
environment for support of the target technology
scenario are developed.
5. The technical implementation of utilities and
full-scaled development availability are
confirmed with integration laboratory tests with
use of real expertise objects - critical IC
systems in language ?/? . The key decisions of
target technology are protected by patents.
20
Target technology of critical software
cost-effective assessment innovation and
advantages.
1. Independent verification of critical software
on the basis of the advanced technology of the
source software static analysis , providing
objective tool measurements of semantic,
interval-precision, logic, etc. software
invariants and an assessment on their basis of
dependability and functional safety
characteristics of software.
2. Increase of trustworthiness of critical
software dependability and functional safety
assessment implementing ths diversification
methods of invariants measurement.
3. Experimental calibration of sensitivity and
variety degree of diverse measurement methods
composition in a framework of the concrete
software project. The calibration is performed
using the method of "crop" of software test
defects as provided by specified defect profiles.
Definition of indicator of decrease in IC
system abnormal functioning probability due to
software defects. Use of the advanced method of
"crop" - the drop injection of test defects
excluding effects of "interference" and
"mutation" of defects during crop in an software
address field.
4. Assessment of test covering completeness for
critical software during independent verification
on the basis of exhaustive (direct and inverse)
traces Specifications of software invariants
with disjuncts (elements) Specifications of
design definitions and justifications.
5. Achievement of a required level of
cost-effective assessment of predicted risks of
IC systems abnormal functioning due to software
defects at the minimal level of resources inputs
on the basis of iterative calibration procedure
of composition of diverse measurement methods of
software invariants. Calibration is controlled by
the parameters ltsoftware invariant typegt -
ltsoftware defects profilegt - ltnumber of test
defectsgt.
21
Summary
The cost-efficiency of critical software
dependability and functional safety assessment is
considered as achievement of reasonable level of
predicted risks of IC systems abnormal
functioning due to residual software defects, at
reasonable resources inputs. The general context
of assessment is implementation of fundamental
practices of the analysis and qualification of
IC systems criticality - FMECA, RAMS, FTA, PHA,
HSIA, etc.
Presented target technology of critical software
cost-effective assessment ?) extends real
opportunities of the developers and regulative
bodies in forecasting the risks of IC systems
abnormal functioning due to software defects at
independent verification, qualification and
certification in frameworks of risk-informed
approaches to safety regulation in critical
spheres of technical activity (atomic
engineering, space technic, etc.).
b) is the important element of long-term programs
of refurbishment, modernization and life time
extension of IC systems important for NPP safety
.
c) provides an opportunity of quantitative
assessment of limit values and decrease of
probability of critical software residual defects
in a range 0 - 100 (in a limit on 100
depending on the software project and acceptable
cost-effectiveness level).
d) Corresponds to the advanced scientific and
technical level (basic project technical
decisions are protected by patents) and general
IAEA policy (state of the art) in sphere of
critical IC system software qualification.
22
TANK YOU FOR YOUR ATTENTION