Balancing Practices: Inspections, Testing, and Others JAXA scenario (formal method)

1
Balancing Practices Inspections, Testing, and
OthersJAXA scenario (formal method)

• Masa Katahira
• Japanese Space Agency

2
Strategy to select methods

• We realize that
• the correct use of particular methods,
• the combination of several methods
• are very important.
• But how?
• Quality Goals
• Budget Limitation
• System Characteristics
• Data availability, Development Phase

• Methods Compliment
• Inspection/Review
• Testing
• Formal Method
• Theorem Proving
• Auto Model Checking

• Inspections/Reviews
• Hard to cover all aspects
• Testing
• Not complete, too late in some case
• Formal Method
• TP Complex for practical use
• MC State explosion possible

3
Selection and Scalability of Methodologies(sample
)
Phases
Full set
Selection (Depth)
Simulation
Light
Modeling/Model Checking
Inspection/Review (Check List)
Completeness/Consistency
Design Coverage Timing
Auto Test Case Generation Robustness Evaluation
Interface Validation
Assessment Attributes (Sample)
Test Case Test Test Result Review
Verification Coverage
4
Still we dont know how to decide methods
selection!
5

• Consideration
• For projects A,B,C, there is not enough time to
  perform model checking. Review with check list
  instead.
• For Project D, a checklist is useful for the data
  correctness and consistency of data handling
  system.
• For Project E, the effective of model checking is
  confirmed due to having enough time.
• Review also shows important role in erroneous
  description in the specification
• For design phase such as C,E, model checking and
  a checklist help finding errors which can not be
  found by review.
• For projects A,B,C, there is not enough time, and
  the review with check list shows efficiency.
• For Project D, Review as well as review with
  checklist shows the efficiency for data handling
  system.
• For such as Project E case, when enough time is
  assigned, the model checking shows good results.

Fig.1 Each methods effectiveness among all
significant issues
Fig.2 Each methods effectiveness among all
Editorial Errors
Fig.3 Each methods effectiveness among all
Significant issues and Editorial errors
6
Lessons Learned Summaryin JAXA case study
7
Boundary of Formal Method Application
Safety Critical
System Characteristics Complexity
Available man-month
Important Border
8
JAXA formal method activity
9
Needs and remaining issues of formal method

• Problem Statement
• Need to assure the high reliability of spacecraft
• Facing to the difficulty to prove the goodness
  only by test and inspection because of system
  complexity and safety requirements such must not
  work
• Large number of defects are introduced mostly at
  the Requirement Phase or the Early Design Phase.
  Unintended or Unexpected system/software behavior
  is difficult to be found at the inspection/review
  by manual.
• Challenge
• Knowledge base inspection/review is still very
  important, but model checking gives a chance to
  detect important findings which are easy to miss
  by the reviewer.
• Modeling task itself gives a chance to think
  enough deeply what the specification really says
  as if the reviewers build the software by
  themselves.
• Remaining Issues
• Quality of Model and model checking task
• Large amount of time is spent to correct the
  erroneous model
• Abstraction and partitioning techniques
• To avoid state explosion, and missing the
  scenarios
• Better Productivity
• Hard to find real problems from thousands of auto
  checking results
• Personnel skill

10
Modeling and model checking in JAXA
Model Checking (Static Analysis)
Requirement Model (SpecTRM1, Uppaal)
Completeness Consistency Reachability
Req. Spec. ICD Hazard Report
Design Model (SpecTRM, Uppaal)
Design Spec. ICD Hazard Report
1SpecTRM Specification Tools and Requirements
Methodology
11
Productivity improvement
Modeling Task Cost
12
Real issues from the results of the modeling and
model checking
Identified issues in the specification
13
Lessons Learned from industry use of modeling and
model checking

• Modeling
• Can organize information and execute the modeling
  in the brain
• Identify lots of basic problems in the
  specification (ambiguous descriptions,
  inconsistency of the contents, unclear data
  definition) as to make the accurate model of the
  specification in the formal language

• Automated Consistency analysis
• Effective to identify the inconsistency in the
  requirement specification
• Identify the inconsistency among the procedures
  in case that multiple tasks executions are
  allowed simultaneously in the design level.
• Automated Completeness analysis
• Check whether the nodes after the transition at
  the branch in the flowchart meet the number of
  the transition conditions and its contents, and
  whether all error handling and exceptional
  procedure are covered in the design level.

• Formal Validation of the functional behavior
  using SPIN
• Effective to validate whether the procedures are
  executed without stagnation and those behavior
  meet the requirements for the procedure flow in
  the detailed design
• Effective to verify whether hazard control
  function/failure recovery functions are working
  without unintended stop in the real time

14
Questions? (Formal Method)

• What is the role of formal method (Theorem
  Proving, Model checking etc.) in many quality
  practices?
• When is a Formal Method necessary or efficient?
• What is a Formal Method useful for? Specific
  Aspects?
• What are most important research issues to deploy
  the method into real projects? Industry Needs?
• What empirical data gathered at the industry will
  be useful to future research?
• What is an expected benefit from use of formal
  method?

15
Backup Slide
16
Findings from each methods(Spacecrafts Projects)
Significant Signification Issues to be modified
such as incorrect or missing functions/logic/data
Editorial Editorial Errors in the
specification No Issue Non real problem
(misunderstanding/modeling mistake)
17
SpecTRM (Model) Based Robustness Test
Environment (SpecRobusT)

• Outline
• By using specification models, the important test
  cases are generated for full software simulation
  during development contractors test phase
  automatically and comparing results.
• Especially, all inputs are verified in the model
  to generate the test cases.
• Auto tests are performed at 10,000 100,000
  cases / sec.
• Results of Project application
• of Test Case 550,870,000,000
• Benefits
• Verification at very early phase
• Introduction to automated test environment
• Introduce Test Before Development paradigm into
  development process

Implementation Procedure