Model-Validation in Model-Based Development

1
Model-Validation in Model-Based Development
Kurt Woodham L-3 Communications
Ajitha Rajan, Mats Heimdahl University of
Minnesota
OSMA SAS 08 September 8-12
2
Problem Model Validation

• Model-Based Development (MBD) is here to stay
• Use of MBD is accelerating
• Estimate 50 of NASA development projects using
  some form of MBD
• Many advantages model-checking, code generation,
  desktop testing, closed-loop simulation
• Enhances early detection of requirement, design,
  or implementation defects
• Executable Specifications enable evaluation of
  behavior that might otherwise be relegated to
  Inspections and Testing
• How do we know the models are right?
• Manually develop black-box tests
• When have we validated enough?
• Measure test coverage on an implementation/model

3
Problem Current Practice

• Measure black-box test coverage over the model
• Indirect measure
• Defects of omission in model not exposed.
• Executable artifact is necessary
• Adequacy can only be determined late in the
  development process

Incomplete Model
Weak Black-Box Test set
4
Goals of Project

• Define metrics for objective, implementation-indep
  endent measure of adequacy of a black-box test
  suite
• Develop tools to measure validation adequacy
  based on the defined metrics
• Provide capability for autogeneration of
  black-box test suites

5
Testing What does it mean?
Assertion Based Testing (ABT) to Validate Model
In General
Our contribution is in providing novel ABT
capabilities
6
What are Assertions?
Properties/ Formal Assertions
Assertions
Can also be over components, interfaces,...
Defined over
System
in1
out1
ink
outm
7
Contributions - ABT
2
Auto-generate
Black-Box Tests
Assertions
Measure Adequacy
1

• We provide the following contributions in the
  Assertion-Based testing domain (indicated by
  in the above figure)
• Objective, implementation-independent measure of
  adequacy of a black-box test suite
• Auto-generation of black-box validation tests
  directly from assertions
• Objective assessment of completeness of model as
  well as assertions

8
Relevance to NASA

• MBD is here - estimate one-half of all NASA
  missions in development or on the books will use
  model-based subsystem development
• Extensive use in avionics industry
• How do we know the models are right?
• Model validation problem
• We provide the capability to
• Objectively measure the quality of
  assertion-based black-box validation tests
• Objectively assess the completeness of a model
• Does the model address all assertions?
• Objectively assess the adequacy of a set of
  assertions
• Are there enough assertions to adequately
  describe the model?
• Automatically generate truly assertion-based tests

9
Achievements to Date

• Formal assertion notation identified
• Most work with LTL
• Extended to work with Live Sequence Charts (LSC)
• Objective validation metrics defined
• Requirements, Antecedent, Unique First Cause, and
  Unique Cause
• Test case generation tool developed
• Developed tool generating tests from LTL
• Capable of generating tests to all metrics
  defined
• Prototype tool working on LSC developed
• Developed test-adequacy measurement tool for the
  defined validation metrics
• Evaluation of metrics and tool
• 12 papers and one PhD dissertation (Ajitha Rajan)

Tool Demo in Tech. Session
10
Next Steps

• Investigate alternative requirements notations to
  LTL
• Complete empirical evaluation of the
  effectiveness in model validation
• Flight Guidance System (FGS) evaluation complete
• Display Manager (DM) evaluation in work
• Coordinate evaluation on NASA IVV project
• Coordinate technology transfer