A Requirement Verification Framework for Realtime Embedded Systems

1
A Requirement Verification Framework for
Real-time Embedded Systems

• By Feng Zhu
• Advisor Prof. W. T. Tsai

2
Agenda

• Introduction
• The Verification Framework
• Requirement and Verification Patterns
• Framework Evaluation
• Summary
• Questions Answers

3
Introduction

• Importance of the Verification of Embedded
  Software
• Failures in reactive applications including
  safety-critical devices such as cardiac rhythm
  management devices can be costly and have to be
  avoided.
• Software is playing more and more important role
  of controlling system components in many
  safety-critical systems
• Its testing is critical to ensure that products
  have higher levels of reliability and
  availability.
• Its testing is the most critical phase of the
  product development where the product is checked
  for its compliance with its requirements through
  its execution

4
Introduction

• The Concepts of Object-Oriented Framework
• A framework is a partially implemented program
  that provides an overall software architecture,
  implemented object and object components for a
  specific application or domain.
• Object-oriented reuse technique data
  abstraction, encapsulation, polymorphism,
  inheritance
• Pre-implemented components
• Design Patterns
• Reuse of analysis, architecture, design,
  implementation

5
Introduction

• Application Frameworks Review
• Earlier Application Frameworks
• The Smalltalk-80 user interface framework,
  Model-View-Controller (MVC), MacApp, InterViews,
  ET etc.
• General Purpose Frameworks
• Taligent Framework, IBM San Francisco
• MFC, CORBA, JavaBean, OpenStep, Rhapsody, Java
  AWT
• Web Applications
• WebObjects (Apple), DIWB (Distributed Interactive
  Web Site)
• Testing Related
• Testing frameworks developed in Tsai 93, Onoma
  98, Tsai 99, Poonawala 97, Shao 98, Poonawala 99,
  Vishnuvajjala 01, Tsai 02.
• JUnit, Cactus, HttpUnit

6
The Verification Framework

• Essential Verification Concepts

7
The Verification Framework

• General Requirements of the Framework
• The test framework should be application-independe
  nt.
• The test framework should be easy to adapt
  platform independent.
• The test framework should be easy to expand and
  maintain.
• The complexities of the verification framework
  should be hidden from test developers.
• The test design and the test framework are
  totally separate entities.

8
The Verification Framework

• Rationale for the Verification Framework

9
The Verification Framework

• Architecture

10
The Verification Framework

• Hardware Abstraction
• Insulate higher level components of the framework
  from the hardware details.
• Each device included in the testing system has an
  abstract interface that remains constant from
  project to project.
• The same tests can be applied to different
  versions of test target, although this requires
  some modification of layers directly interfacing
  with the device.

11
The Verification Framework

• Event Assembling
• The essential data composition mechanism of the
  framework
• Makes it possible for testers to write tests at
  domain level instead of working at the raw data
  level.
• A event assembler listens to all the events it
  needed and composes new, higher level events from
  the events it sees.
• With multi-level event assembling, testers can
  observe test output at various levels.

12
The Verification Framework

• Example of Event Assembling PVC (Premature
  Ventricular Contraction) event assembling

13
The Verification Framework

• Use Case of Architecture Test Execution

14
The Verification Framework

• Use Case of Architecture Behavior Verification

15
The Verification Framework

• Framework Components

16
The Verification Framework

• Whats inside Test Engine

17
The Verification Framework

• The Test
• Scenarios
• Sequential execution
• Multi-level test composition
• Verifiers
• Parallel execution multiple verifiers running
  at the same time to verify multiple features
• Continuously monitor test targets behaviors.
• Separation of Scenarios and Verifiers
• Any verifiers can be executed against any
  scenarios
• Feature interactions can be tested

18
Requirement and Verification Patterns

• Motivation
• Conventionally, requirement verification has
  often been done in a case-by-case approach.
• Each new requirement requires a new verification
  procedure implementation, no matter how similar
  it is to some of the existing requirements.
• If a requirement gets changed, its verification
  procedure has to be changed too.
• Separately implemented verification procedures
  need to be debugged separately, which means
  higher cost for development
• Separately implemented verification procedures
  are often implemented by different people and
  normally in different coding styles, which often
  result in higher cost of maintenance.

19
Requirement and Verification Patterns

• Requirement pattern Definition
• A specific temporal pattern or cause-effect
  relation that can be used in representing a
  number of requirements.
• Describing a requirement pattern
• A timeline depicting the temporal relations of
  the events involved.
• Texts are also helpful for more detailed
  explanations.

20
Requirement and Verification Patterns

• Verification pattern
• A verification pattern is a pre-defined
  verification mechanism that can be used to verify
  a group of behavioral requirements that describe
  similar temporal pattern or cause-effect
  relations, or
• A verification pattern is normally associated
  with a requirement pattern and used to verify all
  the requirements that can be classified as
  belonging to that pattern.

21
Requirement and Verification Patterns

• Describing a verification pattern
• Name
• Description and Purpose
• Class Diagram (Also for the requirement
  representation)
• Verification State Machine (representing the
  verification logic)
• Other constraints (Optional)

22
Requirement and Verification Patterns

• Requirement Patterns and their coverage of
  requirements

23
Requirement and Verification Patterns

• Basic Requirement Pattern
• Simplest
• Most-commonly seen
• Example
• When a threshold test is initiated by command,
  the firmware shall request the TEMPORARY brady
  state within 480ms

24
Requirement and Verification Patterns

• Basic Verification Pattern Class Diagram

25
Requirement and Verification Patterns

• Basic Verification Pattern Verification State
  Machine

26
Formulating Requirements

• To formulate the requirement, we derive our shock
  detection requirement class from the
  VVRequirement as follows

Class ShockDetectionReq public
VVRequirement public ShockDetectionReq()m_Ad
justedTimeout 10000 Virtual
ShockDetectionReq() bool PreCondition() bool
PostCondition()
27
Detect the precondition

• within the PreCondition() method of the
  requirement. When the precondition is found, the
  m_ActivationTime of the requirement is set to the
  timestamp of the current event.

Bool ShockDetectionReqPreCondition() if(m_cur
rentEvent m_DetectionMetEvent) return
TRUE else return FALSE
28
Detect the Postcondition

• within the PostCondition() method of the
  requirement. Here we use m_ShockEvent to verify
  that shock is actually delivered.

Bool ShockDetectionReqPreCondition() if(m_cur
rentEvent m_DetectionMetEvent) return
TRUE else return FALSE
29
Requirement and Verification Patterns

• Using the Basic Pattern Compounding
• Example REQ1
• (Q1) The active cruising shall be terminated upon
  occurrence of any of the following events
• (P1) Braking
• (P2) Manual Cancellation
• (P3) Cruise Control Off
• We can have
• Solution1
• REQ1 (P1 ? P2 ? P3) ? Q1
• solution2
• REQ1-1 P1 ? Q1
• REQ1-2 P2 ? Q1
• REQ1-3 P3 ? Q1

30
Requirement and Verification Patterns

• Key-Event Driven Requirement Pattern
• Upon the occurrence of a key event, a series of
  events (R1,,Rn) are expected to happen with the
  pre-specified order.
• Optionally, a timeout is associated with the
  pre-condition to require that all the expected
  events should happen before a timer expires.

31
Requirement and Verification Patterns

• Key-Event Driven Verification Pattern Class
  Diagram

32
Requirement and Verification Patterns

• Key-Event Driven Verification Pattern State
  Machine

33
Requirement and Verification Patterns

• Timed Key-Event Driven Requirement Pattern
• Requires a duration after the key event occurred.
  
• The pre-condition is a combination of the key
  event and the event P that occurred during the
  specified duration (t1).
• Normally stated as
• Within the duration t1 after the key event, if P
  then R is expected (before t2).

34
Requirement and Verification Patterns

• Timed Key-Event Driven Verification Pattern
  Class Diagram

35
Requirement and Verification Patterns

• Timed Key-Event Driven Verification Pattern
  State Machine

36
Requirement and Verification Patterns

• Key-Event Driven Time-Sliced Requirement Pattern
• After the key-event occurs, a pre-specified time
  interval is partitioned into many small slices
  (t1, t2, tn).
• Each of the post-conditions (Q1, , Qi, R1, ,
  Tm) of this pattern is expected to see in a
  certain interval.
• But there is no specific order for the results
  within each time slice, as long as they all
  finish before the end of the time slice.
• This pattern is commonly used to represent
  requirements that describe repeated or cyclical
  behaviors, or behaviors that fit into a fixed
  time sequence pattern.

37

• Example
• When a brady cardiac cycle occurs, the firmware
  shall perform the steps specified in the CARDIAC
  CYCLE PROCESSING table

38
Requirement and Verification Patterns

• Key-Event Driven Time-Sliced Verification Pattern
  Class diagram

39
Requirement and Verification Patterns

• Key-Event Driven Time-Sliced Verification Pattern
  State Machine

40
Requirement and Verification Patterns

• Command-Response Requirement Pattern
• Very common in embedded systems
• A command could be aborted.
• A response always follows the command.
• A command could be illegal or non-applicable.
• The response is different depending on both the
  command and the timing of the aborting action.

41
Requirement and Verification Patterns

• Command-Response Verification Pattern Class
  Diagram

42
Requirement and Verification Patterns

• Command-Response Verification Pattern
  Verification State Machine

43
Requirement and Verification Patterns

• Look-back Requirement pattern
• To verify causes when results are found
• To verify certain conditions were met before a
  key event occurs.
• The verifier has to look back and check the
  conditions involved.
• Typical statement of requirement
• Prior to event K, condition C1, C2, Cn must be
  met.

44
Requirement and Verification Patterns

• Look-back Verification Pattern Class Diagram

45
Requirement and Verification Patterns

• Look-back Verification Pattern State Machine

46
Requirement and Verification Patterns

• Verifying requirements using patterns

47
Requirement and Verification Patterns

• Pattern and the framework

48
Framework Evaluation

• Framework Effectiveness
• Effectiveness in Test development
• Effectiveness in Simplifying Debugging
• Easy to maintain
• Flatter learning curve
• Effectiveness in system instantiation

49
Framework Evaluation

• Framework Effectives

50
Framework Evaluation

• Framework Stability
• Architectural Stability of the Framework
• Domain Coverage

51
Framework Evaluation

• Handling Changes
• Adding Requirement
• Classify the requirement, or create a new
  requirement pattern
• Represent the requirement using appropriate
  pattern, or implement a special verifier for
  the requirement.
• Removing Requirement
• Delete it from the pattern verifier it belongs,
  or
• Remove its specialized verifier from the test
  list.
• Change in Requirement
• Timing change - Change the timing parameters in
  the requirement representation
• Sequence change - Change the sequence of the
  requirement representation, or Reclassify the
  changed requirement
• Action change - Update the requirement
  representation using events that represent the
  new actions. Normally the requirement item stays
  with the same requirement pattern.

52
Framework Evaluation

• Framework Applicability
• Was initially targeted on real-time embedded
  systems.
• Can be applied to other application domains such
  as business application or GUI application
  testing, as long as the testing is event-driven.
• Can be applied in a simulated testing environment
  instead of real hardware configuration.
• Not suitable for the testing of the continuous
  control systems or scientific calculation
  systems.
• Not suitable for testing of Internet applications
  or client/server applications as contents, not
  reactions, often make differences for these
  applications.

53
Framework Evaluation Case Study Results
54
Summary

• Framework Summary
• Rationale Requirement, test, verifier
• Architecture Layered hardware abstraction,
  support, application
• Mechanism Event assembling and dispatching
• Components Requirement, scenario, verifier,
  event assembler etc.
• Evaluation
• Effectiveness
• Stability
• Applicability

55
Summary

• Main Contributions
• Requirement centered, pattern-based requirement
  verification framework for real-time embedded
  systems.
• Different from any other existing test approaches
  and test frameworks, our verification framework
  makes the requirement an essential component.
• This basic idea is further developed and enhanced
  by requirement patterns and verification patterns.

56
Questions and Answers