Extended DbC Approach to Specification and Conformance Testing of Distributed Software

1
Extended DbC Approach to Specification and
Conformance Testing of Distributed Software

• Victor Kuliamin kuliamin_at_ispras.ru
• Nickolay Pakoulin
• Alexander Petrenko
• ISP RAS, Moscow

2
Outline

• Introduction
• Some Theory
• Application 1 Standards Clean-up
• Application 2 Conformance Testing

3
Distributed Software

• Is urgent for economic and social development
• Gives people more services
• Is very complex
• Unsuitable for human mind, which operates mostly
  sequentially
• Involves lots of possibly heterogeneous
  components
• Errors are getting much more subtle and hard to
  observe
• Need to be checked and tested thoroughly

very complex
tested
4
Outline

• Introduction
• Some Theory
• Application 1 Standards Clean-up
• Application 2 Conformance Testing

5
Testing Fundamentals

• How is testing performed?
• We act upon the system under test
• We watch its reaction
• We check whether that reaction is what should be
• We repeat this until all the reasonable
  situations are exhausted

6
Testing of Distributed Software

• Organize distributed action
• harder then in non-distributed case, but
  possible
• Watch distributed reaction
• harder then in non-distributed case, but
  possible
• Check whether it is expected
• how we know what is expected?
• Ensure reasonable exhaustiveness
• what does this mean at all?

Main sources of complexity
7
How to Describe Requirements?

• The description should be
• Sufficiently expressive
• As clear as possible
• Scalable to rather complex systems
• preferably, component-wise
• Suitable to distributed systems
• include several sides
• How do people describe mutual obligations in
  complex cases including several sides?
• By means of contracts!

8
Contract Specifications

• Pre- and postconditions (Hoare, 1969)
• means for reasoning about program behavior
• augment code elements to enforce rigorous
  development
• Design by Contract (Meyer, 1992)
• software is considered as a set of components
  interacting trough their interfaces
• pre- and postconditions are defined for
  interface operations
• constraints on data integrity are stated in
  invariants
• together they form software contract between
  a component and its environment

9
Design by Contract Pro and Contra

• Advantages
• Component-wise consideration of software
• scalability
• support for reuse, incremental and parallel
  development
• Possibility to use for different aspects and on
  different abstraction levels
• Drawbacks
• Insufficient for distributed systems
• does not consider concurrency and asynchronous
  interaction
• does not consider callbacks

PU 2
PU 2
PL 1
PU 1
PU 1
10
Interaction in Distributed Systems
11
Event Contracts
Obligations of the environment
Obligations of the system
Precondition
says in what states such an event is possible
Pre-state
Input event
Pre-state
Output event
System
System
Environment
Environment
Post-state
Post-state
Postcondition
says what post-states can follow such an event in
such a pre-state
Obligations of the system
Obligations of the system
12
Concurrency

13
Event Contracts Implementation

• Software Contracts
• Pre- and postconditions of events, invariants
• Possibility to specify constraints in form of
  predicates on the results, not the algorithm
• Component-wise consideration of software
• Asynchronous events and callbacks included
• Specifications in extensions of widely-used
  programming languages (C, Java, C)
• Simplifications where possible
• Joint description of call and return if
  intermediate states do not matter

14
Barrier Example
init(3)
Threads
wait()
wait()
wait()
Height 3
15
Barrier Example in Event Contracts

• public specification class Barrier
• int awaitedThreads 0
• int waitingThreads 0
• invariant CountersAreNonnegative() return
  awaitedThreads gt 0 waitingThreads gt 0
• public specification void Init(int n)
• post
• if(n lt 0 waitingThreads gt 0)
  branch NoChanges
• return awaitedThreads pre
  awaitedThreads waitingThreads pre
  waitingThreads
• else
  branch NewHeightSet
• return awaitedThreads n
  waitingThreads 0
• public specification void Wait()
• post
• if(awaitedThreads lt 1)
  branch Immediate
• return awaitedThreads 0
  waitingThreads pre waitingThreads
• else deferred
  branch Waiting

16
Testing Adequacy

• How can we define reasonable exhaustiveness?
• Possibilities
• Fault-based approaches
• Percent of faults detected by tests to all faults
  reported
• Mutants percent of mutants detected
• Source code coverage-based approaches
• Requirements coverage-based approaches
• UniTesK method requirements-based
• Try first to cover structure of postconditions

17
Testing Goals

• post
• if ( a lt 0 c.isActive() )
• else if( a gt 3 !b.closed() )
• else

a gt 0 !c.isActive() a lt 3 a gt 0
!c.isActive() b.closed()
18
Outline

• Introduction
• Some Theory
• Application 1 Standards Clean-up
• Application 2 Conformance Testing

19
Case Studies I

• IPv6 2001
• Parts considered
• Sending datagrams / receiving packets
• Neighbor discovery
• Multicast Listener Discovery
• UPD over IPv6
• Results
• Minor defects found in RFC 2460
• Conformance test suite developed (further)
• IPMP-2 2004
• Results
• Several contradictions between standard parts
  found
• Interoperability flaws detected in Mutual
  Authentication protocol
• 2 accepted submissions on elaboration of the
  standard

20
Approach Used
Testing goals
Formalization
Test Suites
Contract Specifications
Standards
Conformance testing Certification
Interoperability testing Early debugging Requireme
nts traceability
Inconsistencies, ambiguities, interoperability
flaws
Software
21
Outline

• Introduction
• Some Theory
• Application 1 Standards Clean-up
• Application 2 Conformance Testing

22
Testing Scheme
Testing Model
Behavior Model
System under Test
Coverage Model
Single Event Oracles
On-the-fly Test Sequence Generation
Composite Event Oracle
23
Checking Composite Event
Interleaving semantics
?
24
UniTesK Test Construction Tools

• C / Visual Studio 6.0, gcc 2002
• Java / NetBeans 2002
• C / NetBeans MS Visual Studio
  2003specifications in Java extension
• Specialized tool for compiler testing 2003and
  complex data generation
• C / Visual Studio .NET 7.1 2003
• Java / Eclipse 2005

25
Tool Demonstration
26
Case Studies

• ISP RAS Nortel Networks 1994-1997functional
  test suite development for Switch Operating
  System kernel
• IPv6 implementations 2001-2003
• Microsoft Research
• Mobile IPv6 (in Windows CE 4.1)
• Oktet
• Intel compiler optimization units 2001-2003
• IPSec 2004-
• Pilot projects
• Components of TinyOS 2003
• Web-based banking client management system
  (Luxoft) 2004
• Components of billing system (Vympelkom) 2005
• Data Integration Framework (Vympelkom) 2005
• Components of Java 1.4 SE (parts of JDK, JVMTI)
  2005
• Real-Time POSIX-based Operating System 2005
• http//www.unitesk.com

27
References

• V. Kuliamin, A. Petrenko, I. Bourdonov, and A.
  Kossatchev. UniTesK Test Suite Architecture.
  Proc. of FME 2002. LNCS 2391, pp. 77-88,
  Springer-Verlag, 2002.
• V. Kuliamin, A. Petrenko, N. Pakoulin, I.
  Bourdonov, and A. Kossatchev. Integration of
  Functional and Timed Testing of Real-time and
  Concurrent Systems. Proc. of PSI 2003. LNCS 2890,
  pp. 450-461, Springer-Verlag, 2003.
• V. Kuliamin, A. Petrenko. Applying Model Based
  Testing in Different Contexts. Proceedings of
  seminar on Perspectives of Model Based Testing,
  Dagstuhl, Germany, September 2004.
• A. Kossatchev, A. Petrenko, S. Zelenov, S.
  Zelenova. Using Model-Based Approach for
  Automated Testing of Optimizing Compilers. Proc.
  Intl. Workshop on Program Undestanding,
  Gorno-Altaisk, 2003.
• V. Kuliamin, A. Petrenko, A. Kossatchev, and I.
  Burdonov. The UniTesK Approach to Designing Test
  Suites. Programming and Computer Software, Vol.
  29, No. 6 , 2003, pp. 310-322. (Translation from
  Russian)
• S. Zelenov, S. Zelenova, A. Kossatchev, A.
  Petrenko. Test Generation for Compilers and Other
  Formal Text Processors. Programming and Computer
  Software, Vol. 29, No. 2 , 2003, pp. 104-111.
  (Translation from Russian)
• V. Kuliamin, N. Pakoulin, A. Petrenko. Practical
  Approach to Specification and Conformance Testing
  of Distributed Network Applications. Proc. of
  2-nd International Service Availability
  Symposium, April 2005, to be published.

28
Contacts

• RedVerst group web site http//www.ispras.ru/grou
  ps/rv/rv.html
• UniTesK projects web site http//www.unitesk.com
• Group leaderAlexander Petrenkopetrenko_at_ispras.ru
  

29
Thank You!