Interoperability Testing for Communicating Systems

1
Interoperability Testing for Communicating Systems

• Tarot
• Summer School 2005, June 27-July 1, Paris
• Ismail BERRADA
• Richard CASTANET
• Patrick FELIX
• LaBRI, France

2
Contents

• Introduction and Motivations
• Conformance Testing Theory
• From Conformance Testing to Interoperability
  Testing
• Experience with CSMA/CD
• Test Generation with TGSE tool
• Conclusions
• References

3
Introduction and Motivations

• Conformance Testing
• a single protocol entity is tested. Objective
  to determine whether an implementation conforms
  to the specification.
• Interoperability Testing
• two or more protocol entities are tested.
  Objective to determine whether these
  implementations can interact together as expected
  by the specification.

Conformance
Conformance
IP Spec
Vendor B
Vendor A
Router_B
Router_A
Interoperability
4
Introduction and Motivations

• Interoperability testing
• final goal of any product developed in
  communicating systems (example router)
• Large application field distributed systems
  (reactive systems, embedded components,
  protocols)
• Some implementations may be considered conformant
  to their specification, but may not interoperate.
• Why implementations can not communicate or behave
  together as expected ?
• Few frameworks for interoperability testing
  (definitions, methodologies, algorithms)

5
Part 1 Conformance Testing Theory
6
Conformance testing

• Goal to be sure that a single implementation
  (IUT) does what the standard specifies

Test cases
Verdicts Pass Fail Inconclusive
PCO
Specification
Tester
IUT

• Black box architecture
• PCO (Points of control and observation)
  Restricted control and observation with certain
  interfaces

7
Industrial practice

• Manual design of the test cases from informal
  specifications
• long and repetitive process
• important cost
• uninsured correctness of the test cases
• delicate maintenance of the test cases
• ?The automatic generation of test cases from
  formal specifications can be attracting.

8
Conformance testing theory
Properties

• Correctness of test cases

Derivation
Conforms to ?
Verdicts Pass Fail Inconclusive
Specification
Test cases
IUT
Correctness of implementation
IUT
9
Input-Output Labeled Transtion Systems

• IOLTS M(QM,?M,?M,q0M), where
• QM a finite set of states with q0M as the initial
  state,
• ?M ?MI U ?Mo a set of observable events
• ?MI a finite set of inputs (?a),
• ?Mo a finite set of outputs (!a),
• ?M ? QM x (?M U ?) x QM the transition relation
  
• ? an internal action (not in ?M)

q ?? q qq or q ?? q q ?a q
?q1,q2 ?QM q ?? q1 ?a q2 ?? q q ?a.? q
?q1?QM q ?a q1 ?? q q after ? q?QM
q??q OutM(q) a??Mo ?q? QM, q?Ma q
Traces(q) ??(?M) ?q? QM, q??q G(q)
a??M ?q?QM, q?Ma q MaxTraces(M)
??(?M) G(q0 after ?)Ø
Notation MIOLTS q,q?QM a??M ?,??(?M)
10
Test Assumptions

• An IOLTS M is said to be input-complete
  (input-enable) if M accepts all inputs in each
  state. Formally,
• ? q?QM,? a ??Mo,,?q?QM, q ?a q
• Specification model IOLTS S (Qs, ?s,?s,q0s)
• Implementation model IOLTS IUT (QIUT,
  ?IUT,?IUT,q0IUT) with ?sI ? ?IUTI and ?so ?
  ?IUTo and input-complete.

11
Visible behaviors traces quiescence

• A tester observes traces of the IUT, but also
  quiescence.
• Quiescence absence of visible behaviors
• Types of quiescence deadlock (6,7), livelock
  (5,8) and output-lock (1)

12
Characterization of traces of an IOLTS

• Two sequences with the same trace cannot be
  distinguished and their suffix must be considered
  as possible evolution of the system M.
• gt We consider the deterministic IOLTS det(M)
  which has the same traces as M.

• The determinization does not preserve quiescence
  !
• Representation of quiescence in M suspension
  automaton ?(M)

13
Suspension automaton TRE96

• The absence of visible behaviors is modeled by an
  output event !d
• ?(M) M loops of !d on each quiescent states
• Suspended traces of M STraces(M) Traces(?(M))
• det(?(M)) characterizes the visible behaviors of
  M.

?(M)
14
Conformance Relation

• IUT ioco S ?? ? STraces(S) out(?(IUT) after ?)?
  out(?(S) after ?)
• For any observable behavior ? of S, any possible
  observable output of IUT after ? is also a
  possible observable output of S after ?.

?(IUT)
IUT1 ioco S
(IUT2 ioco S)
(IUT3 ioco S)
15
Conformance Relation

• IUT ioconf S ?? ? Traces(S) out(IUT after ?)?
  out(S after ?)
• IUT ioco S ?? ? STraces(S) out(?(IUT) after ?)?
  out(?(S) after ?)

IUT3 ioconf S
(IUT3 ioco S)
?(IUT3 )
16
Test cases

• Why?
• To check the conformance relation
• How?
• Test case describes the interactions between
  Tester and Implementation
• A tester
• executes a test case,
• observes the reactions/behaviors of the IUT,
• compares them with the awaited behaviors of the
  test case and
• deduces the corresponding verdict
• Formally, a Test case TC is an IOLTS with the
  notion of verdicts
• TC (QTC,?TC,?TC,q0TC) with ?TCo ? ?sI et ?TCI ?
  ?IUTo U ?d
• QPass ? QTC and QFail ? QTC and QInconc ? QTC
  (verdict states).
• !d visible quiescence / ?d expiration of the
  timer

17
Test Cases an example

• Assumptions on test cases
• controllable no choice between an output and an
  other action
• Input-complete in all states where an input is
  possible (?otherwise)
• Verdict states are only reachable after inputs

18
Test cases executions

• The execution of a test case TC against an
  implementation IUT is modeled by the parallel
  composition TC//?(IUT) with a synchronization on
  the common visible actions.
• Example Execution of TC against IUT1

?(IUT1)
19
Test cases executions
Example Execution of the test case TC against
IUT2
?(IUT2)
TC // ?(IUT2)
20
Test case execution and verdicts

• TC//?(IUT) gives all possible executions of TC
• TC//?(IUT) may only block in a state of
• (QPass U QFail U QInc)xQIUT
• Execution a maximal trace of TC // ?(IUT).
• Verdict state (of TC) reached at the end of an
  execution.
• Formally, for ?? MaxTraces(TC//?(IUT)) a maximal
  trace of TC // IUT
• verdict(?) fail (resp. pass, inconc)
• (TC after ?) ? QFail (resp Pass, Inconc)

• An execution

21
Test cases and verdicts

• A test case may produce different verdicts
• Example TC1 may produce Inc and Pass
• Formally, a possible rejection of the
  implementation IUT by the test case TC is defined
  by
• TC may reject IUT
• ? ? ? MaxTraces(TC// ?(IUT)), verdict(?)fail
• (may pass and may inconc are defined in the
  same way)
• Example (TC1 may pass IUT1) but (TC1 may fail
  IUT1)

22
Required properties of test cases

• Soundness a test case should not reject a
  conformant IUT.
• A test suite TS is sound for S and ioco if
• ?IUT,?TC?TS,TC may reject IUT ? (IUT ioco S)
• Exhaustively a non-conformant would be rejected
  by a test case.
• A test suite TS is exhaustive for S and ioco if
• ?IUT, (IUT ioco S) ? ?TC?TS,TC may reject IUT

23
Automatic derivation of test suite

• Problem
• Given a specification S, a conformance relation
  R (like ioco), and some properties P, how to
  derive test cases for an eventual implementation
  of S satisfying P.
• ? TSGenTest(S,R,P).
• Properties can be soundness, exhaustively,
  coverage criterion?...

24
Test Suite Derivation

• Test Theory
• Model to specify S, implementation I, test case
  TC and test suite TS
• Formalization of the test correctness with a
  relation (I ioco S for the conformance testing),
  test execution and its verdicts
  (verdicts(exec(I,TC)), I pass TS)
• Definition of required properties soundness,
  exhaustively.
• To define automatic method (algorithm) of test
  suite derivation with the good properties
  TCgen_test(S).
• Other properties coverage
• To define test selection methods

25
An example of Derivation

• Specification S
• Conformance Relation ioco
• Property Test purpose TP

• Reachability of Det(?(S))
• suspended trace of S accepted by TP

!z
?a
!y
!x
Det(?(S))
TP
26
Part 2 From Conformance Testing to
Interoperability Testing
27
Introduction

• Interoperability definitions
• ISO/IEC The ability of two or more systems to
  exchange information and mutually use the
  information that has been exchanged.
• ETSI ETR The ability of a distributed system to
  interchange PDUs via the communicating platform
• Interoperability testing
• should check only external messages that lead to
  inter-component communication (Kim 03).
• should check an interoperability relation (Viho
  01).
• Interoperability architectures
• The degree of observability and controllability,
  and the configuration of the communicating
  systems have a great influence on the
  interoperability architecture to adopt.

28
How to deal with interoperability?

• Not covered by conformance testing
• Interoperability Testing Architectures (Visible
  behaviors,..)
• Model of communication
• Context abstraction, model
• Definition of an interoperability relation R
• Automatic test suite derivation with the good
  properties TCGenTest(S1,..,Si,R,P,) .

• Covered by conformance testing
• Modeling S1, S2, I1, I2 and TC modeled by IOLTS
  
• Definitions verdicts(exec(I,TC)), test suite TS,
  I pass TS
• Properties soundness, exhaustively, coverage.

29
Interoperability Testing Architectures

• Black box / grey box architecture
• Tester observes the IUT through POs (Points of
  observation)
• Tester controls and observes the IUT through PCOs
  (Points of Control and Observation)
• Many testers without procedure coordination /
  only one tester
• Examples of interoperability testing
  Architectures (grey box)

Test System (TS)
Tester


System Under Test (SUT)
PCO

PCO

IUT 2

IUT1
PO

PO

Context

30
Communication model
The considered communications include tested
entities and entities of the context (the context
can be empty)

• Asynchronous

• Synchronous

channels are perfect and bounded FIFO queues
31
Interoperability relations

• Interoperability relation conditions to verify
  if two or more implementations interact
  correctly.
• Such relations depend on
• The testing architecture and the accessibility of
  interfaces
• Communication model
• Quiescence management
• Actually, some interoperability relations have
  been defined BTV01,DV05,BCF04
• BVT01 defines nine relations (Unilateral,
  Bilateral, Global Total Relations) with a
  synchronous communication model.
• BCF04 proposes a parameterized rewriting of
  BVT01 relations
• DV05 extends BVT01 relations to deal with
  quiescence

32
Interoperability relations
Viho et al.01

• Unilateral Total Interoperability Relation
• Architecture
• only access to IUT1
• upper and lower interfaces
• Communication Model
• Synchronous
• Relation

Tester 1



PCO

IUT 2

IUT1
PO


R1(I1,I2) ??1?Traces(S1), ???Traces(S1//S2)
?1?/?S1 ? out?I1(I1//I2,?) ?
out(S1,?1) During the interaction between I1
and I2, at least I1 behaves as expected
according to S1
33
Interoperability relations
Viho et al.01

• Bilateral Total Interoperability Relation
• Architecture
• access to IUT1 and IUT2
• upper and lower interfaces
• Communication Model
• Synchronous
• Relation
• R2(I1,I2) R1(I1,I2) and R1(I2,I1)
• I1 (resp. I2) behaves as expected according to
  its specification S1 (resp. S2)

Tester 1



PCO

IUT 2

IUT1
PO


Tester 2


PCO


IUT 2

IUT1

PO

34
Interoperability relations
Viho et al.01

• Global Total Interoperability Relation
• Architecture
• access to IUT1 and IUT2
• upper and lower interfaces
• Communication Model
• Synchronous
• Relation

coordination
Tester 1

Tester 2

PCO

PCO

IUT 2

IUT1
PO

PO

R3(I1,I2) ???Traces(S1 //S2) out(I1//I2,?) ?
out(S1 //S2,?) The implementation I1//I2
behaves as expected according to its
specification S1 //S2 R3(I1,I2) I1//I2
conforms (ioconf) to S1//S2
35
Part 3 Illustration with CSMA/CD
36
CSMA/CD

• Carrier Sense Multiple Access / Collision
  Detection (Ethernet)
• A station
• listens to the bus
• Sends data if the bus seems to be free
• Detects collision during the emission
• If collision, retries an emission after a random
  delay

37
A variant of CSMA/CD

• Variants of CSMA/CD many senders (here two
  senders) but only one receiver CSMA/CD
  Sender_1//Sender_2//Bus//Receiver
• All messages are sent (through the Bus) to the
  Receiver
• Actions of the environment
• !Mess (to sender_1 or sender_2),
• ?Mess (from Receiver)

38
Sender

• Clock x
• Constants - Lambda emission time of a message
  808
• - Sig propagation delay on the bus 26

39
Bus

• Clock y
• Constants - Lambda emission time of a message
  808
• - Sig propagation delay on the bus 26

40
Receiver
41
CSMA/CD system
Synchronization on common actions (rendez-vous
and broadcast)

42
Untimed CSMA/CD system
Sender

Init
Receiver

?Busy

?

?Mess
!End

!Mess
?End
Finish

Finish
Wait
Transmit


Wait
Transmit
!Begin

?Data
?Busy


?CD
?Busy

Retry


?CD
!Begin


?CD

!End
Bus

!Begin

?Data


Idle
Active1
Active2
!CD
!Begin


!Busy
!Busy
Collision

43
Interoperability of CSMA/CD

• Testing Architecture
• Grey box (PCOs and POs)
• Communication model
• Synchronous
• Relation
• Global Total Interoperability Relation (like
  ioconf)

44
Test case derivation
Test Purpose
Reachability graph of CSMA/CD
S1//S2//Bus//Receiver

?Mess_1
I,I,I,W

SiSender_i RReceiver BBus
S2?Mess
S1?Mess


?Mess_2
W,I,I,W

W,I,I,W

S1//BBegin


S2?Mess

!Mess_1
T,I,A1,W

W,W,I,W

B//RData

S1//BBegin



!Mess_2
T,I,A2,T



S1//B//REnd

S1//BBegin
B//RData



F,I,I,T



S2?Mess
R!Mess
Abstract Test Case
45
Part 4 Test Generation AND TGSE tool
46
Simple Light Controller
Press
Off
Light
Bright
Press
Press
Press
WANT if press is issued twice quickly then
the light will get brighter otherwise the light
is turned off.
47
Simple Light Controller
Press
Off
Light
Bright
x0
Press
Press
xlt3
xgt3
Press
Solution Add a real-valued clock x
Adding continuous variables to state machines
48
Another Simple example
Output b at most 10 time units after receiving
input a.
49
Overview

• Timed Automata Model
• Timed Conformance/Interoperability relations
• Preliminaries
• Timed Input/Output relation
• Test Generation
• Constraint Resolution techniques
• Symbolic Reachability techniques
• TGSE and Case studies

50
Timed Automata
Alur-Dill, 94
Clocks x, y
Guard Boolean combination of comparisons
with Integer/rational bounds
n
Reset Action performed on clocks
Action used for synchronization
xlt5 ygt3
State ( location , xv , yu ) where v,u are
in R
a
x 0
m
51
Timed Automata Syntax
Alur-Dill, 94

• A finite set L of locations
• An initial location l0
• A finite set S of labels (alphabet)
• A finite set C of clocks
• A finite set E of edges. Each edge has
• source location l, target location l
• label a in S (e labels also allowed)
• guard Z (a clock constraint over C)
• a subset r of clocks to be reset

52
Timed Automata Semantics

• A timed automaton A defines an infinite-state
  transition system S(A)
• States Q a state q is a pair (l,v), where l is
  a location, and v is a clock vector, mapping
  clocks in C to R,
• (l0,v) is initial state with v(x)0 for all x of
  C
• Timed transitions for each nonnegative real
  number d, (l,v)-d-gt(l,vd).
• Discrete transitions (l,v)-a-gt(l,v) if there
  is an edge (l,a,Z,r,l) such that v satisfies g
  and vvr0

53
Timed Automata Semantics (contd)
54
Timed Automata

• Automata clock variables.
• Model finite state (control) systems with timing
  constraints.
• Rich theory.
• Practical tools.
• Emerging applications.

55
Overview

• Timed Automata Model
• Timed Conformance/Interoperability relation
• Preliminaries
• Timed Input/Output relation
• Test Generation
• Symbolique Reachability techniques
• Constraint Resolution techniques
• TGSE and Case studies

56
Preliminaries

• Let A a TA and S(A) its semantics
• S e S ? e
• A timed sequence ? over Se is a
  (a0,t0).(a1,t1)(ak,tk) with ti in R, and
  t0ltt1ltlttk (monotonicity of time). time(?)
  tk

• Timed sequence (?a,1.1).(!b,2.3)

57
Preliminaries

• Let A a TA and S(A) its semantics
• S e S ? e
• A timed sequence ? over Se is a
  (a0,t0).(a1,t1)(ak,tk) with ti in R, and
  t0ltt1ltlttk (monotonicity of time). time(?)
  tk
• For a state q of S(A), q (a,t)-gtq will denote
• q t-gtq-a-gtq or
• q a-gtq if t0 or
• q t-gtq if a e

58
Preliminaries (contd)

• Let A a TA and S(A) its semantics
• Run(A) timed sequences ? over Se such that A
  has run over ?

• Run over (?a.1.1).(!b.2.3)

time
0
59
Preliminaries (contd)

• Let A a TA and S(A) its semantics
• Run(A) timed sequences ? over Se such that A
  has run over ?

• Run over (?a.1.1).(!b.2.3)

60
Preliminaries (contd)

• Let A a TA and S(A) its semantics
• Run(A) timed sequences ? over Se such that A
  has run over ?

• Run over (?a.1.1).(!b.2.3)

time
0
61
Preliminaries (contd)

• Let A a TA and S(A) its semantics
• Run(A) timed sequences ? over Se such that A
  has run over ?

• Run over (?a.1.1).(!b.2.3)

time
0
62
Preliminaries (contd)

• Let A a TA and S(A) its semantics
• Run(A) timed sequences ? over Se such that A
  has run over ?

• Run over (?a.1.1).(!b.2.3)

time
0
63
Preliminaries (contd)

• Let A a TA and S(A) its semantics
• Run(A) timed sequences ? over Se such that A
  has run over ?

• Run over (?a.1.1).(!b.2.3)

time
0
64
Preliminaries (contd)

• Let A a TA and S(A) its semantics
• Run(A) timed sequences ? over Se such that A
  has run over ?

• Run over (?a.1.1).(!b.2.3)

1.2
time
0
65
Preliminaries (contd)

• Let A a TA and S(A) its semantics
• Run(A) timed sequences ? over Se such that A
  has run over ?

• Run over (?a.1.1).(!b.2.3)

1.2
Run of A
time
0
66
Preliminaries (contd)

• Let A a TA and S(A) its semantics
• Run(A) timed sequences ? over Se such that A
  has run over ?
• TTrace (A) ? ??? Run(A), ?S ?

67
Timed Input/Output relation

• A timed extension of ioco
• timed input-output conformance relation
  (TIOCO).
• Informally, I TIOCO S if
• For any observable behavior ? of S, any possible
  observable output of I after ? is also a possible
  observable output of S after ?.

68
Timed Input/Output relation (contd)

• Formally

I TIOCO S (I implementation, S
specification) iff ?? ? TTraces(S). out(I, ?) ?
out(S, ?)
69
Timed Input/Output relation (contd)

• where

out(A, ?) (a,t) ? (So ? e) R ??? Run(A).
time(?) time(?) ? ?S ? ? q0-gt? -gtq-gt
(a,t)-gtq

• We recall that q (a,t)-gtq is
• q t-gtq-a-gtq or
• q a-gtq if t0 or
• q t-gtq if a e

70
Examples
Output b at most 5 time units after receiving
input a.
71
Examples (contd)
Output b at most 5 time units after receiving
input a.
72
Examples (contd)
Output b at most 5 time units after receiving
input a.
OK!
73
Examples (contd)
Output b at most 5 time units after receiving
input a.
OK!
74
Examples (contd)
Output b at most 5 time units after receiving
input a.
OK!
OK!
75
Examples (contd)
Output b at most 5 time units after receiving
input a.
76
Examples (contd)
Output b at most 5 time units after receiving
input a.
NOT OK!
77
Examples (contd)
Output b at most 5 time units after receiving
input a.
NOT OK!
78
Examples (contd)
Output b at most 5 time units after receiving
input a.
NOT OK!
OK!
79
Overview

• Timed Automata Model
• Timed Conformance/Interoperability relation
• Preliminaries
• Timed Input/Output relation
• Test Generation
• Constraint Resolution techniques
• Symbolic Reachability techniques
• TGSE and Case studies

80
Timed testers
Tripakis Moez, 05

• Analog-clock testers
• Can measure real-time precisely
• Difficult (impossible) to implement for real-time
  IUTs
• Good (flexible) for discrete-time IUTs with
  unknown time step
• Digital-clock testers
• Can measure ticks of a periodic clock/counter
• Implementable for any SUT
• Conservative (may say PASS when its FAIL)

81
Timed tests
Tripakis Moez, 05

• Analog-clock tests
• They can observe real-time precisely
• Digital-clock tests
• They only have access to a periodic clock

1
2
3
82
Timed tests (contd)

• Digital-clock tests does not mean the
  discretization of time
• The specification is still dense-time
• The capabilities of the tester are discrete-time
• Many dense-time traces will look the same to the
  digital observer

83
Test Cases for Digital-clock

• finite trees

Models the tick of the testers clock
?a
?c
tick
Inc
?a
!b


fail
pass
84
Test Cases for Digital-clock (contd)

• finite trees build the test statically

time
0
85
Test Case Derivation

• Analog-clock tests
• Cannot be represented statically as finite trees
  infinite number of possible inputs delays
• Solution on-the-fly testing compute the test
  while you are testing !
• Digital-clock tests
• Can be generated statically.

86
Test Cases for Analog-clock

• Cannot be represented as finite trees

i

?a
?c
0.1
0.2
Inc
!b
Infinite number of possible delays

fail
pass
87
Test Cases fot Analog-clock (contd)

• Solution build the test on-the-fly

time
0
88
Test Derivation and Reachability Analysis
Exercise 1
Can this automaton A emit the event b? the event
c?
89
Test Derivation and Reachability Analysis (contd)
Exercise 2
The same question as in exercise 1
90
Test Derivation and Reachability Analysis (contd)
Real-Time Test Generation can be reduced to the
Reachability Analysis
91
Constraint Resolution
Let vi, i in 1,2, the instant of firing
transition ti according to a global clock h
(without reset)
Then, (?a,d1).(!b,d2) is a run of A iff d1 and d2
satisfy the system S
92
Constraint Resolution (contd)
Thus, to a linear TA A, we can associate a
system of two-variables by inequalities S The
final state of A is reachable iff S has a
solution
93
Constraint Resolution (contd)
A
0 ? v1 ? v2 v1 ? 2 v2 3 v2 v1 ? 2
94
Constraint Resolution (contd)
A
0 ? v1 ? v2 v1 ? 2 v2 3 v2 v1 ? 2
0 will be considered as a variable v0 equals to
zero
95
Constraint Resolution (contd)
A
0 ? v1 ? v2 v1 ? 2 v2 3 v2 v1 ? 2
0 will be considered as a variable v0 equals to
zero
96
Constraint Resolution (contd)
A
0 ? v1 ? v2 v1 ? 2 v2 3 v2 v1 ? 2
0 will be considered as a variable v0 equals to
zero
97
Constraint Resolution (contd)
A
0 ? v1 ? v2 v1 ? 2 v2 3 v2 v1 ? 2
0 will be considered as a variable v0 equals to
zero
98
Constraint Resolution (Constraint Graph)
v1
v2
v0
99
Constraint Resolution (Constraint Graph)
v1
v2
0
v0
100
Constraint Resolution (Constraint Graph)
v1
v2
0
2
v0
101
Constraint Resolution (Constraint Graph)
v1
v2
0
-3
8
2
v0
102
Constraint Resolution (Constraint Graph)
2
v1
v2
8
0
-3
8
2
v0
Constraint Graph G
103
Constraint Resolution (Constraint Graph)

• S has a solution iff its constraint graph does
  not contain negative cycles

104
Constraint Resolution (minimal Graph)
105
Constraint Resolution (minimal Graph)
For node v0, (dicost(v0?vi))i e1,n and
(di-cost(vi?v0))i e1,n are solutions of S,
and thus runs of A With cost(vi?vk) is the cost
of the edge vi?vk (in the minimal graph)
106
Constraint Resolution (minimal Graph)
2
2
v1
v2
v1
v2
8
-1
0
-3
-1
-3
8
2
2
4
v0
v0
Minimal graph Gm
Graph G

• For node v0,
• d12, d2 4 is a solution of S
• d11, d2 3 is also a solution of S

107
Constraint Resolution (minimal Graph)
A

• For node v0,
• d12, d2 4 is a solution of S
• d31, d4 3 is also a solution of S
• Thus,
• (?a,d1).(!b,d2) is a run of A and
• (?a,d3).(!b,d4) is a run of A

108
Constraint Resolution (Transformation)
2
2
v1
v2
v1
v2
-1
-2
-1
-3
-1
-3
2
1
4
4
v0
v0
Minimal graph Gm
G1 Transformation for v1

• Graph Gi for node vi,
• For every node vk,
• The cost (in Gm) of an incoming edge (vk? vj)
  will be replaced (in Gi) by the opposite cost (in
  Gi) of the outgoing edge (vi? vk)

109
Constraint Resolution (Transformation)
2
2
v1
v2
v1
v2
-1
-2
-1
-3
-1
-3
2
1
4
4
v0
v0
Minimal graph Gm
G1 Transformation for v1
If we compute again the minimal graph of Gi, we
obtain two new runs of the timed automaton A
110
Digital-clock Tests Derivation
In this way, we can derive between 0 and 2(n1)
runs of A
These runs are then used as digital-clock tests
We can show that, if these runs (of A ) are also
runs of an implementation I of A, then every runs
of A is also a run of I (trace inclusion)
111
Symbolic Reachability
Tripakis Moez, 05

• The tester keeps a set of possible states of the
  IUT, according to the specification.
• Updates this set with each new observation.
• Gives inputs to SUT from time to time.
• Announces FAIL if ever set becomes empty.
• Announces PASS when tired of testing

112
Overview

• Timed Automata Model
• Timed Conformance/Interoperability relation
• Preliminaries
• Timed Input/Output relation
• Test Generation
• Symbolique Reachability techniques
• Constraint Resolution techniques
• TGSE and Case studies

113
TGSE Tool
114
Architecture
115
CSMA/CD
Clocks x, y Parameters lambda, Sig

Top
SENDER
116
Experimentation CSMA-CD
117
Conclusions and Discussion

• A precise definition of interoperability is
  somewhat elusive, but
• functionally the meaning seems to be clear!
• If (1) specifications are formals and proved and
  (2) conformance testing is exhaustive, did we
  really need interoperability testing?
• Generally, errors are discovered during
  conformance testing, and errors discovered during
  interoperability testing are often due to the
  context of implementations (CRIL TECHNOLOGY).
• Test Case Derivation
• Measure of the coverage
• Test case minimization
• A relevant case study?

118
References (1)
Real-time systems 1 Moez Krichen, Stavros
Tripakis State Identification Problems for Timed
Automata. TestCom 2005 175-191 2 Moez Krichen,
Stavros Tripakis Black-Box Conformance Testing
for Real-Time Systems. SPIN 2004 109-126 3
Hacène Fouchal, Cyril Rabat, Antoine Rollet,
Abbas Tarhini Experimental Test Generation for
Real-Time Systems. IASSE 2004 284-289 4 Hacène
Fouchal, Eric Petitjean A Test Case Derivation
for Timed Systems. OPODIS 2001 125-140 5
Abdeslam En-Nouaary, Rachida Dssouli A Guided
Method for Testing Timed Input Output Automata.
TestCom 2003 211-225 6 Abdeslam En-Nouaary,
Rachida Dssouli, Ferhat Khendek Timed Wp-Method
Testing Real-Time Systems. IEEE Trans. Software
Eng. 28(11) 1023-1038 (2002) 7 Teruo
Higashino, Akio Nakata, Kenichi Taniguchi, Ana R.
Cavalli Generating Test Cases for a Timed I/O
Automaton Model. IWTCS 1999 197-214 8 Ismail
Berrada, Richard Castanet, Patrick Félix Testing
Communicating Systems a Model, a Methodology,
and a Tool. TestCom 2005 111-128 9 Ismail
Berrada, Richard Castanet, Patrick Félix From
The Feasability Analysis To Real-Time Test
Generation. Stud. Inform. Univ. 3(2) 203-230
(2004) 10 Kim Guldstrand Larsen, Marius
Mikucionis, Brian Nielsen Online Testing of
Real-time Systems Using Uppaal. FATES 2004
79-94 11 Brian Nielsen, Arne Skou Automated
Test Generation from Timed Automata. TACAS 2001
343-357 12 Rachel Cardell-Oliver Conformance
Tests for Real-Time Systems with Timed Automata
Specifications. Formal Asp. Comput. 12(5)
350-371 (2000) 13 Laura Brandán Briones, Ed
Brinksma A Test Generation Framework for
quiescent Real-Time Systems. FATES 2004 64-78
119
References (2)
Untimed systems 1 Jean-Claude Fernandez, Claude
Jard, Thierry Jéron, César Viho An Experiment in
Automatic Generation of Test Suites for Protocols
with Verification Technology. Sci. Comput.
Program. 29(1-2) 123-146 (1997) 2 Mihalis
Yannakakis, David Lee Testing for Finite State
Systems. CSL 1998 29-44 3 Jan Tretmans Test
Generation with Inputs, Outputs, and Quiescence.
TACAS 1996 127-146 4 Ed Brinksma, Jan
Tretmans Testing Transition Systems An
Annotated Bibliography. MOVEP 2000 187-195 5
Machiel van der Bijl, Arend Rensink, Jan
Tretmans Compositional Testing with ioco. FATES
2003 86-100 6 Cédric Besse, Ana R. Cavalli,
David Lee An Automatic and Optimized Test
Generation Technique Applying to TCP/IP Protocol.
ASE 1999 73-80 7 Soonuk Seol, Myungchul Kim,
Sungwon Kang, Jiwon Ryu Fully automated
interoperability test suite derivation for
communication protocols. Computer Networks 43(6)
735-759 (2003) 8 Alexandra Desmoulin, César
Viho Quiescence Management Improves
Interoperability Testing. TestCom 2005
365-379 9 César Viho, S. Barbin, Lénaick
Tanguy Towards a Formal Framework for
Interoperability Testing. FORTE 2001 53-68 10
Soonuk Seol, Myungchul Kim, Samuel T. Chanson
Interoperability Test Generation for
Communication Protocols based on Multiple Stimuli
Principle. TestCom 2002 151-168 11 Richard
Castanet, Ousmane Koné Deriving Coordinated
Testers for Interoperability. Protocol Test
Systems 1993 331-346 (12 Ismail Berrada,
Richard Castanet, Patrick Félix, Generating
Interoperability Test Cases from Conformance Test
Case Generation Tools. FORTE 2004 Work in
progress, Madrid Spain 13 C. Jard and T. Jéron.
TGV Theory, Principles and Algorithms. The sixth
world conference on integrated design and process
technology. Pasadena, California, June 23-28,
2002. 14 ISO/IEC JTC1 DTR-10000. Information
Technology -Framework and Classification of
International Standard Protocol.1994. 15 O.
Rafiq and R. Castanet. From Conformance Testing
to Interoperability Testing. Proceedings of the
3rd IWPCS'90, Oct-Nov, Washington DC,USA, 1990.
120
Thats it!

• Questions?
• Thank you for your time and attention.

121
Deterministic Timed Automata
b
a, xlt1
a, xgt1

• A timed automaton is deterministic if
• No edges labeled with e (some relaxation
  possible)
• Two edges with same source and same label have
  disjoint guards
• At most one run on a given timed sequence