SAbased Code Testing

1
SA-based Code Testing

• Henry Muccini
• Computer Science Department
• Universita dellAquila Italy
• (muccini_at_di.univaq.it)
• http//www.HenryMuccini.com

2
My Research

• Ph.D. Thesis on Software Architecture for
  Testing, Coordination Models and Views Model
  Checking, year 2002.
• Software Architecture and Coordination
• Software Architecture and Model Checking
• Software Architecture and Testing

3
Talk Overview

• Brief introduction on Software Architecture and
  Testing
• SA-based Conformance Code Testing
• SA-based Regression Testing
• PLA-based Testing
• Ongoing and future work
• Some references

4
Software Architecture (SA)
The History PhDThesis, Ch2

• 1992 and 1993
• SA is recognized as an independent area of
  research
• 1993-1995
• SA described through box and line, informal,
  diagrams
• 1995-2002
• Architectural Description Languages (ADLs) are
  introduced to formally describe SA
• 1997-2003
• more interest on dynamic aspects of SA
• SA used in academia and industry
• SA and analysis
• SA recognized as a valid tool to describe
  complex, concurrent, real systems

5
In general terms

• Describes (in a formal language) how a system is
  structured into components and connectors
• ?
• Components
• computation
• Connectors
• interaction
• Ports, Interfaces,
• how these
• components interact

SA Structure (topology)
SA Dynamics (behavior)
6
SA dynamics

• A model of Software Architecture behavior
• Labeled Transition System (LTS)
• Finite State Machine
• Petri Nets
• Given the SA formal description, the model can be
  automatically generated the dynamics is
  expressed in terms of components interaction via
  connectors

7
Motivations on SA-based Analysis

• For Analysis PhDThesis
• SA management is expensive and time consuming
• ? maximize the benefits
• ? Analyze SA as much as possible
• SA and Testing
• SA and Coordination models
• Model checking SA
• SA and Performance Analysis
• Deadlock detection on SA-based specification
• SA and Security
• Refining SA
• SA-based development processes
• ADL- based analysis of SA

8
Testing

• Testing is the process of executing a program
  with the purpose of
• finding errors or defects
• increasing the confidence in the proper
  functioning of the software (dependability).
• Not exaustive approach
• based on the selection of an adeguate set of
  tests

System
Unit
Integration
Code LevelOriented to SyntaxUnit Tests
Functional propertiesFormal specifications
Functional
Structural
9
Architecture-based Integration and Conformance
Testing

• Integration Testing unit tested components are
  integrated to build complex systems, and
• Integration Testing is aimed at exposing problems
  in the interfaces and interactions between the
  system components
• Functional focus on the functional requirements
• Architectural information for testing is
  extracted from the Architectural specification...
  
• Down to the Code and propagated down to the
  Code... Conformance Testing

10
The main idea
Identify SA-Tests
Apply the Tests
SA
Code
Class Client ------- ------ ------ -------
XClient
ClientA
ServerMaster
Recovery
Transform SA-Tests Into Code-level tests
ClientA
ClientB
ClientB
Server Slave
ClientC
ClientC

• Goal
• Derive test cases
• using software architectural description
• run test cases on the Code

11
Working Directions

• 1 - SA-based Conformance Code Testing
• (Joined work with Antonia Bertolino and Paola
  Inverardi)
• 2 - SA-based Regression Testing
• Testing C2-style architectures
• (Joined work with Marcio Dias and Debra
  Richardson)
• 3 - PLA-based Testing
• (Joined work with Andre van der Hoek)

12
1. SA-based Conformance Code Testing PhDThesis,
ch. 4
ICECCS97
ICSE00
Step2 Abstraction
Step1 modeling
SA Specification
ALTS Model
step3
BookChapter03
13
Step1 Formal SA specification

• Architectural Description Language (ADL) that
  produces a Labeled Transition System (LTS)
• Chemical Abstract Machine (CHAM) ICSE00
• Finite State Process (FSP) ICSE01
• Dynamics in terms of Component interactions

14
The Architecture-level Behavior
15
Step2 The Abstraction

• Architectural Views Kruchten
• Flow view
• Component Based view
• Concurrency view
• ...
• Obs_Functions to view the system only from a
  
• perspective that is relevant for testing
• Obs L ? D ? ?
• Abstract LTS (ALTS)
• Minimization and Reduction (trace-equivalence)

16
Flow view

• Alarm Flow
• Check Flow
• Clock Flow

ComponentBased View

• User Component
• Router Comp.
• Server Comp.

17
Step3 - Architectural Test Class
...

• ALTS Path Coverage
• Each complete path corresponds to an
  Architectural Test Class
• McCabe path coverage criteria
• FSP hiding operators

18
Step4 - Testing the Software Implementation

• How are the LTS paths implemented by the Code?
• We can test whether the system as-built
  implements this architectural behavior

19
Applications

• Tele Remote Medical Care System (TRMCS)
  ICSE2000,ICSE2001
• The Whiteboard case study BookChapter
• The Elevator case study Submitted

20
Tool Support

• LTS can be automatically derived from an
  Architectural specification (LTS generator Tool,
  LTSA Tool)
• ALTS can be semi-automatically generated
  (FC2Tools The Abstractor Tool)

ALTS
21
Topics of interest and Considerations

• Step1
• SA specification and modelization
• State explosion problem completeness
• Considerations
• we used Cham and FSP
• Step2
• Observation and Abstraction
• Considerations
• It is an empirical task, based on the software
  architect experience
• Classification of observations could help
• Methods similar to SAAM or SCENT, in which
  architectural information is empirically
  captured, could help in this task
• The ALTS construction has been automated using
  the CAESAR/ALDEBARAN tool

22
Topics of interest and Considerations

• Step3
• Path coverage
• Considerations
• McCabe's test technique seems a reasonable
  coverage criterion
• it may be automated
• Step4
• Traceability and development process
• Deterministic and non deterministic Testing
• Considerations
• it is the most difficult part
• relating SA specification to code
• key concepts traceability, development process

23
Step4 Some Considerations

• Mapping problems
• It is not so obvious which classes and methods
  implement an architectural functionality
• More sequences of method calls can implement the
  desired architectural behavior
• SA description is abstract and hides
  functionalities and objects defined at the code
  level
• The SA model usually describes only the expected
  behaviors, while the code also catches
  exceptional ones
• Test execution nondeterministic or deterministic
  CarverTai_TSE98 approach

24
class MasterRouter ServerConnection allarm
ServerConnection okFunction // the
services ReceiveUserAllarm
serviceReceiveUserAllarm ReceiveUserOkFunz
serviceReceiveUserOkFunz String
name,serverName static public PrintWriter
user_router_ok_funz static public
PrintWriter user_router_alarm ...
???
SA behavior
Source Code
drives
SA (topology and model)
Design and Source Code Def.
drives
SA (model)
Source Code (abstractions)
SA (model)
Source Code
25
2. SA-based Regression Testing Submitted

• Motivations and Goals
• To reduce the testing effort during architecture
  or code maintenance
• To handle the architectural drift
• To maximize benefits in using Software
  Architecture

26
Regression Testing

• Test modified software and provide a certain
  confidence that no new errors are introduced into
  previously tested code.
• Given program P and P, T is a test suite for P,
  regression testing techniques
• provide a certain confidence that P is still
  correct with respect to a set of test T, which
  is a subset of T
• Helps to identify new test cases for T.

27
Regression Test Selection technique

• Select T, subset of T and relevant for P
• Test P with respect to T
• If necessary, create T, to test new
  functionality/structure in P
• Test P with respect to T
• Create T, a new test suite and test history

28
Project Goal
29
From Traditional to SA-based Regression Testing
30
Initial experiment

• The approach has been applied to the Elevator
  case study
• Architecture in the C2 style
• Implemented using the C2 framework
• Tool support
• The SA is formally specified using the C2
  specification language and FSP.
• A behavioral model of the system is automatically
  produced from the FSP specification, using the
  LTSA tool.
• The abstraction over the LTS behavioral model is
  realized again in FSP.
• The mapping between architectural tests and
  code-level tests is provided for by the C2
  Framework.
• Test cases are run over the code using the
  Argus-I environment.
• The selective regression testing step is
  supported by the DejaVOO tool.

31
3. PLA-based Testing Tacos 2003

• A product line architecture precisely captures,
  in a single specification, the overall
  architecture of a suite of closely-related
  products Bosch2000
• A PLA explicitly specifies
• i) elements that are present in all products,
• ii) elements that are optional, and
• iii) elements which may be incorporated in one of
  many forms (variants)
• Whereas a regular architecture defines the
  structure of a single product, a product line
  architecture (PLA) defines the common
  architecture for a set of related products
  Bosch2000

32
Testing PLA

• A new challenge
• how to deal with optional elements or with the
  magnitude of products that may be present?
• The goal of this paper is to highlight the
  challenges and opportunities for software testing
  of PLAs
• We believe that existing mechanisms with which
  SAs are tested can be adapted to PLAs

33
PLA example
mandatory
optional
variant
variant
In total, twenty-four different product
architectures can be formed.
34
Testing PLA- Unit Testing

• SA
• Each SA component need to be unit tested
• PLA
• all components should be unit tested as well
• Including each optional component and each
  variant of a variant component
• However,the order in which they have to be tested
  can be adjusted based on priority.

35
Testing PLA Integration Testing

• SA
• components and connectors are combined together
  according to the architecture configuration
• PLA
• no single architectural configuration exists
• Possible solutions
• Iterative build-up integration approach powerful
  but expensive
• big bang limited but effortless
• Core-first big bang approach
• Integrated with heuristics to test only
  particular combinations

36
Testing PLA Conformance Testing

• SA
• conformance testing has been used to detect
  conformance errors between the SA and its
  implementation.
• PLA
• an implementation I conforms to its PLA when
• I conforms to a single PA
• I conforms to all the possible PAs out of the PLA
• I conforms, at least, to all the constraints and
  functionalities associated to the mandatory, core
  elements of the PLA
• a product architecture PA conforms to its PLA

37
Testing PLA Regression Testing

• SA
• If a new version P1v2 of an implementation P1v1
  is produced, regression testing techniques can be
  used to test the conformance of P1 to the
  initial SA
• If a new version (SAv2) of an architecture SAv1
  is produced, SAv2 test cases may be selected
  reusing SAv1s test cases.
• PLA
• PLA evolution PLA that becomes PLA
• PA becomes PA
• Code becomes Code

38
Future Work
SA-based Code Testing
39

• SA-based Conformance Code Testing
• Testing and Formal Methods
• Integration with the TGV TGV (Test
  Generation and Validation) environment used
  to test formal specifications
• Testing C2 style architectures
• C2 framework and Dradel ongoing work
• Integration with XADL, Menage, Behavioral model
• SA-based Regression Testing
• Implement the second goal
• PLA-based Testing
• Introduce an approach for PLA-based testing
• Integration with XADL, Menage, Behavioral model

40
Integration with XADL, Menage, Behavioral model

• SA specified in XADL
• with an additional behavioral specification
• The XADL architecture is implemented by using the
  new C2 Framework
• May be something different
• Menage features are used to help the regression
  testing steps
• The testing capability is embedded into Menage

41
Some References

• ICSE2000 A. Bertolino, F. Corradini, P.
  Inverardi and H. Muccini. "Deriving Test Plans
  from Architectural Descriptions". In ACM Int.
  Conf. on Software Engineering (ICSE2000), pp.
  220-229, Limerick (Ireland), June 2000.
• ICSE2001 A. Bertolino, P. Inverardi and H.
  Muccini. "An Explorative Journey from
  Architectural Tests Definition downto Code Tests
  Execution". In IEEE Int. Conf. on Software
  Engineering (ICSE2001), Toronto, May 2001.
• PhDThesis H. Muccini. Software Architecture for
  Testing, Coordination Models and Views Model
  Checking, PhD thesis, year 2002.
• Tacos 2003 H. Muccini, A. van der Hoek.
  "Towards Testing Product Line ArchitecturesIn
  Proc. of the ETAPS2003 workshop on "Test and
  Analysis of Component Based Systems" (Tacos),
  Warsaw, Poland, April 2003. In Electronic Notes
  of Theoretical Computer Science, Vol. 82, N. 6
  (2003).
• BookChapter A. Bertolino, P. Inverardi and H.
  Muccini. Formal Methods in Testing Software
  Architectures. Chapter in Formal Methods for
  Software Architectures, SFM-03SA Lectures, Eds.
  M. Bernardo, P. Inverardi, LNCS 2804, 2003, p.
  124-149.
• Submitted H. Muccini, M. Dias and D.
  Richardson. Software Architecture-based
  Conformance and Regression Testing. Submitted for
  publication.
• ongoing work H. Muccini, M. Dias. Systematic
  Testing of C2 style Software Architecture. Under
  Submission for publication.
• CarverTai_TSE98 Carver, R.H., Tai, K.-C.Use of
  Sequencing Constraints for Specification-Based
  Testing of Concurrent Programs. IEEE Trans. on
  Software Engineering 24, 6 (June 1998).
• Kruchten P. Kruchten. Architectural Blueprints
  - The 41 View Model of Software Architecture.
  IEEE Software 12 (6), November 1995, pp. 42-50.
• TGV Fernandez, J.-C., Jard, C., Jeron, T.,
  Nedelka, L., Viho, C. An Experiment in Automatic
  Generation of Test Suites for Protocols with
  Verification Technology. Special Issue of Science
  of Computer Programming, Vol. 29, pp. 123-146,
  1997.