Who wrote the article

1
(No Transcript)
2
Who wrote the article ?
Alan Hartman
Kenneth Nagin
Andrei Kirshin
3
What is it all about ?

• Executing abstract tests on distributed software.
• Deriving tests from specifications abstract
  tests.
• Tester must manually convert those tests into
  their concrete executable form.
• Could be used to verify distributed software.
• The same abstract tests can be extended to verify
  issues related to concurrency, stress and input
  data variations.

4
Main issues

• Introduction.
• Test execution environment design.
• Abstract tests extended with object partitioning.
• Industrial experience.
• Future work.

5
Introduction
6

• What is an abstract test ?
• A test which describes test logic without
  concerning itself with platform or language
  issues.
• In order to run the test, the execution
  environment must deal with the messy details of
  concrete execution.
• The abstract tests may be reused and extended to
  verify additional software requirements such as
  concurrency.

7

• In recent years, software modeling has enjoyed
  tremendous popularity due to the adoption of
  object oriented models.
• Abstract tests are generated from such software
  models.

8

• However, the generation of test suites from a
  specification based modeling has yet to gain
  industrial momentum.
• Mainly because most current test execution
  environments were not designed to run the
  generated tests.

9
System Under Test

• Abstract model of System Under Test (SUT) -

10
System Under Test cont.

• SUT may be perceived of as a set of objects.
• The SUT is not restricted to any programming
  language or operating platform.
• The abstract test describes the objects expected
  behavior, state, and output in response to a
  stimuli from the tester.
• The concrete implementation of the SUT runs
  within different process address space.

11
Test Execution Environment

• Software testing environment are applications
  that assist testers in executing tests and
  collecting results against the SUT.
• Testing environments should run under the same
  operating system and use the same programming
  language in order to interact with the SUT.
• The cost of testing environments is a significant
  part of the overall development cost.

12
Test Execution Environment cont.

• The cost and complexity of the testing
  environment increases when the SUT is
  distributed.
• The main problem is that distributed software may
  run in different host environments, separated by
  a network and its components may be coded in
  different programming languages.

13
Test Execution Environment cont.

• Example of distributed software

Unix C
Linux Java
Windows C
TCP/IP
14
Test Execution Environment cont.

• The main goal of the test execution environment
  is to provide an infrastructure to support the
  heterogeneous application environment and
  languages.
• Of course, it must scale down to support more
  simple non-distributed software testing.

15
Abstract Test Execution

• An abstract test specifies a sequence of tester
  stimuli to the SUT, observation of SUT responses,
  and the assignment of a verdict based on the
  observations.
• An abstract test is designed to exercise a
  particular execution sequence or verify
  compliance with a specific requirement.

16
Abstract Test Execution cont.

• Abstract tests have the advantage that they
  express the testing logic without getting bogged
  down in the details of how to execute the test.
• Thus, the abstract test must be translated in
  order to execute it on the SUT.
• Abstract stimuli is mapped to method invocation,
  mouse clicks, or system commands.
• Abstract observations are mapped to actual value
  checking.

17
Abstract Test Execution cont.

• The test execution environment has 2 inputs
• Abstract test, telling what to execute against
  the SUT.
• Test execution directives, telling how to execute
  the abstract test.

Test Driver
Abstract Test
Test Exec. Directives
18
Test Execution Environment Design
19

• The test execution environment design supports
  the basic functionality for testing distributed
  systems.

20

• The main components of the test execution
  environment are
• Test Driver manages the test execution.
• Host Manager replicated on each target host
  (each host can have different operating system).
• Process Controllers interact directly with the
  SUT objects.
• The test driver and SUT need not be coded in the
  same programming language.

21

• The test driver communicates with host managers
  and process controllers in order to accomplish
  the following tasks
• Create, destroy and locate process controllers.
• Create and destroy SUT objects.
• Control input to the SUT objects.
• Observe output and state of the SUT objects.

via process controllers
via host manager
22

• Only the test driver understands the syntax and
  semantics of the abstract test cases.
• All other components handle real world
  interactions.
• This separation of concerns makes the test
  execution environment design well-suited for the
  testing of distributed software.

23

• The test driver is implemented only once since it
  has no SUT platform or language dependencies.
• Host manager and process controller
  implementation is dependent on the target SUT.
• Host managers must be implemented in all target
  SUT platforms (windows, Linux, ).
• Process controller runs in the same process as
  the SUT object and they are both coded in the
  same programming language.

24
Abstract Tests Extended with Object partitioning
25
Partitioning

• The SUT consist of different entities whose
  behavior is described in the abstract test.
• The tester can partition the abstract test into a
  set of executable objects.

• The object partitioning idea is everywhere.

26

• Motivation for object partitioning
• Extend the test usage to verify issues related to
  concurrency, stress and input data variations.

27

• The test modeling language does not have to be
  object oriented.
• However, it is required that the abstract stimuli
  and observable responses be identifiable so they
  can be mapped to concrete objects.

28
Example

• Consider an SUT that is client/server
  application.
• Server role - connect its clients to a database
  of accounts.
• Unique keys index the database of accounts.
• Client role create, remove, or update accounts
  based on the keys.

29
Example cont.

• We illustrate a simple test case
• Start the application server.
• Observe that no exception occurred.
• Create an account with key1.
• Observe that no exception occurred.
• Create another account with key1.
• Observe that an exception is thrown.
• Stop the application server.

30
Example cont.

• If all the test case observations are as
  expected, the test case passes. otherwise, it
  fails.
• The test can be partitioned into a client object
  and a server object in the test execution
  directives.
• Create(), remove(), update() mapped to a client
  object.
• Start(), stop() mapped to server object.

31
Example cont.

• When execution the test, our execution
  environment does the following
• Creates both the application server and the
  client object.
• Calls the client objects methods create(),
  remove() and update().
• Calls the application server objects methods
  start() and stop().
• Compare the responses from both the server and
  client objects with expected responses.

32
Object distribution

• When the SUT is a distributed application, the
  test runs on multiple hosts.
• The test driver coordinates all interaction
  between or within the test objects , based on the
  abstract test.
• This includes object initialization and cloning.

33
Object multiplication and cloning

• The abstract test may be re-used for concurrency
  and stress testing by multiplying the SUT objects
  and the environment in which they execute.

34

• Abstract test is executed in a stepwise fashion.
• Multiple independent operations may run
  concurrently or sequentially.
• At each step, the test driver runs the required
  number of independent operations and waits for
  their completion. Only then it checks the
  results
• If as expected continue to the next test step.
• If failed the exact sequence that triggered the
  fault is known.

35

• The sequence can be replayed when a fix is
  provided for the fault.
• The test driver also allows the user to step
  through a test case interactively when debugging
  a fault.

36
Mutual independence of objects

• When the number of objects does not affect the
  test behavior, these objects are said to be -

37
Mutual independence of objects cont.

• We refer to the creation of many instances of the
  same SUT object as cloning.
• Clones are created alike but they are made
  distinct by initializing each clone with
  different values.

38
Mutual independence of objects cont.

• The value that is used to initialize each clone
  must be set according to the following rule
• The value must be a shared attribute that
  distinguishes the SUT objects, but allow the test
  behavior to remain invariant when many objects
  are instantiated.

39
Mutual independence of objects cont.

• In the client/server system there may be many
  clients and servers referring to the same
  database.
• It is necessary to verify that the SUT works
  properly in the presence of many clients and
  servers.
• Object mutual-independence is achieved by
  initializing each client clone with a different
  set of keys used to access the database.

40
Mutual independence of objects cont.

• Object multiplication and cloning is an efficient
  way to validate concurrency and stress since it
  reuses an existing abstract test.
• Once the test has run successfully without object
  multiplication, any problems discovered
  subsequently can be attributed to the
  introduction of more objects into the system.

41
Object synchronization

• The test driver can reproduce the precise
  synchronization scenario that introduced the
  fault, thereby facilitating the fixing of the
  bug.
• The test driver implements two types of
  synchronization of cloned objects concurrent
  and sequential.

42
Concurrent synchronization.

• Concurrent synchronization causes each abstract
  stimulus and observation to start on all clones
  at the same time.
• The next stimulus and observation does not start
  until the previous one is complete on all clones.

43
Concurrent synchronization.
44
Sequential synchronization

• Sequential synchronization causes each abstract
  stimulus and observation to start and complete on
  each clone, before proceeding to the next one.

45
Sequential synchronization
46
Object initialization

• Object initialization is used to create mutual
  independence between objects.
• The test driver creates process controllers and
  objects dynamically during testing.
• It uses a hierarchical structure to organize them.

47
Object initialization cont.

• Each host manager, process controller or object
  is initialized with initialization sets
  containing name-value pairs specified in the test
  directives.
• Value pairs may reference values belonging to
  other members that are further up in the
  hierarchy.

48
Object initialization cont.

• Example
• Host initialization set includes
• IP 9.148.32.112 .
• One of its processes initialization set includes
• USER JOE .
• Any object of the process may include
• IP HOST.IP USER PROCESS.USER .
• In some cases those pairs may not be enough and
  there should be at least one more unique pair,
  why ?

49
Industrial Experience
50
GOTCHA - TCBeans

• IBM internal test automation toolset.
• Developed at the IBM Research Laboratory in
  Haifa.
• GOTCHA modeling tool used to generate abstract
  tests.
• TCBeans prototype of IBMs test execution
  environment.
• Has been used successfully to test IBM
  applications for the past four years.

51
Customer support center

• SUT dispatcher of an electronic customer
  support center.
• The customer support center is a component based
  distributed application written in java.
• Customers of the support center contact agents
  and request support.

52
Customer support center cont.

• Agents are represented by software components
  called - access points.
• Each access point generates interaction objects
  to represent its communications.
• A dispatcher schedules interactions to be
  serviced by access points.
• The dispatcher has a complex scheduling policy
  based on the skills required to deal with the
  communication and the skills possessed by the
  available agents.

53
Customer support center cont.

• GOTCHA was used to generate a set of abstract
  tests.
• TCBeans was used by an independent testing group
  to execute the test.
• 35 defects were discovered.
• Most of the defects were attributed to sequencing
  problems.

54
Customer support center cont.

• Testing with many access points was a requirement
  in the system test to validate the dispatcher
  under stress.
• The test team reused the abstract test but
  created many access points using object
  distribution, multiplication and cloning.
• Without any additional coding, they discovered an
  additional 5 defects.

55
Future Work
56
Future work

• Go beyond the stepwise concurrency scheme and run
  more complex synchronization patterns generated
  from an abstract model.
• Example synchronize the execution of more than
  one instance of the abstract tests.
• When to synchronize the stimuli and observations
  between the multiple test instances.

57
THE END