15.1 Introduction

1
15.1 Introduction

• Test execution is situated on the critical path
  to product introduction. Test automation is used,
  for instance, to minimize the time needed for
  test execution. Automation can be profitable in
  terms of time, money, and/or quality.
• In principle, the execution of almost every test
  can be automated. In practice, only a small part
  of the tests are automated. Test automation is
  often used in the following situations
• tests that have to be repeated many times
• a basic test with a huge range of input
  variations the same steps have to be executed
  every time but with different data (for instance,
  evolutionary algorithms)
• very complicated or error-prone tests
• testing requires specialized equipment that
  generates the appropriate input signals,
  activates the system, and/or captures and
  analyzes the output.

2
15.1 Introduction

• Changes in the system, design, and requirements
  are common and are also a threat to the usability
  of the automated tests. The consequences of these
  changes can be
• additional test cases
• changed test cases
• different result checks
• changed system interface
• changed functionality
• different signals
• different target platform
• different internal technical implementation.

3
15.2 The technique of test automation-15.2.1
Design for maintainability

• To avoid a test suite from becoming shelfware it
  is designed to be as independent as possible of
  changes changes in the test object, the
  technical connection between the test suite and
  the system under test, and the tests and data
  used.
• This is established by reducing the dependencies
• test data is stored outside the test suite
• the description of test actions is separated
  from the technical implementation of these
  actions
• the process of how to handle an automated test
  is implemented independent of its environment and
  system under test this explains why any test
  suite needs several essential components
• the communication between the system under test
  and the test suite is implemented independent of
  the central process of the test suite.
• These issues will be discussed in more detail in
  the following sections. The following terms will
  be used
• Automated testing. Executing tests using an
  automated test suite.
• Test suite. Everything necessary to execute a
  test by pushing just one button.
• Data driven. An approach where physical test
  actions are separated from test data, and the
  test data is the impulse for the test.
• Framework. A library of reusable modules and
  design.

4
15.2.1.1 Test cases

• The test cases are stored outside the test suite
  (see Figure 15.1). They can be stored in a
  database or spreadsheet. An additional test case
  means adding only this test case to the storage
  of the test cases. The test suite doesnt have to
  be changed.

5
15.2.1.2 Test actions

• This implementation should be hidden from the
  normal user of the test suite the only thing to
  know is which test actions are available (see
  Figure 15.2).
• By choosing the right test actions the right
  test script can be composed. If the functionality
  of the system under test stays the same but the
  technical implementation is changed (for
  instance, another interface is used) only the
  technical implementation must be changed. The
  user can still use the same script as if nothing
  has happened. Table 15.1 shows an example where
  test data and actions are combined in one test
  script. This combination is quite common.

6
15.2.1.2 Test actions
7
15.2.1.3 Essential components

• Those basic components are part of the generic
  design of a test suite (see Figure 15.3) which is
  called a blueprint this has three major
  sections.
• The first is the input side with test scenarios
  and scripts.
• The last is the output side with status reports
  and progress reports. The processing unit, or
  test suite kernel, is situated in between.
• This kernel makes it possible to test with one
  push of a button.
• The test suite continues working without human
  intervention until the complete test scenario
  (see section 6.4.3) is executed. Appendix C
  provides a more detailed description of the
  blueprint.

8
15.2.1.4 Communication

• The system under test can have the following
  forms
• executable code on the same machine as that on
  which the test suite is installed
• executable code on a test target (separated from
  the test suite environment)
• system under test in a simulation environment
• system under test as pre-production system.

9
15.2.1.4 Communication

• The test case will stay the same while the
  communication, or technical implementation of the
  communication, is changed. There has to be a
  communication layer (see Figure 15.3) for all
  forms.

10
15.2.2 Maintenance

• Due to certain changes, the automated tests may
  not run correctly any more and maintenance
  becomes necessary. The blueprint is developed to
  keep maintenance to a minimum. The parts possibly
  affected by changes are
• the test data
• synchronization, error recovery, checks and
  initialization
• application specific modules
• the communication layer.

11
15.2.2 Maintenance

• In the introduction to this chapter, a list of
  possible changes was presented. The impact of
  those changes on the test suite are listed below.
• Adding test cases. Only the data store has to be
  changed.
• Changed test cases. The data store has to be
  changed. Sometimes it is necessary to add new
  test actions to the framework.
• Different result checks. Only the data store has
  to be changed if only the output prediction is
  changed. Otherwise checks and or test actions
  have to be changed or added.
• Changed system interface. The communication layer
  has to be changed. If the signal definition is
  changed, the test data should also be changed if
  these test data contain information about the
  signals.
• Changed functionality. This change can have an
  effect on the test data because new test cases
  have to be added. The synchronization, error
  recovery, checks, and initialization can be
  changed because the system dialog and sequence of
  functionality can have changed. New functionality
  means new test actions. The new functionality can
  also affect the communication layer.
• Different signals. There have to be changes in
  the communication layer. Sometimes it is also
  necessary to change the test data. Different
  target platform. This can affect the
  communication layer.
• Different internal technical implementation. If
  the external interface and functionality are not
  changed then this can only affect the software
  implementation which bypasses the external
  interface.

12
15.2.3 Testability

• To make test automation easier it is essential
  that testability is a quality attribute
  recognized during the design and implementation
  of the system.
• It is rather time consuming and expensive to
  built a dedicated communication layer because the
  behavior of the system is impossible to monitor.

13
15.3 Implementing test automation

• Test tools are used to enhance the quality, lower
  the costs of test execution, and/or save time.
• The automated test suite must be repaired and
  upgraded to make it run once again and cover
  the changed system behavior that must be tested.
  In bad cases, the repair costs more than
  executing the test manually.
• The objectives of quality, cost, and time are no
  longer realized and the overall perception is one
  of failure.

14
15.3 Implementing test automation

• Investment in the implementation of these kinds
  of tools is high and so the complete test set
  should be executed three to six times at least to
  achieve pay back.
• Most problems can be overcome by using a
  lifecycle model with proper analyzing activities
  and decision points (see Figure 15.4). This
  lifecycle has three consecutive phases
• initiation
• realization
• exploitation.

15
15.3 Implementing test automation
16
15.3 Implementing test automation

• Test automation objectives are quantified, the
  test objectives are described in detail, and the
  technical implications of the system under test
  are investigated.
• The output of this process biases the design
  process. According to the design, the automated
  test environment is developed.
• The end product of this phase will be an
  implemented test automation environment capable
  of realizing the test automation objectives.

17
15.3.1 Initiation

• The test automation project starts with a
  feasibility study. The main objective of this is
  to get enough information to decide if test
  automation is the proper thing to do. In order to
  get the right information, three items are
  analyzed
• test automation objectives
• system under test
• test organization.
• The test automation objectives are the
  expectations of the organization about test
  automation.
• The objectives are described in terms of quality,
  cost, and time there should be a balance
  between these. High quality generally leads to
  lower cost and time saving.
• The objectives are also the criteria for the
  determination of the success of the test
  automation project.
• Other objectives include in which development
  stage should automated testing be implemented,
  and what should be tested with this automated
  test environment.

18
15.3.1 Initiation

• The system under test is investigated technically
  to find out how it should be tested and whether
  additional equipment, such as simulators and
  signal generators, is necessary.
• It is also necessary to know whether the
  interfaces, internal technical structure, and the
  functionality of the system are regularly changed
  during development or consecutive releases.

19
15.3.1 Initiation

• It would be rather nice if this format could be
  used in the test automation environment.
• The proper use of test tools demands a repeatable
  test process. This means that there are
  well-described test cases derived by using test
  design techniques and a test strategy which
  decides on the relative importance of the system
  parts and quality attributes.
• To use and develop the automated test
  environment, special knowledge is necessary. If
  this knowledge is not available then employees
  have to be trained.
• Another consequence can be that the usage of the
  automated test environment is made as simple as
  possible by developing an intuitive user
  interface. This will hide the complexity of the
  automated test environment.

20
15.3.1 Initiation

• In addition to all these primary conditions one
  should also know if there is a tool available to
  use for the system itself. This will depend on
  the system itself, the target, the instantiation
  of the system, and also the program language.
• Figure 15.5 shows the different stages in the
  test tool selection process.

21
15.3.1 Initiation

• The ultimate consequence is the development of a
  dedicated tool or no automated test execution.
  The following activities have to be carried out.
• 1. Set goals and define the scope. Goals can
  include improving the test process, shortening
  test execution time, diminishing test execution
  capacity, or enhancing quality test execution. Be
  aware that goals such as solving organizational
  problems or attention-free testing can not be
  realized by tools. The scope shows whether the
  automated test environment should be used for
  only one particular system, a range of systems,
  or across the company.
• 2. Define knockout criteria. These criteria are
  mostly based on the technical implications of the
  system under test. A tool that cannot meet these
  criteria is useless and is immediately deleted
  from the list.
• 3. Long list and RFI. With the knockout criteria
  in mind, a long list of tools is prepared. All
  tool vendors get a Request For Information (RFI).
  This is a list of questions a vendor should
  answer. The accuracy of this answer is also an
  indicator of the vendors service level tool
  vendors that do not answer are discounted.
• 4. Detailed criteria. A detailed list of
  selection criteria is prepared. There are three
  kinds of criteria
• technical (compatibility, stability, script
  language, etc.)
• functional (object recognition, synchronization,
  report functionality, operating system, target
  platforms, etc.)
• demands to tool vendors (continuity, education,
  support, installed base, etc.). The criteria are
  grouped by knockout criteria, demands, and wishes.

22
15.3.1 Initiation

• 5. Short list and RFP. Based on the detailed
  criteria, tools are selected from the long list.
  The selected tools are placed on a short list
  which is sent to the vendors with detailed
  criteria and a Request for Proposal (RFP). This
  gives an indication of the costs, and the
  possibility of support and training. Based on
  this information, two or three tool vendors are
  invited to show the capabilities of their tools
  on a demonstration test environment. A list of
  capabilities each vendor should show with their
  tool should be prepared.
• 6. Proof of concept. Before using the tool in
  practice, and investing much effort in its
  implementation, a pilot can be started. If this
  pilot is successful the tool can be implemented.
  Many tool vendors offer the chance to use the
  tool on trial for a certain period.
• 7. Decision. The results of the proof of concept
  give enough information to decide if an
  implementation of the tool will meet
  expectations. The outcome could be that the
  objectives can only be met with a dedicated
  developed test tool.

23
15.3.1 Initiation

• The benefits can also be quantified in terms of
  saving money by spending less time and labor on
  testing.
• The enhancement of the quality of the test
  process is the subjective part of the benefits.
  Be aware that the direct cost of tools is high
  but the cost of labor during implementation,
  usage, and maintenance are usually much higher.
• This whole process should aid the enhancement of
  awareness and positive support for implemention
  as a major side effect.

24
15.3.2 Implementation

• The first is the preparation of a plan of action
  the purpose being to show the activities,
  people involved, and time estimation. Then the
  design and development will start. In this
  process, six activities are identified (see
  Figure 15.6).

25
15.3.2 Implementation

• Based on this design, the engineer develops the
  automated test environment and is responsible for
  implementing the necessary functionality to
  execute the delivered test cases. The functional
  testers develop the test cases based on test
  design techniques. In the last step, a final test
  of the automated test environment is performed.
• Now, the design and all other documentation about
  the automated test environment is brought up to
  date. Together with the automated test
  environment, this is deployed and used in the
  maintenance phase.
• In additional to these major activities, parallel
  activities can be executed as necessary.

26
15.3.3 Exploitation

• The automated test environment is now ready for
  use. The impact on the test process of the
  availability of an automated test environment is
  shown in Figure 15.7.

27
15.3.3 Exploitation

• During specification, the test cases should be
  described in a readable format for the automated
  test environment, and the environment should be
  adjusted due to changes in the test object.
• The test execution is partly covered by the
  automated test environment and in the completion
  phase all changes are frozen and stored. It is
  essential that a complete copy of the automated
  test environment is subject to configuration
  management.

28
15.3.3 Exploitation

• The major objective of the exploitation phase is
  to maintain and use the automated test
  environment.
• During this phase, the organization can profit
  from the benefits of the automated test
  environment.
• To meet the test automation objectives and to be
  able to benefit from this environment, in the
  next release the focus must be on maintenance of
  the environment. By neglecting this part of the
  job the environment will be useless after just
  one or two releases and that is probably before
  the investment is paid back.