15.1 Introduction - PowerPoint PPT Presentation

1 / 28
About This Presentation
Title:

15.1 Introduction

Description:

15.1 Introduction Test execution is situated on the critical path to product introduction. Test automation is used, for instance, to minimize the time needed for test ... – PowerPoint PPT presentation

Number of Views:76
Avg rating:3.0/5.0
Slides: 29
Provided by: JayM151
Category:

less

Transcript and Presenter's Notes

Title: 15.1 Introduction


1
15.1 Introduction
  • Test execution is situated on the critical path
    to product introduction. Test automation is used,
    for instance, to minimize the time needed for
    test execution. Automation can be profitable in
    terms of time, money, and/or quality.
  • In principle, the execution of almost every test
    can be automated. In practice, only a small part
    of the tests are automated. Test automation is
    often used in the following situations
  • tests that have to be repeated many times
  • a basic test with a huge range of input
    variations the same steps have to be executed
    every time but with different data (for instance,
    evolutionary algorithms)
  • very complicated or error-prone tests
  • testing requires specialized equipment that
    generates the appropriate input signals,
    activates the system, and/or captures and
    analyzes the output.

2
15.1 Introduction
  • Changes in the system, design, and requirements
    are common and are also a threat to the usability
    of the automated tests. The consequences of these
    changes can be
  • additional test cases
  • changed test cases
  • different result checks
  • changed system interface
  • changed functionality
  • different signals
  • different target platform
  • different internal technical implementation.

3
15.2 The technique of test automation-15.2.1
Design for maintainability
  • To avoid a test suite from becoming shelfware it
    is designed to be as independent as possible of
    changes changes in the test object, the
    technical connection between the test suite and
    the system under test, and the tests and data
    used.
  • This is established by reducing the dependencies
  • test data is stored outside the test suite
  • the description of test actions is separated
    from the technical implementation of these
    actions
  • the process of how to handle an automated test
    is implemented independent of its environment and
    system under test this explains why any test
    suite needs several essential components
  • the communication between the system under test
    and the test suite is implemented independent of
    the central process of the test suite.
  • These issues will be discussed in more detail in
    the following sections. The following terms will
    be used
  • Automated testing. Executing tests using an
    automated test suite.
  • Test suite. Everything necessary to execute a
    test by pushing just one button.
  • Data driven. An approach where physical test
    actions are separated from test data, and the
    test data is the impulse for the test.
  • Framework. A library of reusable modules and
    design.

4
15.2.1.1 Test cases
  • The test cases are stored outside the test suite
    (see Figure 15.1). They can be stored in a
    database or spreadsheet. An additional test case
    means adding only this test case to the storage
    of the test cases. The test suite doesnt have to
    be changed.

5
15.2.1.2 Test actions
  • This implementation should be hidden from the
    normal user of the test suite the only thing to
    know is which test actions are available (see
    Figure 15.2).
  • By choosing the right test actions the right
    test script can be composed. If the functionality
    of the system under test stays the same but the
    technical implementation is changed (for
    instance, another interface is used) only the
    technical implementation must be changed. The
    user can still use the same script as if nothing
    has happened. Table 15.1 shows an example where
    test data and actions are combined in one test
    script. This combination is quite common.

6
15.2.1.2 Test actions
7
15.2.1.3 Essential components
  • Those basic components are part of the generic
    design of a test suite (see Figure 15.3) which is
    called a blueprint this has three major
    sections.
  • The first is the input side with test scenarios
    and scripts.
  • The last is the output side with status reports
    and progress reports. The processing unit, or
    test suite kernel, is situated in between.
  • This kernel makes it possible to test with one
    push of a button.
  • The test suite continues working without human
    intervention until the complete test scenario
    (see section 6.4.3) is executed. Appendix C
    provides a more detailed description of the
    blueprint.

8
15.2.1.4 Communication
  • The system under test can have the following
    forms
  • executable code on the same machine as that on
    which the test suite is installed
  • executable code on a test target (separated from
    the test suite environment)
  • system under test in a simulation environment
  • system under test as pre-production system.

9
15.2.1.4 Communication
  • The test case will stay the same while the
    communication, or technical implementation of the
    communication, is changed. There has to be a
    communication layer (see Figure 15.3) for all
    forms.

10
15.2.2 Maintenance
  • Due to certain changes, the automated tests may
    not run correctly any more and maintenance
    becomes necessary. The blueprint is developed to
    keep maintenance to a minimum. The parts possibly
    affected by changes are
  • the test data
  • synchronization, error recovery, checks and
    initialization
  • application specific modules
  • the communication layer.

11
15.2.2 Maintenance
  • In the introduction to this chapter, a list of
    possible changes was presented. The impact of
    those changes on the test suite are listed below.
  • Adding test cases. Only the data store has to be
    changed.
  • Changed test cases. The data store has to be
    changed. Sometimes it is necessary to add new
    test actions to the framework.
  • Different result checks. Only the data store has
    to be changed if only the output prediction is
    changed. Otherwise checks and or test actions
    have to be changed or added.
  • Changed system interface. The communication layer
    has to be changed. If the signal definition is
    changed, the test data should also be changed if
    these test data contain information about the
    signals.
  • Changed functionality. This change can have an
    effect on the test data because new test cases
    have to be added. The synchronization, error
    recovery, checks, and initialization can be
    changed because the system dialog and sequence of
    functionality can have changed. New functionality
    means new test actions. The new functionality can
    also affect the communication layer.
  • Different signals. There have to be changes in
    the communication layer. Sometimes it is also
    necessary to change the test data. Different
    target platform. This can affect the
    communication layer.
  • Different internal technical implementation. If
    the external interface and functionality are not
    changed then this can only affect the software
    implementation which bypasses the external
    interface.

12
15.2.3 Testability
  • To make test automation easier it is essential
    that testability is a quality attribute
    recognized during the design and implementation
    of the system.
  • It is rather time consuming and expensive to
    built a dedicated communication layer because the
    behavior of the system is impossible to monitor.

13
15.3 Implementing test automation
  • Test tools are used to enhance the quality, lower
    the costs of test execution, and/or save time.
  • The automated test suite must be repaired and
    upgraded to make it run once again and cover
    the changed system behavior that must be tested.
    In bad cases, the repair costs more than
    executing the test manually.
  • The objectives of quality, cost, and time are no
    longer realized and the overall perception is one
    of failure.

14
15.3 Implementing test automation
  • Investment in the implementation of these kinds
    of tools is high and so the complete test set
    should be executed three to six times at least to
    achieve pay back.
  • Most problems can be overcome by using a
    lifecycle model with proper analyzing activities
    and decision points (see Figure 15.4). This
    lifecycle has three consecutive phases
  • initiation
  • realization
  • exploitation.

15
15.3 Implementing test automation
16
15.3 Implementing test automation
  • Test automation objectives are quantified, the
    test objectives are described in detail, and the
    technical implications of the system under test
    are investigated.
  • The output of this process biases the design
    process. According to the design, the automated
    test environment is developed.
  • The end product of this phase will be an
    implemented test automation environment capable
    of realizing the test automation objectives.

17
15.3.1 Initiation
  • The test automation project starts with a
    feasibility study. The main objective of this is
    to get enough information to decide if test
    automation is the proper thing to do. In order to
    get the right information, three items are
    analyzed
  • test automation objectives
  • system under test
  • test organization.
  • The test automation objectives are the
    expectations of the organization about test
    automation.
  • The objectives are described in terms of quality,
    cost, and time there should be a balance
    between these. High quality generally leads to
    lower cost and time saving.
  • The objectives are also the criteria for the
    determination of the success of the test
    automation project.
  • Other objectives include in which development
    stage should automated testing be implemented,
    and what should be tested with this automated
    test environment.

18
15.3.1 Initiation
  • The system under test is investigated technically
    to find out how it should be tested and whether
    additional equipment, such as simulators and
    signal generators, is necessary.
  • It is also necessary to know whether the
    interfaces, internal technical structure, and the
    functionality of the system are regularly changed
    during development or consecutive releases.

19
15.3.1 Initiation
  • It would be rather nice if this format could be
    used in the test automation environment.
  • The proper use of test tools demands a repeatable
    test process. This means that there are
    well-described test cases derived by using test
    design techniques and a test strategy which
    decides on the relative importance of the system
    parts and quality attributes.
  • To use and develop the automated test
    environment, special knowledge is necessary. If
    this knowledge is not available then employees
    have to be trained.
  • Another consequence can be that the usage of the
    automated test environment is made as simple as
    possible by developing an intuitive user
    interface. This will hide the complexity of the
    automated test environment.

20
15.3.1 Initiation
  • In addition to all these primary conditions one
    should also know if there is a tool available to
    use for the system itself. This will depend on
    the system itself, the target, the instantiation
    of the system, and also the program language.
  • Figure 15.5 shows the different stages in the
    test tool selection process.

21
15.3.1 Initiation
  • The ultimate consequence is the development of a
    dedicated tool or no automated test execution.
    The following activities have to be carried out.
  • 1. Set goals and define the scope. Goals can
    include improving the test process, shortening
    test execution time, diminishing test execution
    capacity, or enhancing quality test execution. Be
    aware that goals such as solving organizational
    problems or attention-free testing can not be
    realized by tools. The scope shows whether the
    automated test environment should be used for
    only one particular system, a range of systems,
    or across the company.
  • 2. Define knockout criteria. These criteria are
    mostly based on the technical implications of the
    system under test. A tool that cannot meet these
    criteria is useless and is immediately deleted
    from the list.
  • 3. Long list and RFI. With the knockout criteria
    in mind, a long list of tools is prepared. All
    tool vendors get a Request For Information (RFI).
    This is a list of questions a vendor should
    answer. The accuracy of this answer is also an
    indicator of the vendors service level tool
    vendors that do not answer are discounted.
  • 4. Detailed criteria. A detailed list of
    selection criteria is prepared. There are three
    kinds of criteria
  • technical (compatibility, stability, script
    language, etc.)
  • functional (object recognition, synchronization,
    report functionality, operating system, target
    platforms, etc.)
  • demands to tool vendors (continuity, education,
    support, installed base, etc.). The criteria are
    grouped by knockout criteria, demands, and wishes.

22
15.3.1 Initiation
  • 5. Short list and RFP. Based on the detailed
    criteria, tools are selected from the long list.
    The selected tools are placed on a short list
    which is sent to the vendors with detailed
    criteria and a Request for Proposal (RFP). This
    gives an indication of the costs, and the
    possibility of support and training. Based on
    this information, two or three tool vendors are
    invited to show the capabilities of their tools
    on a demonstration test environment. A list of
    capabilities each vendor should show with their
    tool should be prepared.
  • 6. Proof of concept. Before using the tool in
    practice, and investing much effort in its
    implementation, a pilot can be started. If this
    pilot is successful the tool can be implemented.
    Many tool vendors offer the chance to use the
    tool on trial for a certain period.
  • 7. Decision. The results of the proof of concept
    give enough information to decide if an
    implementation of the tool will meet
    expectations. The outcome could be that the
    objectives can only be met with a dedicated
    developed test tool.

23
15.3.1 Initiation
  • The benefits can also be quantified in terms of
    saving money by spending less time and labor on
    testing.
  • The enhancement of the quality of the test
    process is the subjective part of the benefits.
    Be aware that the direct cost of tools is high
    but the cost of labor during implementation,
    usage, and maintenance are usually much higher.
  • This whole process should aid the enhancement of
    awareness and positive support for implemention
    as a major side effect.

24
15.3.2 Implementation
  • The first is the preparation of a plan of action
    the purpose being to show the activities,
    people involved, and time estimation. Then the
    design and development will start. In this
    process, six activities are identified (see
    Figure 15.6).

25
15.3.2 Implementation
  • Based on this design, the engineer develops the
    automated test environment and is responsible for
    implementing the necessary functionality to
    execute the delivered test cases. The functional
    testers develop the test cases based on test
    design techniques. In the last step, a final test
    of the automated test environment is performed.
  • Now, the design and all other documentation about
    the automated test environment is brought up to
    date. Together with the automated test
    environment, this is deployed and used in the
    maintenance phase.
  • In additional to these major activities, parallel
    activities can be executed as necessary.

26
15.3.3 Exploitation
  • The automated test environment is now ready for
    use. The impact on the test process of the
    availability of an automated test environment is
    shown in Figure 15.7.

27
15.3.3 Exploitation
  • During specification, the test cases should be
    described in a readable format for the automated
    test environment, and the environment should be
    adjusted due to changes in the test object.
  • The test execution is partly covered by the
    automated test environment and in the completion
    phase all changes are frozen and stored. It is
    essential that a complete copy of the automated
    test environment is subject to configuration
    management.

28
15.3.3 Exploitation
  • The major objective of the exploitation phase is
    to maintain and use the automated test
    environment.
  • During this phase, the organization can profit
    from the benefits of the automated test
    environment.
  • To meet the test automation objectives and to be
    able to benefit from this environment, in the
    next release the focus must be on maintenance of
    the environment. By neglecting this part of the
    job the environment will be useless after just
    one or two releases and that is probably before
    the investment is paid back.
Write a Comment
User Comments (0)
About PowerShow.com