How to make test cases for integration test

1
How to make test cases for integration test
Hans Schaefer hans.schaefer_at_ieee.org http//home.c
2i.net/schaefer/testing.html
2
Contents

• 1 - What we test
• 2 - Typical errors
• 3 - How to make test cases
• 4 - Strategies for integration

3
1 - What to test

• Interfaces between modules, subsystems, systems
• Common functionality
• Consistency between parts

This we test

4
Candidates for finding what to test

• Interfaces
• Sequence- and collaboration charts
• APIs
• Communication protocols
• Files and their formats
• Call-pairs, argument lists
• Event lists
• Use and flow of global data (Init, write, read,
  search, change, delete)
• Interaction Online - batch - online
• Checking consistency between interfacing parts
• Synchronization of parallel processes and
  transactions. Especially queue handling.
• Multitasking Capacity of shared resources (place
  and speed)
• Handling of failure in other components and
  recovery
• Special cases at interfaces (not existing data,
  bad data, empty files, network down etc.)

5
2 - Typical defects we can find

• Wrong module or function called
• Wrong state of a called object when it is called
• Wrong boundary values
• Wrong data type
• Speed problems Too fast or slow transmission
• Format inconsistency
• Conventions or restrictions misused
• Wrong interpretation of names, data, codes etc.
• Transfer and output files too large, empty,
  missing

6
More defects...

• Order of parameters
• References How many times indirect?
• Cleaning up Are reserved things freed,
  superfluous data deleted etc.?
• Data flow problems Data not found, disappear
  (overwritten), never used
• Wrong use of default (Standard values)
• Problems with synchronization and queues
• Problems with shared resources
• Performance, speed
• Wrong use of communication channels (not
  connected, disconnect, wrong port, confusion
  about who initiates, ...)
• Problems across platforms
• Start orders of parallel programs or processes,
  restart an recovery
• and more...

7
The first step in test execution

• Just integration of the parts
• Will they install?
• Do they start and communicate at all?
• Correct version?
• Complete?
• Does the test bed work?

8
3 - Test design methods

• Data variation test
• State transition test
• Entity life history test
• Data flow test
• Using Heuristic rules

9
Method 1 Data variation test

• (Class partitioning and boundary values)
• Find the equivalence classes for the data
  crossing the interface to be integrated !
• Equivalence classes for the called module input
• Equivalence classes for the calling module output
• If signal sequences are sent, and a dialogue
  between caller and called has to be run, find the
  classes for data returned to the caller.
• Often, this is a subset of the caller equivalence
  classes.

10
More about data variation

• To test real time aspects, add the classes
• Called is awaiting the signal,
• Called is not waiting
• Fast and slow traffic (to test timeouts and
  stress).
• To test shared resources add
• High volumes of data
• High communication rate

11
More about data variation

• Boundary values have good chances of finding
  errors.
• Especially if receiving module does not accept
  the whole value range possibly sent by the
  sender.
• Test at minimum, maximum and just around
• Test at zero
• Test for the maximum and minimum values possible
  in your hardware
• Send signals as fast as possible, as slow as
  possible, around and after time-out.
• Test with large data volumes (volume test) and
  with empty data storage.

12
Method 2 Testing state transitions

• For programs where behavior is state dependent.
• To test such a program, the following is a
  minimum
• Combine State transition diagrams of both
  modules.
• From the start state, traverse all states and all
  transitions
• Test every feasible input (input class) in every
  state
• For higher reliability, cover combinations of
  transitions (2,3, up to the number of states).
  (switch coverage)

13
Method 3 Entity life history testEvent
perturbations

• Events may arrive in certain different orders.
  The program may need to handle them all.
• Design notation Entity life history diagram.
• In principle an alternative notation for state
  transition diagrams.

14
Diagram to be read like JSP diagrams Left to
right, as far down as possible. o means
alternatives. means repetition. Alternative
notation Finite state machine or state
transition diagram.
15
More about entity life history test

• What to test?
• Test all (or most important) possible lives of
  all entities
• Starting at birth event and moving to death event
  list all valid sequences of events affecting
  entitys life.
• Translate each event sequence into a sequence of
  operations.
• Every allowed sequence, every alternative. For
  repetitions zero, one, several times.

16
Example

• The following sequences of events represent the
  possible life histories for the call
• 1. E1 - E3
• 2. E1 - E4
• 3. E1 - E2 - E3
• 4. E1 - E2 - E4
• 5. E1 - E2 - E2 - E3
• 6. E1 - E2 - E2 - E4
• The operations are
• 1. Connect - Disconnect
• 2. Connect - cut line, restore system
• 3. Connect - transmit - Disconnect
• 4. Connect - transmit - cut line, restore system
• 5. Connect - transmit - transmit - Disconnect
• 6. Connect - transmit - transmit - cut line,
  restore system
• Number one is a bit special, and would incur that
  the caller hangs up just exactly at connection
  time.

17
Event perturbations for parallel events

• Five tests
• Only event 1
• Only event 2
• Event 1 before 2
• Event 2 before 1
• Both at the same time
• Special tests
• Nothing at all happens (testing the timeout)
• Stress test Repeated event arriving fast.

18
Testing queues

• Every time waiting schemes are involved
• Is it a queue or a stack?
• Several parallel queues with different (correct?)
  priorities?
• Empty queue?
• Full queue?
• Correct priorities?

19
Method 4 Tests for a data flow model
Roper and Ab.Rahim, Software Testing Using
Design-based Techniques, EuroSTAR 93
20
What to test

• Every connection from one program to another, or
  to itself.
• Chain program calls after each other.
• Generally Combine module tests where one
  process is the continuation of another.
  Connections in data flow diagrams.

21
What to test for this example

• Get reservation , success, then Get ticket
• Get reservation, success, then Get reservation
  once more
• Get reservation, unsuccessful, then Get ticket -
  check that no reservation was done.
• Do this for all ticket types!

22
Test design in detail

• 1.) Identify the inputs and outputs in every
  component.
• For example
• READ, GET, ACCEPT, WAIT FOR, FIND, SEARCH,
  LOCATE, ....
• WRITE, PUT, SEND, SIGNAL, APPEND, DISPLAY, ...
• 2. Find the different processing paths through
  every component or find the equivalence classes
  at the interface.

23
The remaining test design process

• 3. Produce test cases
• For every component All output classes must be
  generated.
• But for integration only the output classes used
  by the receiving process are important.
• Generate test cases to execute the path from
  every definition of a variable in one process to
  every use of it in the other process.
• (Definition-use-pairs)
• 4. Combine the tests by putting together several
  components. Make the next component use all the
  output classes from the components before.
• Continue putting on components until the input
  comes from outside the system and the output is
  observable outside.

24
Method 5 Heuristic rules

• Make all data fields different, if possible.
• (You may then find wrong order of parameters)
• Dont use default-values - BUT, test them also.
• (Zero, blank, etc. - You cannot see if wrong
  defaults are chosen. But you also have to test
  defaults!)
• Try to make field values wrong at places they
  should not be.
• (Thus, if wrong order occurs, the program will
  probably protest.)
• Try excluded values, troublesome values and
  boundary values.
• Try any values derived from common errors or
  rules in this section.

25
More about Heuristic rules

• Choose input such that all outputs are different
  from all inputs.
• This finds errors where output fields are not
  filled, or input values just put out again.
• Example Computing the payment date and sending
  it back. Payment is today 30 days. Use a day in
  a month, which is not 30 days long. Best is
  December, as the year will also change if day one
  is not chosen.
• In test sequences, the next test should have
  different values from the one before in every
  data position.
• This may detect that output is not generated.
• Generate all different output messages.
• Especially test functions that stress hardware.

26
More heuristic rules File I/O

• From Myers, The Art of Software Testing
• 1. File attributes correct?
• 2. OPEN/CLOSE statements correct?
• 3. Format specification matches I/O statement?
• 4. Buffer size matches record size?
• 5. Files opened before use?
• 6. End-of-file conditions handled?
• 7. I/O errors handled?
• 8. Any textual errors in output information?

27
4 - Integration strategies

• Function wise integration
• Integrate functional groups belonging together
• Word-processor
• 1-File functions, 2-Writing, 3-Editing,...
• Data directed integration
• For every global data element, integrate all
  functions working with it.
• Bank system
• Everything you can do with a customer
• Everything you can do with an account
• Everything you can do with a loan

28
Integration strategies

• Layer wise integration
• Lowest layer first
• Then every layer further up, together with
  layer(s) below

29
Integration strategies (sequential programs)

• Top down integration
• Integrate functional groups starting at the
  control flow top down to what is called
• Web-pages
• 1-Index file, 2-all linked pages from
  index.html, 3-further down,...
• Bottom up integration
• Integrate following control flow from what is
  called (library, utilities) and up to who calls
• Start on the bottom
• integrate one level up with who calls the
  bottom
• Integrate further up until you reach the top.

30
Integration strategies (continued)

• Skeleton integration
• Integrate functions which make the remaining
  integration easier.
• Word-processor
• File handling
• Writing
• Editing
• Formatting
• ...
• Run newly integrated functions by calling them
  through the skeleton existing from before.

31
Integration strategy for object oriented programs

• Integration of inheritance hierarchies
• Top-down
• Test the super class first, then every subclass
  with the super class. Retest the working together
  of existing and new functions. (consistency).
• Integration of calls between objects
• Server first, then client
• A class calling another class is tested after the
  called class has been tested by itself.
• Problem with stubbing of used classes.
• Best to use tools to help with stubbing. (e.g.
  Cantata from IPL)
• Very important to keep the test cases for
  (automatic) retest.

32
Priorities for integration

• Important modules
• Unstable system parts
• Modules which make the test easier
• After a good module test you may integrate larger
  groups of modules at a time.
• Dont integrate one and one module!
• (A kind of Big-Bang-Test)

33
Exercise

• A DATE is to be transferred over an interface.
  The date consists of three two digit fields for
  day, month and year.
• The modules have been module tested and work
  fine.
• Year 2000 and leap year are no issues here!
• Which dates do you want to try to test the
  interface?

34
Solution

• The largest risk is that someone has screwed up
  the order of day, month and year. A good test
  data has every of the three values different, a
  day greater than 12 and a year greater then 12
  and greater than 31 OR less than 1.
• For example 31 August 00
• If you doubt the quality of the module tests, you
  might test the first and last of every month,
  especially 29th of February.

35
Test coverage

• Depends on risk
• If the subsystem is more used, test more
• If you know little about the parts to be
  integrated, test more
• Impossible to test everything
• Important to define
• what you test
• how thoroughly
• what you dont test
• why

36
Test coverage metrics

• Interface coverage
• Call pair coverage
• Data flow coverage (data given or defined in the
  one program to data use in the other)
• Combined state transition coverage (cover the
  combined state diagram of both programs)

37
Literature and references

• Boris Beizer, Black Box Testing, 1995
• The book describes test design techniques.
  Especially data flow and transaction flow testing
  is useful.
• Kaner, Falk, Nguyen, Testing Computer Software,
  1999.
• Generally good ideas for test design and
  priorities.
• Glenford Myers, The Art of Software Testing, 1979
• The classical book. A good chapter on low level
  integration.
• Roper and Ab.Rahim, Software Testing Using
  Design-based Techniques, Presentation at
  EuroSTAR 93
• Look at tools from ATTOL testware, Testwell Oy,
  IPL, Telelogic, Software Research, LDRA,
  junit.org, xunit.org.