Test Plans and Test Implementation

1
Test Plans and Test Implementation
EEE492.04
Royal Military College of Canada Electrical and
Computer Engineering

• Maj JW Paul
• Jeff.Paul_at_rmc.ca
• 613-541-6000 ext. 6656

Dr. Terry Shepard shepard_at_rmc.ca 613-541-6000
ext. 6031
Ref HS Ch 10
2
Systematic Testing

• Ad-hoc approach to testing is
• ineffective too late to influence design
  decisions
• expensive tests not reused during maintenance
• Systematic testing
• planned design for testability
• documented tests can be understood and evaluated
• maintained tests executed after every change
• Two useful work products
• Test Plan (TP) strategy for selecting
  executing tests
• Test Implementation (TI) implements the test plan

3
One Version of Testing Tasks

• Build the test harness
• Generate the test inputs (TP and TI)
• Determine the expected outputs
• Execute the test cases, monitoring the behaviour
• Compare the actual outputs to the expected
  outputs
• Evaluate the test results
• Focus automation on frequently required tasks (4
  - 6)

4
Test Plan (TP)

• TP serves as a specification for the TI
• Required sections
• Assumptions
• assumptions on which the testing depends
• Test environment
• environment in which the testing is performed
• Test case selection strategy
• guides the choice of test cases
• Test implementation strategy
• describes drivers, stubs, auxiliary functions,

5
Test Drivers

• Interactive drivers
• Prompts user for access routine name, parameters
• Prints return values and exceptions
• Good for debugging
• Tedious and error prone when there are many cases
• Tedious and error prone when cases must be
  repeated many times
• PGMGEN driver
• Simple test script language
• Described in Hoffman Strooper
• for educational use only
• Automates execution and checking of results
• Customized batch drivers
• Modules that interact with the environment

6
Module and System Testing

• Hard to test a module in its production
  environment
• poor controllability and observability
• Module testing
• test module in isolation using stubs and drivers
• Integration testing
• test combination of modules as subsystems
• System testing
• test the entire system

7
Test Case Selection

• Functional testing
• tests are based on specification
• largely ignore the implementation
• Structural testing
• tests are based on the implementation
• if code is missing, so are tests!

8
Functional Testing

• Choose special values for access routine
  parameters
• both exceptional and normal case values
• Choose special values for MID state
• only normal case values, on the basis that
  exception handling should prevent exceptional
  states from occurring
• eg. a full stack, an empty stack, but not an
  overflowed stack
• Test combinations of special values
• watch out for combinatorial explosion
• there may be dependencies

9
Functional Testing - Example

• Access routine f(p1,p2)
• p1 is an integer restricted to interval 0,100
• P2 belongs to enumerated type red, green, blue
• Special values for p1 the interval rule
• normal case values 0, 50, 100
• exceptional values -100, -1, 101, 1000
• Special values for p2
• normal case values red, green, blue
• no exceptional values
• Test cases cover all 21 combinations

10
(No Transcript)
11
Structural testing - limitations

• Path testing
• number of test cases often infinite
• many paths are not feasible
• In general, structural coverage is not sufficient
• usually measured by statement coverage
• statements can be covered in many different ways
• need to test all the ways that matter
• if statements are missing, they cannot be tested

12
Build a TP

• What are the critical sections?
• Assumptions
• Test environment
• Test case selection strategy
• Test implementation strategy
• pseudo-code

13
Lab 1 Starting Point
Existing TP and TI

• Functional testing for test case selection
• based on the symtbl MIS
• special values for parameters and module state
• MID and MI used for other special values
• Structural analysis as a cross-check
• measure statement coverage with tcov
• 100 coverage when possible, but not sufficient

14
Test Plan for existing Symbol Table
Assumptions Test environment

• Assumptions
• ST_MAXSYMLEN gt length of ST_MAXSYMS - 1 in
  string form
• ST_MAXINTLEN gt 0
•  
• Test environment
• PGMGEN driver
• no stubs

15
Test Plan (test case selection strategy)

• Test-case selection strategy
•  
• special values
•  
• module state
• number of symbols in table 0,1,ST_MAXSYMS/2,ST_M
  AXSYMS
• symbol length short, ST_MAXSYMLEN
•  
• access routine parameters
• st_s_add strings of length 0, 1,ST_MAXSYMLEN
  1,
• 2ST_MAXSYMLEN
• st_s_add, st_s_loc, st_g_loc, st_g_exsym empty
  string
•  

16
Test Plan (test cases exceptions)

• test cases
•  
• exceptions
•  
• for each special module state
• add overlength symbols
• if the table is full
• add a symbol not in the table
• set and get locations for symbols not in the
  table
• add every symbol in the table

17
Test Plan (test cases normal)

• test cases
•  
• normal
•  
• check st_g_exsym for empty string in empty table
• add the empty string, check and change its
  location
• for each special module state
• check table length
• check that a very long symbol is not in table
• for each i in 0..ST_MAXSYMS-1
• if i in 0..t_siz - 1
• check t_sym(i) in table with correct location
• check st_s_loc resets location
• else
• check t_sym(i) not in table

18
Design for Testability

• To make testing affordable and effective
• testability must be considered at design time
• key considerations controllability and
  observability
• Controllability
• ease with which a given module state can be
  reached
• example call to initialize a module with
  specific values
• Observability
• ease with which the module state may be observed
• example use of get for each value in a sequence
• example instrument the code to allow internal
  values to be observed

19
Work product criteria Test Plan

• For the module under test, M
• Audience
• those considering running or modifying Ms
  testing
• Prerequisites
• a thorough understanding of Ms MIS and some
  understanding of Ms MID
• Purpose
• serves as a planning tool for the TI
• documents the TI

20
Test Implementation (TI)

• TI implements the TP as simply and inexpensively
  as possible
• Includes
• test scaffolding such as drivers and stubs
• test data files
• manual and automated procedures for executing the
  tests

21
Work product criteria Test Implementation

• For the module under test, M
• Audience
• those considering running or modifying Ms
  testing
• Prerequisites
• a thorough understanding of Ms MIS and TP and
  some understanding of Ms MID
• Purpose
• Implement the TP simply and inexpensively

22
The PGMGEN testing tool

• Test case lttrace, expexc, actval, expval, typegt
• trace
• A trace used to exercise this module
• expexc
• Exception that the trace is expected to generate
• actval
• An expression evaluated after trace
• expval
• The value that actval is expected to have
• type
• The data type of actval and expval

23
(No Transcript)
24
PGMGEN Test Driver Generation

• For each case lttrace,expexc,actval,expval,typegt
• invoke c1 cn, monitoring exception occurences
• compare the actual ocurrances to expexc
• if there are any differences
• Print a message
• else
• If actval is not equal to expval
• Print a message
• if exceptions occurred since cn was invoked
• Print a message
• update summary statistics

25
Embedded C code

• Arbitrary C code delimited by and may
  appear
• between test cases
• as a call in a trace of a test case
• as the actval and expval field in a test case
• as a parameter of any call
• example

for (i1 ilt10 i) lts_init().s_push(1
0i), noexc, g_top(), 10i, intgt lts_init().s_push
(10i), noexc, g_depth(), 1, intgt