Test Coverage

1
Test Coverage

• We have studied various black box and white box
  test techniques involving boundary values and
  test paths.
• Various Quality Goals may be articulated with
  test cases
• 100 of input variables are boundary value tested
• 100 of code statements are executed
• 95 of branches (dd paths) are tested
• These goals are set in such a manner that we can
  also use them as criteria to stop testing.

2
Test Result Analysis
accumulative of defects found
Slow down testing?
More testing !
test time
Number of problems found tend to slow down at
some point in the over-all testing - - - that
point is known as the knee of the curve when we
start considering stopping the test.
3
Buggy Areas

• There have been historical data that shows the
  buggy areas tend to be buggy all the way
  through the development life cycle (from
  review/inspection results of requirements to
  design to code through testing of code).
• Also there has been historical indication that
  the code that has a large number of defects at
  the beginning of test tend to have a high
  probability of having a lot more defects.

probability of more defects exist
the higher the number of defects found in a
module, the more likely it contains more defects!!
balance against finding more bugs will clean up
the module
of problems found in a module
4
What to focus on and When to end testing

• After monitoring the test results, one might
  consider
• Testing more in the buggy code areas
• Continue testing until the slowing down knee
  starts to appear. (tracking the test results also
  allows one to re-adjust the test schedule if
  necessary ---- very hard fight because testing is
  the last major activity before release)

5
Fault/Defect seeding

• Since we can never be sure that all the defects
  are found and fixed, we are looking for
  indicators such as the testing knee to help us
  determine when to terminate testing.
• Another methodology is to seed the code with
  defects
• Compare the number of seeded defects found
  against the total number of seeded defects.
• Stop testing stop testing when that ratio is an
  acceptable quality level for the organization.
• The key question is what type of seeded defects
  should one put in where
• Look at the history of defects of similar
  software produced by similar groups
• Look at the early defects found in the buggy
  code areas - - - insert more seeded defects in
  those buggy areas.

6
Example of Defect Seeding

• Suppose we decided to generate 50 seeded defects
  and insert them into the code.
• Further suppose, through testing, we have found
  45 of these seeded defects along with 392
  indigenous (original unseeded) defects.

Assuming a linear relationship of seeded and
unseeded defects, we can use the ratio
(detected seeded) / (total
seeded) (detected unseeded) / (total
unseeded) to estimate the total number of
indigenous or unseeded defects.
For the example, we have 45
/ 50 392 / total unseeded total unseeded
435.5 or 436 defects Estimated remaining
defects in the code is 436-392 44
unseeded defects
Question Is 45/50 .9 or 90 discovery rate
good enough? What was the Testing and Quality
Goal? Can you live with possibly 44 more
bugs? Is your maintenance-support team ready?
7
Confidence in Software with Defect Seeding

• Our confidence that there are N defects in the
  software can be tested with seeding of S defects
  and running the test until we find all S seeded
  defects. Further suppose we find n actual
  non-seeded defects. Then our confidence level
  (CL) of there are N non-seeded defects can be
  defined as
• CL 100. if n gt
  estimated N
• CL S/(S N 1), if
  n lt or estimated N

This looks a bit strange. ( Look at page 407 of
your text book ) 1) We would always have to use
S gt or N to maintain a positive CL. 2) As long
as n lt N, CL does not change. Wont you be
concerned if n got pretty close to N?
8
Confidence Level

• With the previous example of seeding, we found
  392 non-seeded defects. We estimated that there
  are a total of Z 436 defects.
• We claim that there are (436 392) 44 defects
  remaining in the software. What is our
  confidence?
• Lets put in 44 seeded defects and run this
  software until we find all 44 seeded defects.
• A) In the process assume we found 32 unseeded
  defects. Then our confidence level that estimated
  total remaining defects of N44 is
• S 44 and N 44, with n32 lt N
• CL 44/ (44 44 1) 44 / 1 44
  confidence level
• B) Assume we found 1 unseeded defects. Then our
  confidence level that the estimated remaining
  defects of N44 is
• n 1 lt N
• CL 44/ (44 44 1) 44 confidence level
• C) Assume that we found 53 unseeded defects. Then
  our confidence level that there are N44 defects
  is
• n 53 gt N 44
• CL 100

(Same n 1 or n 32 !! )
Take a look at the improved formula for CL on
page 408, taking into account that you may never
find all the S seeded defect. Yes --- but not
much better.
9
Multiple Team Testing of The Same Code

• Only in very special cases can a software
  organization afford to use multiple teams to test
  the same code - - - (life critical?)
• also assume that one group is just effective
  in finding defects in all parts of the code.

Assume that there are 2 test groups, T1 and T2,
who each have found x and y respectively from a
potential of n defects in the code. Also there is
a common number q from x and y that is the same
set of defects in x and y. Then the
effectiveness of T1 and T2 may be expressed as

E1 x / n
E2 y / n If T1 is uniform in
effectiveness in finding defects, then its
effectiveness in finding those defects that T2
found is the same. So, for T1, x / n q / y
and for T2, y / n q / x ( from a big
assumption above) Then we have E1 E2 (x /
n) (y / n) (q / y) (y / n) q / n
n q / (E1E2)
estimated total number of defects in the code
10
Example of 2 Teams testing

• Consider T1 found 34 defects and T2 found 45
  defects and there were 24 common defects found by
  both teams.

So, E1 34 / n and E2 45 / n. But E1 is also
24/45 and E2 24/34
E1 24/45 .533 53 effective E2
24/34 .705 71 effective n q / (E1
E2) 24/ (.53 .71) 24 / (.376) 63.8 or
64 So we estimate that the total number of
defects in the code is 64. T1 found 34 of the 64
and T2 found 45 of the 64. Does this satisfy the
Quality Goal where T2 found estimated 71 of
defects? There is an estimated of 64 45 19
more defects in the code. Is your support
maintenance team ready for this many defects?
11
Where were the defects found?

• Besides the numbers, we also know from before
  that buggy area tends to continue to be buggy
  - - -
• So if the defects found tend to concentrate in
  certain areas, continue testing with a more
  stringent quality goal for those areas.