A Guest Lecture

1
CSE 494 Software Testing in Real-Time Systems
A Guest Lecture On Testing-Based Software
Reliability Modeling
Yinong Chen
2
Terminology
Reliability
Availability
Safety
Attributes
Confidentiality
Data integrity
Maintainability
Prevention (Methodology)
Tolerance (Redundancy)
Means
Dependability
Software Hardware
Removal (Testing)
Forecasting (Modeling)
Fault (defects, bugs)
Impairments
Error (Manifestation of fault)
Trustworthiness
Failure (impact to output)
Business Integrity
3
Failure Probability and Reliability Function
Let T be a random variable for the time interval
from time 0 to the time of the first
failure. The Cumulative Distribution Function
F(t) ProbT t. The Probability Density
Function f(t) dF/dt. F(t) is called failure
probability in the interval 0, t. Reliability
function R(t) of a system is the probability that
the system has survived in the time interval 0,
t, given that it is operational at time 0. R(t)
1 F(t) 1 ProbT t ProbT gt t.
4
Reliability Function and Failure Rate
The failure rate, z(t), is defined as It is the
rate at which the system changes from working
state to failed state at time t. z(t) Dt is
the probability that the system is in working
state at time t, but in failed state at time t
Dt.
5
Reliability Function for Computer Hardware
?
6
Availability Function and Repair Rate
Availability A(t) of a system is the probability
that the system is working at time t. The repair
rate m(t) is the rate at which the system changes
from failed state to working state at time
t. m(t) Dt is the probability that the system
is in failed state at time t, but in working
state at time t Dt.
7
Availability Function and Repair Rate
Availability is the probability that a system
stays in a "working" state. A(t Dt) A(t) -
A(t)z(t)Dt (1 A(t))m(t)Dt
8
Example
For l 10-4/hour, R(t) e-lt , MTTF 1/l
104 hours
A(t)
1
0.9
0.8
0.7
R(t)
0.6
0.5
Probability
0.4
0.3
MTTF
0.2
0.1
0
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
13000
14000
15000
16000
17000
18000
19000
20000
Time in Hours
9
Software Reliability Models Overview
Reliability Models
Design phase
Testing debugging phase
Operation phase
Validation phase
(reliability predication)
(reliability estimation)
(reliability assessment)
(reliability validation)
10
Time-Between-Failures (Time Domain) Models
Software reliability is usually defined as the
probability R(t)  Probno of failures within
time period 0, t where t is the exposure
period whose time unit is the calendar or CPU
time. R(t) is assumed to follow certain
probability distribution, for example, R(t)
e-z(t), where z(t) is the failure rate. The
main concern of these reliability growth models
is to estimate the value of the failure rate
function z(t) by testing.
11
Shooman Model Shooman 1973
Error model where Er number of errors remains
at time T E0 number of errors at T 0 in the
program under test I0 number of
instructions Ec number of errors corrected in
the time interval 0, T E0, I0, and Ec are
decided by testing
12
JM Model Jelinski and Moranda 72

• Assumption of the model
• There are N independent software faults in the
  program at the beginning of testing
• Each fault is equally likely to cause a failure
  during testing
• A detected fault is removed with certainty in a
  negligible time and no new faults are introduced
  during testing and debugging process
• The software failure rate at time t, after the
  ith fault is removed, is proportional to the
  current faults and is given by

where, f is a constant
13
Fault Count An Axiomatic Model

• Software reliability is postulated to obey
  certain universal laws.
• One of the well-known models Software Science
  Model
• The number of bugs
• where
• K constant
• V volume of program
• E0 number of errors at T 0 in the program
  under test
• Axiomatic models a special issue IEEE Trans.
  Soft. Eng. 1979.

14
Fault Seeding Model
How do we estimated the number of fishes in a
lake?
1) Insert a certain number of faults into the
program under test 2) Test the program How
many detected faults are inserted? How many
detected faults are original? How many original
faults are detected? 3) Calculate the total
number of original faults according statistics.
15
Example
X Original faults (unmarked fishes) in the
program under test The goal is to estimate X. Y
faults (marked fishes) are seeded/inserted into
the program We test the program and we found M
N faults, where M Original faults
found N Seeded faults found Assume that the
probability of finding an original fault and a
seeded fault is the same, then
M Y N
M N X Y
X

16
Basic Input Domain Models
Examples MacWilliams73, BrownLipow75, Nelson78.
Software reliability is defined as the
probabilityR(N) Probno of failures over N
application runs where N is the exposure period
whose time unit is the number of application
runs. Assuming that input cases are selected
independently, then R(N) can be expressed
as R(N) (R(1))N RN where, R ? R(1) is the
expected reliability per application run. Now
the question is how to estimate R.
17
Estimate R
R, the reliability per test run, can be defined
by the ratio of the number of test runs in which
failures are observed and the total number of
test runs when infinite number of different input
cases are applied for test runs R 1 F 1
Because of test time limit only a subset of
the entire input domain can be applied to test
the program in practice. Thus the reliability per
test run, R, is usually estimated by
18
Example
Test the program 10 000 time (test runs) Five (5)
failures are observed F 5 / 10 000 1 / 2
000 R 1 - 1 / 2 000 0.9995 Reliability in N
application runs are R(N) RN
19
MacWilliams 73 and BrownLipow 75 With partition
and profiling
MacWilliams 73 the s input cases are selected
randomly from the input domain. BrownLipow 75
the input domain is partitioned into m classes.
If si input cases are selected from class Ci and
fi failures are observed, the reliability can be
calculated by where P(Ci) is a probability
function reflecting the input profile in terms of
classes.
20
Example
The input domain is partitioned into 10
sub-domains.



-

1 0.020938 0.979063
R
1
F
21
Case Study 1Anti-lock Braking System (ABS)

• Requirement
• To obtain the maximum braking effect
• Algorithm
• Define (or measure) the wheel diameter
• Measure the wheel rotations per seconds rps
• Compute the wheel velocity wv
• Measure the body velocity bv
• Error detection and action
• if (bv gt wv), reduce braking force
• else if (bv lt wv), reduce acceleration force
• else no action

22
Sample Code in C
include ltiostreamgt using namespace std const
float mile_inch 63360 const float pi
3.1416 float wheel_diameter 15 //
inches float wheel_sensor() float rps cout
ltlt "get rotations per second " ltlt endl rps
ReadWheelRotationSensor() return rps
23
float wheel_velocity(float rps) float wv wv
(pi wheel_diameter rps 3600)/mile_inch r
eturn wv float body_velocity() float
bv cout ltlt "get miles per hour " ltlt endl bv
ReadBodySpeedSensor() return bv
24
void error_detection(float wv, float bv) if
(abs(bv - wv) lt 0.01) cout ltlt "no action" ltlt
endl else if (bv gt wv) cout ltlt "reduce
brake force!" ltlt endl else cout ltlt "reduce
acceleration force!" ltlt endl void evaluation()
float rps, wv, bv rps wheel_sensor() wv
wheel_velocity(rps) bv body_velocity() err
or_detection (wv, bv) void main() for (i
1, I lt 10000, i) evaluation()
25
Apply the input domain reliability model

• Write the random function that simulates
  ReadWheelRotationSensor()
• Write the random function that simulates
  ReadBodySpeedSensor()
• Test the program and collect date
• Apply the input domain model to evaluate the
  reliability of the program

26
Testing the greatest common divisor program
include ltstdio.hgt int gcd (int n0, int m0) //
n0 ? 0, m0 ? 0 int n, m // n0 or m0 ? 0 n
n0 m m0 while (n ! 0 n ! m) if (n lt
m) m m - n else n n - m return
m void main() int i, j, k scanf("d\nd",
i, j) //input k gcd(i, j) // call
gcd printf("d\n", k) // output
27
Random Test Case Generation
Input Output k (i, j) (6, 9) 3 (i, j) (10,
5) 5 (i, j) (0, 4) 4 (i, j) (5, 7) 1 (i, j)
(8, 29) 1
The program produces correct outputs for all
these different test cases. Can we claim the
program is correct?
28
Example Input Domain Partition Analysis

• Input Domain Analysis
• The program takes two integers as input.
• The branches of the program are controlled by the
  relative values of the two integers.
• Input Domain Partitioning
• Partition each integer input into three groups
  lt 0, 0, and gt 0.
• For this program, lt 0 is not allowed. The group
  that has only one value is called boundary value.
• Considering the semantics of the program, prime
  numbers and nonprime numbers play a role in the
  program. Thus, the positive integers are further
  divided into prime and nonprime numbers.

29
Input Domain Partition
Based on the analysis, we have a partition of
i 0 2, 3, 5, 7, 11, 4, 6, 8, 9, j
0 2, 3, 5, 7, 11, 4, 6, 8, 9, The
combination of the two inputs generates
following cases (0, 0) This case is not allowed
according to the specification. (0, 2), (0, 3),
(0, 5), (0, 7), (0, 11), (0, 4), (0, 6), (0,
8), (0, 9), (2, 0), (2 2), (2, 3), (2, 5), (2,
5), (2, 4), (2, 6), (2, 8), (2, 9), (3, 0),
(3, 2), (3, 3), (3, 5), (3, 7), . . . (9, 0),
(9, 2), (9, 3), (9, 5), (9, 7),
30
Coverage Consideration Branch Coverage
?
?
no
n ! 0?
?
yes
?
no
n ! m?
yes
?
?
no
n lt m?
?
yes
m m - n
return m
n n - m
?
?
?
Exit
(0, 2),(0, 3),(0, 9),(0, 10) ??? (2 2),(3, 3),
(9, 9),(10, 10) ??? (2, 3),(2, 9), (2,
10) ?????... (3, 2),(9, 2),10, 2),(10, 3),(10,
9) ?????... (2, 0),(3, 0), (9, 0),(10,
0) ?????...
31
Testing the Program Using Test Cases
Inputs Output k (i, j) (0, 2) 2 (i, j) (2,
2) 2 (i, j) (2, 3) 1 (i, j) (3, 2) 1 (i, j)
(2, 0) ?
32
Apply the partition-based input domain
reliability model

• Modify the main program, so that it
  systematically generate (large number of) input
  cases from different sub-domain
• Test the gcd function and collect date
• Apply the partition-based input domain model to
  evaluate the reliability of the program

33
SUMMARY SOFTWARE RELIABILITY MODELS

• Basic concepts and terminology
• Faults and failure rates
• Reliability R(t) and availability A(t)
• Software reliability models
• Classifications
• Time-domain models
• Fault count model
• Input-domain models
• Fault seeding model
• Sample code of real-time ABS software
• Sample code of real-time ABS software
• Partition-based input domain testing