SENG 521 Software Reliability

1
SENG 521Software Reliability Testing

• Preparing for Test
• (Part 6b)

Department of Electrical Computer Engineering,
University of Calgary B.H. Far (far_at_enel.ucalgary.
ca) http//www.enel.ucalgary.ca/far/Lectures/SENG
521/06b/
2
What is a Test Case?

• A Test Case is a partial specification of a run
  through the naming of its direct input variables
  and their values.
• Test case is specified with its direct input
  variables.
• Test case is independent of operational mode. The
  same test case can execute in different
  operational modes.

3
Procedure

• The procedure involves two steps
• Preparing test cases (Test case management)
• Using either field data recording or operational
  profiles
• Preparing test procedures
• Prepare one test procedure for each operational
  mode using operational profile and the operation
  occurrence rate

4
Test Cases Management

• The procedure for preparing test cases involve
• Estimate the number of new test cases needed for
  the current release
• Allocate the number of new test cases among the
  systems to be tested
• Allocate the number of new test cases for each
  system among its new operations
• Specify the new test cases
• Adding the new test cases to the ones already
  available (may be from a previous release)

5
1. Estimate New Test Cases

• Affected by two factors time and cost
• Method
• Compute the number of test cases for the given
• Time
• (available time X available staff)/
• (average time to prepare a test
  case)
• Cost
• (available budget) / (average preparation cost
  per test case)
• Select the minimum number of the two

6
2. Test Case Allocation System

• Allocate the bulk of the test cases to the
  product itself.
• Give weight to the differing operations that have
  high occurrence probabilities.
• Do not allocate test cases to acquired components
  unless the newly acquired components are
  substituted for previous ones.
• Allocate test cases to acquired components based
  on their size relative to the product and the
  estimated risk.

7
3. Test Case Allocation Operations

• Convert graphical representation of the
  operational profile to the tabular representation
  by walking through all the paths.
• Identify the rarely occurring critical new
  operations and determine how many test cases to
  pre-assign to each.
• Determine the allocation probabilities for the
  other new operations.
• Pre-assign one test case to each infrequent other
  new operation.
• Assign the remaining test cases to the remaining
  other new operations in accordance with the
  allocation probabilities.

8
Critical Operations

• A critical operation is one for which successful
  execution adds a great deal of extra value and
  failure causes a great deal of impact with
  respect to human life, cost, or system
  capability.
• Example SCRAM operation in nuclear power plants
  that shuts down a reactor when it starts to
  overheat. The operation that handles this is very
  rarely used, but it is extremely critical.
• Identify only the rarely occurring critical
  operations because sufficient test cases will be
  allocated to the frequently occurring operations
  because of their substantial occurrence
  probabilities.

9
New Operations

• In the case of a new release, set the allocation
  probabilities for the other new operations equal
  to the occurrence probabilities of the system
  operational profile.
• This is a satisfactory approximation the sum of
  the occurrence probabilities for the other new
  operations will be very close to 1.
• For subsequent releases, divide the system
  occurrence probabilities for the other new
  operations by the total of the system occurrence
  probabilities for the other new operations.

10
Infrequent Operations

• Infrequent operations are those that would not
  normally be assigned test cases by virtue of
  their occurrence probabilities.
• Assign each infrequent operation at least one
  test case.
• The occurrence probability of an operation is
  often such that, because of the fact that runs
  usually substantially outnumber test cases, the
  operation might well invoke a run even though the
  occurrence probability might not normally result
  in test case selection.
• Thus determine the threshold occurrence
  probability below which you would not normally
  assign a test case to an operation.
• This is 0.5 divided by the number of new test
  cases.
• We then pre-assign one test case to each other
  new operation whose occurrence probability falls
  below the threshold.

11
Example /1

• Number of test cases 500
• First release All test cases are new
• One critical operation with 2 test cases assigned
  to it
• Threshold 0.5 / 500 0.001
• Number of infrequent operations with occurrence
  probability below threshold 2
• Assign 1 test case to each
• Assign the remaining 496 test cases among the
  rest of operations based on their probability
  distribution

12
Example /2

• Example Occurrence probabilities for normal
  operation mode.

13
Example /3
14
4. Specify Test Cases

• Quantify the value of direct input variables
  based on levels for which the same behavior will
  be expected.
• A test case will be specified using a combination
  of levels of direct input variables.

15
Test Procedures

• Test procedure is a controller that sets up
  environmental conditions and invokes randomly
  selected test cases at random times.
• Prepare one test procedure for each operational
  mode.
• To specify the test operational profile for an
  operational mode, start with
• Operational mode operational profile (for the
  first release)
• Operational mode and test operational profile
  (for subsequent releases)
• Then modify the occurrence probabilities to
  account for rarely occurring critical new
  operations.

16
Example /1

• Unadjusted operational profile
• Operation mode Peak hour
• 2 test cases (out of 500) assigned to the rarely
  occurring critical operation

17
Example /2

• Occurrence probabilities for the rarely occurring
  critical operation 2/500 0.004
• Occurrence probabilities are adjusted to account
  for this.

18
Example /3

• Occurrence probabilities for the 2nd release
  after adding 2 new operations
• Operational interaction factor 0.2