Software Testing Using Model Program

1
Software Testing Using Model Program

• DESIGN BY
• HONG NGUYEN SHAH RAZA
• Dec 05, 2005

2
Overview Software Testing Using Model Program

• Briefly introduction of M-mp testing
• What is M-mp testing?
• How it work?
• Why we need program base testing
• The problem of software testing
• Oracle test
• N-Version programming
• What is N-Version Programming?
• Disadvantages
• Advantages

3
Overview (continues)

• M-mp testing
• Framework for M-mp testing
• Asymptotic behavior of M-mp testing
• Input domain partitioning
• An M-mp Experiment
• Background and motivation
• Selecting the application
• Phases of the experiment
• Conclusion

4
Introduction

• What is M-mp testing?
• Testing using M Model program
• Abbreviated to M-mp testing
• It is alternative to software testing base on
  manual outcome prediction
• M-mp testing is base on N-Version programming
  technique
• Which is apply specially in the well-known
  software fault tolerant technique.

5
Introduction (Continues)

• How it work
• Model program implements selected part of
  functional specification of the software to be
  tested.
• The M-mp testing strategy requires that M (Mgt 1)
  
• Model programs as well as the program under test,
  should be independently developed
• P and M model program are then subjected to the
  same test data.
• Difference analysis is conducted on the outputs
  and appropriate corrective action taken
• P and the M model program jointly constituted an
  approximate test oracle (out come correct or not)

6
Introduction (Continues)

• Why we need program base testing
• For small number of test data, it may be feasible
  to work out the expected output manually
• For thorough testing which required large test
  datasets manual testing could be time consuming
  and error prone
• That is why M-mp base testing may be more
  feasible for those kinds of data set

7
The problem of software testing

• The difficult task in software testing is to
  assess the correctness to the outcome of a
  program that is subjected to particular test
  input
• In testing theory, the mechanism to adjudicate on
  whether or not an output is correct, somewhat
  neglected this problem is referred to as the
  oracle problem.
• Testing theory concern with the choice of test
  and testing methods, usually ignore the oracle
  problems.

8
The problem of software testing (continues)

• Oracle Test
• It means of determining whether a program passed
  or failed a test.

9
The problem of software testing (continues)

• Oracle test (continues)
• P is the program to be tested
• Test strategy determines the set of test data to
  be used
• To get the test result we compare Ps output with
  the oracle s output
• For small out put oracle manually base oracle
  technique is feasible
• But for large industry base application manually
  base oracle testing is not feasible
• One of alternative of manually base oracle is N-
  Version programming

10
N-Version Programming

• N-Version is comprises N independently written
  version of the software
• All has the same functional specification
• At runtime voting base on majority agreement is
  used to decided the probable output
• Disadvantage
• Concern in N-Version systems is the fact that
  correlated failures may limit the practical gain
  in reliability
• N-Version systems should not be casually assumed
  to be completed reliable.

11
N-Version (continues)

• Disadvantage (continues)
• Is the increased development cost because at
  least three version being required for a voting
  system to work
• Advantage
• N-Version design appear to offer more reliability
  that we can gain from any other way
• Even though the cost of failure is high
• It would be more cost effective to build
  N-Version systems rather than focusing on
  building one good version

12
M-mp Testing

• Framework for M-mp testing

13
M-mp Testing (continues)

• Framework (continues)
• In M-mp testing P is primary program (the program
  under test)
• mp1 mpM are M so called model program of P
• M-mp testing treat P and mp1-mpM to be same test
  input
• Any disagreement on the output indicates the
  presence of specification defects Or software
  faults in at least one of program
• Once defects are detected and removed. The
  program are re-run

14
M-mp Testing (continues)

• Framework (continues)
• The cycle is repeat until all disagreements for a
  particular dataset are resolved.
• Asymptotic behavior of M-mp testing
• Difference between the M-mp and manual approaches
  to testing is the cost of outcome verification

15
M-mp Testing (continues)

• Asymptotic behavior of M-mp testing (continues)
• On the graph the cost of M-mp outcome
  verification is high initially but it increase
  only slightly.
• The cost growth is modeled as linear
• In manual approach, the growth of the output
  verification cost may be expected to be much
  steeper.
• Input domain partitioning
• Divided into a standard domain and an exception
  domain
• Exception domain may be split into an incomplete
  data domain and an invalid data domain.
• The standard domain consists of inputs that a
  program can process as is

16
M-mp Testing (continues)

• Input domain partitioning (continues)
• The invalid domain contains the input that a
  program can not process it should be rejected
  with appropriate error message
• The incomplete data domain is between the
  standard and invalid data domains
• It is made up of inputs that the program has to
  completed with default values before processing
  them
• An M-mp Experiment
• Background and motivation

17
M-mp Testing (continues)

• Background and motivation (continues)
• The primary program is a scheduling application,
  a non-interactive data processing subsystem of a
  large logistics management system
• The program can be characterized as
  algorithmically complex
• In practice, the testing process in the company
  was well defined and supported by standards and
  procedures but had several drawback
• Designing test cases based on manual outcome
  prediction was not only difficult, error prone
  and time consuming
• But consequently also unattractive and
  demoralizing

18
M-mp Testing (continues)

• Selecting the application
• The scheduling application was chosen as the
  primary program for following reasons
• Developer was not familiar with the scheduling
  domain and application
• The cost of developing from scratch could be
  assesses more authentically
• The risk of correlated failures caused by common
  design error was reduces
• scheduling application is representative of the
  family of operation management program
• The time the experiment started, the scheduling
  subsystem had been in use for more than year
• This seem a good opportunity to test the fault
  detection ability of M-mp testing, as the
  likelihood of finding defect was expected to be
  low

19
M-mp Testing (continues)

• Phase of the experiment

20
M-mp Testing (continues)

• Phase of the experiment
• The experiment entailed a test design, a test
  execution and test evaluation phase as depicted
• Test design phase comprised test data selection
• Which includes activities related to test data
  generation, Model program design and model
  program implantation
• The test execution phase includes activities
  relating to setting up the test environment and
  executing the program
• The test evaluation phase includes the
  disagreement analysis procedures whereby
  disagreements between the program need to be
  arbitrated.

21
Conclusion

• M-mp approach could test a scheduling program
  more adequately than manually designed test and
  at lower cost
• M-mp testing should be judged on a case by case
  basic
• M-mp approach will tend to be better option than
  the manual one when the number of tests required
  to achieve a particular adequacy level become
  large

22
Question

• Finished
• Please be easy on us.Thank you