Testing the System

1
Testing the System
ECE 668 Fundamentals of Computer
Engineering Module II Software Engineering
Andrzej Rucinski Khrystyna Pysareva Spring 2004
2
System Testing

• Unit Integration Testing vs. System Testing
• Accurate Code Implementation vs. Customer
  Requirements

3
Sources of Software Faults
4
System Testing Process
5
System Testing Checklist

• Function testing does the integrated system
  perform as promised by the requirements
  specification?
• Performance testing are the non-functional
  requirements met?
• Acceptance testing is the system what the
  customer expects?
• Installation testing does the system run at the
  customer site(s)?

6
Example A Telecommunications System That Routes
Calls

• Routing calls within an exchange
• Routing calls within an area code
• Routing calls within a state, province, or
  district
• Routing calls within a country
• Routing international calls

7
Configuration Management

• Definitions
• Version (or configuration)
• A collection of system components delivered to a
  particular customer
• One in a sequence of phased products
• Release improved system to replace the current
  one
• Production system (during a phased development)
  a version (2) that has been tested and performs
  according to only a subset of the customers
  requirements
• Development system the next upcoming version
  that is being built and tested at the moment
• Configuration management coordinates efforts
  among the testers and developers

8
Configuration Management

• Regression test applied to a new version or
  release to verify that it still performs the same
  functions in the same manner as an older version
  or release

9
Version/Release Control Techniques

• Separate files
• Deltas
• Conditional compilation

10
Test team

• Professional testers organize and run the tests
• Analysts who created requirements
• System designers understand the proposed
  solution
• Configuration management specialists to help
  control fixes
• Users to evaluate issues that arise

11
Function Testing

• Thread set of actions associated with a
  function (thread testing function testing)
• Choose the order in which functions are tested
  carefully
• Guidelines
• Use a test team independent of the designers and
  programmers
• Know the expected actions and output
• Test both valid and invalid input
• Never modify the system just to make testing
  easier
• Have stopping criteria
• Cause and effect graphs automatic generation of
  test cases from the requirements (IBM 1974).
  Produces a Boolean graph reflecting
  cause-and-effect relationships.

12
Performance Tests

• Stress tests
• Volume tests
• Configuration tests
• Compatibility tests
• Regression tests
• Security tests
• Timing tests

• Environmental tests
• Quality tests
• Recovery tests
• Maintenance tests
• Documentation tests
• Human factors (usability) tests

13
Reliability, Availability, Maintainability

• Software reliability the probability that the
  system will operate without failure under given
  condition for a given time interval (01)
• Software availability the probability that a
  system is functioning completely at a given
  instant in time, assuming that the required
  external resources are also available (1 system
  is up and running)
• Software maintainability the probability that,
  for a given condition of use, a maintenance
  activity can be carried out within a stated time
  interval and using stated procedures and
  resources

14
Levels of failure severity

• According to US Military standard
• Catastrophic a failure that may cause death or
  system loss
• Critical a failure that may cause severe injury
  or major system damage that results in mission
  loss
• Marginal a failure that may cause minor injury
  or minor system damage that results in delay,
  loss of availability, or mission degradation
• Minor a failure not serious enough to cause
  injury or system damage, but that results in
  unscheduled maintenance or repair

15
Evaluating Reliability, Availability, and
Maintainability

• MTTF Mean Time To Failure
• MTTR - Mean Time To Repair
• MTBF Mean Time Between Failures
• MTBF MTTF MTTR
• R MTTF/(1MTTF)
• A MTBF/(1MTBF)
• M 1/(1MTTR)

16
Acceptance Tests

• Benchmark test actual user prepares a set of
  test cases
• Pilot test install on experimental basis
• --------
• Alpha test in-house test
• Beta test customer pilot
• ---------
• Parallel testing new system operates in
  parallel with old system

17
Test Documentation
18
Problem Report Form

• Location Where did the problem occur?
• Timing When did it occur?
• Symptom What was observed?
• End result What were the consequences?
• Mechanism How did it occur?
• Cause Why did it occur?
• Severity How much was the user or business
  affected?
• Cost How much did it cost?