How does software fail

1
How does software fail?

• Wrong requirement not what the customer wants
• Missing requirement
• Requirement impossible to implement
• Faulty design
• Faulty code
• Improperly implemented design

2
Terminology

• Fault identification what fault caused the
  failure?
• Fault correction change the system to remove
  faults
• Fault removal take out the fault

3
Types of faults

• We need to identify the types faults to help us
  locate them.

4
Types of faults

• Algorithmic fault
• a. Branching too soon or too late
• b. Test wrong condition
• c. Forgetting to initialize variables, set
  loop
• variables, test a particular condition
  
• like division by zero
• d. Comparing variables of inappropriate
  types

5
Types of faults cont.

• Syntax fault uncaught syntax by compiler
• Computation and precision fault due to mixing
  data types
• Documentation fault
• Stress or overload fault data structures fill
  passed capacity
• Capacity or boundary fault When using more than
  the required limit, of devices.

• Timing or coordination fault Inaccurate
  coordination code
• Throughput or performance fault Below
  anticipated level of performance.
• Recovery fault from failure
• Hardware and system software fault
• Standards and procedures fault

6
Orthogonal classification

• Orthogonal classification of faults.
• faults are classified to belong to exactly
  one category. It helps identify faults of
• a) omission when part of the code is missing.
  int x ( ????) x not initialized variable
• b) commission int x 6.781 x assigned wrong data
  type.

7
Table 8.1. IBM orthogonal defect classification.
Fault type
Meaning
Function
Fault that affects capability, end-user
interfaces, product interfaces,
interface with hardware architecture, or global
data structure
Interface
Fault in interacting with other components or
drivers via calls, macros,
control blocks or parameter lists
Checking
Fault in program logic that fails to validate
data and values properly before
they are used
Assignment
Fault in data structure or code block
initialization.
Timing/serialization
Fault that involves timing of shared and
real-time resources
Build/package/merge
Fault that occurs because of problems in
repositories, management changes,
or version control
Documentation
Fault that affects publications and maintenance
notes
Algorithm
Fault involving efficiency or correctness of
algorithm or data structure but
not design
8
Hewlett-Packard fault Classification

• Page 336
• It depends on the origin of fault, type of fault
  and mode of fault.

9
Test Organization
Unit test
Unit test
Integration test
Functiontest
Performance test
Acceptance test
Installation test
Unit test
Other Software requirement
System functional Requirement From specification
Installation test
Design specification
Acceptance test
10
Testing

• Attitude towards testing Every aspect of program
  works. Egoless testing encouraged
• Who performs testing A separate group from the
  programming staff.
• The choice between closed and open-box views of
  testing.

11
Unit testing

• Code review By technical experts
• (does not include the
  customer)
• Code walkthrough informal review where
  programmer leads and control discussion
• Code inspection formal review of code against a
  list of concerns.
• a) Introduction meeting review group
  meets the program team
• b) group looks for faults in code
• c) group discuss faults with the
  programmer.

12
Table 8.2. Typical inspection preparation and
meeting times.
Development artifact
Preparation time
Meeting time
Requirements document
25 pages per hour
12 pages per hour
Functional specification
45 pages per hour
15 pages per hour
Logic specification
50 pages per hour
20 pages per hour
Source code
150 lines of code per hour
75 lines of code per hour
User documents
35 pages per hour
20 pages per hour
Table 8.3. Faults found during discovery
activities.
Discovery activity
Faults found per thousand lines of code
Requirements review
2.5
Design review
5.0
Code inspection (gets the most faults)
10.0
Integration test
3.0
Acceptance test
2.0
13
Proving code correct

• Done after code inspection
• Formal proof techniques Using logical series of
  step.
• Symbolic execution involves simulated execution
  of code using symbols
• Automated theorem-proving using an automated
  theorem prover. It accepts inputs data, output
  data and code component to determine problem
  correctness.

14
To test code thoroughly

• Statement testing all statements are tested
• Branch testing test all branches
• Path testing check all paths
• Definition-use path testing Every path from
  every definition of every variable to every use
  of that definition is exercised in some test.
• All-uses testing The test set includes at least
  one path from every definition to every use that
  can be reached by that definition.
• All-predicate-uses/some-computational-uses
  testing
• All-computational-uses/some-predicate-uses testing

15
Example on Page 354
Statement testing 1-2-3-4-5-6-7 Branch
testing may be 1-2-3-4-5-6-7
1-2-4-5-6-1 Path testing
1-2-3-4-5-6-7
1-2-3-4-5-6-1
1-2-4-5-6-7
1-2-4-5-6-1
16
Comparing techniques
Table 8.5. Fault discovery percentages by fault
origin.
Discovery technique
Requirements
Design
Coding
Documentation
Prototyping
40
35
35
15
Requirements review
40
15
0
5
Design review
15
55
0
15
Code inspection
20
40
65
25
Unit testing
1
5
20
0
Table 8.6. Effectiveness of fault discovery
techniques. (Jones 1991)
Requirements
Design faults
Code faults
Documentation
faults
faults
Fair
Excellent
Excellent
Good
Reviews
Prototypes
Good
Fair
Fair
Not applicable
Testing
Poor
Poor
Good
Fair
Poor
Poor
Fair
Fair
Correctness
proofs
17
Integration testing

• Bottom-up Use component driver to call and test
  component
• Top-down Use stubs to simulate the missing
  components that is been called.
• Big-bang Testing and combining all components at
  the same time.

18
Integration test cont

• Sandwich testing A combination of bottom-up and
  top-down approaches.
• Modified top-down components are tested
  individually at each level before merging.
• Modified sandwich Upper level components are
  tested before merging them with others

19
Table 8.7. Comparison of integration strategies.
Bottom-up
Top-down
Modified
Big-bang
Sandwich
Modified
top-down
sandwich
Integration
Early
Early
Early
Late
Early
Early
Time to
Late
Early
Early
Late
Early
Early
basic
working
program
Component
Yes
No
Yes
Yes
Yes
Yes
drivers
needed
Stubs
No
Yes
Yes
Yes
Yes
Yes
needed
Work
Medium
Low
Medium
High
Medium
High
parallelism
at beginning
Ability to
Easy
Hard
Easy
Easy
Medium
Easy
test
particular
paths
Ability to
Easy
Hard
Hard
Easy
Hard
Hard
plan and
control
sequence
20
Example
The figure below illustrates the component
hierarchy in a software system. Describe the
sequence of tests for integrating the components
using Buttom-up Top-down Modified
top-down Big-bang Sandwich Modified sandwich
A
E
B
C
D
J
H
G
F
I
K
L
M
N
21
(No Transcript)
22
Test planning

• Establish test objectives
• Design test cases
• Write test cases
• Test test cases
• Execute tests
• Evaluate test results

23
Automated testing tools

• Code analysis
• Static analysis
• code analyzer
• structure checker
• data analyzer
• sequence checker
• Dynamic analysis
• program monitor

• Test execution
• Capture and replay
• Stubs and drivers
• Automated testing environments
• Test case generators

24
When to stop testing

• Coverage criteria
• Fault seeding
• Confidence in the software