Test vs. inspection Part 1

1
Test vs. inspectionPart 1

• Tor Stålhane

2
What we will cover

• Part 1
• Introduction
• Inspection processes
• Testing processes
• Part 2
• Tests and inspections some data
• Inspection as a social process two experiments
  and some conclusions

3
Introduction
4
Adams data - 1
Mean Time to Problem Occurrence years Mean Time to Problem Occurrence years Mean Time to Problem Occurrence years Mean Time to Problem Occurrence years Mean Time to Problem Occurrence years Mean Time to Problem Occurrence years Mean Time to Problem Occurrence years Mean Time to Problem Occurrence years
Product 1.6 5 16 50 160 500 1600 5000
1 0.7 1.2 2.1 5.0 10.3 17.8 28.8 34.2
2 0.7 1.5 3.2 4.3 9.7 18.2 28.0 34.3
3 0.4 1.4 2.8 6.5 8.7 18.0 28.5 33.7
4 0.1 0.3 2.0 4.4 11.9 18.7 28.5 34.2
5 0.7 1.4 2.9 4.4 9.4 18.4 28.5 34.2
6 0.3 0.8 2.1 5.0 11.5 20.1 28.2 32.0
7 0.6 1.4 2.7 4.5 9.9 18.5 28.5 34.0
8 1.1 1.4 2.7 6.5 11.1 18.4 27.1 31.9
9 0.0 0.5 1.9 5.6 12.8 20.4 27.6 31.2
5
Adams data 2

• The main information that you get from the table
  on the previous slide is that
• Some defects are important because they will
  happen quite often.
• Most defects are not important since they will
  happen seldom.
• How can we tell the difference?

6
Testing alone is not the solution

• As can be seen from the next slide, testing is
  not an acceptable solution alone. It will
• Take too long time
• Cost too much
• We can generate tests automatically, but would
  never the less have to use large resources to
  check the result the oracle problem

7
A limit result

• The following relation holds under a rather wide
  set of conditions
• The initial number of defects N0 must be
  estimated e.g. based on experience from earlier
  projects as number of defects per KLOC.

8
An example from telecom
9
Testing and inspection the V model
10
Testing and inspection 1

• The important message here is that testing cannot
  always be done.
• In the first, important phases, we have nothing
  to execute and will thus always have to do some
  type of inspection.
• This might be considered one of the weaknesses of
  traditional software engineering over Agile
  development.

11
Testing and inspection 2

• In order to understand the main differences
  between testing and inspection, we should
  consider Fits list.
• Based on this, we will give a short discussion of
  the relative merits of testing and inspection.

12
Area of competence Man Machine
Understanding Good at handling variations in written material Bad at handling variations in written material
Observe General observations, multifunctional Specialized, good at observing quantitative data, bad at pattern recognition
Reasoning Inductive, slow, imprecise but good at error correction Deductive, fast, precise but bad error correction
Memory Innovative, several access mechanisms Copying, formal access
Information handling Single channel, less than 10 bits per second Multi channel, several Megabits per second
Consistency Unreliable, get tired, depends on learning Consistent repetition of several actions
Power Low level, maximum ca. 150 watt High level over long periods of time
Speed Slow seconds Fast
13
Man vs. machine 1

• Good when we need the ability to
• Handle variation
• Be innovative and inductive
• Recognize and handle patterns
• Not so good when we need the ability to
• Do the same things over and over again in a
  consistent manner
• Handle large amount of data

14
Man vs. machine 2

• In order to do the best job possible we need
  processes where we let each part
• Do what they are best at
• Man is innovative
• Machine handles large amounts of data
• Support the other with their specialties.
• Machine supports man by making large amounts of
  information available
• Man support machine by providing it with
  innovative input

15
General considerations - documents

• Architecture, system, sub-system and component
  design plus pseudo code. Here we can only use
  inspections.
• Man will use experience and knowledge to identify
  possible problems
• Machine can support by identifying information
  e.g. find all occurrences of a string.

16
General considerations code (1)

• For executable code, we can use inspection,
  testing or a combination of both.
• The size and complexity degree of dynamism of
  the code will, to a large degree, decide our
  choice.
• Other important factors are the degree of
  experience with
• The programming language
• The algorithms used

17
General considerations code (2)

• Simple code
• Start with inspection all code
• Design and run tests
• Complex code
• Start with inspection focus on algorithm and
  logic
• Decide test completeness criteria we cannot
  test everything
• Design and run tests

18
Inspection processes
19
Inspections 1

• The term inspection is often used in a rather
  imprecise manner. We will look at three types of
  inspection
• Walkthrough
• Informal inspection also called informal review
• Formal inspection also called formal review or
  just inspection
• The first two types are usually project internal
  while the last one is used as a final acceptance
  activity for a document.

20
Inspections 2

• For all types of inspections
• The quality of the results depends on the
  experience and knowledge of the participants.
  Garbage in Garbage out
• It might be a good idea to involve customer
  representatives.

21
The walkthrough process

• Walkthrough is a simple process mostly used for
  early decisions for an activity. The document
  owner
• Makes a rough sketch of the solution
  architecture, algorithm etc.
• Presents explain the sketch to whoever shows
  up.
• Registers feedback improvements.

22
Walkthrough pros and cons

• Pros
• Easy and inexpensive. Needs no extra preparation.
• Collect ideas at an early stage of development.
• Cons
• No commitment from the participants
• May collect many loose or irrelevant ideas

23
The informal inspection process
24
Informal inspections pros and cons

• Pros
• Is simple and inexpensive to perform.
• Can be used at all stages of development
• Usually has a good cost / benefit ratio
• Needs a minimum of planning
• Cons
• No participant commitment
• No process improvement

25
The formal inspection process

• The formal inspection process described below is
  with small variations the most commonly used.
  The version shown on the following slides stem
  from T. Gilb and D. Graham.
• We recommend this process as the final acceptance
  process for all important documents

26
Formal inspection process overview
27
Distribution of resources
Activity Range Typical value
Planning 3 5 4
Kick-off 4 7 6
Individual checking 20 30 25
Logging 20 30 25
Editing 15 30 20
Process brainstorming 15 30 16
Leader overhead, follow up, entry, exit 3 5 4
28
Initiating the inspection process

• The inspection process starts with a request for
  inspection from the author to the QA
  responsible.
• The QA responsible appoints an inspection leader.
• First step is always to check that the document
  is fit for inspection.

29
Planning

• Important planning points are
• Who should participate in the inspections
• Who is interested?
• Who have time available for preparation and
  meetings?
• Who has the necessary knowledge concerning
  application, language, tools, methods?

30
Kick-off

• Important activities here are
• Distribution of necessary documents
• Documents that shall be inspected
• Requirements
• Applicable standards and checklists
• Assignment of roles and jobs
• Setting targets for resources, deadlines etc.

31
Individual checking

• This is the main activity of the inspection. Each
  participant read the document to look for
• Potential errors - inconsistencies with
  requirements or common application experience
• Lack of adherence to company standards or good
  workmanship

32
Logging meeting

• The logging meeting has three purposes
• Log issues already discovered by inspection
  participants
• Discover new issues based on discussions and new
  information that arises during the logging
  meeting.
• Identify possible improvement to the inspection
  or development process.

33
Improve the product - 1

• The author receives the log from the inspection
  meeting. All items - issues - in the log are
  categorised as one of the following
• Errors in the authors document.
• Errors in someone elses document.
• Misunderstandings in the inspection team.

34
Improve the product - 2

• Errors in own documentMake appropriate
  corrections
• Errors in someone elses documentsInform the
  owner of this document.
• Misunderstandings in the inspection teamImprove
  document to avoid further misunderstandings.

35
Checking the changes

• This is the responsibility of the inspection
  leader. He must assure that all issues raised in
  the log are disposed of in a satisfactory manner
• The documents that have been inspected
• Related documents - including standards and
  checklists
• Suggested process improvements

36
Formal inspection pros and cons

• Pros
• Can be used to formally accept documents
• Includes process improvement
• Cons
• Is time consuming and expensive
• Needs extensive planning in order to succeed

37
Testing processes
38
Testing

• We will look at three types of testing
• Unit testing does the code behave as intended.
  Usually done by the developer
• Function verification testing also called
  systems test. Does the component or system
  provide the required functionality?
• System verification testing also called
  acceptance test. Does the hardware and software
  work together to give the user the intended
  functionality?

39
The unit testing process

• Unit testing is done by the developer one or more
  times during development. It is a rather informal
  process which mostly run as follows
• Implement (part of) a component.
• Define one or more tests to activate the code
• Check the results against expectations and
  current understanding of the component

40
Unit testing pros and cons

• Pros
• Simple way to check that the code works.
• Can be used together with coding in an iterative
  manner.
• Cons
• Will only test the developers understanding of
  the spec.
• May need stubs or drivers in order to test

41
The system test process

• A systems test has the following steps
• Based on the requirements, identify
• Test for each requirement, including error
  handling
• Initial state, expected result and final state
• Identify dependencies between tests
• Identify acceptance criteria for test suite
• Run tests and check results against
• Acceptance criteria for each test
• Acceptance criteria for the test suite

42
Systems test pros and cons

• Pros
• Tests systems behavior against customer
  requirements.
• Cons
• It is a black box test. If we find an error, the
  systems test must be followed by extensive
  debugging

43
The acceptance test process

• The acceptance test usually has three activities
  all involving the customer or his
  representatives
• Rerun the systems test at the customers site.
• Use the system to solve a set of real-world
  tasks.
• Try to break the system by stressing it or by
  feeding it large amounts of illegal input

44
Acceptance test pros and cons

• Pros
• Creates confidence that the system will be useful
  for the customer
• Shows the systems ability to operate in the
  customers environment
• Cons
• Might force the system to handle input that it
  was not designed for, thus creating an
  unfavorable impression.