Verification and Validation

1
Verification and Validation

• Reference Software Engineering, Ian
  Sommerville, 6th edition, Chapter 19

2
Objectives

• To introduce software verification and validation
  and to discuss the distinction between them
• To describe the program inspection process and
  its role in V V
• To explain static analysis as a verification
  technique

3
Verification vs. Validation

• Verification "Are we building the product
  right?"
• The software should conform to its specification.
• Validation "Are we building the right
  product?"
• The software should do what the user really
  requires.

4
The V V Process

• Is a whole life-cycle process - V V must be
  applied at each stage in the software process.
• Example Peer document reviews
• Has two principal objectives
• The discovery of defects in a system
• The assessment of whether or not the system is
  usable in an operational situation

5
V V Goals

• Verification and validation should establish
  confidence that the software is fit for its
  purpose.
• This does NOT mean completely free of defects.
• Rather, it must be good enough for its intended
  use. The type of use will determine the degree
  of confidence that is needed.

6
V V Confidence

• Depends on the systems purpose, user
  expectations, and marketing environment
• System purpose
• The level of confidence depends on how critical
  the software is to an organization (e.g., safety
  critical).
• User expectations
• Users may have low expectations of certain kinds
  of software
• Marketing environment
• Getting a product to market early may be more
  important than finding defects in the program

7
Static vs. Dynamic V V

• Code and document inspections - Concerned with
  the analysis of the static system representation
  to discover problems (static v v)
• May be supplement by tool-based document and code
  analysis
• Software testing - Concerned with exercising and
  observing product behavior (dynamic v v)
• The system is executed with test data and its
  operational behavior is observed

8
Program Testing (Dynamic V V)

• Can reveal the presence of errors, NOT their
  absence
• A successful test is a test that discovers one or
  more errors.
• The only validation technique for non-functional
  requirements
• Should be used in conjunction with static
  verification to provide full V V coverage

9
V V Planning

• Careful planning is required to get the most out
  of inspection and testing processes.
• Planning should start early in the development
  process.
• The plan should identify the balance between
  static verification and testing.

10
The V-model of Testing
11
The Structure of a Test Plan

• The testing process
• Requirements traceability
• Tested items
• Testing schedule
• Test recording procedures
• Hardware and software requirements
• Constraints

12
Code Inspections (Static V V)

• Involves examining the source code with the aim
  of discovering anomalies and defects
• Defects may be logical errors, anomalies in the
  code that might indicate an erroneous condition
  (e.g., an uninitialized variable), or
  non-compliance with coding standards.
• Intended for defect detection, not correction
• Very effective technique for discovering errors
• Saves time and money the earlier in the
  development process an error is found, the better

13
Inspection Success

• Many different defects may be discovered in a
  single inspection, whereas with testing one
  defect may mask another so that several
  executions/tests are required.
• Inspections reuse domain and programming
  knowledge so reviewers are likely to have seen
  the types of errors that commonly arise.

14
Inspections and Testing

• Inspections and testing are complementary and not
  opposing verification techniques.
• Both should be used during the V V process.
• Inspections can check conformance with a
  specification, but not conformance with the
  customers real requirements.
• Inspections cannot check non-functional
  characteristics such as performance, usability,
  etc.

15
Inspection Preparation

• A precise specification must be available.
• Team members must be familiar with the
  organizations standards.
• Syntactically correct code must be available.
• An error checklist should be prepared.
• Management must accept that inspection will
  increase costs early in the software process.
• Management must not use inspections for staff
  appraisal.

16
Ethics Question
A manager decides to use the reports of program
inspections as an input to the staff appraisal
process. These reports show who made and who
discovered program errors. Is this ethical
management behavior? Would it be ethical if the
staff were informed in advance that this would
happen? What difference might it make in the
inspection process?
17
Inspection Procedure

• The inspection procedure is planned.
• A system overview is presented to the inspection
  team.
• Code and associated documents are distributed to
  inspection team in advance.
• Inspection takes place and discovered errors are
  noted.
• Modifications are made to repair discovered
  errors.
• Re-inspection may or may not be required.

18
Inspection Teams

• Made up of
• Author of the code being inspected
• Inspector who finds errors, omissions, and
  inconsistencies
• Reader who reads the code to the team
• Moderator who chairs the meeting
• Scribe who makes detailed notes regarding errors
• Roles may vary from these (e.g., Reader).
• Multiple roles may be taken on by the same member.

19
Inspection Checklist

• Checklist of common errors should be used to
  drive the inspection
• Error checklist is programming language
  dependent
• The weaker the language type checking, the
  larger the checklist
• Examples variable initializations, constant
  naming, loop termination, array bounds, etc.

20
Inspection checks
21
Inspection Rate

• Measurements at IBM by M. E. Fagan
• 500 statements/hour during overview
• 125 source statements/hour during individual
  preparation
• 90-125 statements/hour can be inspected
• Inspection is, therefore, an expensive process
• Inspecting 500 lines costs about 40 person-hours
  effort

22
Automated Static Analysis

• Static analyzers are software tools for source
  text processing.
• They parse the program text and try to discover
  potentially erroneous conditions and bring these
  to the attention of the V V team.
• Very effective as an aid to inspections. A
  supplement to, but not a replacement for,
  inspections.

23
Static Analysis Checks
24
Stages of Static Analysis

• Control flow analysis. Checks for loops with
  multiple exit or entry points, finds unreachable
  code, etc.
• Data use analysis. Detects uninitialized
  variables, variables assigned twice without an
  intervening use, variables that are declared but
  never used, etc.
• Interface analysis. Checks the consistency of
  procedure declarations and their use

25
Stages of Static Analysis (cont)

• Information flow analysis. Identifies the
  dependencies of output variables. Does not
  detect anomalies itself but highlights
  information for code inspection or review
• Path analysis. Identifies paths through the
  program and sets out the statements executed in
  that path. Potentially useful in the inspection
  and testing processes.
• Both of these stages generate vast amounts of
  information. Must be used with care.

26
LINT static analysis
138 more lint_ex.c include ltstdio.hgt printarray
(Anarray) int Anarray printf(d,Anarray)
main () int Anarray5 int i char c
printarray (Anarray, i, c) printarray
(Anarray) 139 cc lint_ex.c 140 lint
lint_ex.c lint_ex.c(10) warning c may be used
before set lint_ex.c(10) warning i may be used
before set printarray variable of args.
lint_ex.c(4) lint_ex.c(10) printarray, arg. 1
used inconsistently lint_ex.c(4)
lint_ex.c(10) printarray, arg. 1 used
inconsistently lint_ex.c(4) lint_ex.c(11) print
f returns value which is always ignored
27
Use of Static Analysis

• Particularly valuable when a language with weak
  typing, such as C, is used and hence many errors
  are undetected by the compiler
• Less cost-effective for languages like Java that
  have strong type checking and can, therefore,
  detect many errors during compilation