Code Review, Test Data, and Code Comparison

1
CHAPTER 17

• Code Review, Test Data, and Code Comparison

2
Evidence -collection techniques

• Three evidence -collection techniques auditors
  can use primarily during substantive testing are
• (a) program code review,
• (b) test data, and
• (c) code comparison.

3
Program code review

• Program code review involves reading source-code
  listings with a view to identifying defective
  program code.
• Test data involves executing a program using a
  sample of data to determine whether the program
  is defective.

4
Code comparison

• Code comparison involves using software to
  compare a source code version or object-code
  version of a program against another version of
  the program that has known characteristics (e.g.,
  auditors believe it to be error free)

5
Use of techniques - Coordination

• Program code review, test data, and code
  comparison can be used as independent evidence
  -collection techniques.
• Sometimes, however, auditors might use them in a
  coordinated way.

6
Use of techniques - Coordination

• First, they can undertake code review to generate
  hypotheses about defects in the program.
• Next, they can use test data to determine whether
  their flaw hypotheses are confirmed.
• They can then have the defects in the program
  corrected to produce a "blueprint.
• Finally, auditors can use code comparison to
  compare production versions of the program
  against this blueprint to identify undocumented
  changes that have been made to the program.

7
Defects in Programs

• Empirical studies, defects occur in
• a small number of modules will have a large
  number of faults, and a large number of modules
  will have none or a small number of faults
• requirements specification and design errors are
  just as likely to occur as coding errors
• design errors seem to relate primarily to the
  user interface, input/output devices, and the
  database
• coding errors seem to related primarily to
  incorrect computation, incorrect indexing, and
  incorrect control flow and
• defects relating to data management are more
  common than defects relating to computation.

8
Objectives of program source-code review

• The objectives of program source-code review are
  to
• identify erroneous code,
• unauthorized code,
• ineffective code,
• inefficient code, and
• nonstandard code.

9
Steps In Source-code review

• Source-code review involves seven steps
• on the basis of materiality and risk, selecting
  the source code to be examined
• reviewing the organization's programming
  standards to develop expectations about the
  characteristics of the code and likely defects in
  the code

10
Steps In Source-code review

• obtaining an understanding of the program's
  specifications as a basis for determining whether
  the program has the correct functionality
• obtaining an up-to-date listing of the source
  code,
• reviewing the programming language used to
  implement the code to develop expectations about
  where defects are likely to occur in the code
• reviewing the source code in a systematic way to
  identify program defects. and
• formulating flaw hypotheses that form the basis
  for further testing of the program.

11
Primary Benefit

• The primary benefit of program source-code review
  is that it provides a level of detailed knowledge
  about a program that is difficult to acquire
  using other evidence -collection techniques.

12
Primary Disadvantages

• The primary disadvantages are that it is time
  consuming and often difficult and costly to
  undertake.

13
Test data

• Test data is reliable if it reveals a defect in a
  program when the program contains a defect.
• Unfortunately, we know of no way to ensure that
  reliable test data has been designed.
• Auditors should use a systematic approach to the
  design of test data, however, to achieve high
  levels of reliability in their test-data designs.

14
Approaches to Test-data Design
15
Equivalence Classes
16
Systematic Approaches

• Two types of systematic approaches to the design
  of test data exist.
• Black Box Specification Based
• White box or program Based

17
Black-box

• Black-box, or specification-based testing
  methods, view the program to be tested as a black
  box.
• Test data is designed on the basis of the
  program's specifications rather than its internal
  control structure.

18
White-box

• White-box, or program-based testing methods, rely
  on the internal control structure of the program.
  
• Test data is designed on the basis of a direct
  examination of the program's source code.

19
Design Methods

• Two types of black-box test-data design methods
  that auditors might use are
• equivalence partitioning and
• boundary value analysis.
• With equivalence partitioning, auditors divide
  the input domain of the program into classes
  where each element of a class is treated
  identically to all other members of the class, at
  least according to the specifications.

20
Decision tables and Decision trees

• Auditors can use decision tables and decision
  trees to help them identify the equivalence
  classes in a program's input domain.
• They then choose one element from each of the
  material equivalence classes to include within
  their test-data design.
• Providing a program is not defective in terms of
  its treatment of this element, auditors assume it
  is not defective in terms of its treatment of
  other elements in the equivalence class.

21
Boundary Value Analysis

• With boundary value analysis, auditors again
  focus on the equivalence classes in a program's
  input domain.
• They choose test data items, however, that test
  how the program treats the boundary values of
  each material equivalence class.
• For example, if a program tests whether a value
  is greater than a certain amount, auditors will
  choose a test data value that is equal to the
  amount, just above the amount, and just below the
  amount.
• They will also test the boundary values for any
  material output of the program.

22
Objectives of White-box Test-data Design

• There are three objectives that auditors might
  seek to achieve when they undertake white-box
  test-data design.
• First, they can seek to achieve full statement
  coverage of a program, whereby they try to ensure
  that each statement in a program is executed at
  least once.
• Second, they can seek to ensure full branch
  coverage, whereby they seek to ensure that each
  statement is executed at least once and each
  branch is executed at least once.
• Third, they can seek to achieve full path
  coverage, whereby every logic path in the program
  is executed at least once. Full statement
  coverage is the easiest objective to achieve. For
  moderately complex programs, achieving full path
  coverage is often practically impossible.

23
Two types of white-box test-data design methods

• Two types of white-box test-data design methods
  that auditors might use are
• basis path testing and
• loop testing.

24
Basis path testing

• Basis path testing is an approach designed to
  achieve full branch coverage in a program.
• Auditors first convert the program into a
  flowgraph and then design test data to traverse
  each of the independent paths in the flowgraph.
• An independent path is one that introduces at
  least one node in the flowgraph that has not been
  traversed by another path.
• The number of independent paths in a program can
  be determined by calculating a measure called
  McCabe's cyclomatic complexity number which, as
  its name implies, is also a measure of the
  complexity of the program.

25
Loop testing

• Loop testing seeks to test whether the material
  loops in a program have any defects.
• Because the number of paths through a loop or a
  set of nested loops can be large or practically
  infinite, loop testing focuses on those aspects
  of a loop or set of loops that are likely to be
  defective.

26
Creating test data

• Creating test data is often difficult and
  time-consuming.
• One approach auditors can use is to rely on
  existing production data and to modify this data
  so it complies with test-data design.
• Another approach auditors might use is to rely on
  test data that has been designed by the
  organization they are auditing and again modify
  this test data so it complies with test-data
  design.
• If auditors cannot obtain high-quality test data
  using either of these two methods, they will have
  to create all the test data themselves.

27
Automated aids

• Whatever approach auditors use, a number of
  automated aids are now available that facilitate
  their design, creation, and use of test data-
• for example, test data generators to assist with
  the creation of test data that complies with
  their test-data design.

28
Benefit of using test data

• The major benefit of using test data as an
  evidence-collection technique is that it allows
  auditors to examine the quality of program code
  directly.
• They do not infer the quality of the code based
  on other evidence, such as the quality of
  production output.

29
Major disadvantage of using test data

• The major disadvantage of using test data is that
  it is often time-consuming and costly.
• Auditors use program code comparison for two
  reasons.
• First, it gives them some assurance that they are
  auditing the correct version of the software.
• Second, it gives them some assurance that any
  software they use as an audit tool is the correct
  version of that software.

30
Software

• Software is available to undertake two types of
  code comparison
• (a) source-code comparison and
• (b) object-code comparison.
• With source-code comparison, auditors should be
  able to obtain a meaningful list of any
  discrepancies between two versions of a program's
  source code.

31
Version of the source code

• They need to determine, however, whether the
  version of the source code they are evaluating is
  the one used to prepare the production object
  code.
• With object-code comparison, it is unlikely that
  auditors will be able to identify the nature of
  any discrepancies found between two versions of a
  program's object code.
• They can better determine, however, whether
  changes have been made to the current version of
  a program's production object code.

32
Effectiveness of Source-code and object-code
comparison

• Source-code and object-code comparison are often
  most effective when they are used in conjunction
  with one another.
• First, auditors compare the audit version of a
  program's source code with the version that the
  organization they are auditing contends is the
  source code used to compile the production
  version of the object code.

33
Discrepancies between the two versions

• Any discrepancies identified between the two
  versions of the source code must be reconciled.
• Second, auditors compile either the audit version
  or the organizational version of the source code
  to produce the production version of the object
  code.
• Third, auditors compare the object code produced
  via this compilation with the production object
  code in use to identify whether discrepancies
  exist.

34
Primary benefit of code comparison

• The primary benefit of code comparison is that it
  is easy to undertake,
• The software used to perform code comparison is
  neither costly to purchase nor difficult to use.
• The primary limitation of code comparison is that
  it does not provide evidence directly on the
  quality of the code being compared.