Software Analysis Call Trees, Statement Coverage and Dynamic Analysis

1
Software Analysis Call Trees, Statement
Coverage and Dynamic Analysis

• Chris Greenough and David J WorthSoftware
  Engineering Group
• Computational Science Engineering Department
• Rutherford Appleton Laboratory
• c.greenough_at_rl.ac.uk

2
What is software analysis?

• Software analysis is wide ranging. It has
  elements in
• Software quality
• Testing
• Performance
• Software metrics
• and many others
• We will consider only a very small subset of
  these
• Testing data coverage analysis
• Documentation call tree generation
• Build/Configuration dependency analysis

3
Software Testing!

• You can never do enough!
• This can range over many forms and takes place in
  all parts of the software life cycle.
• Implementation and Testing phases are where the
  maximum concentration is going to be.
• Software Testing is the process of executing a
  program or system with the intent of finding
  errors.
• It involves any activity aimed at evaluating an
  attribute or capability of a program or system
  and determining that it meets its required
  results.
• Detecting all of the different failure modes for
  software is generally infeasible.

4
Testing classifications

• There is a plethora of testing methods and
  testing techniques, serving multiple purposes in
  different life cycle phases.
• Classified by purpose, software testing can be
  divided into
• correctness testing
• performance testing
• reliability testing
• security testing.
• Classified by life-cycle phase, software testing
  can be classified into the following categories
• requirements phase testing
• design phase testing
• program phase testing
• evaluating test results
• installation phase testing
• acceptance testing
• maintenance testing
• By scope, software testing can be categorized as
  follows
• unit testing
• component testing
• integration testing
• system testing.

5
Summary on Testing (Pan 1999)

• Software testing is an art. Most of the testing
  methods and practices are not very different from
  20 years ago. It is nowhere near maturity,
  although there are many tools and techniques
  available to use.
• Good testing also requires a tester's creativity,
  experience and intuition, together with proper
  techniques.
• Testing is more than just debugging. Testing is
  not only used to locate defects and correct them.
  It is also used in validation, verification
  process, and reliability measurement.
• Testing is expensive. Automation is a good way to
  cut down cost and time. Testing efficiency and
  effectiveness is the criteria for coverage-based
  testing techniques.
• Complete testing is infeasible. Complexity is the
  root of the problem. At some point, software
  testing has to be stopped and product has to be
  shipped. The stopping time can be decided by the
  trade-off of time and budget. Or if the
  reliability estimate of the software product
  meets requirement.
• Testing may not be the most effective method to
  improve software quality. Alternative methods,
  such as inspection, and clean-room engineering,
  may be even better.

6
plusFort spag for Coverage and Dynamic Analysis
7
Coverage Analysis

• Use coverage analysis for
• Identifying untested code
• Profiling execution count
• Detecting hotspots
• Items in spag.fig
• item 6 set to 2
• item 234 set to directory for analysis files
• (items 230 and 232 for other output files)

8
Process for Coverage Analysis

• Run spag
• creates new source...
• ...with calls to probe routines added
• Compile new source
• can your compiler handle names like PRENT?
• Run test cases
• with instrumented executable
• Run cvranal
• cvranal FIGfile TOfile cvr_files

9
The CVRANAL Tool

• cvranal FIGfile TOfile cvr_files
• cvranal.fig
• Item 501 Set to 0 for normal run, 1 to zero all
  execution counts
• Item 513 Set to column where counts added to
  source (gt72)
• Outputs
• Log file
• Coverage data per routine
• NO ( 0) Untested code blocks in
  subprogram CPMIX in file cap.f
• 3 ( 50) Untested code blocks in
  subprogram HEATCAP in file cap.f
• at lines 74 75 77
• Hotspots
• 807780 tot totcomp(i)
• in s/prog MWHT at line
  1816 of singch.f
• Execution count per line added to source

10
Dynamic Analysis

• AKA Run-time analysis
• Use dynamic analysis for
• Identifying variables unset when used
• Identifying incorrect values in a variable
• Items in spag.fig
• item 1 set to 4
• item 36 set to 1 to assume local variables are
  static
• item 234 set to directory for analysis files
• (items 230 and 232 for output files)

11
Process for Dynamic Analysis

• Run spag
• creates new source...
• ...with calls to probe routines added
• Compile new source
• can your compiler handle names like QDD8?
• Run test cases
• with instrumented executable
• View file PROBES.LOG

12
Dynamic Analysis Output

• File PROBES.LOG
• GAMCT value is undefined
• subprogram AA0001 line 155 file
• /home/wksh10/examples/plusFORT/model/singch.f
• V(I) value is undefined
• subprogram DDANRM line 1688 file
• /home/wksh10/examples/plusFORT/numerical/dassl.
  f

13
NagWare Tools for Coverage Dynamic Anlaysis
14
NagWare Transformation Tools

• The NAGWare Fortran Tools provide users with a
  set of anlaysis tools for Fortran 77 and Fortran
  95 code.
• The NAGWare f95 Tools accept as input Fortran 77
  and fixed or free format Fortran 95.
• Output from the transformational tools is always
  free format, so these tools are effectively fixed
  to free format translators.
• These tools can be used in a range of ways
• Standardisation
• Enforcing coding standards
• Converting from fixed format Fortran 77 to free
  format Fortran 95
• Re-engineering code

15
f95 Tools Contents Summary

• Analysers
• nag_coverage95 - coverage analysis
• nag_depend95 - dependency analysis
• nag_fcalls95 - call tree generation
• nag_modules95 - module generation and language
  checking
• nag_xref95 - cross-reference generation

16
f77 Tools Contents Summary

• Analysers
• nag_fcalls - call tree generator
• nag_fxref - cross reference generator
• nag_libdoc - library documentor
• nag_metrics - software metrics analysis
• nag_pfort - portability analyser

17
Coverage with nag_coverage95

• nag_coverage95 is the NAGWare f95 Coverage tool
  which instruments Fortran 95 source file(s)
  producing code that has the same effect but with
  extra monitoring code added.
• The tool breaks down the input Fortran code into
  segments of straight-line code and each segment
  is then instrumented with a counter. When
  compiled and executed, the instrumented program
  will produce a trace detailing how many times a
  particular segment of code was executed.
• Those segments which are executed most may be the
  best places to concentrate efforts in optimising
  the code if speed of execution is required. If
  some segments are not executed at all when
  running with test data, the testing strategy may
  need to be improved.
• The tool also reports segments which contain dead
  code (code that cannot be reached).

18
Coverage with nag_coverage95

• nag_coverage95 options infile ... -first
  ... -last ...
• -single_run filename - The default operation of
  the tool is to produce an instrumented program
  that writes single run statistics to the file
  base.out', (see FILES' below). This option
  allows the output file to be named. Only one of
  -cumulative and -single_run may be specified.
• -cumulative filename - This option causes the
  cumulative trace output across multiple
  executions to be output to filename. It enables
  the instrumented program to be invoked with
  different input data so that a more general view
  of its behaviour can be obtained. The file
  filename.hist' is created, if it does not
  already exist, when the executable program is
  run. It contains the internal tables of
  cumulative execution frequencies and is not
  intended to be viewed. To reset the cumulative
  counts to zero, simply remove this file. Only one
  of -cumulative and -single_run may be specified.
• -first obj1 - Instruct the tool to load with
  obj1 ... before the instrumented object files.
  This option may appear only after the list of
  source files to be instrumented (see also -last).
  
• -L dir - Add dir to the list of directories to be
  searched by the loader when producing an
  executable.

19
Coverage with nag_coverage95

• -l lib - Instruct the loader to load with lib
  when producing an executable. The space between
  the option and its argument may be omitted
  provided that the result does not clash with
  other options (such as -load).
• -last obj1 - Instruct the tool to load with obj1
  ... after the instrumented object files. This
  option may appear only after the list of source
  files to be instrumented (see also -first).
• -load - Instruct the tool to load all object
  files into an executable. This is the default if
  the -source and -objects options are omitted.
• -objects - Compile the instrumented source to
  produce object files, but do not load them into
  an executable unless -load is also specified.
  Without this option, -load will cause
  intermediate object files to be deleted after
  loading. This option may be useful if the
  instrumented object files are to be placed in a
  library archive.
• -source - Create the instrumented source, but do
  not compile them unless -load or -objects is also
  specified. Without this option, -load and
  -objects will delete the instrumented files after
  compilation.

20
nag_coverage95 example

• All files must be compiled with nag_modules95 or
  the NagWare Fortran 95 compiler to the module
  (.mod) files.
• Seg3p1.f90 is a simple finite element program
  requiring libfelib90.a and a definitions file
  def3p1.f90 (a module).
• Steps
• Ensure whole library compiled with f95 or
  nag_modules95
• Compile local files def3p1.f90 ? def3p1.mod and
  def3p1.o
• Locate necessary libraries and module files
• nag_coverage95 I../modules90 L../lib90
  I../felib90 seg3p1.f90 last def3p1.o
• Many messages about source files but links an
  executable called seg3p1.inst.exe which can be
  run.
• Execution produces seg3p1.out which is the
  coverage
• Run coverage95 to view combined source listing
  and coverage results.