Testing: A Roadmap

1
Testing A Roadmap

• Mary Jean Harrold
• Sunita Bodigutla
• CS 589 Fall 2003

2
Definition

• A verification method that applies a controlled
  set of conditions and stimuli for the purpose of
  finding errors.
• The paper says
• Its a process that is performed to support
  quality assurance

3
Quality Assurance

• ISO 90002000
• "Part of quality management focused on providing
  confidence that quality requirements will be
  fulfilled
• It is a planned program consisting of all
  actions necessary to provide adequate confidence
  that an item or product conforms to established
  technical requirements and, activities designed
  to evaluate the process by which products are
  developed or manufactured.
• A product does what it is expected to.

4
Testing

• Designing test cases
• executing the software with those test cases
• examining the results produced by those executions

5
Expensive

• Testing accounts for 50 of the cost of software
  development
• Higher for critical systems
• Efficient ways to perform testing to control the
  cost

6
What this paper is about

• A report written in 2000
• assesses the state of the art in software testing
• outlines future directions in software testing
• gives pointers to software testing resources

7
Status of Testing Methods, Tools, and Processes
Fundamental Research
Testing Component based Systems
Testing based on Precode artifacts
Methods and Tools
Testing Evolving Software
Demonstrating effectiveness of testing techniques
creating effective testing processes
Empirical Studies
Using testing artifacts
Practical Testing Methods, Tools, and Processes
for Development of High-Quality Software
Other testing approaches
fig 1. Software Testing Roadmap
8
Status of Testing Methods, Tools, and Processes
Fundamental Research
Testing Component based Systems
Testing based on Precode artifacts
Methods and Tools
Testing Evolving Software
Demonstrating effectiveness of testing techniques
creating effective testing processes
Empirical Studies
Using testing artifacts
Practical Testing Methods, Tools, and Processes
for Development of High-Quality Software
Other testing approaches
fig 2
9
Component Based Systems

• Increased size and complexity of software systems
  has led to component based systems
• Component Developer vs. Component user

10
Component User
Reusable Service
Repository
Select
Semantics
Component
Understand
Composible
Assembly
System Under Assemble
Executable
Compile
System
fig 3
11
Component Based Systems (contd.)

• COTS Commercial off the shelf software
• e.g. Microsoft Active X, MFC, Java Applets etc
• Why COTS-
• saves money
• saves time
• component reuse
• standardization
• simplified design

12
Problems with Testing COTS

• Unavailability of source code
• Technical Heterogeneity
• Enterprise Heterogeneity
• Dynamic evolution
• Component Trust Problem

13
Component Based Software

• Techniques for testing by Component provider- a
  number of them
• Techniques available to component user-very few
• Theory of test adequacy by Rosenblum

14
Test Adequacy

• A test adequacy criterion is a systematic
  criterion that is used to determine whether a
  test suite provides an adequate amount of testing
  for a component under test
• C-adequate-for-P for adequate unit testing of a
  component
• C-adequate-on-M for adequate integration testing
  of a component-based system

fig 4
15
Future Directions

• Efficient techniques and tools for
• testing aspects like security, dependability and
  safety
• representing and computing the types of testing
  information (meta-data) that the component-user
  needs
• Java Beans BeanInfo
• storing the information
• using the information
• A Framework for component deployment 1 called
  component deployment testing (CDT) framework
• component runtime introspection
• reflection API of Java

16
Status of Testing Methods, Tools, and Processes
Fundamental Research
Testing Component based Systems
Testing based on Precode artifacts
Methods and Tools
Testing Evolving Software
Demonstrating effectiveness of testing techniques
creating effective testing processes
Empirical Studies
Using testing artifacts
Practical Testing Methods, Tools, and Processes
for Development of High-Quality Software
Other testing approaches
fig 5
17
Integration test strategies and architectural
styles
fig 6
18
Fault type and Integration Test Strategy
fig 7
19
Testing based on Architecture specifications

• Formal notation for architecture specification
  CHAM Chemical Abstract Machine.
• Architecture a set of components (molecules)
• states and interactions governed by rules
  (reactions)
• CHAM is specified by defining molecules m1,m2,,
  solutions s1, s2,.. and transformation rules T,
  T,..
• Solutions are multisets of molecules interpreted
  as defining the states of CHAM
• Transformation rules describe how the system
  dynamically evolves through reaction steps

20
Testing based on Architecture specifications
(contd.)

• 4 Using the formal description of SA, a
  transition graph depicting all possible behaviors
  of the system is arrived at.
• A suitable set of reduced graphs are identified,
  each subgraph specific architectural properties
  of the system.
• For each subgraph, test cases are derived to
  cover all the transitions in it.

21
Future Directions

• Techniques for
• automatic generation of test cases from formal
  specification of software architecture
• evaluating software architectures for testability

22
Status of Testing Methods, Tools, and Processes
Fundamental Research
Testing Component based Systems
Testing based on Precode artifacts
Methods and Tools
Testing Evolving Software
Demonstrating effectiveness of testing techniques
creating effective testing processes
Empirical Studies
Using testing artifacts
Practical Testing Methods, Tools, and Processes
for Development of High-Quality Software
Other testing approaches
fig 8
23
Testing Evolving Software

• Regression Testing ensuring that no new errors
  are introduced into previously tested code
• Expensive
• Selective Regression Testing which subset of a
  test suite must be rerun to test a new version of
  the system

24
Future Directions

• Techniques to
• assist in managing the test suites
• reduce the size of test suites
• prioritize test cases

25
Status of Testing Methods, Tools, and Processes
Fundamental Research
Testing Component based Systems
Testing based on Precode artifacts
Methods and Tools
Testing Evolving Software
Demonstrating effectiveness of testing techniques
creating effective testing processes
Empirical Studies
Using testing artifacts
Practical Testing Methods, Tools, and Processes
for Development of High-Quality Software
Other testing approaches
fig 9
26
Test Criteria

• Effective for certain classes of faults
• data flow
• complex boolean expressions Modified
  Condition/decision coverage

27
Modified Condition/Decision Coverage

• Definition
• Every point of entry and exit in the program has
  been invoked atleast once
• every condition in a decision in the program has
  taken on all possible outcomes atleast once
• each condition has been shown to independently
  affect the decisions outcome

28
Modified Condition/Decision Coverage (contd.)

• Simple example
• A and B
• Extension of truth table to pairs table
• The test set (T T),
• (T F), (F T) is adequate for MC/DC coverage

fig 10
29
Future directions

• Find ways
• to determine interaction between the various
  test- selection criteria and find ways to combine
  them to perform more effective testing

30
Status of Testing Methods, Tools, and Processes
Fundamental Research
Testing Component based Systems
Testing based on Precode artifacts
Methods and Tools
Testing Evolving Software
Demonstrating effectiveness of testing techniques
creating effective testing processes
Empirical Studies
Using testing artifacts
Practical Testing Methods, Tools, and Processes
for Development of High-Quality Software
Other testing approaches
fig 11
31
Effective Processes for Testing

• Process used for planning and implementing it
• Test plan
• Perpetual testing
• Microsoft uses a different model
• synchronize and stabilize
• Future directions
• validate the existing models
• integration of various quality techniques and
  tools

32
Using Testing Artifacts

• Execution Traces
• Information about which
• statements were executed
• paths were executed
• values particular variables acquired
• Artifacts can be stored for use in retesting the
  software after it is modified
• dynamic program slices
• heuristic approaches
• List of invariants
• Regression Testing
• Visualization technique

33
Other topics

• Methods and Tools
• Scalability
• Computational tradeoffs
• Automatic generation
• Empirical Studies
• Testing Resources

34
Conclusion

• Overview
• List of techniques
• No comparison
• Future directions
• Testing is a wide area
• Dynamic field

35
References

• 1 Antonia Bertolino, Andrea Polini, A
  Framework for Component Deployment Testing
• 2 A.Orso, M.J.Harrold and D.Rosenblum,
  Component Metadata for Software Engineering
  Tasks, in W.Emmerich and S.Tai(Eds) EDO 2000,
  LNCS 1999, pp. 129-144
• 3 J.A.Stafford and A.L.Wolf, Annotating
  components to support component based static
  analysis of software systems , Proc. the Grace
  Hopper Celeb. of women in computing 2001
• 4 Antonia Bertolino, Paola Inverardi, Henry
  Muccini, Andrea Rosetti, An Approach to
  Integration Testing based on Architectural
  Descriptions

36
fig 12. Sequential Multiphase Compiler
Architecture
37
fig 13.Transition Graph for the Sequential Compile
38
fig 14. Concurrent Reduced Graph
39
Thank You