CS527: Advanced Topics in Software Engineering (Software Testing and Analysis)

1
CS527 Advanced Topics in Software
Engineering(Software Testing and Analysis)

• Darko Marinov
• September 23, 2008

2
Schedule

• TODAY Sep 23 Test clustering
• Sep 25 Test granularity
• Your initial project idea due
• Sep 30 More on testing? Preferred topic?
• Oct 2 discussion of project ideas (no lecture)
• Your project proposals due
• Oct 7 on your presentations of papers related to
  your projects
• Any volunteers to present?

3
Class Info

• Reports You should have got comments
• Most of them are quite good
• If nothing pointed out, everything is fine
• Project ideas Please email them so far
• Automated GUI testing
• Checkpointing
• Improved testing of refactoring engines
• Testing databases
• Testing Photran refactoring engine

4
Paper Today

• Finding Failures by Cluster Analysis of Execution
  Profiles by William Dickinson, David Leon, Andy
  Podgurski (ICSE 2001)

5
Brief Summary

• Motivation manual inspection of test runs
  (without exact oracles) is expensive
• Problem filter runs to inspect
• Solution use clustering analysis
• Evaluation three series of experiments
• Results clustering helps

6
Some Terminology

• Observation-based testing
• Test test case, test input, test data
• Which one does this paper consider?
• Failure, fault, error
• Which one does this paper consider?
• Execution profiles
• Filtering procedure (based on clustering)

7
Clustering Use in One Slide

• Cluster analysis
• Split objects into groups based on
  (dis)similarity
• Dissimilarity metric Euclidean distance
• Approaches partitioning vs. hierarchical
  (addlomerative vs. divisive)
• Cluster filtering
• Key if failures are infrequent and have unusual
  profiles, then failures are in small clusters
• Clustering technique, of clusters, sampling

8
Clustering Technique

• Agglomerative hierarchical clustering
• Execution profiles
• Function caller/calllee (for each pair f and g)
• Simple function call (for each function f)
• Dissimilarity metrics based on n-dimensional
  Eucledian distance
• Transformations Binary, proportional, standard
  deviation, histogram, linear regression
• Hybrid count-binary, proportional-binary

9
Number and Sampling

• Number of clusters
• Fraction of test runs, from 1 to 30
• Sampling strategy
• Simple random sampling (of all runs)
• Doesnt require clustering
• N-per-cluster sampling (special case N1)
• More executions come from smaller clusters
• Adaptive sampling
• If a failure found in a cluster, select all runs
  from it
• Small clusters all runs from small clusters

10
Experimental Questions

• Q1 Is a significant number of failures often in
  small clusters?
• Q2 How likely is a sampling strategy to find at
  least one failure, if any present?
• Q3 How many failures do sampling strategies
  find?
• Q4 Which dissimilarity metrics and cluster
  counters can find most failures?

11
Answers

• Q1 Is a significant number of failures often in
  small clusters?
• Yes, more than 50 in 50 of runs
• Q2 How likely is a sampling strategy to find at
  least one failure, if any present?
• Table 2 more clusters, more likely
• Q3 How many failures do sampling strategies
  find?
• failures found/ runs sampled gt 1
• Q4 Which dissimilarity metrics and cluster
  counters can find most failures?
• Metrics that focus on rare events

12
Questions for Discussion (1)

• How do clustering and prioritization relate?
• What new clustering or sampling techniques would
  you propose? DH
• How the operational profile can be used for
  characterizing patterns of operational usage and
  generating inputs? ST
• Why in GCC simple random selection performs
  better than 1-per-cluster? ST
• Why choose those subject programs? PP

13
Questions for Discussion (2)

• Can we use execution profiles of the Eclipse's
  refactoring engine with ASTGen? YL
• Can we find hidden bugs using filtered execution
  information? KB
• Is it useful to get numerous test case runs that
  reveal the same bug?
• Does this favor finding more (rare) bugs over
  finding important (frequent) bugs?
• What is hot topic in testing test selection,
  test minimization, test prioritization, test
  clustering, automatic test generation...?

14
Questions for Discussion (3)

• What other clustering/prioritization techniques
  exist?
• What types of attribute to use in cluster
  analysis for software testing?
• Would another clustering algorithm but
  agglomerative hierarchical be better?
• How to fit clustering/prioritization in TDD?
• Is extra effectiveness of clustering and
  prioritization techniques worth the extra cost
  (compared to random)?