Tools for Automated Testing

1
Tools for Automated Testing

• Todd L Miller
• UW-Madison

2
We want you to run more tests!

• Only half of our users have run tests.
• But its the Build and Test Lab!
• Better testing makes better software.
• Regular and automated tests
• Help pinpoint bad commits.
• Help prevent regressions.
• Ease additional types of testing.

3
Direction of Talk

• Splitting the build and test runs.
• Binary compatibility testing.
• Using remote_declare.
• Advanced topics.

4
Why split building and testing?

• Decouples build and test development.
• Faster test development.
• Allows new tests to run against old builds.
• Allows different kinds and frequencies of tests
  to run against the same build.
• Allows for binary compatibility testing.

5
Notes on a Test Submit Files

• Copy build submit file.
• Change run_type to test.
• Change remote_task add timeout.
• Add an NMI-method input to fetch binaries from a
  build.

6
Automating Test Submission

• Use nmi_gid2runid _NMI_GID in the build.
• When to submit?
• With the build?
• After a delay?
• As part of the end of the build?

7
Binary Compatibility Testing

• Test a binary built on one platform on another,
  e.g., different Red Hat versions.
• Simply add
• platforms ltsourcegtlttargetgt
• to your NMI-method input file.
• Very simple with split build and test runs.

8
Using remote_declare

• Writes tasklist.nmi, which tells Metronome about
  a set of subtests.
• Can easily set test-specific timeouts.
• Individual tests have individual result rows on
  the webpage, including their own output and error
  logs.
• Individual database records increase data-mining
  opportunities.

9
Advanced TopicVersion Sweeps

• Test your code against different versions of its
  dependencies.
• nmi_meta_create.pl can automate version sweeps of
  formal prereqs.
• Can use other Metronome builds for cutting-edge
  dependencies.
• Unmanaged platforms to test against patched
  versions of distributions.

10
Advanced TopicCollating Results Across Runs

• At submit time
• Append test_id date Y-M-d to the submit
  file. This helps to keep results from different
  days of testing separate.
• In remote_post
• Add results to results.tar.gz.
• In platform_post
• Unpack ./results.tar.gz and move the contents to
  /results/NMI_test_id/NMI_PLATFORM.
• In post_all
• Tar up /results/NMI_test_id.

11
Advanced TopicsIdeas for Future Work

• You tell us!
• Fuzz testing automation.
• Purify and/or Coverity integration.
• Scalability testing.
• Different network regimes.

12
Questions about Testing?

• Ill be available for more in-depth questions
  this afternoon, during the tutorial session.