Test Automation II

1
Test Automation II
EE491.18
Maj JW Paul
2
A child comes home from his first day at school.
His Mother asks, Well, what did you learn
today? The kid sighs, Apparently not enough.
They want me to come back tomorrow.
3
Stages from Ch 13 KFN

• first functionality
• almost alpha Maurice
• alpha Oldford
• pre-beta
• internal beta Girard
• outside beta McLellan
• UI freeze Toenders
• pre-final Pollard
• final integrity test Hutter

DELAYED Thank You OCdt Pollard
4
NSB - Refired - Refried
5
Is OO dead?

• What is the purpose of OO?

MAINTENANCE
wear-out
software curve (actual)
break-in period
change
hardware curve
software curve (ideal)
6
NSB vs Bugs

• What was the theme of the EIS paper?
• what were your worst bugs
• did not talk about design/implementation (any of
  the other work products)
• Lesson
• interesting comparison, but
• be careful what you compare

7
Other Comments

• Essential vs Accidental
• what do you think is the current ratio?
• Who cares - once a program works
• Philosophers Stone Pipe Dreaming
• What problems are left?
• why does software fail?

8
Review - Syntax Testing

• Name an advantage of this technique
• helps select test cases

9
Todays Class

• Test Automation - refired
• Advantages of automation
• Maintaining test suites

10
Test AutomationAdvantages
11
Why Automate Tests?

• Some tasks are easier
• stress/load, regression testing
• Tests are more repeatable
• Tightens test cycle (means more testing)
• Can capture expertise
• Test battery can be continuously improved

http//www.io.com/wazmo/success.html
12
How to maintainthese advantages
13
Build Maintainable Tests

• Dont let the test suite become too big
• before adding any new test ask what is the test
  contribution to defect finding capability, likely
  maintenance cost, etc
• Limit resources for test designers/builders
• large amounts of test data have adverse impact on
  test failure analysis debugging
• Keep functional test cases as short in time and
  as focused as possible

14
Build good test suites

• Design software tests with debugging in mind -
  what would I like to know when this test fails?
• Start cautiously when designing tests that chain
  together - use snapshots/checkpoints
• Adopt a naming convention for tests
• Test cases need documentation too
• who - what - how - why
• overview of test items
• annotations in scripts

15
Be careful

• Limit number of complex test cases
• difficult to understand even for minor changes
• effort needed to automate and maintain may wipe
  out any savings
• Use flexible/portable formats for test data
• time taken to convert data often less than cost
  of maintaining large amounts of data in a
  specialized format

16
Lab 3 - Testing Tools

• You must evaluate the tool -
  which means.
• You must give a purchase go/nogo
• Your scenario should support this

17
Evaluating Testing Tools

• has all the critical features needed
• works consistently without failures
• handles industrial environments
• reasonable learning curve or support
• results are correct and repeatable
• ease of use of interface
• faster than manual/other tools
• ease of integration with environment
• not altering the behaviour of the
• software under test/instrumentation

• Capability
• Reliability
• Capacity
• Learnability
• Dependability
• trustworthiness
• Operability
• Performance
• Compatibility
• Non-intrusiveness

18
Summary
19
Review
Pop Quiz!!!!
20
Questions?
21
Next Class

• Lab 3
• OO Testing - is it different?
• Review
• GL the last
• Oh, and we will finish Ch 13