Automation Testability Issues

1
Automation TestabilityIssues

• Jamie L. Mitchell
• Principal QA Consultant
• BenchmarkQA, Inc.

2
Agenda

• What is testability?
• Observability and Controllability
• Current practices
• Where we need to go
• Specific hints

3
What Does Testability Mean?

• Incompatibilities between
• Testing Environment
• System under test
• Test automation tool
• Good testing processes
• Lack of testability means reduced ROI

4
Automation Specific

• Two facets of testability for tools
• Observability ability of the tool to capture
  data and information from system under test
• Controllability ability of tool to drive system
  in same way a manual tester would, simulating
  mouse and keyboard actions.

5
Observability

• Each GUI object is a metaphor
• Window, edit, list, button
• Humans interact with metaphor based on visual cue
• Tool does not understand metaphor
• Requests properties from OS
• Control characteristics
• Data

6
Lack of Observability?

• Many factors can affect observability
• Custom / aggregate / changed controls
• Automation tool design
• Add-ins needed
• Development tool design
• Power Builder windows
• Environment design
• Thin client / Citrix

7
Controllability

• Usually less problematic than observability (can
  program it)
• Context sensitive control
• Simulates metaphor control
• edit_text(), list_select(), etc
• Low-level
• Mouse movements, clicks
• Keyboard action

8
Lack of Controllability?

• Incorrect context
• Power Builder data windows
• Missing messages
• Custom secondary messages
• Often caused by incorrect capture
• If capture bogus, playback will likely fail

9
Historical Fix for Testability Problems

• Automator workarounds
• Modify behavior by programming in tool
• Low level simulation (mouse, keyboard)
• Encapsulate in functions
• Direct API calls from tool
• Custom DLL, functions called from tool
• Ignore and dont automate
• Leads to shelfware

10
Problems With Workarounds

• Expensive to create
• Assumes high level of technical expertise
• Expensive to maintain
• Brittle
• Requires refresh from release to release
• Come late in cycle
• Reduced ROI

11
We Must Understand

• Automation Strategic Investment
• High investment costs
• High potential payback
• Not just a testing problem!
• Stop treating automation as a stepchild
• Automation must become mainline
• Whole organization must contribute

12
Get Management Involved

• Management concerns
• Functionality
• Cost
• Schedule
• Risk
• Must present automation in these terms
• May need management leverage

13
Test Tool

• Start with correct test tool
• One that works best in environment
• Understands most controls
• Use implant if available
• Use tool features
• Programming facilities
• Add-ins

14
High Level Design

• If possible
• Use standard Windows objects that tool
  understands
• If not
• Get early form with all controls to be used
• Test all controls against the tool
• Work out problems before coding phase

15
For Each Control

• Choice (or design, if custom) must include
  testability concerns
• Object sub-classes must be checked
• If not compatible
• Add properties and methods to control
• May be directly coded
• May be custom DLL written by developers

16
Naming Conventions

• Must be tool friendly
• Persistent, meaningful control names
• Each window uniquely named
• Single, common error dialog
• Avoid single dialog / multiple window
• If not, ensure all controls are uniquely named
• Emulator windows need unique numbers
• Attached text must be close to control

17
Processes

• Absolute change control
• There are no small changes (to tool)
• Automators need to review before changes are made
• Testability bugs should have same status as other
  defects
• Version control of all test artifacts
• Build and Change Notes

18
Other Considerations

• Use debug builds with special hooks and logging
  for tool
• Each state must be unique
• Visual cues which tool can pick up this is good
  design methodology for users, too
• Exception handling should be triggerable by tool

19
Summary

• Automation is strategic investment
• Management buy-in
• Developer buy-in (or coercion if needed)
• Must be treated as any other phase of development
• Start early

20
Contact Me

• Jamie L. Mitchell CSTE
• BenchmarkQA
• 3800 West 80th St
• Suite 1580
• Minneapolis, MN 55431
• jamie.mitchell_at_benchmarkQA.com