Acceptance Testing vs. Unit Testing: A Developer

1
Acceptance Testing vs. Unit TestingA
Developers Perspective

• Owen Rogers
• orogers_at_thoughtworks.com

Rob Styles rob_at_objectagile.com
2
What are Acceptance Tests?

• Tests owned and defined by the customer to verify
  that a story is complete and correct.

3
Why we use Acceptance Tests

• Increase team confidence that the system is
  correct.
• Structures communication between customer and
  developers.
• Help customer think through requirements.
• Informs developer estimates.
• Indicates how to split a story.
• Concrete definition of story completion (knowing
  when were done).
• Early involvement of QA.
• Automated regression testing for free.

4
Why other people dont

• Customer participation.
• Management buy-in.
• QA support.
• More tests for developers to implement.
• No common framework.

5
ATs and UTs are the same though, right?

• Both can and should be written first, before
  implementation.
• Both should be automated by containing assertions
  for validating expected results.
• Both typically use fixture setup and teardown
  code to run before and after the execution of the
  tests.
• Both can be implemented using an xUnit framework.

6
Difference ATs are written by the customer.

• UTs only need to be read and understood by other
  developers.
• ATs must be defined in a language familiar to the
  customer.
• ATs require a customer meta-language
• Generic enough to capture requirements
• Abstract enough to be maintainable
• Simple enough to be easily understood

7
Example Acceptance Tests.
FIT Acceptance Test
FAT Acceptance Test
fit.ActionFixture fit.ActionFixture fit.ActionFixture
start eg.Page
enter location http//google.com
check title Google
enter link Jobs
check title About Google
enter link Press
enter link Review
check title Google Press Room

• Request http//google.com
• Check title Google
• Click link Jobs
• Check title About Google
• Click link Press
• Click link Review
• Check title Google Press Room

8
Difference Customers dont use an IDE.

• UTs are implemented in the language of
  development using an IDE.
• ATs should be defined using a tool accessible to
  the customer (Excel, web page, wiki).
• ATs must be retrieved from the tool and parsed.
• Ideally, the customer should be able to run the
  ATs from this environment.

9
Interacting with customer tool.
Extract Test (.csv)
Excel
AT Framework
Execute Test
Write Test
Wiki
Extract Test (.html)
10
Difference ATs are not directly executable.

• UTs invoke code-under-test directly.
• AT definition is abstracted from its
  implementation.
• ATs specify what needs to be done, but not how to
  do it.
• ATs require an implementation layer to invoke the
  application.
• AT definitions must be interpreted parsed and
  mapped onto corresponding implementation.
• Tests must be definable and executable prior to
  implementation.

11
Test Definition / Test Implementation
Test Verify Google Links
Class WebPageFixture
AT Framework
Interpreter

• Request http//google.com
• Check title Google
• Click link Jobs
• Check title About Google
• Click link Press
• Click link Review
• Check title Google Press Room

12
Difference ATs test through the same interface.

• UTs test at level of a single unit (class).
• ATs test at level of user/client interaction.
• All tests go through a common gateway (Http,
  Soap, UI controls or application façade)
• Considerable opportunity for reuse across test
  implementations.
• ATs need a framework in the technology of the
  interface it drives Web, GUI, Web Service,
  Messaging etc.

13
Testing through the application gateway.
Application
Definition
Implementation
Application Gateway Test Client
Acceptance Test Interpreter
Application Gateway
ATD
ATD
ATD
14
Difference ATs test the Users Experience.

• UTs should test all application logic in detail.
• ATs encourage the customer to consider the all
  aspects of user experience.
• ATs should test
• Interaction Flow
• Performance
• Error Handling
• Security
• ATs dont test
• Usability
• Look Feel

15
Difference ATs are end-to-end.

• UTs test single unit in isolation.
• ATs test interaction of all layers in the system.
• ATs require external systems to be operating
  correctly.

16
Difference ATs are process-driven.

• Each UT typically tests a single public method.
• ATs consist of a sequence of steps simulating a
  client interacting with the system.
• ATs may test the same steps repeatedly in
  different variations with different data.
• Considerable overlap across different ATs.
• Single point of failure will break multiple ATs.

17
Difference ATs have serious side-effects.

• For UTs, setup and teardown code should be common
  only to a single fixture.
• ATs interact extensively with external systems
  and the environment.
• ATs typically require separate setup and teardown
  per test as different tests will have different
  side-effects.
• Setup and teardown routines are commonly shared
  across tests and even across stories.

18
Difference ATs suffer concurrency issues.

• UTs should be largely independent of their
  environment.
• ATs require that the environment is setup as
  expected by the tests.
• Problems can result from different users running
  ATs concurrently.
• Ensure sufficient variation in the data used for
  each test to prevent collisions.

19
Difference Failing ATs dont break the build.

• UTs must pass when checked in failure breaks
  the build.
• ATs may be written at the start of a story and
  may not pass until the very end of the iteration.
• We mark ATs as Ignore until they have been
  fully implemented.

20
Difference ATs dont go quietly

• UTs are designed to run silently, with all
  verification encoded in the assertions.
• Customer requires visual feedback to give
  reassurance that test does what is expected.
• ATs need to provide step-by-step visibility of
  the contents and results of running the tests.

21
Difference ATs are useful outside of dev
environment.

• UTs are primarily useful to developers for
  ensuring the code does what was intended of it.
• ATs on the other hand can provide a good
  assessment of the overall health of the system.
• This makes them useful for assuring deployments
  to testing environments are sound.
• To do this usually requires environmental
  parameters to be extracted to a configurable
  location.

22
Difference ATs mock out systems, not units.

• Introducing mocks for dependencies in UTs helps
  to decouple the code.
• We often have the same needs in ATs to mock third
  party interfaces that are not always available.
• Mocking is very hard because the AT is outside of
  the application context so a specific interface
  for the system being mocked has to be created.
• Mocking is still important to maintain customers
  trust in the solution.
• Introduce mocks to help simulate failure
  conditions.

23
Implementation
Unit TestFixture
Unit Implementation
AcceptanceTest Definition
Application Interface
Acceptance Test Client
Mock