Acceptance Testing vs. Unit Testing: A Developers Perspective Owen Rogers Rob Styles

2 What are Acceptance Tests? Tests owned and defined by the customer to verify that a story is complete and correct.

3 Why we use Acceptance Tests: Increase team confidence that the system is correct. Structures communication between customer and developers. Help customer think through requirements. Informs developer estimates. Indicates how to split a story. Concrete definition of story completion (knowing when were done). Early involvement of QA. Automated regression testing for free.

4 Why other people dont: Customer participation. Management buy-in. QA support. More tests for developers to implement. No common framework.

5 ATs and UTs are the same though, right? Both can and should be written first, before implementation. Both should be automated by containing assertions for validating expected results. Both typically use fixture setup and teardown code to run before and after the execution of the tests. Both can be implemented using an xUnit framework.

6 Difference: ATs are written by the customer. UTs only need to be read and understood by other developers. ATs must be defined in a language familiar to the customer. ATs require a customer meta-language: Generic enough to capture requirements Abstract enough to be maintainable Simple enough to be easily understood

7 Example Acceptance Tests. fit.ActionFixture starteg.Page enterlocationhttp://google.com checktitleGoogle enterlinkJobs checktitleAbout Google enterlinkPress enterlinkReview checktitleGoogle Press Room FIT Acceptance Test:FAT Acceptance Test: Request: Check title: Google Click link: Jobs Check title: About Google Click link: Press Click link: Review Check title: Google Press Room

8 Difference: Customers dont use an IDE. UTs are implemented in the language of development using an IDE. ATs should be defined using a tool accessible to the customer (Excel, web page, wiki). ATs must be retrieved from the tool and parsed. Ideally, the customer should be able to run the ATs from this environment.

9 Interacting with customer tool. Excel Wiki AT Framework Extract Test (.html) Extract Test (.csv) Execute TestWrite Test

10 Difference: ATs are not directly executable. UTs invoke code-under-test directly. AT definition is abstracted from its implementation. ATs specify what needs to be done, but not how to do it. ATs require an implementation layer to invoke the application. AT definitions must be interpreted – parsed and mapped onto corresponding implementation. Tests must be definable and executable prior to implementation.

11 Test Definition / Test Implementation Request: Check title: Google Click link: Jobs Check title: About Google Click link: Press Click link: Review Check title: Google Press Room Interpreter public void Request(string url) { } public void CheckTitle(string title) { } public void ClickLink(string link) { } Test: Verify Google LinksClass: WebPageFixtureAT Framework

12 Difference: ATs test through the same interface. UTs test at level of a single unit (class). ATs test at level of user/client interaction. All tests go through a common gateway (Http, Soap, UI controls or application façade) Considerable opportunity for reuse across test implementations. ATs need a framework in the technology of the interface it drives: Web, GUI, Web Service, Messaging etc.

13 Testing through the application gateway. Application Gateway ApplicationDefinitionImplementation Application Gateway Test Client Acceptance Test Interpreter ATD AT impl ATD

14 Difference: ATs test the Users Experience. UTs should test all application logic in detail. ATs encourage the customer to consider the all aspects of user experience. ATs should test: Interaction & Flow Performance Error Handling Security ATs dont test: Usability Look & Feel

15 Difference: ATs are end-to-end. UTs test single unit in isolation. ATs test interaction of all layers in the system. ATs require external systems to be operating correctly.

16 Difference: ATs are process-driven. Each UT typically tests a single public method. ATs consist of a sequence of steps simulating a client interacting with the system. ATs may test the same steps repeatedly in different variations with different data. Considerable overlap across different ATs. Single point of failure will break multiple ATs.

17 Difference: ATs have serious side-effects. For UTs, setup and teardown code should be common only to a single fixture. ATs interact extensively with external systems and the environment. ATs typically require separate setup and teardown per test as different tests will have different side-effects. Setup and teardown routines are commonly shared across tests and even across stories.

18 Difference: ATs suffer concurrency issues. UTs should be largely independent of their environment. ATs require that the environment is setup as expected by the tests. Problems can result from different users running ATs concurrently. Ensure sufficient variation in the data used for each test to prevent collisions.

19 Difference: Failing ATs dont break the build. UTs must pass when checked in – failure breaks the build. ATs may be written at the start of a story and may not pass until the very end of the iteration. We mark ATs as [Ignore] until they have been fully implemented.

20 Difference: ATs dont go quietly UTs are designed to run silently, with all verification encoded in the assertions. Customer requires visual feedback to give reassurance that test does what is expected. ATs need to provide step-by-step visibility of the contents and results of running the tests.

21 Difference: ATs are useful outside of dev environment. UTs are primarily useful to developers for ensuring the code does what was intended of it. ATs on the other hand can provide a good assessment of the overall health of the system. This makes them useful for assuring deployments to testing environments are sound. To do this usually requires environmental parameters to be extracted to a configurable location.

22 Difference: ATs mock out systems, not units. Introducing mocks for dependencies in UTs helps to decouple the code. We often have the same needs in ATs to mock third party interfaces that are not always available. Mocking is very hard because the AT is outside of the application context so a specific interface for the system being mocked has to be created. Mocking is still important to maintain customers trust in the solution. Introduce mocks to help simulate failure conditions.

Implementation Mock Application Interface Unit Test Fixture Setup Teardown Unit Implementation impl UT Acceptance Test Definition AT impl ATD AT impl ATD AT impl ATD setup teardown setup teardown setup teardown Acceptance Test Client