Designing Reusable Frameworks for Test Automation Presenter: Byron Goodman National Director of Quality Assurance Neudesic

Overview Some thoughts on automation Successful test automation Components of a framework Some coding standards Standard routines

Automated Testing: Why We Love It Let the machines do the tests Easy to create alternate scenarios by leveraging existing tests Using tools can create more interest in testing activities Potential to reduce testing cycle time Effective at catching regression bugs

Automated Testing: Why We Hate It Tests break too often! Pass or fail results may not be reliable Tools are expensive, although there are some good open source solutions Tools may oversimplify or overcomplicate the degree of effort required to test applications

Have you Experienced Any of These Symptoms? More time spent working on automated tests than time needed to do the manual testing Sporadic interest in implementing automated testing Statements by others that automated testing is: a sham, ineffective, waste of time Shelving of test automation in the interest of project success

What is Successful Test Automation? As with all project related work, the time spent creating automated testing must offset this effort by the return of value. This is demonstrated in various ways: Less time expended during the testing cycle than if manual testing is the only solution More thorough and consistent regression coverage Reduce human error and boredom Frees up people to test new functionality while computers test existing functionality

Basics of Beginning Strategy – start simple and work towards a more complete and complex solution Timing – start using tests as soon as they are written, Do not wait for a completed suite Buy-in – Demonstrate to others that each test does exactly what it is intended to so that there is confidence in results Traceability – Each automated test must trace back to a specific automated test

Start Somewhere Generally speaking, some automated testing is better than no automated testing Select the most simple of tests to automate Begin by using record and playback to minimize time investment but to show that the tool does prove to be effective

Record and Playback For years, I have insisted that no tests will be created using record and playback I will admit that there is a place for R&P Very light use to prove out the tool and to minimize the reluctance people may have to use complicated technology However, R&P is does not have much gain in efficiencies over manual testing

Testing Framework Many of us have experience with testing frameworks Other names– DLLs, namespace, test library Its just a fancy name for code that is used to automate tests A testing framework is a set of routines that are reused by tests that will speed up test development while increasing test reliability

Testing Framework Goals Needs to be recognized to have the same rigor as any development effort Adds value to the overall test effort Implements complex and redundant routines for tests to leverage Minimizes the impact of a changing application to the tests, which reduces test maintenance

Layers & Design Each layer is developed to promote portability and reusability The more generic a layer, the more reusable it needs to be i.e. the Generic Routines should be able to work for everyone all the time regardless of the environments and AUTs

General Data Flow

Major Components of a Framework Common routines that do not interact with any application Generic UI layer, non-technology-specific manipulation routines Technology-specific layer routines Application-specific layer routines Application-specific Scenario/transactional layer routines Tests layer

Common routines that do not interact with any application Not related to the application or an interface Build more capability and utility that the test tool may not have Functionality that a tester may need for any reason If written properly, these routines will work with test tools that use the same programming language

Generic UI layer routines Some control types, like grids, that are generic and rather independent from the technology that they are written in This is a good place to create additional or alternate logging

Technology-specific layer routines For handling specific technologies – i.e. Browsers, WinForms and WPF. Custom controls will be wrapped with calls that are identical to similar standard controls Create interfaces for new versions of technology if the tool vendor does not yet have the new technology versions integrated

Application-specific layer routines Routines that are specific to the AUT Repository for UI mappings Defines a handler for every individual form/page AUT service interfaces defined and wrapped Serves the calls needed for tests to interact with the AUT

Application-specific Scenario/transactional layer Use this as a distinguishable layer if the routines for scenarios are complex and extensive May include multiple forms/pages In cases where there will not be a lot of routines, this can be combined into the Application-specific manipulation routines

Tests layer This layer will contain only tests No test calls any other test Tests call into the other layers Tests should be very simple as all the complexity is implemented in the other layers

General Data Flow – Strictly Enforce

Standards Using consistent coding standards promoted reusability and maintainability Standards need to be documented and followed similar to what project development would do Use code reviews to enforce standards

Typical Coding Standards Naming conventions for variables and methods/functions Use human readable names! Limit code complexity Separation of layers Comments

Automation Coding Standards Do not use the control type in the name, Address, not AddressEdit No duplication of control names for any specific window if the tools would normally allow this Consistent naming of similar controls form to form, even if the forms label the controls differently

Automation Coding Standards Execute anywhere – tests need to written so that they do not rely upon a very specific environment configuration, unless that is the purpose of the test Unattended testing – tests that need human interaction to execute should be segregated so that the other tests can all be run unattended

Automation Coding Standards No hard coding – use configuration files or similar mechanisms to store information or environment values In tests, avoid using the automation tool syntax Tests should call into the other layers This will make tests less likely to fail if the tool vendor modifies their syntax Can make it easier to port tests to new automation tools

Automation Coding Standards Overuse logging Design for distributed testing and for concurrent execution of tests An automated test will test just a single test No test ever depends upon the successful actions or completion of a previous test Computers don’t care if they do repetitive actions

Standard Routines Baseline (Initialization) Navigation Editing Verification Tools already have their own routines defined for some of this – however, you will get more flexibility and maintainability using these patterns

Baseline Will set the environment to the proper state to start a test This state is defined for each AUT and test team Can be simple to very complex Determined by defining a known state that all tests can reliably execute and get consistent results

Navigation Used to determine if a page/form is active and ready for interaction Will navigate to the desired page/form if the steps required are not complex, or the steps will not alter the outcome of any test Every page/form will have an unique navigation routines created

Editing Enters data into fields, pushes buttons, sets checkboxes, selects values in lists, etc. The control types should not matter when calling this routine – it will figure out what the control type is during test execution and interact with the controls appropriately Defined as an unique routine for every individual page/form

Verification Reads data from all controls types Determines if the values read result in a pass or fail Responsible for logging results Defined as an unique routine for every individual page/form

Areas we did not cover Solution layout Actual code examples Dealing with particularly difficult controls Configuration management Skill sets Specific tools’ capabilities

What we did cover Beginning automated tests Test framework goals Layers of a framework Coding standards Initial routines that every page/form requires

Questions? Thanks for your participation Byron.goodman@neudesic.com