Automation Framework Development -Best Practices Mohd. Umair Siddiqui - Technical Leader Girish Chandra - Technical Leader Aricent Technologies

Abstract These day’s systems are complex with rich User Interface, backend and that too require multiple capabilities. Test Automation Framework are complex to develop and maintain. This white paper gives an idea about the implementation of single framework for front-end and back-end which covers all aspects of testing. The major purpose of this document is to define a common set of Automation development guidelines and how the Selenium with Java can be used to develop a single automation framework. This white paper looks at applying the Hybrid (KDF and Data Driven) framework approach for automating the test design. The KDF approach not only saves effort and cost but increases the usability of framework across the different teams and reduces the dependency on the resources with automation skills. This paper sets out the steps involved in building the automation framework.

The Urge for Automation Testing   Relying solely on manual testing to execute repetitive testing steps is not only costly but also time-consuming and ineffective. Testing with the use of tools or automation testing is the best solution when you have a lot of regression cases. For better accuracy and speed, test automation is vital to cover all corners of the application.

Test Automation Frameworks Architecture

Benefits of Automation Framework

Test Automation Life Cycle vs. Application Development Feature Development Defect Fixing System Build Defects Intermediate Build Knowledge Transfer Test Strategy Test Planning Test Execution Test Report Quality Gate Lab setup Automation strategy Regression plan Test case writing Automation Tool PoC Exit Criteria It is very important to consider the Test Automation development as a separate project all together from the Application development Execution Cycle 1 All test cases execution Automation Framework Development Test case script writing Defects validation Failed test cases Small set of regression Automation Script Execution Final regression Test Execution Execution Cycle 2 Execution Cycle 3

Automation Tool Selection Factors to be consider before selecting the right tool: 1. Meeting Requirements 2. Testing Scope 3. Availability of required skills set 4. Management Aspects Following figures depicts how the appropriate tool is selected

Challenges In Automation The choice of the tool is often restricted by the technology underlying the application under test. In web applications, multi‐window test cases are usually difficult to automate. Pop‐ups, single child windows are not a problem. Integrations between applications under test are sometimes difficult to automate. Frequent changes in feature/UI causes rework of already automated test cases. Development UI changes need to be stable enough (at least on objects level) to avoid this challenge

Automation Best Practices Since automation frameworks are essentially about abstraction, an important set of best practices deals with ensuring loose coupling between – The test data and the test scripts, Test scripts themselves, The automation framework and the application under test. The test cases and the automation framework, and The automation framework and the automation tool. Hybrid Framework (Keyword + Data Driven approach) should be developed to provide less scripting efforts for the manual engineers. Keyword names should be carefully chosen, so that human readability is also high. This enables gradual transitioning from manual testing to automated testing

Automation Best Practices Contd. Verification points should be judiciously inserted into the scripts. In case of test case failure, these points accelerate the process of zeroing in on the reason of the failure Framework should be integrated with the Test Management tool Generic Libraries should be developed to reuse the modules in different keywords. The development of an automation framework is similar to the development of an application in several respects, and hence should be planned and tracked as a (sub) project in itself. It should be noted that framework creation and test case design are distinct activities (and require different skills). Figure on next slide shows the different layers of ideal Framework

Business Challenges Solved Test Management tool Integration Test Management tools like RQM ,QC integration with the execution server. Batch/Shell script to receive the execution command from tools to its adaptor. Test Cases Orchestration Layer Keyword Driven and Data Driven Test Scenarios excel sheets Setup Specific Configuration excel sheets Framework Tools Library Layer Technologies specific testing tools like Selenium ,QTP, Appium etc. Multiple tools can co –exist Modular and Reusable libraries in any language/scripts like Java, Perl, python. VB etc. System Under Test Wireline, Wireless, Convergence or Next Generation infrastructure components Web Application /Web Portals Devices and Mobile Applications Business Challenges Solved Easy to use across the different teams Dependency on team competency for a specific tool/language. Modular/Reusable Framework

ROI on Automation Framework Automation should be seen as an overall quality and productivity improvement initiative, rather than merely as a cost saving exercise The monetary values of the following can be compared to estimate ROI: Return: Saving of manual testers’ time, converted to a monetary value Investment Cost of the automation tool Cost of the automation effort

Intelligence Quotient Case Study on ROI of Automation Framework Manual Automated Average Execution Time per Test Case (excluding REST API testing) 70 minutes 18 minutes Average Execution Time per REST API (covering all possible combinations of input parameters) 4320 minutes 10 minutes Key Highlights Increase in Test Coverage (Manual v/s Automation) 50% reduction in the time taken for a verification cycle and 5 folds increases in the number of test cases executed per cycle 35 % Reduction the defects trends seen from the production releases Break-even for automation investment achieved in 7 regression cycles Intelligence Quotient Test Data Creation : Learns from the configuration the API’s which are to be used and creates the data to test the respective API’s for the all possible parameter combinations Log Analyzer: Finds subroutine causing the major failures and can prioritize to resolve those first. Setup Analyzer: analyses the setup configuration and provides details about number of devices which is needed for a particular test suite execution Setup Creation : A Java and Rest API based tool which configure pre-requisite data required to execute the test suite 85% Regression Effort Saved 35% Reduction in field defects 50% Reduced Validation Cycles Automated test cases 11255 (85%) 88% Regression Time Saved

Summary Test automation offers a promising way of quality and productivity improvement in software testing. While manual testing is required and desired (except perhaps for a product that is purely in sustenance mode), the time and cost required for it can be significantly reduced. Moreover, a part of this saving can be invested for better quality. Commercial/Open source tools and a rapidly growing body of knowledge have led to a reduction in the time needed for monetary returns to be seen, thus accelerating the adoption of test automation in the industry.

www.google.co.in www.softwaretestinghelp.com http://www.impactqa.in References www.google.co.in www.softwaretestinghelp.com http://www.impactqa.in

Author Biography Girish Chandra Umair Siddiqui Girish Chandra has 7 + years of testing experience which includes automation and manual . Girish is Quality Assurance and Software testing specialist , managing testing efforts , developing test methodologies and creating standard procedures for quality assurance and testing .He has worked for different telecom technologies like 2G , 3G(UMTS),4G (LTE) and SIP .And has good experience on real time protocol testing . Umair Siddiqui Umair Siddiqui has 7+ years of experience of working in Automation Framework and manual testing of medium to large sized projects in Telecom domain. He is currently working as a Technical Leader in Aricent Technologies. He is awarded with Aricent Individual Excellence annual award. He has strong exposure of various software development life cycle including Requirement Understanding, Design, Feature Analysis, Test Strategy, Test Planning, Test Execution and Test Automation of assigned features

Thank You