Addressing Test coverage in Continuous Testing Barkha Tyagi – Test Architect Sameer Chawla – Sr. Test Architect 1

Release times are reducing but “Test Coverage” is increasing Abstract Continuous Testing is on the rise and increasingly significantly every year Waterfall Agile DevOps Months / Years Usability Customer Experience Time per Release Weeks / Months Multiple Devices Security Accessibility Multiple Browsers Performance Application Requirements Functional / Integration testing Days / Weeks Expansion in Test Coverage Release times are reducing but “Test Coverage” is increasing

Analysis – Increasing test coverage in Continuous Testing (CT) Technical users Increased need of frequent changes ‘Mandate security and performance Talent to test Lack of skilled testers. Inter personal and analyzing skills. Rising digital platforms Multiple browsers Multiple devices are in market How and when to start test Need to maintain speed with balance Squeezed timeline More mind and more ideas. If idea not floated on time it would be “stale” soon.

Production Environment Snapshot: Handling test coverage challenges in CT Product in Production Crowd Source testing Automation Focused Regression Suite Enhanced Talent Automation Mutation testing Start early – Plan idle periods of team members Continuous Integration Continuous Testing Dev Environment Unit Testing Test Environment Integration Test / Functional Test / Performance Test / Multi device Test / Compatibility Test / Security Test / Regression Test Production Environment Deploy in Production / Feedback Staging Environment Acceptance Test Product Backlog Sprint Backlog Rigorous practices with brainstorming sessions – to handle multiple environment issues

Automation is the integral part of Continuous Delivery Techniques and Best Practices: Integrating automated testing in all CT phases to achieve delivery with quality on time Build Build creation process can be automated using different tools. Automation is the integral part of Continuous Delivery Develop In this phase Unit and other required test cases automation will help Integration Jenkins, Marven Sonar etc. tools are available to automate integration process. Functional, Security, Compatibility ,performance etc. test cases can be automated using for faster execution Deploy Build package creation and deployment in production can be done through tools only.

Pass Mutant 1 Fail Source Code Mutant 2 Techniques and Best Practices: Mutation Testing Creating Mutants - Change your source Code by adding fake error in it. Errors can be added by changing ‘Value, Definition or Statement” Source Code Mutant 1 Pass Fail Mutant 2 Run Test Result: High quality test suite which may help in stable and reliable automation.

Techniques and Best Practices: Crowd Sourced Testing Not a hired test team but the crowd is testing the application Software is tested on diverse and realistic platforms. Specific groups can be selected through the crowd to test software. Can be useful for compatibility, browser, device and usability testing

Adopt new testing techniques “Focused” regression Suite Lessons Learnt: Overcoming test coverage challenges in CT ..1/2 Automation Adopt new testing techniques “Focused” regression Suite Automated test buckets for UI and API test cases help in saving time and covering more. Mutation Testing Crowd Source Testing The “Focused” Test coverage, ‘Prioritization of test scenarios’ on the basis of revenue generation User experience is a valuable approach that straightforwardly cops up with continuity of design, develop and test in CD.

Start early and work together Enhance talent Rigorous practices Lessons Learnt: Overcoming test coverage challenges in CT ..2/2 Start early and work together All team members should start work together. Plan Idle periods of team members due to any dependency. Effectively utilize time by starting test planning, including designing and testing Automate test cases at the right time parallel to dev/unit coding phase. Enhance talent Work on manual power to bring manual testers close to the technical skills that is a key area to enhance test quality, coverage and reduce time to test. Rigorous practices Understand environment issues and set up rigorous practices with brainstorming sessions.

References & Appendix Mutation testing information available on "https://en.wikipedia.org/wiki/Mutation_testing" Two-Week Sprint: referanced from my own current project cycle and from "https://agile101.wordpress.com/2009/07/24/scrum-meeting-schedule-two-week-sprint/ https://en.wikipedia.org/wiki/Crowdsourcing Reference - Automation “Automagic"...Gartner’s Approach to IT Automation in the Data Center. http://www.cigniti.com/devops-testing/

Author Biography Barkha Tyagi – Test Architect 10+ years experience in IT Industry in managing the end to end testing of multiple projects operated in Agile/Scrum. Working as Test Architect in NIIT Technologies Masters in computers. Having an \'A\' Level Certification from Software Technology Group Delhi. Sameer Chawla – Sr. Test Architect HP Certified automation test professional with 11+ years of rich industry experience Presently working in the role of a Senior Test Architect with responsibilities of: Test Automation, PoC Creation Test Team Management & Technical Training MCA from M.D.U.