IMPACTED TESTS BASED ON FUNCTION COVERAGE Shagufta, Software Engineer Shubhi Gupta, Software Engineer Rakesh Jha, Quality Engineering Manager Adobe Systems India Pvt Ltd.

ABSTRACT Rapid Release model of Software Development demands an expeditious shipment of product to market. The continuous integration process in turn requires frequent certification of the product build which necessitates the execution of a large number of tests. An optimization to this can be attained by an intelligent selection of a test suite for execution. Code changes can serve as reliable indicator for identifying the tests which need to be run for a particular build of the product. Assessment of the probability of failure can be done by running only the tests which invokes THE FUNCTIONS THAT HAVE BEEN CHANGED in the code (based on Code Coverage). This leads to a shorter test execution window and saving of computing resources. So, the extent of testing / the set of test cases that are to be chosen for a particular requirement are the ones which have their test requirements impacted by the function changes. In products like Adobe Acrobat that have 8k+ test cases, running the full test suite is a time consuming and redundant process. Executing only a limited set of tests thus reduces the test count significantly which can be run in no time, like in Acrobat test count reduces to approximately 300, and any break caused by code change is immediately caught. Thus, change based automation not only optimizes the test execution but also significantly reduces the turnaround time for the developer to be reported any regression caused by the change. The implementation of “Impacted Tests based on function coverage" in Adobe Acrobat team has proven to be very useful in obtaining accurate results in a short span of time.

CURRENT PROCESS Automation Framework New Build ` Scope of Improvement CL 3: Version bump CL 5: Dummy CL CL 1: Test fix CL 2: Bug Fix CL 4: DLL Change Build System Automation Framework Executes 8K+ Test cases Time: ~24 hours ‘N’ Machines Test results available in 24 hours Entire QE team triages the results New Build Fixes and Changes Turn around time in reporting regressions ~ 30hours. Can we improve this??? Based on triage and testing, new bugs logged.

Component Based Testing (Executing Feature Specific Tests) Create PDF PDF View Security Acrobat Components Acrobat Test Suite Create PDF Tests (500) PDF View Tests (700) Security Tests (300) CONS Tests dependent on other Components are not executed

Function Change Based Testing (Executing Tests invoking the functions changed) Function level coverage Product Build Identify functions being changed F1, F2,…….,F(n) Query DB to get test list {T} that covers the changed functions {F1…F(n)} Execute Tests {T} affected by the changed functions PRO’s Ensures testing of all components impacted by code change. Only required tests executed, so time is reduced.

Architecture of “IMPACTED” Test Tool Creates the Functional Profile of a test which is actually a list of functions covered by a given test. TEST PROFILER PROVIDER IMPACTED TEST TOOL Provides list of tests which covers the functions changed in a given submission.

(TESTS TO FUNCTION MAPPING) TEST PROFILER (TESTS TO FUNCTION MAPPING) Tests C++ Function Name 1. Commenting_Main void comments() 2. Highlight_Text void highligtText() 3.Commenting_TODO int toDoCommenting() 4. CreatePDF_Maker bool makeNewPDF() 5. …. … 8000+ tests Executed on Code Coverage files Code coverage product build Parse code coverage file Database COLLECT DATA Fortnightly Task Determining Functions invoked by a test can be done easily by Code Coverage Tools

TEST PROVIDER New Product Build Parse the files of the new build using Code Difference Tool Identify the lines which have been changed Query DB to get the tests corresponding to functions changed CTAG library determines the Functions Changed . List of Tests to be executed due to function changes

EXAMPLE – TEST RESULTS Before After Implementing Function Based Automation Before

Return of Investment Saves Computing Resources: Runs only the tests impacted by the function change without wastage of computing resources. Stable build available all the time. Enables frequent Beta releases. Reduction in regression turn around time: Shorter Test Execution Window provides quick results to the developer enabling him to promptly fix the issue when the code change context is fresh in his mind. Saves QE hours in isolating the change which caused test failure (Only the failures due to a particular code change are reported).

REFERENCES & APPENDIX https://stackify.com/code-coverage-tools/ https://confluence.atlassian.com/clover/comparison-of-code-coverage-tools-681706101.html https://msdn.microsoft.com/en-IN/library/dd537628.aspx

AUTHOR BIOGRAPHY The author is Shagufta, Software Engineer in Acrobat Team. Involved in Security Testing, Automation Tools, and Feature Testing. The co-author is Shubhi Gupta, Software Engineer in Acrobat Team. Involved into developing Automation Tools and writing Automation Tests. The co-author is Rakesh Jha, Quality Engineering Manager in Acrobat Team. Involved majorly in Acrobat security, commenting and Automation projects. We are collectively responsible for developing Impacted Tests Based On Function Coverage.

THANK YOU!!!