Measure Reliability of Automation – using Machine learning Varun Bhal – Lead Software Engineer Adobe Systems, Noida

Why do we need to measure the reliability? How can we measure it? Abstract With the technological advancements, we are moving towards Automation of various stages of our SDLC lifecycle, including testing of the product / software. At the same time, it becomes equally important to measure / judge the reliability of the Automation framework, testing infrastructure, test cases, other aspects involved. Why do we need to measure the reliability? How can we measure it? How its going to help us?

Find new failing test cases Detect deviations in the run in terms of : Objective : Find new failing test cases Detect deviations in the run in terms of : Total test cases executed Failed test cases Test execution time, etc. Graphs comparing the historical results for different builds across different platforms Auto triaging of known failures

How to implement auto triaging? Find previous instance of failed test case on same configuration Check if its triaged Compare the failure reason Triage with same bug Not Found New Failure Found No Yes Not matched Matched

How to implement deviation notifications ? Start to capture below data per run : Total test cases Failed test cases Untriaged test cases (new failures) Test execution time While pushing new results, compare its data values with averaged data of last few runs (eg. 3 runs). Measure the difference in the values, compare it with threshold values If deviation is more than threshold, send notification with details.

How to implement deviation graphs ? Database would have below data per run : Total test cases Failed test cases Untriaged test cases (new failures) Test execution time Plot graphs based on this data for different suites In case of multiple runs of 1 configuration, values should be averaged for all the runs

Example Graphs: 1. Failing test cases Suite name

Example Graphs: 2. Execution time Suite name

Example Graphs: 3. Total test cases Suite name

Example deviation emails: Hidden Suite name

Increased confidence in the product as well as automation. Benefits : Early issue detection (esp. injections) will thereby help us achieve timelines easily. Increased confidence in the product as well as automation. Trends of various matrices over a period of time will help us take project critical decisions. To know the reliability of what we do is always great. It gives us answers for various W-H family questions 

Author Biography Varun is a Lead Software engineer at Adobe with 5.5 years of industry experience in automation and tools development. He is a part of Flash Runtime team at Adobe and have exposure of working in installation, deployment and runtime test automation across different platforms, including Mobile technology. This is where he came across with this practice and got successful in implementing it in automation.

Thank You!!!