1. Measure Reliability of Automation – using Machine learning
Varun Bhal – Lead Software Engineer
Adobe Systems, Noida

2. Why do we need to measure the reliability? How can we measure it?
Abstract
With the technological advancements, we are moving towards Automation of various stages of our SDLC lifecycle, including testing of the product / software.
At the same time, it becomes equally important to measure / judge the reliability of the Automation framework, testing infrastructure, test cases, other aspects involved.
Why do we need to measure the reliability?
How can we measure it?
How its going to help us?

3. Find new failing test cases Detect deviations in the run in terms of :
Objective :
Find new failing test cases
Detect deviations in the run in terms of :
Total test cases executed
Failed test cases
Test execution time, etc.
Graphs comparing the historical results for different builds across different platforms
Auto triaging of known failures

4. How to implement auto triaging?
Find previous instance of failed test case on same configuration
Check if its triaged
Compare the failure reason
Triage with same bug
Not Found
New Failure
Found No
Yes
Not matched
Matched

5. How to implement deviation notifications ?
Start to capture below data per run :
Total test cases
Failed test cases
Untriaged test cases (new failures)
Test execution time
While pushing new results, compare its data values with averaged data of last few runs (eg. 3 runs).
Measure the difference in the values, compare it with threshold values
If deviation is more than threshold, send notification with details.

6. How to implement deviation graphs ?
Database would have below data per run :
Total test cases
Failed test cases
Untriaged test cases (new failures)
Test execution time
Plot graphs based on this data for different suites
In case of multiple runs of 1 configuration, values should be averaged for all the runs

7. Example Graphs:
1. Failing test cases
Suite name

8. Example Graphs:
2. Execution time
Suite name

9. Example Graphs:
3. Total test cases
Suite name

10. Example deviation emails:
Hidden
Suite name

11. Increased confidence in the product as well as automation.
Benefits :
Early issue detection (esp. injections) will thereby help us achieve timelines easily.
Increased confidence in the product as well as automation.
Trends of various matrices over a period of time will help us take project critical decisions.
To know the reliability of what we do is always great. It gives us answers for various W-H family questions 

12. Author Biography
Varun is a Lead Software engineer at Adobe with 5.5 years of industry experience in automation and tools development. He is a part of Flash Runtime team at Adobe and have exposure of working in installation, deployment and runtime test automation across different platforms, including Mobile technology. This is where he came across with this practice and got successful in implementing it in automation.

13. Thank You!!!