“Its All In The Numbers” – Predictive Analytics in Software Testing Name of Speaker : Wayne Sinclair Company Name : ThoughtWorks Place: Hilton, Sandton http://1point21gws.com/devops/2017/DevOps_Microservices_and_Containers_Summit_Johannesburg/

Agenda Introduction Typical approach in QA What is Predictive Analytics Proposed Approach Model Building Challenges Conclusion

World Quality Report - Introduction Key trends driving QA and testing 1600 executives, 32 countries Use of predictive analytics is increasing We are innovating with predictive analytics to help software teams to tap into historical data to learn and anticipate the future, further streamlining their delivery. A striking development in the approach to QA is the uptake of continuous monitoring with predictive analysis. Drive more efficient test cycles with predictive analytics and the ability to leverage big data, including production and user data. 40% of respondents foresee using predictive analysis as one of their automation techniques in the coming year. Leveraging predictive analytics to bring speed efficiencies in all areas of quality decisions is one of the major emerging opportunities The use of PA to determine or optimise test coverage emerges alongside TDD and BDD as the two most used approaches https://www.capgemini.com/thought-leadership/world-quality-report-2016-17

World Quality Report - Trends Use Big Data to learn from the past and predict the future Speed efficiencies in decision making Optimise test coverage Shift right, use production Analytics to feedback into testing https://www.capgemini.com/thought-leadership/world-quality-report-2016-17

Typical Approach in QA Testing Tools Junit Cucumber Jira UFT RIT Jira Test Management Jira UFT Defects Jira UFT Performance LoadRunner Jmeter RPT

Typical Approach in QA - Challenges Unable to identify issues that can lead to challenges in future Unable to provide different stakeholder reports in preferred view Constantly changing reporting needs Different formats for different expectations Changing expectations Where are costs incurred Are we able to meet deadline Cannot generate expected report because data is in different sources

Typical Approach in QA - Expectations Gives rise to a number of expectations that is difficult of impossible to meet Which area of testing practice has greatest cost Can we meet deadline at current velocity Unable to generate expected report

What is Predictive Analytics? Predictive analytics uses statistical algorithms to Identify patterns in the data that you have Accurately forecast how the environment will behave in the future We need to understand how to make better decisions Gives meaningful insight into testing process Tells us how far off we are from reaching our goals in the delivery process Complements Dev Ops Leverage knowledge React quickly Reduce chance of failure Use same techniques as other industries

Examples of Predictive Analytics Google Suggest Amazon Product Recommendations Facebook News Feeds Other Email spam filtering Behavioural advertising *(Adwords, Facebook)

Commercial Use Cases Insurance Transport Telecoms Insurance Underwriting Pricing Claims / Fraudulent Marketing Transport Improve operations Reduce costs Better serve travellers Likelihood of breakdown Telecoms

Proposed Approach Testing Tools E T L Data Mart Predictive Analytics Test Management Defects Performance E T L Data Mart Predictive Analytics Test Manager Financial Manager Programme Manager Business Analyst

Paradigm Shift Testing Phase Traditional Way Test Planning What is the test strategy for this project Test Design What are the test cases to be prepared and reviewed Test Execution What defects are identified Test Closure What are the metrics in the reports

Predictive Analytics Approach Paradigm Shift Testing Phase Traditional Way Predictive Analytics Approach Test Planning What is the test strategy for this project What will be the effective test strategy for the upcoming sprint based on the historical data Test Design What are the test cases to be prepared and reviewed What will be the priority test cases to be designed pertaining to the focus areas for testing Test Execution What defects are identified What is the likelihood of defects arising due code changes Test Closure What are the metrics in the reports Test reports tailor made to different stakeholders

Cyclomatic Complexity LOC Branch Count Halstead Difficulty Cyclomatic Complexity No. Defects Historical Data Training Set Testing Set LOC BC HD CC ND 20 3 11.63 2 4 28 9 6.43 5 15 14.57 LOC BC HD CC ND ? 36 4 16.8 11 3 40 5 18.1 2 35 6 16.9 9 Halstead Difficulty Difficulty of understanding the program when reading or writing it and effort can be translated into the time it takes to code an application Cyclomatic Complexity A software metric (measurement), used to indicate the complexity of a program. It is a quantitative measure of the number of linearly independent paths through a program's source code. Algorithm LOC BC HD CC ND ? 36 4 16.8 11 3 40 5 18.1 2 35 6 16.9 9 Model

Predictive Model Building Historical data with known outcome Training set (build the model) Testing set (test the model) Build the model using an algorithm Result is predictive model Test the model (to check accuracy)

Commercial Applications HP Enterprises Infosys ClicTest HPE Performance testing – data from prod Defect convergence algorithm (ALM) InfoSys Defects and production incidents Identify an optimal regression suite Clic Test ClicTest also helps to perform Predictive Analytics by using data to find out the root cause of the problems at the earliest. ClicTest helps to generate stake-holder based reports which help in making rapid decisions

Pentaho – Open Source Testing Tools E T L (Pentaho Data Integration) Test Management Defects Performance E T L (Pentaho Data Integration) Data Mart Predictive Analytics Test Manager Financial Manager Programme Manager Business Analyst

Benefits Proactively identify likely defects Identify and prioritise critical test cases Faster time to market Improving quality in delivery Using right skill set to achieve goal Helps to reduce testing costs Reduces risk

Challenges Management and Team buy in Planning Cost Skills Need data (not a crystal ball)

Conclusion Predictive Analytics is data driven technology that can be leveraged to: Improve the testing process Assist with decision-making It can be used to: Predict defect spread Predict risk areas Plan the priority of test cases

THANK YOU World Conference Next Generation Testing 2015 Speaker Name: Wayne Sinclair Email ID: sinclair@thoughtworks.com Twitter: @tester_RSA Organized by: 1.21GIGA WATTS http://1point21gws.com/devops/2017/DevOps_Microservices_and_Containers_Summit_Johannesburg/ http://1point21gws.com/BlockchainSummit/2017/Blockchain_Summit_Dub