1. “Its All In The Numbers” – Predictive Analytics in Software Testing
Name of Speaker : Wayne Sinclair
Company Name : ThoughtWorks Place: Hilton, Sandton

2. Agenda Introduction Typical approach in QA
What is Predictive Analytics
Proposed Approach
Model Building
Challenges
Conclusion

3. World Quality Report - Introduction
Key trends driving QA and testing
1600 executives, 32 countries
Use of predictive analytics is increasing
We are innovating with predictive analytics to help software teams to tap into historical data to learn and anticipate the future, further streamlining their delivery.
A striking development in the approach to QA is the uptake of continuous monitoring with predictive analysis.
Drive more efficient test cycles with predictive analytics and the ability to leverage big data, including production and user data.
40% of respondents foresee using predictive analysis as one of their automation techniques in the coming year.
Leveraging predictive analytics to bring speed efficiencies in all areas of quality decisions is one of the major emerging opportunities
The use of PA to determine or optimise test coverage emerges alongside TDD and BDD as the two most used approaches

4. World Quality Report - Trends
Use Big Data to learn from the past and predict the future
Speed efficiencies in decision making
Optimise test coverage
Shift right, use production
Analytics to feedback into testing

5. Typical Approach in QA Testing Tools Junit Cucumber Jira UFT RIT Jira
Test Management
Jira
UFT
Defects
Jira
UFT
Performance
LoadRunner
Jmeter
RPT

6. Typical Approach in QA - Challenges
Unable to identify issues that can lead to challenges in future
Unable to provide different stakeholder reports in preferred view
Constantly changing reporting needs
Different formats for different expectations
Changing expectations
Where are costs incurred
Are we able to meet deadline
Cannot generate expected report because data is in different sources

7. Typical Approach in QA - Expectations
Gives rise to a number of expectations that is difficult of impossible to meet
Which area of testing practice has greatest cost
Can we meet deadline at current velocity
Unable to generate expected report

8. What is Predictive Analytics?
Predictive analytics uses statistical algorithms to
Identify patterns in the data that you have
Accurately forecast how the environment will behave in the future
We need to understand how to make better decisions
Gives meaningful insight into testing process
Tells us how far off we are from reaching our goals in the delivery process
Complements Dev Ops
Leverage knowledge
React quickly
Reduce chance of failure
Use same techniques as other industries

9. Examples of Predictive Analytics Google Suggest
Amazon Product Recommendations
Facebook News Feeds
Other
spam filtering
Behavioural advertising *(Adwords, Facebook)

10. Commercial Use Cases Insurance Transport Telecoms Insurance
Underwriting
Pricing
Claims / Fraudulent
Marketing
Transport
Improve operations
Reduce costs
Better serve travellers
Likelihood of breakdown
Telecoms

11. Proposed Approach Testing Tools E T L Data Mart Predictive Analytics
Test Management
Defects
Performance
E T L
Data Mart
Predictive Analytics
Test
Manager
Financial
Manager
Programme Manager
Business Analyst

12. Paradigm Shift Testing Phase Traditional Way Test Planning
What is the test strategy for this project
Test Design
What are the test cases to be prepared and reviewed
Test Execution
What defects are identified
Test Closure
What are the metrics in the reports

13. Predictive Analytics Approach
Paradigm Shift
Testing Phase
Traditional Way
Predictive Analytics Approach
Test Planning
What is the test strategy for this project
What will be the effective test strategy for the upcoming sprint based on the historical data
Test Design
What are the test cases to be prepared and reviewed
What will be the priority test cases to be designed pertaining to the focus areas for testing
Test Execution
What defects are identified
What is the likelihood of defects arising due code changes
Test Closure
What are the metrics in the reports
Test reports tailor made to different stakeholders

14. Cyclomatic Complexity
LOC
Branch Count
Halstead Difficulty
Cyclomatic Complexity
No. Defects
Historical
Data
Training
Set
Testing
Set
LOC BC HD CC ND 20 3
11.63 2 4 28 9
6.43 5 15
14.57
LOC BC HD CC ND ? 36 4
16.8 11 3 40 5
18.1 2 35 6
16.9 9
Halstead Difficulty
Difficulty of understanding the program when reading or writing it and effort can be translated into the time it takes to code an application
Cyclomatic Complexity
A software metric (measurement), used to indicate the complexity of a program. It is a quantitative measure of the number of linearly independent paths through a program's source code.
Algorithm
LOC BC HD CC ND ? 36 4
16.8 11 3 40 5
18.1 2 35 6
16.9 9
Model

15. Predictive Model Building
Historical data with known outcome
Training set (build the model)
Testing set (test the model)
Build the model using an algorithm
Result is predictive model
Test the model (to check accuracy)

16. Commercial Applications
HP Enterprises
Infosys
ClicTest
HPE
Performance testing – data from prod
Defect convergence algorithm (ALM)
InfoSys
Defects and production incidents
Identify an optimal regression suite
Clic Test
ClicTest also helps to perform Predictive Analytics by using data to find out the root cause of the problems at the earliest.
ClicTest helps to generate stake-holder based reports which help in making rapid decisions

17. Pentaho – Open Source Testing Tools E T L (Pentaho Data Integration)
Test Management
Defects
Performance
E T L (Pentaho Data Integration)
Data
Mart
Predictive Analytics
Test
Manager
Financial
Manager
Programme Manager
Business Analyst

18. Benefits Proactively identify likely defects
Identify and prioritise critical test cases
Faster time to market
Improving quality in delivery
Using right skill set to achieve goal
Helps to reduce testing costs
Reduces risk

19. Challenges Management and Team buy in Planning Cost Skills
Need data (not a crystal ball)

20. Conclusion
Predictive Analytics is data driven technology that can be leveraged to:
Improve the testing process
Assist with decision-making
It can be used to:
Predict defect spread
Predict risk areas
Plan the priority of test cases

21. THANK YOU World Conference Next Generation Testing 2015
Speaker Name: Wayne Sinclair
ID:
Organized by: 1.21GIGA WATTS