1. Performance Testing In Agile
Abhilasha Vyas
Deep Pandey

2. Contents
Performance Testing In Agile : Challenges And Risks, Opportunities
Performance Test In Agile : Key Enablers
Progressive and Adaptive Baseline
Demo and discussion

3. Performance Testing In Agile : Challenges And Risks
Performance testing requires a stable code and that would mean doing it at sprint hardening but that would be a deviation from Agile methodology
There is no one fit work load model identification in Agile as business and Infrastructure workload could change based on functionalities delivered
Shorter Development Cycles Require More Tests in Less Time thus need automation tools which could closely bind with application's native language
Agile also promises more closer/frequent interaction with the capacity teams and hence performance testers need to factual data to ensure meaningful discussion.

4. Performance Testing In Agile : Opportunities
Agile allows frequent revisits at the baselines and SLA
It provides opportunity for closer interaction with developers and Infrastructure teams thereby bringing great deal of increase in skill set and overall process
It allows opportunity component based SLA and addressing performance issues at various integration issues early
Allows more opportunity to be proactive and leverage/implement Performance Modelling

5. Performance Test In Agile : Key Enablers used in SGM
Adaptive and Progressive Baseline techniques
Central Reporting Dashboard providing KPI in Sprint by Sprint comparison mode and bolstering Common performance goals
Aggregate & Structure Metrics to move from real time analytics to predictive performance modelling.
Test tools Familiar to Dev and capture performance from presentation as well as internal layers.
More Open source tool adoption to eliminate the tools dependency.

6. Key Take Away
Baseline techniques that could help improve overall process.
Reporting ideas which work well within Agile Practices.
Ideas to leverage log analytics and develop accelerator which capture performance from presentation as well as internal layers
Open source framework for Aggregating & Structure Metrics monitoring data ( open source)

7. Adaptive and Progressive baseline

8. Progressive Baseline New Baseline for Next Sprint SC1 SC2 SC3 SC4
Baseline (Sprint A )
Sc1
Sc2
New Baseline for Next Sprint
SC1
SC2
SC3
SC4
New functionality
Sc3
Sc4

9. Adaptive Baseline
SLA by Business  ERT (Expected Response Time) Actual performance in production (based on historical data) ART(Actual Response Time) Transaction response time  TRT Ideal scenario TRT<ART<ERT Risk ART<TRT < ERT : Implies that although application is performing within the SLA , there is degradation in performance against historical trends. Deviation = ART – TRT , is positive and increasing  application performance is deteriorating on every release. How do we negate the risk?

10. Adaptive Baseline (De Mystified)
Weekly monitoring data

11. Adaptive Baseline (De Mystified)
Weekly monitoring data

12. Adaptive Baseline (De Mystified)
Weekly monitoring data

13. Reporting Dashboard within Agile

14. Challenge
Reporting process is often weak and is managed in mails/file servers/share points which makes it difficult to make historical trending and analysis of performance over extended period of time.
Inaccurate baselines and performance testing goals. Various stakeholders have different view of performance and hence collaboration is hindered between teams.
Agile integration: Often teams needed framework in which reporting could be quick and give clear indication of application performance on key KPI on release on release basis . But this was hindered by the fact that it involves specialized skills of performance testers for reporting.

15. Tools capability
Reporting dashboard (Plug and Play with most automation tools) allows performance against the historical trends and compare performance of key transactions against various sprint cycles. This allows teams being proactive and identify key areas to address performance issues.
In built Performance Analytics on Key KPI ( Key Performance Indicators) to view average and percentile figures.
Feed performance test summary results out of current performance tools ( Load runner/J meter/QTP/Selenium) to the dashboard for effective reporting and easy understanding of performance in CI environment.

16. REPORTING DASHBOARD (DEMO)
Sprint comparison view for Two Transactions

17. REPORTING DASHBOARD (DEMO)
Single Sprint Multiple Transaction View

18. REPORTING DASHBOARD (DEMO)
Multi Sprint Average/Percentile View

19. Aggregating & Structure Metrics monitoring data
Grafana/Kibana based GUI representation for messaging system

20. ELK implementation
Collect & Store performance metrics in dashboard and use basic analytics .
Leverage Log analytics to aggregate performance from presentation and internal layers for messaging system.

21. “Advantage Analytics” In Performance Management
Often test and production environment differ in terms of infrastructure and performance testing results differ in two environments. Analytics could be used to predict the performance under varying parameters. Key steps would involve;
Identify the key resource parameters from monitoring data which are directly related to measured performance. Not all parameters contribute to performance equally.
Split the monitoring data into test and train data. Run more combination of tests to gather more test data.
Create a model for predicting the performance based on varying parameter's on the train data. Test thoroughly for various data sets and compute the accuracy.
Identify the KPI in production which vary and test the predicted results from model created in lower environment.

22. Q&A

23. SOURCE
Deploy a Multivendor Strategy for Availability and Performance Monitoring, Jonah Kowall, 4 Dec 2012 Retrieved from
EMA Radar for “Advanced Performance Analytics (APA) Use Cases, Dennis Drogseth, Dec 2012 Retrieved from
Analytics: The real-world use of big data. IBM Global Services, Available at ibm.com/services/us/gbs/thoughtleadership/ibv-big-data-at-work.html and supported by unpublished BMC analysis