Performance Testing In Agile Abhilasha Vyas Deep Pandey

Contents Performance Testing In Agile : Challenges And Risks, Opportunities Performance Test In Agile : Key Enablers Progressive and Adaptive Baseline Demo and discussion

Performance Testing In Agile : Challenges And Risks Performance testing requires a stable code and that would mean doing it at sprint hardening but that would be a deviation from Agile methodology There is no one fit work load model identification in Agile as business and Infrastructure workload could change based on functionalities delivered Shorter Development Cycles Require More Tests in Less Time thus need automation tools which could closely bind with application's native language Agile also promises more closer/frequent interaction with the capacity teams and hence performance testers need to factual data to ensure meaningful discussion.

Performance Testing In Agile : Opportunities Agile allows frequent revisits at the baselines and SLA It provides opportunity for closer interaction with developers and Infrastructure teams thereby bringing great deal of increase in skill set and overall process It allows opportunity component based SLA and addressing performance issues at various integration issues early Allows more opportunity to be proactive and leverage/implement Performance Modelling

Performance Test In Agile : Key Enablers used in SGM Adaptive and Progressive Baseline techniques Central Reporting Dashboard providing KPI in Sprint by Sprint comparison mode and bolstering Common performance goals Aggregate & Structure Metrics to move from real time analytics to predictive performance modelling. Test tools Familiar to Dev and capture performance from presentation as well as internal layers. More Open source tool adoption to eliminate the tools dependency.

Key Take Away Baseline techniques that could help improve overall process. Reporting ideas which work well within Agile Practices. Ideas to leverage log analytics and develop accelerator which capture performance from presentation as well as internal layers Open source framework for Aggregating & Structure Metrics monitoring data ( open source)

Adaptive and Progressive baseline

Progressive Baseline New Baseline for Next Sprint SC1 SC2 SC3 SC4 Baseline (Sprint A ) Sc1 Sc2 New Baseline for Next Sprint SC1 SC2 SC3 SC4 New functionality Sc3 Sc4

Adaptive Baseline SLA by Business  ERT (Expected Response Time) Actual performance in production (based on historical data) ART(Actual Response Time) Transaction response time  TRT Ideal scenario TRT<ART<ERT Risk ART<TRT < ERT : Implies that although application is performing within the SLA , there is degradation in performance against historical trends. Deviation = ART – TRT , is positive and increasing  application performance is deteriorating on every release. How do we negate the risk?

Adaptive Baseline (De Mystified) Weekly monitoring data

Adaptive Baseline (De Mystified) Weekly monitoring data

Adaptive Baseline (De Mystified) Weekly monitoring data

Reporting Dashboard within Agile

Challenge Reporting process is often weak and is managed in mails/file servers/share points which makes it difficult to make historical trending and analysis of performance over extended period of time. Inaccurate baselines and performance testing goals. Various stakeholders have different view of performance and hence collaboration is hindered between teams. Agile integration: Often teams needed framework in which reporting could be quick and give clear indication of application performance on key KPI on release on release basis . But this was hindered by the fact that it involves specialized skills of performance testers for reporting.

Tools capability Reporting dashboard (Plug and Play with most automation tools) allows performance against the historical trends and compare performance of key transactions against various sprint cycles. This allows teams being proactive and identify key areas to address performance issues. In built Performance Analytics on Key KPI ( Key Performance Indicators) to view average and percentile figures. Feed performance test summary results out of current performance tools ( Load runner/J meter/QTP/Selenium) to the dashboard for effective reporting and easy understanding of performance in CI environment.

REPORTING DASHBOARD (DEMO) Sprint comparison view for Two Transactions

REPORTING DASHBOARD (DEMO) Single Sprint Multiple Transaction View

REPORTING DASHBOARD (DEMO) Multi Sprint Average/Percentile View

Aggregating & Structure Metrics monitoring data Grafana/Kibana based GUI representation for messaging system

ELK implementation Collect & Store performance metrics in dashboard and use basic analytics . Leverage Log analytics to aggregate performance from presentation and internal layers for messaging system.

“Advantage Analytics” In Performance Management Often test and production environment differ in terms of infrastructure and performance testing results differ in two environments. Analytics could be used to predict the performance under varying parameters. Key steps would involve; Identify the key resource parameters from monitoring data which are directly related to measured performance. Not all parameters contribute to performance equally. Split the monitoring data into test and train data. Run more combination of tests to gather more test data. Create a model for predicting the performance based on varying parameter's on the train data. Test thoroughly for various data sets and compute the accuracy. Identify the KPI in production which vary and test the predicted results from model created in lower environment.

Q&A

SOURCE http://www.evolven.com/blog/downtime-outages-and-failures-understanding-their-true-costs.html http://www.evolven.com/blog/2011-devastating-outages-major-brands.html http://www.informationweek.com/it-downtime-costs-$265-billion-in-lost-revenue/d/d-id/1097919? https://en.wikipedia.org/wiki/2012_RBS_Group_computer_system_problems#cite_note-14 https://scrumalliance.org/community/articles/2013/2013-may/agile-performance-testing-an-experimental-approach http://betanews.com/2015/03/13/apples-app-store-outage-cost-25-million-in-lost-revenue/ Deploy a Multivendor Strategy for Availability and Performance Monitoring, Jonah Kowall, 4 Dec 2012 Retrieved from http://www.gartner.com/id=2263415 EMA Radar for “Advanced Performance Analytics (APA) Use Cases, Dennis Drogseth, Dec 2012 Retrieved from http://media.cms.bmc.com/documents/EMA-APA_Q4_2012_Radar-BMC_Profile.pdf Analytics: The real-world use of big data. IBM Global Services, 2012. Available at http://www- 935.ibm.com/services/us/gbs/thoughtleadership/ibv-big-data-at-work.html and supported by unpublished BMC analysis http://www.it-director.com/business/costs/content.php?cid=12043