Computer Measurement Group, India CLOUD PERFORMANCE TESTING - KEY CONSIDERATIONS Abhijeet Padwal, Persistent Systems

Computer Measurement Group, India 2 Contents Cloud Case study – Performance Testing Requirements – Approach – Selection And Setup of Test Environment – Execution and Results – Challenges Conclusion

Computer Measurement Group, India 3 Cloud A style of computing in which scalable and elastic IT-enabled capabilities are delivered as a service using internet technologies. [Gartner 2014] – Scalable and elastic – Accessibility over the internet – Service-Oriented Types – Private – Public – Hybrid

Computer Measurement Group, India 4 Cloud (Contd.) Types of Services – Infrastructure as a Service (IaaS) – Platform as a Service (PaaS) – Software as a Service (SaaS) Service Providers – Amazon, Google, Microsoft Azure, Openstack Market Outlook – 17.7% growth is expected from 2011 through 2016

Computer Measurement Group, India 5 Case Study Customer – – A leading software company delivering Retail Solutions – Solutions includes POS, CRM, SCM and ERP. Application under test (AUT) – An enterprise class retail solution to manage the front end and backend operations within a retail store and – controlling the stores from the head office through a single application.

Computer Measurement Group, India 6 Performance Testing Requirement Performance Benchmarking for 3000 stores 5 most critical business scenarios spread across all components – Scenario 1 –Replication Replication of billing data from store to AS for 3000 stores. – Scenario 2 – Billing counter Multi user (minimum 25 parallel counters) performing billing transactions which include the Bill, Sales Return, Bill Cancellation, Lost Sales (in order of execution priority) with max line item not above 200 and minimum of 20 line items with Cash and Credit card as payment. – Scenario 3 – AS Access the reports to be checked while data from Store (minimum 20+ stores) is being updated to AS. – Scenario 4 – Operations Access stock management functions with line items namely with 5/10 users – Scenario 5 –Updates Download of patch for more than 100 stores simultaneously. Various patch sizes to be tested namely 50MB, 80MB, 100MB

Computer Measurement Group, India 7 Scenario 1 - Replication Load Test Requirement- Replication of data from Store to Server for 3000 stores. Each store would have 100 billing counters and each counter generating bill with 200 line items. 3 Parts of the scenario – 1.Collation of billing data from all the counters and generate the xml message files. 2.Transfer the xml message files from store to server (replication client -> replication server). 3.Extract the xml files and store the extracted billing data on the head office database.

Computer Measurement Group, India 8 Test Harness Setup To simulate this scenario a test harness was created which had 5 parts, – xml messages folders on injector machine – Vb based replication client (.exe) on injector machine – IIS and sql server based replication server – xml message folder on the head-office server and – Perfmon setup for monitoring the resource consumption on the AS as well as load injectors. Batch Response Time Finding out the number of Load injectors

Computer Measurement Group, India 9 Server Machine Configurations

Computer Measurement Group, India 10 Computer Measurement Group, India Setup Selection Usage Pattern Comparison *Other advantages of Azure – High amount of data and No cost for OS and S/W

Computer Measurement Group, India 11 Computer Measurement Group, India Azure Setup

Computer Measurement Group, India 12 Computer Measurement Group, India Initial Test Run Results on Azure

Computer Measurement Group, India 13 Computer Measurement Group, India Resource Utilization

Computer Measurement Group, India 14 Computer Measurement Group, India Moving Back to Local Setup Upgrade, CPU power 4 –>8 core and disk speed – 10k rpm -> 15 krpm Azure – CPU upgrade possible however disk array has been shared so no changes Local Setup Server with exact configuration available

Computer Measurement Group, India 15 Computer Measurement Group, India Results on Local Setup

Computer Measurement Group, India 16 Computer Measurement Group, India Challenges/ issues faced in Cloud during execution Managing multiple injectors and Switching between injectors to initiate the test Test data setup Monitoring Data transfer

Computer Measurement Group, India 17 Computer Measurement Group, India Conclusion Cloud provides an edge over the conventional on- premises test setups however… There are few disadvantages or challenges with the cloud environment which could severely hamper the purpose. All pros and cons should be studied in details and that too in the context of the load testing requirement and define the holistic strategy to perform load testing on cloud.

Computer Measurement Group, India 18 Computer Measurement Group, India Q&A

Computer Measurement Group, India 19 Computer Measurement Group, India Thanks Abhijeet Padwal Persistent Systems