1. Cultivating Software Quality In Cloud Via Load Testing Tools
SOFTWARE QUALITY CONFERENCE
PACIFIC NW
Cultivating Software Quality In Cloud Via Load Testing Tools
Presented by
Pooja Jain (Intel)
Kingsum Chow* (Alibaba)
October 2016
*work performed while at Intel

2. Cultivating Software Quality In Cloud Via Load Testing Tools
SOFTWARE QUALITY CONFERENCE
PACIFIC NW
Cultivating Software Quality In Cloud Via Load Testing Tools
Pooja Jain, Emad Guirguis, Pranita Maldikar, Li Chen
Intel Corporation
Kingsum Chow
Alibaba Infrastructure Service
October 2016

3. OUTLINE Motivation Introduction to Load Testing 3-Step Methodology
SOFTWARE QUALITY CONFERENCE
PACIFIC NW
OUTLINE
Motivation
Introduction to Load Testing
3-Step Methodology
Case Study
Key Learnings

4. Software quality in the cloud
SOFTWARE QUALITY CONFERENCE
SOFTWARE QUALITY CONFERENCE
PACIFIC NW
PACIFIC NW
Application Monitoring (JMeter/Faban)
Load Testing
Application
Application Monitoring (logs)
Operating System
System Monitoring (sar)
Hardware
CPU Monitoring (perf)

5. Application Monitoring
SOFTWARE QUALITY CONFERENCE
SOFTWARE QUALITY CONFERENCE
PACIFIC NW
PACIFIC NW
Throughput focuses on maximizing the amount of work completed by an application in a specific period of time. Examples of how throughput might be measured include:
The number of transactions completed in a given time
The number of jobs that a batch program can complete in an hour
The number of database queries that can be completed in an hour
Responsiveness refers to how quickly an application or system responds with a requested piece of data. Examples include:
How quickly a desktop UI responds to an event
How fast a website returns a page
How fast a database query is returned

6. Load TESTING Determining the Goals of Load Testing
SOFTWARE QUALITY CONFERENCE
SOFTWARE QUALITY CONFERENCE
PACIFIC NW
PACIFIC NW
Determining the Goals of Load Testing
Hardware Resource Utilization: CPU, MEMORY, I/O, NETWORK
Operations/Use cases to test: Determining one or more set of transactions
Throughput: maximum throughput, transaction mix…
Response time: maximum response time, 99th %, 95th %, …
Think Time: Zero, range, cycle-time, …

7. Open Source Load Testing Tools
SOFTWARE QUALITY CONFERENCE
SOFTWARE QUALITY CONFERENCE
PACIFIC NW
PACIFIC NW
Faban is an open-source framework developed by Sun for developing and running benchmarks
Faban supports both multi-tier server and simple micro-benchmark
Features:
Specific support for HTTP
Easily configurable runtime, injection rates, ramp-up & steady state times.
Faban comes with built-in system-level monitoring.
Faban ideal for simulation of realistic workloads.

8. Open Source Load Testing Tools: Apache JMeter
SOFTWARE QUALITY CONFERENCE
SOFTWARE QUALITY CONFERENCE
PACIFIC NW
PACIFIC NW
The Apache JMeter™ application is open source software, a 100% pure Java application designed to load test functional behavior and measure performance of many server/protocol types: Web - HTTP, HTTPS, SOAP/ REST, Database via JDBC, LDAP, JMS, SMTP , etc
Highly Extensible core:
Pluggable Samplers allow unlimited testing capabilities for data analysis, visualization and data manipulation.
Scriptable Samplers (BeanShell, BSF-compatible languages and JSR223-compatible languages)
JMeter is ideal for cases where testing of many protocols and complex scenarios is required.

9. Load Testing SOFTWARE QUALITY CONFERENCE SOFTWARE QUALITY CONFERENCE
PACIFIC NW
PACIFIC NW
Design the Experiment
Initiate requests with # simulated users
Wait for Server Response
Record performance data
Save responses and generate stats & graphs
Report

10. 3-Step Process Determine the saturation point
SOFTWARE QUALITY CONFERENCE
SOFTWARE QUALITY CONFERENCE
PACIFIC NW
PACIFIC NW
Determine the saturation point
Correlate Performance data
Identify Software Solutions

11. Correlate system performance data with load testing statistics
3-Step Process
SOFTWARE QUALITY CONFERENCE
SOFTWARE QUALITY CONFERENCE
PACIFIC NW
PACIFIC NW
Load Testing
System Performance
Application Monitoring
#Users
Think-time
100
200
300
CPU %
N/w (Mbps) 30
400 60
800 95
1000
Through-put (ops/s)
Response Time (s) 50
0.2
100
0.7
125
2.5
Correlate system performance data with load testing statistics

12. Case STUDY: Ticket Monster
SOFTWARE QUALITY CONFERENCE
SOFTWARE QUALITY CONFERENCE
PACIFIC NW
PACIFIC NW
A java web application hosted on JBoss Application Server.
Accepts HTTP requests and responses with dynamic or static web page.
Throughput is measured : Total Requests served / Overall-Time
Think time = 0 seconds
Driver : JMeter Load Testing Tool
Very chatty

13. Ticket Monster Determining the saturation point
SOFTWARE QUALITY CONFERENCE
SOFTWARE QUALITY CONFERENCE
Ticket Monster
PACIFIC NW
PACIFIC NW
Determining the saturation point
Application performance with increase in load.
What are the scalability issues.

14. (Load Testing Tool- CLient)
SOFTWARE QUALITY CONFERENCE
SOFTWARE QUALITY CONFERENCE
Ticket Monster
PACIFIC NW
PACIFIC NW
Correlate the performance Data
System Data
Server)
Application Data
(Load Testing Tool- CLient)
Application Data
(Server logs)
System Data
(Client)

15. Ticket Monster Identify a solution and test it.
SOFTWARE QUALITY CONFERENCE
SOFTWARE QUALITY CONFERENCE
Ticket Monster
PACIFIC NW
PACIFIC NW
Identify a solution and test it.
In this case: Increased the Heap size 85

16. Ticket Monster ? ? Iteration2: Looking in to the details
SOFTWARE QUALITY CONFERENCE
SOFTWARE QUALITY CONFERENCE
Ticket Monster
PACIFIC NW
PACIFIC NW
Iteration2: Looking in to the details ? ?
Why is the dip? What is causing this?

17. Ticket Monster Identifying Solutions with experts.
SOFTWARE QUALITY CONFERENCE
SOFTWARE QUALITY CONFERENCE
Ticket Monster
PACIFIC NW
PACIFIC NW
Identifying Solutions with experts.
In this case: A data collection tool!
Cpu is saturated in both cases, disabling the tool, subsides the over head and recovery time.
Perhaps better throughput

18. Key Learnings Discover performance issues with load testing tools
SOFTWARE QUALITY CONFERENCE
SOFTWARE QUALITY CONFERENCE
PACIFIC NW
PACIFIC NW
Discover performance issues with load testing tools
Correlate system performance data with load testing statistics
Identify solutions for software quality issues

19. Any Questions? Thank you!
SOFTWARE QUALITY CONFERENCE
PACIFIC NW
Any Questions?
Thank you!

20. 3-Step Process/ Methodology
SOFTWARE QUALITY CONFERENCE
SOFTWARE QUALITY CONFERENCE
PACIFIC NW
PACIFIC NW
Determine the saturation point
Correlate Performance data
Identify Software Solutions

21. Ticket Monster Identifying Solutions with experts.
SOFTWARE QUALITY CONFERENCE
SOFTWARE QUALITY CONFERENCE
Ticket Monster
PACIFIC NW
PACIFIC NW
Identifying Solutions with experts.
In this case: A data collection tool!
300