Performance Testing Design By Omri Lapidot Symantec Corporation Mobile: At SIGiST Israel Meeting November 2007

Agenda Why Test Performance? How not to test Performance Testing Phases –Designing a Usage Model –Tests and environments creation –Load and tune The Politics of Performance Tests Use Case: Symantec I3 Summary

System under test assumptions User Interface which multiple users manipulate concurrently Core applications Core database with data loading mechanism Dynamic components customers choose and install

–Without performance testing, functionality is likely to suffer in increased loads Most QA is done on under-loaded environments and unrealistic configurations designed for locating specific functional problems –Performance tuning is a repetitive, cyclic process Performance problems can not be addressed via normal patching process unlike functional problems –Determine realistic configuration recommendations Customers and field personnel need to know what are the recommended hardware and software configurations –Let the field people know what they are up against If we release the product with known performance issues, field personnel should know what problems should they expect and how to work around them Why test for performance?

–“just load the system with users” Set up a system Blast it with gazillion of virtual users Check V next to “Performance Tests” –The problems of overloading Chasing ghosts wastes dev resources Reduce QA accountability –The problems of under loading We won’t find real performance issues Reduce QA accountability How not to test

–Usage Model design –Tests and environments creation –Load and Tune Performance testing phases

Load is generated by three factors: –User activity User actions on user interface Typically: select operations, configuration changes –External automated activity Data flows into the system Typically: insert operations –Internal automated activity Data manipulation Typically: aggregation and data purge activity, internal processes Performance testing phases Designing a Usage Model

Mapping the loading metrics of the system we want to test –What causes the load on the system (Loading Parameters)? –Settle for a finite number of metrics Obtaining the loading metrics –Objective sources Customer logs Customer databases Customer support calls –Subjective sources Field personnel Support personnel Product marketing Selected customers Performance testing phases Designing a Usage Model

For each customer size: –Users How many concurrent users? What is the user activity distribution? How long does a typical session last? –Hardware What is the estimated hardware? How is it configured? –Software How many Alerts are defined? How many are set off each second? How many SLAs? Etc’ Performance testing phases Designing a Usage Model

–Hardware –Monitors –Data emulation –User emulation Performance testing phases Tests and environments creation

–(Optional) Run a baseline run –Load and Tune Performance testing phases Load and Tune Increase Load Yes Finished YesNo Run load Does the system handle the load? Tune Code No Did we reached expected thresholds ?

–The three phases of QA-Dev interaction Cooperation Shock Retaliation –Back yourself up Involve dev personnel in test design Involve field personnel in test design Know how each metric you use is relevant to real life The Politics Of Performance Tests

–A Symantec I3 system consists of the following components: Customer’s monitored applications and servers Data collectors Data Loaders Database (PW) GUI Use case: Symantec I3

–Who are the large, medium and small customers? –Support file analysis PW team had size estimation for each monitored technology Support files hold hundreds of customers each with the amount of instances in each monitored technology When the analysis was through, we knew what is the average configuration for small, medium and large customers Use case: Symantec I3

–Field interaction Product experts feedback Subjective data source –Performance Test Plan Formalize our goals, tests, tools and schedule Use case: Symantec I3

–Test preparation Synthetic Vs Real data User activity patterns Monitoring –No thresholds Unable to open performance bugs No measurable software standards –Testing Set up a system Transmit data Generate virtual users Measure user experience, internal processes, hardware resource usage Use case: Symantec I3

–Summary Our reports pressured dev to improve GUI response time by 40% We issued a table of hardware recommendations based on the expected load Two most problematic application within the system identified and will be dealt with We changed the way load is measured in our organization Dev and support teams now contact us with performance problems, performance tests are now integral part of QA tests Use case: Symantec I3

Agenda Why Test Performance? How not to test Performance Testing Phases –Designing a Usage Model –Tests and environments creation –Load and tune The Politics of Performance Tests Use Case: Symantec I3 Summary

–Releasing software without performance testing is like releasing new running shoes without running with them. –Always design a usage model What are the loading parameters? Retrieve data from objective source Verify data with subjective source

Thank You Omri Lapidot