HPHC - PERFORMANCE TESTING Dec 15, 2015 Natarajan Mahalingam

PERFORMANCE TESTING REQUESTS AT HPHC 2 © 2014 Harvard Pilgrim Health Care, Inc.  Applications & Services that fall under following category need to be evaluated for performance testing: – Newly Built Applications – Enhancements - major or minor – Upgrades (Hardware/Software) - major or minor – Change in overall Application Architecture – Implementation of new tools/layered products (e.g.: web analytics sensors) – Production incidents/failures related to Application performance.

CTS PERFORMANCE TEST REQUESTS 3 © 2014 Harvard Pilgrim Health Care, Inc. Monthly Release Activities Payload/XML Changes New Apps/ServicesProduction Incidents Infrastructure/ Patch Upgrades Service/App Enhancements

INITIATIVES 4 © 2014 Harvard Pilgrim Health Care, Inc.  Identify the list of Applications/Services that were never performance tested  Take new baseline for the apps/services listed as part of Gap Analysis  Find out list of apps/services currently experiencing performance issues in production  Focus on System Level Performance Testing  Conduct periodic ‘Awareness Workshops’ to all application owners, project managers, Dev and QA team about performance testing processes and capabilities  Continue to conduct cross training sessions to improve/enhance skills within the team to share the best practices and effective usage of HP Performance Center

WHAT IS SYSTEM PERFORMANCE TESTING 5 © 2012 Harvard Pilgrim Health Care, Inc.  Performance Testing carried predominantly at system level rather than at individual component level  Each and every component, interfaces and their integration points are carefully monitored and profiled for its parallel or un parallel nature of usage as it happens to operate in the production in any typical business day  Helps us to expose performance issues at overall system level especially during peak concurrent usage

NEED FOR SYSTEM LEVEL PERFORMANCE TESTING 6 © 2012 Harvard Pilgrim Health Care, Inc.  The performance testing carried against the individual component wouldn’t determine the system level performance  These measurements become invalid when multiple components/applications are working together (integrated fashion) wherein they all share common resources most of the time  The online user experience may not be same and consistent across any typical business day  It varies based on various background jobs running in parallel like Batches, essential system maintenance jobs (Because they share the resources)

SYSTEM PERFORMANCCE TESTING APPROACH 7 © 2012 Harvard Pilgrim Health Care, Inc. Application - A Application Pool - B Interaction - 2Interaction - 1 Batch Jobs - 1 Batch Jobs - 2 Interaction - 3Interaction - 4 Interaction - 5 System Maintenance Jobs

APPLICATION LEVEL PERFORMNCE TESTIG 8 © 2012 Harvard Pilgrim Health Care, Inc. Application - A Application Pool - B Interaction - 2Interaction - 1 Batch Jobs - 1 Batch Jobs - 2 Interaction - 3Interaction - 4 Interaction - 5 System Maintenance Jobs Application Performance Testing Project - 1

APPLICATION LEVEL PERFORMNCE TESTIG… 9 © 2012 Harvard Pilgrim Health Care, Inc. Application - A Application Pool - B Interaction - 2Interaction - 1 Batch Jobs - 1 Batch Jobs - 2 Interaction - 3Interaction - 4 Interaction - 5 System Maintenance Jobs Application Performance Testing Project - 2

SYSTEM PERFORMANCCE 10 © 2012 Harvard Pilgrim Health Care, Inc. Application - A Application Pool - B Interaction - 2Interaction - 1 Batch Jobs - 1 Batch Jobs - 2 Interaction - 3Interaction - 4 Interaction - 5 System Maintenance Jobs App[A] Business Scenarios Batch/System Maintenance Jobs App[C] Business Scenarios App[B] Business Scenarios

IDENTIFYING CONCURRENT SCENARIOS (Server vs Services Mapping) 11 © 2014 Harvard Pilgrim Health Care, Inc.

WHAT’S NEXT? 12 © 2014 Harvard Pilgrim Health Care, Inc.  Identify all the apps/services that were not performance tested in the past and add them to the “In Scope” list  Determine the number of applications /services physically running on the same server and/or running against the same server  Develop a new test scripts, suite and scenarios  Run a baseline testing  Provide necessary tuning recommendations and next course of action items  Work with development team to tune the service or app  Compare the results every month against baseline/benchmark  Retest and establish new baseline

HOW DO WE BASLEINE? 13 © 2012 Harvard Pilgrim Health Care, Inc. Planning Phase Identify the Applications/Components resides in the overall “SYSTEM” Collect the performance requirements Identify all the internal and external interfaces and dependencies Gather work load model across each and every components and their integration points Define and document “High Level System Performance Test Strategy” Development Phase Develop detailed system performance test strategy with all internal and external dependencies Conduct feasibility study to automate every interactions between different components and their access or integration points Develop the automation solution using any of the performance automation tools Identify and develop monitoring strategy for a overall end to end system at each component and integration level Implementation Phase Implement performance monitoring at all level Verify all developed automated solution (scripts) for performance testing Conduct smoke test to ensure the solution is smooth and cover the overall end to end system Baseline the performance metrics with respect to the overall system Document all baseline performance metrics for future releases comparison

NEW APPS/SERVICES REQUESTS 14 © 2014 Harvard Pilgrim Health Care, Inc. Review & Analyze Collect detailed requirements and understand the proposed changes Analyze the changes and assess the impacts to come up with right performance test strategy Develop Load Model & Scripts Analyze usage pattern and develop Load model Capture Business Scenarios, Develop scripts and Setup Test Data Execute Tests, Analyze & Report Execute appropriate load test s – Load, Stress and Endurance and Monitor/Capture the server metrics Analyze the results and report to respective project team or app team to decide on the next steps Tune/Rerun Work with Dev Team and DBA for any tuning activities Rerun the test until the issue is resolved Baseline Execute final round of tests Update and Keep the results as “Baseline” for any future reference

MONTHLY RELEASE/ENHANCEMENT REQUESTS 15 © 2014 Harvard Pilgrim Health Care, Inc. Review & Analyze Collect detailed requirements and understand the proposed changes Analyze the changes and assess the impacts to come up with right performance test strategy Validate/Update Scripts & Data Validate the existing scripts/data /load model and update them if required Execute some smoke test to make sure scripts are up to date and ready to start the performance test Execute Tests, Analyze & Report Execute appropriate load test s – Load, Stress and Endurance and Monitor/Capture the server metrics Analyze the results and report to respective project team or app team to decide on the next steps Tune/Rerun Work with Dev Team and DBA for any tuning activities Rerun the test until the issue is resolved Baseline Execute final round of tests Update and Keep the results as “Baseline” for any future reference

PRODUCTION INCIDENT REQUESTS 16 © 2014 Harvard Pilgrim Health Care, Inc. Review & Analyze Collect the required data and problem background information Analyze the problem statement and all detailed information Develop Load Model & Scripts Analyze usage pattern and develop Load model Capture Business Scenarios, Develop scripts and Setup Test Data Execute Tests, Analyze & Report Replicate production performance issue in test environment Monitor and capture server metrics Analyze the results and report to respective project team or app team to decide on the next steps Tune/Rerun Work with Dev Team and DBA for any tuning activities Rerun the test until the issue is resolved Baseline Execute final round of tests Update and Keep the results as “Baseline” for any future reference

BASIC PERFORMANCE ENGINEERING 17 © 2014 Harvard Pilgrim Health Care, Inc.  Define ‘Performance Requirements’ precisely  Arrive ‘Right’ Load Model  Configure ‘Monitors’ across all tiers  Collect all the metrics from all tiers  Enable JMX port and configure JVM monitoring for sure  Whether or Not the performance is good, always include captured metrics from all tiers while preparing the report  Include all metrics and graphs even though the performance is good (always explain why the performance is good or bad)  Start with ‘Transaction Response Time’ graph pattern analysis  Review all server resource graphs and review the pattern for the test duration  Always capture server resource usage metrics across all tiers before and after the test

Vertical Scaling:  Scaling by adding more JVM, CPU & RAM to your existing machine  adding more power (CPU, RAM) to your existing machine  limited to the capacity of a single machine Vertical Scaling & Horizontal Scaling 18 © 2014 Harvard Pilgrim Health Care, Inc. Horizontal Scaling:  Scaling by adding more machines/nodes into the existing pool/cluster JVM-4 CPU RAM JVM-4 CPU RAM JVM-4 CPU RAM Node - ANode - B

19 © 2014 Harvard Pilgrim Health Care, Inc. Q & A ?