FroNtier Stress Tests at Tier-0 Status report Luis Ramos LCG3D Workshop – September 13, 2006

Luis Ramos – September 13 th, 2006 FroNtier Stress Tests - 2/18 Outline 1.Test Plan 2.Test Setup 3.Main Results 4.Conclusions

Luis Ramos – September 13 th, 2006 FroNtier Stress Tests - 3/18 Objectives Develop a benchmark for Frontier servers –DB schema independent Build a tool that identifies performance bottlenecks of a given setup Performance analysis of the software stack –CORAL / Frontier plugin –Frontier Client –Squid –Frontier Servlet

Luis Ramos – September 13 th, 2006 FroNtier Stress Tests - 4/18 Test Plan How fast are the individual components? –Database –Application Server –Cache Server –Network Explore performance impact of: –Different data (strucutre, content, size, storage type, compression) –Different caching policies –Different access methods How do DB throughput, network bandwidth, payload size, # of clients or server CPU correlate?

Luis Ramos – September 13 th, 2006 FroNtier Stress Tests - 5/18 Metrics and Parameters Metrics: –Individual and total throughput –Server errors –CPU consumption/load (clients, frontier server, squid server) –Memory usage and disk space needs –Network bandwidth usage Parameters: –# of client nodes –# of test clients –Payload sizes –Database structure and content –Caching policy

Luis Ramos – September 13 th, 2006 FroNtier Stress Tests - 6/18 FroNtier Test Setup

Luis Ramos – September 13 th, 2006 FroNtier Stress Tests - 7/18 Server Test Setup Hardware setup: –1 server running Frontier & Squid: Dual Intel Xeon CPU 2.80GHz 2Gb RAM HD 150Gb Fast Ethernet (100Mbps) –1 Backend Oracle Database 10gR2 (cooldev) Software Setup: –FroNtier v3.3 –Frontier Squid v1.0rc4

Luis Ramos – September 13 th, 2006 FroNtier Stress Tests - 8/18 Client Test Setup Hardware: –Dedicated lxplus nodes Dual Pentium III 1GHz 500Mb RAM HD 6Gb Fast Ethernet (100Mbps) Software: –CORAL_1_5_3 –FrontierClient v2.5.1

Luis Ramos – September 13 th, 2006 FroNtier Stress Tests - 9/18 The Test Client C++ CORAL/FrontierPlugin test –Queries the server constantly –Gathers results –Outputs measures Until shutdown message received Python controller script –Starts a number of clients Manages test client ramp up –Gathers measures –Generates structured data for plotting

Luis Ramos – September 13 th, 2006 FroNtier Stress Tests - 10/18 Test Cases Network Analysis Throughput analysis –CORAL Oracle Plugin –CORAL Frontier Plugin Directly to FroNtier Through SQUID

Luis Ramos – September 13 th, 2006 FroNtier Stress Tests - 11/18 Network Analysis Test tool that checks network performance between multiple given client nodes and a single server –Generates TCP/IP traffic between clients and server –Each client shows an individual throughput Done using the netcat utility

Luis Ramos – September 13 th, 2006 FroNtier Stress Tests - 12/18 Throughput analysis Frontier Server Up to 150 clients running against a single server (direct FroNtier server access, no Squid involved) Old version of FroNtier -> no compression! New Frontier version with compression

Luis Ramos – September 13 th, 2006 FroNtier Stress Tests - 13/18 Throughput analysis Oracle, FroNtier and Squid Oracle vs Frontier Server vs Squid Cache Hits

Luis Ramos – September 13 th, 2006 FroNtier Stress Tests - 14/18 Throughput analysis Notes on previous plots Direct Frontier access –NOZIP version: 3MBps (bottleneck is the database) –ZIP version: 0,3MBps (bottleneck is the server CPU) –ZIP version can get 10 times slower than NOZIP version Production setup with 3 FroNtier nodes will perform better! Squid access –NOZIP version: 8MBps –ZIP version: 14MBps user preceived throughput can be bigger than the network throughput (due to compression) –ZIP version can get 2 times faster than NOZIP version Oracle access - 1,34MBps –First guess, should be faster then FroNtier direct access in any case! –Second thought, each client is repeatedelly creating DB connections which is quite heavy for OraclePlugin and not so much for Frontier because frontier servlet reuses connections

Luis Ramos – September 13 th, 2006 FroNtier Stress Tests - 15/18 Throughput analysis Some predictions CMS predicted real world access patern –10% direct FroNtier access –90% Squid access Factors (from previous slide) –SQUID_ZIP_time = SQUID_NOZIP_time / 2 –FRT_ZIP_time = FRT_NOZIP_time * 10 Some calculations: –Real NOZIP query time = 181% ZIP query time Prediciton test: –Tests run from FNAL nodes will produce new data in different network conditions

Luis Ramos – September 13 th, 2006 FroNtier Stress Tests - 16/18 Future work Run multi client throughput tests: –Using experiment DB content –Using COOL generated queries –Changing the ratio of cached queries –From FNAL to measure the impact of a poorer network connection –Analyzing the Frontier server error rate with the new Frontier v3.3

Luis Ramos – September 13 th, 2006 FroNtier Stress Tests - 17/18 Conclusions FroNtier is ready for production Some performance indicators were obtained –More real performance indicators should be now obtained from the production setup Test scripts developed –Next step: turn scripts easily reusable by others (Richard Hansen is running the test suite for ATLAS) Tests will continue!

Luis Ramos – September 13 th, 2006 FroNtier Stress Tests - 18/18 Questions? Ideas?