1. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t COOL Conditions Database for the LHC Experiments Development and Deployment Status Andrea Valassi (CERN IT-DM) R. Basset, G. Pucciani (CERN IT-DM) M. Clemencic (CERN PH / LHCb) S. A.Schmidt, M. Wache (Mainz / ATLAS) IEEE-NSS 2008, 23rd October 2008 Data Management Group

2. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t NSS 2008 – Andrea Valassi COOL Status - 2 Outline Introduction Deployment overview Ongoing developments Performance tests and optimization –Query optimization on small data samples –Scalability tests on large simulated samples –Support of actual deployment with real data Conclusions

3. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t NSS 2008 – Andrea Valassi COOL Status - 3 What is the COOL software? Manage conditions data of Atlas and LHCb –Time variation (validity) and versioning (tags) e.g. calibration, alignment –Common project of Atlas, LHCb, CERN IT Support for several relational databases –Oracle, MySQL, SQLite, Frontier –Access to SQL from C++ via the CORAL libraries

4. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t NSS 2008 – Andrea Valassi COOL Status - 4 COOL deployment overview Similar setups in Atlas and LHCb –“3D” distributed DB model – not specific to COOL Two separate Oracle servers at CERN (online, offline) Distributed Oracle replicas at the experiment Tier-1 sites –Replication via the Oracle Streams technology Capture changes at source, propagate, apply at target (G. Dimitrov, F. Viegas) 3D Distributed Database Deployment model (D. Duellmann)

5. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t NSS 2008 – Andrea Valassi COOL Status - 5 Deployment status Setup is complete for both experiments –T0 online/offline DBs, T1 sites (6 LHCb, 10 Atlas) Distributed tests are very useful for COOL –Several lessons from Atlas tests in 2007 already Most T0 and T1 databases were up by Q4 2006 –New issues identified and addressed in 2008 e.g. user-level read access during Streams write activity COOL Status - 5 Much larger data rates in ATLAS NSS 2008 – 23rd October 2008

6. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t NSS 2008 – Andrea Valassi COOL development status Mature functionality and code base –First release in April05, latest (2.5.0) in June08 –Test-driven development, automated nightly tests for all supported relational database backends Maintenance and code consolidation –Internal refactoring of existing functionalities –New platforms (OSX/Intel, gcc43, VS9, SLC5…) –New versions of external software –Fix bugs/issues identified in real-life deployment A few new developments too –Functionality enhancements (e.g. transactions) –Performance optimization COOL Status - 6

7. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t NSS 2008 – Andrea Valassi Performance optimization Main focus: performance for Oracle DBs –Master T0 database for both Atlas and LHCb Proactive performance test on small tables –Test main use cases for retrieval and insertion –Query times should be flat as tables grow larger e.g. avoid full table scans Oracle performance optimization strategy –Basic SQL optimization (fix indices and joins) –Use hints to stabilize execution plan for given SQL Instability from unreliable statistics, bind variable peeking COOL Status - 7

8. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t NSS 2008 – Andrea Valassi Performance optimization example Good SQL strategy (COOL231). Good Oracle statistics. Bad execution plan due to “bind variable peeking” (no hints). Systematic tests of known causes of instabilities –Bind variable “peeking”, missing or stale “statistics” –Instabilities observed in the Atlas 2007 tests (e.g. CNAF vs. Lyon) –Stable performance after adding Oracle hints Bad SQL strategy (COOL230). Retrieval time for 10 IOVs is larger for IOVs at the end of the relational table (full table scan). Good SQL strategy (COOL231). Stable execution plan thanks to the use of hints.

9. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t NSS 2008 – Andrea Valassi Romain Basset Scalability tests Proactive performance test on large tables –Stable insertion and retrieval rates (>1k rows/s) –Simulate data sets for 10 year of LHC operation Test case: Atlas “DCS” data –Measured voltages, currents... –Largest Atlas data set 1.5 GB (2M IOVS) / day To do next: data partitioning –Goal: ease data management –Evaluating Oracle partitioning Test possible performance impact

10. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t NSS 2008 – Andrea Valassi Oracle DB Server CoralServer Oracle Plug-in Oracle Client Connection Pool CORAL API FIREWALL Oracle OCI protocol (OPEN PORTS) CORAL protocol Oracle OCI protocol (NO OPEN PORTS) COOL Status - 10 Future deployment model COOL API Oracle Plugin Oracle OCI Connection Pool CORAL API User Code DB access via CORAL server –Address secure authentication and connection multiplexing –Development still in progress See next talk by Zsolt Molnar Only minimal changes in COOL User Code Coral Plugin COOL API Connection Pool CORAL API

11. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t NSS 2008 – Andrea Valassi COOL Status - 11 Conclusions COOL: conditions DB for Atlas and LHCb –Support for several relational database backends Mature code, but development is not over –Performance optimization is the highest priority Proactive tests and support for real deployment issues –Evaluating models for data partitioning Distributed deployment setup is ready –Waiting for more data from LHC!

12. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t NSS 2008 – Andrea Valassi COOL Status - 12 Reserve slides

13. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t NSS 2008 – Andrea Valassi COOL Status - 13 COOL collaborators Core development team Andrea Valassi (CERN IT-DM) –80% FTE (core development, project coordination, release mgmt) Marco Clemencic (CERN LHCb) –20% FTE (core development, release mgmt) Sven A. Schmidt (Mainz ATLAS) –20% FTE (core development) Martin Wache (Mainz ATLAS) –80% FTE (core development) Romain Basset (CERN IT-DM) –50% FTE (performance optimization) + 50% FTE (scalability tests) On average, around 2 FTE in total for development since 2004 Collaboration with users and other projects Richard Hawkings and other Atlas users and DBAs The CORAL, ROOT, SPI and 3D teams Former collaborators G. Pucciani, D. Front, K. Dahl, U. Moosbrugger

14. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t NSS 2008 – Andrea Valassi COOL data model Modeling of conditions data objects –System-managed common “metadata” Data items: many tables, each with many channels Interval of validity - “IOV” [since, until] Versioning information - with handling of interval overlaps –User-defined schema for “data payload” Support for fields of simple C++ types Main use case: event reconstruction –Lookup data payload valid at a given event time COOL Status - 14

15. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t NSS 2008 – Andrea Valassi Functionality enhancements (work in progress) Tagging enhancements –“Partial tag locking” (prevent tag modifications) Data retrieval enhancements –Payload queries (fetch time for given calibration) Default use case: fetch calibration at given validity time Database connection enhancements –User control over database transactions –DB session sharing between COOL sessions COOL Status - 15