ATLAS Distributed Database Services Client Library D. Malon, A. Vaniachine ANL T. Wenaus BNL R. Hawkings, Yu. Shapiro CERN A. Pérus, RD Schaffer LAL, Orsay J. Boudreau, V. Tsulaia U. Pittsburgh XV International Conference on Computing in High Energy and Nuclear Physics T.I.F.R., Mumbai, India February 13-17, 2006

CHEP06, Mumbai, India February 13-17, 2006 Alexandre Vaniachine (ANL)2 Outlook Client Library rationale and architecture Client Library services integration in ATLAS software framework Athena Use of these services by ATLAS major database applications - the Geometry DB and Conditions DB Integration through the ConnectionService module in the CORAL layer of the common LCG persistency project POOL

CHEP06, Mumbai, India February 13-17, 2006 Alexandre Vaniachine (ANL)3 Coordinated Development In the ATLAS Computing Model access to terabytes of data stored in relational databases is required for applications widely distributed on the grids As grid computing technologies mature, development must focus on database and grid integration New developments are required to bridge the gap between data accessibility and the increasing power of grid computing In ATLAS coordinated changes occurs on both sides: Server-side technology developments (contribution id #38) Client-side technology developments are described in this presentation

CHEP06, Mumbai, India February 13-17, 2006 Alexandre Vaniachine (ANL)4 Complex Data Access Patterns Athena Applications Online Applications User Applications Algorithms COOL/POOL/CORAL Services D a t a b a s e C o n n e c t i o n s Database Servers In ATLAS Computing Model various applications require access to the data resident in the relational databases

CHEP06, Mumbai, India February 13-17, 2006 Alexandre Vaniachine (ANL)5 Foundation Layer ATLAS Database Project is responsible for ensuring the integration and operation of the full distributed database and data management infrastructure To achieve the integration goal the Distributed Database Services client library serves as a unique foundation layer for enforcing policies, following rules, establish best practices and encode logic to deliver efficient, secure and reliable database connectivity to applications in a heterogeneous Distributed Database Services environment

CHEP06, Mumbai, India February 13-17, 2006 Alexandre Vaniachine (ANL)6 Driven by Experience In preparation for data taking, the ATLAS experiment has run a series of large-scale computational exercises to test and validate multi-tier distributed data grid solutions under development ATLAS experience in prototypes and production systems of Data Challenges and Combined Test Team provided various database connectivity requirements for applications: connection management, online-offline uniformity, server indirection, etc. For example, the dynamics of ATLAS distributed database services requires a single point-of-control over server indirection: The logical-to-physical database server mapping which is similar to the logical-to-physical mapping of file names on the grids To address these requirements we developed, tested and deployed ATLAS Database Client Library

CHEP06, Mumbai, India February 13-17, 2006 Alexandre Vaniachine (ANL)7 At the beginning of the Client Library project we collected the requirements in the Client Library document Connection Management Indirection Layer Robust Remote Access Scalability Lightweight, etc Through emphasis on a practical approach, and by avoiding too much abstraction and flexibility at early stage we concentrate on solving actual problems of ATLAS operations workflow Client Library Document

CHEP06, Mumbai, India February 13-17, 2006 Alexandre Vaniachine (ANL)8 Layer Decomposition The outer layer provides management of database drivers, database connections and Connections/Servers lists The extensible inner library layer is composed of a number of technology specific database drivers Currently providing support for MySQL and Oracle Oracle MySQL ODBC To hide the complexity of heterogeneous database technologies the library is separated in two layers:

CHEP06, Mumbai, India February 13-17, 2006 Alexandre Vaniachine (ANL)9 Layered Architecture A finer structure of the library architecture Application Interface Connections Pool Manager Singleton Oracle Driver MySQL Driver Policies Connections Pool Drivers Common Interface Server Indirection User Credentials

CHEP06, Mumbai, India February 13-17, 2006 Alexandre Vaniachine (ANL)10 Connection Management This module provides client-side connection management capabilities including: connection pooling logical-physical mapping for database connection string implements policies of retries and connection timeout failover to replica

CHEP06, Mumbai, India February 13-17, 2006 Alexandre Vaniachine (ANL)11 Connections Logic Application Call DBConnect obj Load Drivers Set Policies Get Connection Release Connection Connections Pool Manager No conn for This driver Different Serv Similar ConnConn Exist Check out Idle Connections PoolConnections-in-use Pool Check ActiveSwitch UserCheck no of ConnRedirection Establish Connection Redirect to replica If overloaded

CHEP06, Mumbai, India February 13-17, 2006 Alexandre Vaniachine (ANL)12 Library Configuration Configuration parameters can be changed through the interface for specific run using calls to: setRetries, setTimeout, setMaxconn These parameters can be changed permanently via configuration file For indirection mechanism data can be added to IndirectionMap file (XML) Plans for a dynamic (e.g. DB-based) IndirectionMap mechanism that wouldn't require user intervention updating a local file also preserving local file for failover if the DB connection fails

CHEP06, Mumbai, India February 13-17, 2006 Alexandre Vaniachine (ANL)13 “User” Interface Note: this is a low level library - here “User” is core software developer User interface consists of just two calls: getConnection (string conn_str, void *conn_ptr) releaseConn(string conn_str, void *conn_ptr) Connection string URI syntax from POOL/RAL/CondDB: technology://host=;user=;pass=;db=;specific_initialization=; At the end of the string a technology specific initialization listed A concise/elegant but very powerful interface: A plethora of use cases covered on the next slide

CHEP06, Mumbai, India February 13-17, 2006 Alexandre Vaniachine (ANL)14 Client Library Packaging Client Library available at ClientLibrary package for modules implementation: plug-ins abstract interface indirection module connections pool management configuration module library interface Plus technology specific implementation packages : MySQLConnection OracleConnection (don’t even dare to look into this)

CHEP06, Mumbai, India February 13-17, 2006 Alexandre Vaniachine (ANL)15 Server Indirection Many data persistency problems are similar for all technologies: POOL files or relational databases Thus, we re-use a familiar grid concept of logical-to-physical mapping of files and file replicas Implementation based on the POOL catalog indirection architecture MySQL and Oracle catalogues hold logical-to-physical mapping Logical server names and its replicas Both catalogues are synchronized: Change in one causes change in the other Per default time user application updates the XML Catalogue in its local working area from the central MySQL/Oracle catalogue

CHEP06, Mumbai, India February 13-17, 2006 Alexandre Vaniachine (ANL)16 Indirection Mechanism Now in COOL/POOL/CORAL A single point-of-control over server indirection The ConnectionService module represents integration of ATLAS Client Library components with the software from the LCG POOL project

CHEP06, Mumbai, India February 13-17, 2006 Alexandre Vaniachine (ANL)17 Framework Integration Athena service resolves logical-physical database server mapping and access to the connection management library Resolution can be done against the local file or directly against the Catalogue residing on Oracle It is now in use by Geometry DB application by default Replica server failover was found to be practical during Oracle service downtime Another (unexpected) indirection benefit – protection against the backward incompatible Oracle service name changes that affects distributed releases

CHEP06, Mumbai, India February 13-17, 2006 Alexandre Vaniachine (ANL)18 Failover Experience Campus security was using specialized hardware/software to detect and block intrusion attacks on campus Oracle servers which has student records for all 25,000 students Inadvertently, they also blocked outbound Oracle connections for ATLAS jobs the hardware cannot selectively block in one direction since it is port based Configurable number of connection retries: POOL/RelationalPlugins/oracle: level[Error] ORA-12152: TNS:unable to send break message (authenticating)... POOL/RelationalPlugins/oracle: level[Error] ORA-12152: TNS:unable to send break message (authenticating)... Failover to configurable replica: POOL/odbc/ODBCDriverSelector: level[Info] [MySQL] POOL/RelationalPlugins/mysql/odbc: level[Info] Connection URI : mysql://dbdevel1.usatlas.bnl.gov/ATLASDD POOL/RelationalPlugins/mysql/odbc: level[Info] Connected via MySQL driver... Follow-up queries repeat the retries/failover sequence: POOL/RelationalPlugins/oracle Error ORA-12152: TNS:unable to send break message (authenticating)... Improvements in Client Library failover strategy are now identified P.S. Campus security now resolved the issue (by excluding ATLAS Oracle RAC CERN IP's)

CHEP06, Mumbai, India February 13-17, 2006 Alexandre Vaniachine (ANL)19 Conclusions As grid computing technologies mature, development must focus on database and grid integration New technologies are required to bridge the gap between data accessibility and the increasing power of grid computing used for distributed event production and processing Changes must happen both on the server side and on the client side Server-side developments are reported in another track (id #38) Client technology developments were encapsulated in the ATLAS Database Client Library and are now integrated in COOL/POOL/CORAL software (contribution id #329)

CHEP06, Mumbai, India February 13-17, 2006 Alexandre Vaniachine (ANL)20 Recommendation The steady progress and scheduled deliverables in ATLAS Client Library development would not be possible without contributions by Yulia Shapiro Until her recent departure from CERN Yulia was the main ATLAS Client Library developer Yulia is an accomplished strong programmer with proven track record of solid contributions to ATLAS software Please let us know of opportunities for her return to ATLAS or other LHC experiments