Database Deployment on OSG Yuri Smirnov BNL US ATLAS DDM operations and MC production Workshop, BNL September 28-29, 2006

September 28,2006 Yuri Smirnov (BNL) 2 Outline TATLAS MySQL services for OSG at US ATLAS Tier1 TPanda Database services on OSG TDatabase replication CERN -> BNL TMonitoring tools for MySQL servers: MySQLStat TDatabase release deployment on OSG TProduction: 11.0.X and database deployment on OSG TProduction: 12.0.X (project-built) and database deployment T New DB deployment approach in

September 28,2006 Yuri Smirnov (BNL) 3 BNL MySQL DB services for US ATLAS and OSG  BNL Tier1 not only contributes to ATLAS OSG production (half of U.S. production in 2006 was done at Tier1, remainder at Tier2 sites), but provides many different reliable services, including MySQL DataBases support  ATLAS production and development MySQL servers for OSG at BNL:  dbpro production cluster (PandaDB)  adbpro development cluster (PandaDevDB, grid-enable MySQL development)  db1 (ConditionsDB IOV+payload, DBs for subdetectors LAr, TileCal, etc., LRC, DataSetDB, DialDB, Geometry DB Nova,etc.)  dbdevel1 (ATLASDD, MagdaFC)  dbdevel2 (TAG DB, PandaArchiveDB, PandaLogDB)  dms01/dms02 ( DQ2 services and catalogs, will support Production LRC soon)  gums/vo (GUMS and VOMS db for OSG and LCG)

September 28,2006 Yuri Smirnov (BNL) 4 Panda database production service Tdbpro MySQL cluster: 2 nodes dbpro01 and dbpro02 Teasy recovery: if one node goes down, the 2-nd node contunues to work without interruption in providing the database service T in-memory technology for Panda DB (NDB engine) Tto provide quick reliable access to data PandaDB keeps information about production and analysis jobs completed withing ~ one week. TPandaArchiveDB on dbdevel2 server:  Uses the same structure as in-memory PandaDB,  Disk-resident tables (MyISAM engine)  Keeps track of all production and user analysis jobs completed/failed on OSG since the beginnig 2006  Size is ~1GB at the moment TPandaLogDB on dbdevel2 stores log-file extracts of failed jobs. TExcellent monitoring tool Panda-monitor to both PandaDB and PandaArchiveDB developed by Torre Wenaus

September 28,2006 Yuri Smirnov (BNL) 5 Panda-monitor

September 28,2006 Yuri Smirnov (BNL) 6 Database replication at BNL TOracle – Oracle stream-replication (presentation of Iris) TMySQL – MySQL replication:  DataBases: Geometry DB, Conditions DB  collected the first experience CERN-BNL ATLAS DB replication  procedure using both mysqldump and on-line replication  thanks to Sasha Vaniachine and Vakho Tsulaia for participation TOracle – MySQL replication:  DataBase: TAG DB.  use case : Oracle CERN to MySQL BNL (push)  tool: Octopus replicator ( Java-based extraction, transformation and loading)  thanks to Julius Hrivnac, Ed May and Kristo Karr for successful collaboration  More details in Twiki:

September 28,2006 Yuri Smirnov (BNL) 7 MySQL servers monitoring We use different monitoring tools for MySQL servers and services: -MySQLStat (status of different BNL, CERN, OSG, LCG servers ) -Nagios -Ganglia

September 28,2006 Yuri Smirnov (BNL) 8 MySQL servers monitoring

September 28,2006 Yuri Smirnov (BNL) 9 Database release deployment on OSG TCSC Panda production on OSG uses ATLAS software releases installation (dist.kit) at Tier1 and Tier2s TTwo types of production releases:  11-branch: , ; txt jobOptions, shell-based trfs;  12-branch: , , (soon): project-built releases, python-based trfs and jobOptions. TProduction jobs work with 2 ATLAS core DataBases:  Geometry Database;  Conditions Database (both IoV and payload)

September 28,2006 Yuri Smirnov (BNL) X DB releases deployment T11.0.X release itself and job-transformations use two different areas:  $APP/atlas_app/atlas_rel/11.0.X/  $APP/atlas_app/atlas_rel/kitval/KitValidation/JobTransforms/JobTransforms X-Y TGeometry Database (in SQLite-files) is a part of ATLAS-release installation (dist. kit), for instance:  in geomDB_sqlite file, size ~11MB  Location: $APP/atlas_app/atlas_rel/11.0.5/atlas/offline/data TConditions Database runs on MySQL sever:  Master at CERN: atlasdbdev.cern.ch  Replica at BNL: db1.usatlas.bnl.gov

September 28,2006 Yuri Smirnov (BNL) X DB releases deployment T12.0.X releases use job-transformations as a part of the installation (and new versions can be easily added):  $APP/atlas_app/atlas_rel/12.0.X/  $APP/atlas_app/atlas_rel/12.0.X/AtlasProduction/12.0.X.Y TGeometry DB (Sqlite-files) and Conditions DB come with the installation as a Database Release 2.Z (default version):  $APP/atlas_app/atlas_rel/12.0.X/DBRelease/2.Z  DB release cycle may be not synchronized with ATLAS release (one ATLAS release can work with several DB releases) TIf a new version of DB release is available it can be installed on OSG as a new subdirectory under the same ATLAS-release, for instance:  $APP/atlas_app/atlas_rel/12.0.2/DBRelease/2.6  $APP/atlas_app/atlas_rel/12.0.2/DBRelease/2.7

September 28,2006 Yuri Smirnov (BNL) 12 New DB deployment approach in T New technology for production DB deployment in release is prepared by Martin and Vakho. T Can be tested already now with under transformations (DB-release 2.7.1). T Idea of dynamic DB-release installation:  trf itself (not the installation script) should check if the corresponding Database Release (SQLite files) is available and install it (through DQ2 subscriptions) directly on the WN where the job using this trf runs. Can be removed later when necessary.