Misha Borodin: ATLAS Computing ATLAS S&C week – 02 December Conditions DB access from Python tools M. Borodin (MEPhI/CERN)

Misha Borodin: ATLAS Computing ATLAS S&C week – 02 December 2010 Outlines lPython tools nList of tools nNew tool: AtlCoolIOVRunCheck.py nNew default logical connections nExpert tasks lObsolete tags lReduce MC DBRelease size lCondition DBRelease news nStatistics nCOOL access during reprocessing 2

Misha Borodin: ATLAS Computing ATLAS S&C week – 02 December 2010 Python tools lAtlCoolCopy.exe nhttps://twiki.cern.ch/twiki/bin/viewauth/Atlas/AtlCoolCopy lAtlCoolMerge, AtlCoolTransfer nhttps://twiki.cern.ch/twiki/bin/viewauth/Atlas/CoolPublishing lAtlCoolConsole: nhttps://twiki.cern.ch/twiki/bin/viewauth/Atlas/AtlCoolConsolehttps://twiki.cern.ch/twiki/bin/viewauth/Atlas/AtlCoolConsole lAtlCoolTag: nhttps://twiki.cern.ch/twiki/bin/viewauth/Atlas/CoolTagging lCondDSMgr nhttps://twiki.cern.ch/twiki/bin/viewauth/Atlas/CondDSMgr lNightly Tasks nhttps://twiki.cern.ch/twiki/bin/viewauth/Atlas/CoolNightlyTasks 3

Misha Borodin: ATLAS Computing ATLAS S&C week – 02 December 2010 New AtlCoolIOVRunCheck.py tool lObject: quick check IOV completeness for all folders for all run list before reprocessing Head tags closed. lInput nRun list nFolder nHead or particular tags lOutput nList of run and folder that haven’t data for input tags. lNote nScript doesn’t check that data correct. Only that some data exist. 4

Misha Borodin: ATLAS Computing ATLAS S&C week – 02 December 2010 New AtlCoolIOVRunCheck.py tool (Cont.) lWill be ready for users very soon. lTool was successfully tested for the COMCOND-REPP* HeadTags for autumn reprocessing lExample of using: /afs/cern.ch/user/a/atlcond/utils/AtlCoolIOVRunCheck.py –r./runlist.txt –f /TRT/Calib/RT –ht COMCOND-REPP lExample output: RUN : schema: ATLAS_COOLOFL_TRT Error No data found: folder : /TRT/Calib/RT tag : TrtCalibRt-Physics-REP-FieldOff

Misha Borodin: ATLAS Computing ATLAS S&C week – 02 December 2010 New default logical connections lIf you use for connection / ("COOLONL_TRT/COMP200”), the correspondence of this physical servers and schema will reads from a file dblookup.xml pointed to by the path variable CORAL_DBLOOKUP_PATH. lUsing asetup like `asetup ` is setup environment to ‘releases’ area, where dblookup.xml are taken from DBRelease (it has SQLIte connection by default) nFor access to Oracle/Frontier you need to use ‘;readoracle’: "COOLONL_TRT/COMP200;readoracle” lStarting from tag CoolConvUtilities connection to “ / ” will skip SQLIte by default n"COOLONL_TRT/COMP200” will connected to Oracle/Frontier lDirect link should used for connection to SQLite: n"sqlite://;schema=mysqlitefile.db;dbname=COMP200” 6

Misha Borodin: ATLAS Computing ATLAS S&C week – 02 December 2010 Expert task lExpert tasks: nCreating a new folder in COOL DB nModifying a folder description nAdding a payload column to a folder nRenaming a payload column nRenaming a folder nDeleting channels nCloning and creating BFieldMap tag lFor all question about expert task, send to me, Richard and Mathieu. 7

Misha Borodin: ATLAS Computing ATLAS S&C week – 02 December 2010 Obsolete tags lThere are now 172 Head tags for the real data and 188 Head tags for the MC data nDifficult to manage nSome tags are never used nSome tags are corresponding to obsolete software release l(Proposed by Paul), Obsolete Head tags should be marked and do not show by COOL tools: nAtlCoolConsole.py nAtlCoolTag.py (backtrace) nCOOL tag browser n…. lWorks in progress 8

Misha Borodin: ATLAS Computing ATLAS S&C week – 02 December 2010 Reduce MC DBRelease size lContinued growth in size and quantity of DBRelease can lead to problems in the future lSeveral steps were proposed (Vakho, Richard): nExclude obsolete Head tags from DBRelease nExclude pool condition files which are not connected to any instance in OFLP200 nExclude all tags which are not corresponded to any Global Tags  For folders with tags that are necessary, but are not corresponded to any Global tags, string “ ” should be added to the folder description. In this case folder will be fully replicated. lReplication scripts almost ready for new approach lIt will break backward compatibility 9

Misha Borodin: ATLAS Computing ATLAS S&C week – 02 December 2010 DB Release for reprocessing and HI lAutumn reprocessing statistics: nTwo sets of DB Release pairs were used: first-pass & main-pass n149 runs (124 in phase I + 25 in phase II), 171 Folders nDataset size: 57 GB in phase I + 13 GB in phase II lComparing with previous reprocessing, the build time increased for long runs, but it is fast enough nThe build time is proportional to the duration of the run lReprocessing experience identified further improvements for DB Release packaging procedures to assure fault tolerance to intermittent site- specific problems lDb-on-demand service was extended for supporting DB Releases build for bulk processing of HI runs at Tier-1s lDB Release assures robust Conditions DB access on the Grid nFurther Conditions DB Release improvements are in progress 10

Misha Borodin: ATLAS Computing ATLAS S&C week – 02 December 2010 Oracle access in reprocessing lDuring Autumn reprocessing several problems were observed during direct Oracle access lTo eliminate direct Oracle access form Athena python code we decided to read all Conditions DB data from SQLite only nAll connections from python code and IOVDBSvc will use SQLite from the Condition DB Release nThe solution was implemented to CoolConvUtilities (CoolConvUtilities ) and to db-on-demnd service lNew database access code was successfully tested lAthena DB connections circumventing IOVDBSvc are still the problem nIt is not yet clear how to create SQLite replicas for them 11

Misha Borodin: ATLAS Computing ATLAS S&C week – 02 December 2010 Conclusion lFor work with ATLAS Condition DB we have several very robust utilities. lCreating new tools for new use cases, improvement of usability and efficient existing one is ongoing. 12