2nd September 2008 1Richard Hawkings / Paul Laycock Conditions data handling in FDR2c  Tag hierarchies set up (largely by Paul) and communicated in advance.

Slides:



Advertisements
Similar presentations
Michigan State University 4/15/ Simulation for Level 2 James T. Linnemann Michigan State University NIU Triggering Workshop October 17, 1997.
Advertisements

David Adams ATLAS DIAL Distributed Interactive Analysis of Large datasets David Adams BNL March 25, 2003 CHEP 2003 Data Analysis Environment and Visualization.
CSC DQA and Commissioning Summary  We are responsible for the online and offline DQA for the CSC system, a US ATLAS responsibility  We are ready for.
F Fermilab Database Experience in Run II Fermilab Run II Database Requirements Online databases are maintained at each experiment and are critical for.
Database Deployment on OSG Yuri Smirnov BNL US ATLAS DDM operations and MC production Workshop, BNL September 28-29, 2006.
Abstract The automated multi-platform software nightly build system is a major component in the ATLAS collaborative software organization, validation and.
LHCb Quarterly Report October Core Software (Gaudi) m Stable version was ready for 2008 data taking o Gaudi based on latest LCG 55a o Applications.
Alexandre A. P. Suaide VI DOSAR workshop, São Paulo, 2005 STAR grid activities and São Paulo experience.
1 G A A new Document Control System “A new system to manage LIGO documents” Stuart Anderson Melody Araya David Shoemaker 29 September, 2008
D0 Farms 1 D0 Run II Farms M. Diesburg, B.Alcorn, J.Bakken, T.Dawson, D.Fagan, J.Fromm, K.Genser, L.Giacchetti, D.Holmgren, T.Jones, T.Levshina, L.Lueking,
ATLAS Metrics for CCRC’08 Database Milestones WLCG CCRC'08 Post-Mortem Workshop CERN, Geneva, Switzerland June 12-13, 2008 Alexandre Vaniachine.
Alignment Strategy for ATLAS: Detector Description and Database Issues
LHC: ATLAS Experiment meeting “Conditions” data challenge Elizabeth Gallas - Oxford - August 29, 2009 XLDB3.
ATLAS Scalability Tests of Tier-1 Database Replicas WLCG Collaboration Workshop (Tier0/Tier1/Tier2) Victoria, British Columbia, Canada September 1-2, 2007.
Plans for Trigger Software Validation During Running Trigger Data Quality Assurance Workshop May 6, 2008 Ricardo Gonçalo, David Strom.
Bookkeeping Tutorial. Bookkeeping & Monitoring Tutorial2 Bookkeeping content  Contains records of all “jobs” and all “files” that are created by production.
ATLAS Database Operations Invited talk at the XXI International Symposium on Nuclear Electronics & Computing Varna, Bulgaria, September 2007 Alexandre.
Databases E. Leonardi, P. Valente. Conditions DB Conditions=Dynamic parameters non-event time-varying Conditions database (CondDB) General definition:
Web application for detailed real-time database transaction monitoring for CMS condition data ICCMSE 2009 The 7th International Conference of Computational.
ATLAS Detector Description Database Vakho Tsulaia University of Pittsburgh 3D workshop, CERN 14-Dec-2004.
David Adams ATLAS DIAL status David Adams BNL November 21, 2002 ATLAS software meeting GRID session.
CERN Physics Database Services and Plans Maria Girone, CERN-IT
V.Sirotenko, July Status of Online Databases Currently there are 2 online Oracle Databases running on d0online cluster: 1.Production DB, d0onprd,
Misha Borodin: ATLAS Computing ATLAS S&C week – 02 December Conditions DB access from Python tools M. Borodin (MEPhI/CERN)
Organization and Management of ATLAS Nightly Builds F. Luehring a, E. Obreshkov b, D.Quarrie c, G. Rybkine d, A. Undrus e University of Indiana, USA a,
1 Database mini workshop: reconstressing athena RECONSTRESSing: stress testing COOL reading of athena reconstruction clients Database mini workshop, CERN.
3rd November Richard Hawkings Luminosity, detector status and trigger - conditions database and meta-data issues  How we might apply the conditions.
Trigger Offline Expert Report Simon George (RHUL), Ricardo Gonçalo (RHUL) Trigger General Meeting – 25 th November 2009.
CERN - IT Department CH-1211 Genève 23 Switzerland t COOL Conditions Database for the LHC Experiments Development and Deployment Status Andrea.
Process Architecture Process Architecture - A portion of a program that can run independently of and concurrently with other portions of the program. Some.
Conditions Metadata for TAGs Elizabeth Gallas, (Ryan Buckingham, Jeff Tseng) - Oxford ATLAS Software & Computing Workshop CERN – April 19-23, 2010.
Karsten Köneke October 22 nd 2007 Ganga User Experience 1/9 Outline: Introduction What are we trying to do? Problems What are the problems? Conclusions.
David Adams ATLAS DIAL: Distributed Interactive Analysis of Large datasets David Adams BNL August 5, 2002 BNL OMEGA talk.
Online (GNAM) and offline (Express Stream and Tier0) monitoring produced results during cosmic/collision runs (Oct-Dec 2009) Shifter and expert level monitoring.
Michele de Gruttola 2008 Report: Online to Offline tool for non event data data transferring using database.
Database authentication in CORAL and COOL Database authentication in CORAL and COOL Giacomo Govi Giacomo Govi CERN IT/PSS CERN IT/PSS On behalf of the.
Bookkeeping Tutorial. 2 Bookkeeping content  Contains records of all “jobs” and all “files” that are produced by production jobs  Job:  In fact technically.
Pixel DQM Status R.Casagrande, P.Merkel, J.Zablocki (Purdue University) D.Duggan, D.Hidas, K.Rose (Rutgers University) L.Wehrli (ETH Zuerich) A.York (University.
Oracle for Physics Services and Support Levels Maria Girone, IT-ADC 24 January 2005.
Prompt Calibration Loop 11 February Overview Prompt calibration loop in SCT –Provides ATLAS with conditions data used for the bulk reconstruction.
November 1, 2004 ElizabethGallas -- D0 Luminosity Db 1 D0 Luminosity Database: Checklist for Production Elizabeth Gallas Fermilab Computing Division /
11th November Richard Hawkings Richard Hawkings (CERN) ATLAS reconstruction jobs & conditions DB access  Conditions database basic concepts  Types.
Victoria, Sept WLCG Collaboration Workshop1 ATLAS Dress Rehersals Kors Bos NIKHEF, Amsterdam.
1 A Scalable Distributed Data Management System for ATLAS David Cameron CERN CHEP 2006 Mumbai, India.
Status of tests in the LCG 3D database testbed Eva Dafonte Pérez LCG Database Deployment and Persistency Workshop.
ATLAS FroNTier cache consistency stress testing David Front Weizmann Institute 1September 2009 ATLASFroNTier chache consistency stress testing.
Conditions Metadata for TAGs Elizabeth Gallas, (Ryan Buckingham, Jeff Tseng) - Oxford ATLAS Software & Computing Workshop CERN – April 19-23, 2010.
ID Week 13 th of October 2014 Per Johansson Sheffield University.
Distributed Data Management Miguel Branco 1 DQ2 status & plans BNL workshop October 3, 2007.
M.Frank, CERN/LHCb Persistency Workshop, Dec, 2004 Distributed Databases in LHCb  Main databases in LHCb Online / Offline and their clients  The cross.
The NA62RunControl: Status update Nicolas Lurkin School of Physics and Astronomy, University of Birmingham NA62 TDAQ Meeting – CERN, 10/06/2015.
ATLAS The ConditionDB is accessed by the offline reconstruction framework (ATHENA). COOLCOnditions Objects for LHC The interface is provided by COOL (COnditions.
Initial Planning towards The Full Dress Rehearsal Michael Ernst.
GGUS summary (3 weeks) VOUserTeamAlarmTotal ALICE7029 ATLAS CMS LHCb Totals
Joe Foster 1 Two questions about datasets: –How do you find datasets with the processes, cuts, conditions you need for your analysis? –How do.
Dario Barberis: ATLAS DB S&C Week – 3 December Oracle/Frontier and CondDB Consolidation Dario Barberis Genoa University/INFN.
AliRoot survey: Calibration P.Hristov 11/06/2013.
ALICE Full Dress Rehearsal ALICE TF Meeting 02/08/07.
David Adams ATLAS DIAL Distributed Interactive Analysis of Large datasets David Adams BNL May 19, 2003 BNL Technology Meeting.
Software Release Build Process and Components in ATLAS Offline Emil Obreshkov for the ATLAS collaboration.
L1Calo Databases ● Overview ● Trigger Configuration DB ● L1Calo OKS Database ● L1Calo COOL Database ● ACE Murrough Landon 16 June 2008.
L1Calo DBs: Status and Plans ● Overview of L1Calo databases ● Present status ● Plans Murrough Landon 20 November 2006.
DB and Information Flow Issues ● Selecting types of run ● L1Calo databases ● Archiving run parameters ● Tools Murrough Landon 28 April 2009.
Database Replication and Monitoring
3D Application Tests Application test proposals
Remaining Online SW Tasks
Conditions Data access using FroNTier Squid cache Server
Offline Database Report
ATLAS DC2 & Continuous production
Presentation transcript:

2nd September Richard Hawkings / Paul Laycock Conditions data handling in FDR2c  Tag hierarchies set up (largely by Paul) and communicated in advance  No real problems uploading data to the correct tag  Calibration experts starting to deal with ‘real’ IOVs (data valid for calib n period)  New POOL file registration scripts worked fine  Calibration users need to be in AFS group atlcond:poolcond  Consider doing calibration uploads from a ‘calibration’ account, not personal ones?  No instances of data in COOL without corresponding (or wrong) POOL file upload  No use of run-signoff database pages yet  System was not ready and integrated yet (holidays; too busy with other things)  But only one set of runs, and all calibrations were ‘accepted’ - no real test  Handling of detector status information works technically  Merging and transfer to LBSUMM folder (for ESD/AOD) still done by hand  Limited mapping of DQ histograms to status flags restricts usefulness  Need to make sure this improves for real data  Need to clarify how detector status flags are dealt with in ES1, ES2 processing

2nd September Richard Hawkings / Paul Laycock Conditions DB access problems  Big problems in Tier-0 conditions DB access Thursday night/ Friday morning  Combination of several factors  2/4 of Oracle server nodes got into trouble and restarted  Kernel patch being applied this week, some interdependencies not fully understood yet  Server full of ‘stuck’ connections which were never released or cleaned up - deadlock  Very high load due to FDR2 bulk reprocessing and cosmics reprocessing going on in parallel, plus FCT, ATN, RTT, TCT tests, plus user jobs  All jobs accessing Oracle directly, no use of SQLite replicas at present  Replica only useful once the run is ended online - applicable to ES2, bulk reco only  Vulnerability in that ALL Athena jobs accessing Oracle use same reader account  Limit of 800 concurrent sessions, now changed to 4 x 800  Each Athena job holds O(10) connections in parallel until end of first event (one per subdetector schema) - typically for 5 minutes or so. Vulerable to ‘deadlock’  Further actions being pursued  Deploy SQLite replica for bulk processing (but not for cosmics / express stream)  Use a dedicated COOL reader account for Tier-0 jobs - guarantee # connections  Reduce connection load from Athena jobs (short/long term actions)

2nd September Richard Hawkings / Paul Laycock Next steps - discussion needed  Work on conditions DB access problems  Deployment of SQLite replicas to be used where possible  Start to setup tag hierarchies for first data  Separate top-level tags to be used by HLT, monitoring, Tier-0, reprocessing  Define calibration loop model for first data  Cosmics processing has no calibration loop, and several ‘express’ streams  Same plan for single beam running, or move to ‘calibration loop’  Calibration 24hrs might be needed for code fixes even if no prompt calibration can be done yet, might have multiple processings at Tier-0  What to do for first collisions  Sign-off tool and Tier-0/conditions integration to support all this..?