TAG and iELSSI Progress Elisabeth Vinek, CERN & University of Vienna on behalf of the TAG developers group.

Slides:



Advertisements
Similar presentations
Distributed databases
Advertisements

The Latest news … and Future of ATLAS Databases Elizabeth Gallas - Oxford ATLAS Software & Computing Workshop CERN November 29 to December 3, 2010.
Transaction.
MIS 385/MBA 664 Systems Implementation with DBMS/ Database Management Dave Salisbury ( )
Grid and CDB Janusz Martyniak, Imperial College London MICE CM37 Analysis, Software and Reconstruction.
ABCSG - Distributed Database 1 Data Management Distributed Database Data Replication.
Extensible Scalable Monitoring for Clusters of Computers Eric Anderson U.C. Berkeley Summer 1997 NOW Retreat.
Overview Distributed vs. decentralized Why distributed databases
Conditions Metadata for TAGs Elizabeth Gallas, (Ryan Buckingham, Jeff Tseng) - Oxford ATLAS Software & Computing Workshop CERN – April 19-23, 2010.
Chapter 12 Distributed Database Management Systems
Systems Analysis and Design in a Changing World, 6th Edition 1 Chapter 6.
A tool to enable CMS Distributed Analysis
LHC Experiment Dashboard Main areas covered by the Experiment Dashboard: Data processing monitoring (job monitoring) Data transfer monitoring Site/service.
CERN - IT Department CH-1211 Genève 23 Switzerland t Monitoring the ATLAS Distributed Data Management System Ricardo Rocha (CERN) on behalf.
LCG 3D StatusDirk Duellmann1 LCG 3D Throughput Tests Scheduled for May - extended until end of June –Use the production database clusters at tier 1 and.
Database Design – Lecture 16
ATLAS DQ2 Deletion Service D.A. Oleynik, A.S. Petrosyan, V. Garonne, S. Campana (on behalf of the ATLAS Collaboration)
SITools Enhanced Use of Laboratory Services and Data Romain Conseil
ATLAS Data Periods in COMA Elizabeth Gallas - Oxford ATLAS Software and Computing Week CERN – April 4-8, 2011.
David Adams ATLAS ATLAS Distributed Analysis David Adams BNL March 18, 2004 ATLAS Software Workshop Grid session.
OSG Area Coordinator’s Report: Workload Management February 9 th, 2011 Maxim Potekhin BNL
Update on Database Issues Peter Chochula DCS Workshop, June 21, 2004 Colmar.
- Iain Bertram R-GMA and DØ Iain Bertram RAL 13 May 2004 Thanks to Jeff Templon at Nikhef.
Grid Status - PPDG / Magda / pacman Torre Wenaus BNL U.S. ATLAS Physics and Computing Advisory Panel Review Argonne National Laboratory Oct 30, 2001.
10 1 Chapter 10 Distributed Database Management Systems Database Systems: Design, Implementation, and Management, Sixth Edition, Rob and Coronel.
Database Systems: Design, Implementation, and Management Ninth Edition Chapter 12 Distributed Database Management Systems.
David Adams ATLAS ATLAS Distributed Analysis Plans David Adams BNL December 2, 2003 ATLAS software workshop CERN.
John Gordon STFC-RAL Tier1 Status 9 th July, 2008 Grid Deployment Board.
 Three-Schema Architecture Three-Schema Architecture  Internal Level Internal Level  Conceptual Level Conceptual Level  External Level External Level.
ATLAS in LHCC report from ATLAS –ATLAS Distributed Computing has been working at large scale Thanks to great efforts from shifters.
And Tier 3 monitoring Tier 3 Ivan Kadochnikov LIT JINR
Distributed Database. Introduction A major motivation behind the development of database systems is the desire to integrate the operational data of an.
Towards a Global Service Registry for the World-Wide LHC Computing Grid Maria ALANDES, Laurence FIELD, Alessandro DI GIROLAMO CERN IT Department CHEP 2013.
Conditions Metadata for TAGs Elizabeth Gallas, (Ryan Buckingham, Jeff Tseng) - Oxford ATLAS Software & Computing Workshop CERN – April 19-23, 2010.
EGEE is a project funded by the European Union under contract IST “Interfacing to the gLite Prototype” Andrew Maier / CERN LCG-SC2, 13 August.
SC4 Planning Planning for the Initial LCG Service September 2005.
Integration of the ATLAS Tag Database with Data Management and Analysis Components Caitriana Nicholson University of Glasgow 3 rd September 2007 CHEP,
Claudio Grandi INFN-Bologna CHEP 2000Abstract B 029 Object Oriented simulation of the Level 1 Trigger system of a CMS muon chamber Claudio Grandi INFN-Bologna.
The ATLAS TAGs Database - Experiences and further developments Elisabeth Vinek, CERN & University of Vienna on behalf of the TAGs developers group.
MND review. Main directions of work  Development and support of the Experiment Dashboard Applications - Data management monitoring - Job processing monitoring.
CASTOR project status CASTOR project status CERNIT-PDP/DM October 1999.
David Adams ATLAS Datasets for the Grid and for ATLAS David Adams BNL September 24, 2003 ATLAS Software Workshop Database Session CERN.
Pavel Nevski DDM Workshop BNL, September 27, 2006 JOB DEFINITION as a part of Production.
CERN IT Department CH-1211 Genève 23 Switzerland t COOL Performance Tests ATLAS Conditions Database example Romain Basset, IT-DM October.
1 A Scalable Distributed Data Management System for ATLAS David Cameron CERN CHEP 2006 Mumbai, India.
Network integration with PanDA Artem Petrosyan PanDA UTA,
ELSSISuite Services QIZHI ZHANG Argonne National Laboratory on behalf of the TAG developers group ATLAS Software and Computing Week, 4~8 April, 2011.
Status of tests in the LCG 3D database testbed Eva Dafonte Pérez LCG Database Deployment and Persistency Workshop.
Distributed Analysis Tutorial Dietrich Liko. Overview  Three grid flavors in ATLAS EGEE OSG Nordugrid  Distributed Analysis Activities GANGA/LCG PANDA/OSG.
Conditions Metadata for TAGs Elizabeth Gallas, (Ryan Buckingham, Jeff Tseng) - Oxford ATLAS Software & Computing Workshop CERN – April 19-23, 2010.
David Adams ATLAS ATLAS Distributed Analysis (ADA) David Adams BNL December 5, 2003 ATLAS software workshop CERN.
Operations model Maite Barroso, CERN On behalf of EGEE operations WLCG Service Workshop 11/02/2006.
1 Information Retrieval and Use De-normalisation and Distributed database systems Geoff Leese September 2008, revised October 2009.
WLCG Operations Coordination report Maria Alandes, Andrea Sciabà IT-SDC On behalf of the WLCG Operations Coordination team GDB 9 th April 2014.
OSG Area Coordinator’s Report: Workload Management February 9 th, 2011 Maxim Potekhin BNL
ATLAS TAGs: Tools from the ELSSI Suite Elizabeth Gallas - Oxford ATLAS-UK Distributed Computing Tutorial Edinburgh, UK – March 21-22, 2011.
Dario Barberis: ATLAS DB S&C Week – 3 December Oracle/Frontier and CondDB Consolidation Dario Barberis Genoa University/INFN.
POOL Based CMS Framework Bill Tanenbaum US-CMS/Fermilab 04/June/2003.
Computing Operations Roadmap
Database Replication and Monitoring
U.S. ATLAS Grid Production Experience
Umbrella Update Björn Abt.
Elizabeth Gallas - Oxford ADC Weekly September 13, 2011
Database Readiness Workshop Intro & Goals
Efficiently Searching Schema in SQL Server
Farida Fassi, Damien Mercie
AMI – Status November Solveig Albrand Jerome Fulachier
Readiness of ATLAS Computing - A personal view
Data Lifecycle Review and Outlook
SNC/SCANA – New Build Configuration Management
Presentation transcript:

TAG and iELSSI Progress Elisabeth Vinek, CERN & University of Vienna on behalf of the TAG developers group

TAG data management news  ( 1 ) Average for latest events (first pass is a mix of merged and unmerged TAG and TAG_COMM data)  ( 2 ) We are now applying vertical partitioning to reprocessed data.  Details can be found here (ATLAS S&C Week July): resId=0&materialId=slides&confId= resId=0&materialId=slides&confId=76895  ( 3 ) MC data is not compressed (>255 columns) TAG & iELSSI Progress2 TAG TYPE# events (M)DB Storage (GB)Avg. space/event (kB) First Pass ( 1 ) September Reprocessing ( 2 ) Monte Carlo ( 3 )

iELSSI news: hiding data distribution  TAG DB sites: CERN, DESY, PIC, BNL, TRIUMF  Previously, the user had to select a site  Now, this selection is done automatically, based on:  data locality  proximity to iELSSI browser TAG & iELSSI Progress3

iELSSI news: period-based selection  TAG data selection based on periods and sub-periods  PERIOD -> RUN mapping in COMA tables  based on information from Data Preparation TAG & iELSSI Progress4

iELSSI news: RunBrowser Integration  For details on RunBrowser, see talk „Conditions Metadata and RunBrowser Update” by E. Gallas TAG & iELSSI Progress5

Architecture news: service catalog  Site-transparency as first step towards a distributed system – achieved with the „data catalog“.  Next steps: put service catalog into production catalog in order to:  Make service distribution transparent  Make the system more robust as regarding responding to outages  Need to catalog the TAG services in terms of:  Functionality (service groups)  Available resources (at sites)  Compatibility  Several non-functional attributes  Components of a distributed system:  Sites  Resources  Services  Deployments  Links TAG & iELSSI Progress6

Architecture news: service catalog (2)  Further service catalog use cases:  Improve software management by:  Defining dependencies between deployments/versions  Keeping configuration parameters centrally  Use the service catalog schema to log service activity:  Gather statistics  Analyze and use them to:  Optimizing caching strategies  Optimizing data and services distribution  Ultimately, optimize request distribution TAG & iELSSI Progress7

Conclusion  First step (data distribution transparency) completed and changes implemented on iELSSI.  Second step (services distribution transparency) currently in development, to make the TAG system:  Modular  Transparent  easily configurable  Continuous effort to improve on covering the TAG use cases. Thanks to the modular architecture new use cases are easy to implement.  TAG database at RAL is ready and waiting for data.  Now waiting for December TAG reprocessing TAG & iELSSI Progress8