POOL Status and Plans Current developments and proposed workplan for 2005 Ioannis Papadopoulos, CERN IT/ADC 31/3/2005 LCG Applications Area Internal Review.

Slides:



Advertisements
Similar presentations
Object Persistency & Data Handling Session C - Summary Object Persistency & Data Handling Session C - Summary Dirk Duellmann.
Advertisements

CERN - IT Department CH-1211 Genève 23 Switzerland t LCG Persistency Framework CORAL, POOL, COOL – Status and Outlook A. Valassi, R. Basset,
The POOL Persistency Framework POOL Summary and Plans.
D. Düllmann - IT/DB LCG - POOL Project1 POOL Release Plan for 2003 Dirk Düllmann LCG Application Area Meeting, 5 th March 2003.
POOL Project Status GridPP 10 th Collaboration Meeting Radovan Chytracek CERN IT/DB, GridPP, LCG AA.
Object Storage into RDBMS through the POOL framework Ioannis Papadopoulos, CERN IT/ADC January 27 th, 2005 January 27 th, 2005.
Introduction to Databases Transparencies 1. ©Pearson Education 2009 Objectives Common uses of database systems. Meaning of the term database. Meaning.
SEAL V1 Status 12 February 2003 P. Mato / CERN Shared Environment for Applications at LHC.
GRID job tracking and monitoring Dmitry Rogozin Laboratory of Particle Physics, JINR 07/08/ /09/2006.
Framework for Automated Builds Natalia Ratnikova CHEP’03.
Digital Object Architecture
5 November 2001F Harris GridPP Edinburgh 1 WP8 status for validating Testbed1 and middleware F Harris(LHCb/Oxford)
Nightly Releases and Testing Alexander Undrus Atlas SW week, May
EGEE is a project funded by the European Union under contract IST Testing processes Leanne Guy Testing activity manager JRA1 All hands meeting,
Conditions DB in LHCb LCG Conditions DB Workshop 8-9 December 2003 P. Mato / CERN.
Software Solutions for Variable ATLAS Detector Description J. Boudreau, V. Tsulaia University of Pittsburgh R. Hawkings, A. Valassi CERN A. Schaffer LAL,
CHEP 2006, Mumbai13-Feb-2006 LCG Conditions Database Project COOL Development and Deployment: Status and Plans Andrea Valassi On behalf of the COOL.
CHEP 2003 March 22-28, 2003 POOL Data Storage, Cache and Conversion Mechanism Motivation Data access Generic model Experience & Conclusions D.Düllmann,
ATLAS Detector Description Database Vakho Tsulaia University of Pittsburgh 3D workshop, CERN 14-Dec-2004.
CERN Physics Database Services and Plans Maria Girone, CERN-IT
POOL Status and Plans Dirk Düllmann IT-DB & LCG-POOL Application Area Meeting 10 th March 2004.
Feedback from the POOL Project User Feedback from the POOL Project Dirk Düllmann, LCG-POOL LCG Application Area Internal Review October 2003.
SEAL Core Libraries and Services CLHEP Workshop 28 January 2003 P. Mato / CERN Shared Environment for Applications at LHC.
Metadata Mòrag Burgon-Lyon University of Glasgow.
CERN - IT Department CH-1211 Genève 23 Switzerland t COOL Conditions Database for the LHC Experiments Development and Deployment Status Andrea.
The POOL Persistency Framework POOL Project Review Introduction & Overview Dirk Düllmann, IT-DB & LCG-POOL LCG Application Area Internal Review October.
CERN - IT Department CH-1211 Genève 23 Switzerland t CORAL COmmon Relational Abstraction Layer Radovan Chytracek, Ioannis Papadopoulos (CERN.
7/6/2004 CMS weekZhen Xie 1 POOL RDBMS abstraction layer status & plan Zhen Xie Princeton University.
CHEP 2004, Core Software Integration of POOL into three Experiment Software Frameworks Giacomo Govi CERN IT-DB & LCG-POOL K. Karr, D. Malon, A. Vaniachine.
INFSO-RI Enabling Grids for E-sciencE ARDA Experiment Dashboard Ricardo Rocha (ARDA – CERN) on behalf of the Dashboard Team.
WP3 Information and Monitoring Rob Byrom / WP3
G.Govi CERN/IT-DB 1 September 26, 2003 POOL Integration, Testing and Release Procedure Integration  Packages structure  External dependencies  Configuration.
SEAL Project Overview LCG-AA Internal Review October 2003 P. Mato / CERN.
Database authentication in CORAL and COOL Database authentication in CORAL and COOL Giacomo Govi Giacomo Govi CERN IT/PSS CERN IT/PSS On behalf of the.
D. Duellmann - IT/DB LCG - POOL Project1 The LCG Pool Project and ROOT I/O Dirk Duellmann What is Pool? Component Breakdown Status and Plans.
Oracle for Physics Services and Support Levels Maria Girone, IT-ADC 24 January 2005.
LCG Distributed Databases Deployment – Kickoff Workshop Dec Database Lookup Service Kuba Zajączkowski Chi-Wei Wang.
ATLAS Database Access Library Local Area LCG3D Meeting Fermilab, Batavia, USA October 21, 2004 Alexandre Vaniachine (ANL)
David Adams ATLAS ATLAS-ARDA strategy and priorities David Adams BNL October 21, 2004 ARDA Workshop.
D. Duellmann - IT/DB LCG - POOL Project1 The LCG Dictionary and POOL Dirk Duellmann.
Overview of C/C++ DB APIs Dirk Düllmann, IT-ADC Database Workshop for LHC developers 27 January, 2005.
The SEAL Component Model Radovan Chytracek CERN IT/DB, LCG AA On behalf of LCG/SEAL team This work received support from Particle Physics and Astronomy.
CHEP 2004, POOL Development Status & Plans POOL Development Status and Plans K. Karr, D. Malon, A. Vaniachine (Argonne National Laboratory) R. Chytracek,
CORAL CORAL a software system for vendor-neutral access to relational databases Ioannis Papadopoulos, Radoval Chytracek, Dirk Düllmann, Giacomo Govi, Yulia.
A. Aimar - EP/SFT LCG - Software Process & Infrastructure1 SPI Infrastructure for LCG Software Projects Status and work plan for H July 2003 A.Aimar.
Summary of persistence discussions with LHCb and LCG/IT POOL team David Malon Argonne National Laboratory Joint ATLAS, LHCb, LCG/IT meeting.
1 A Scalable Distributed Data Management System for ATLAS David Cameron CERN CHEP 2006 Mumbai, India.
G.Govi CERN/IT-DB 1GridPP7 June30 - July 2, 2003 Data Storage with the POOL persistency framework Motivation Strategy Storage model Storage operation Summary.
Status of tests in the LCG 3D database testbed Eva Dafonte Pérez LCG Database Deployment and Persistency Workshop.
CPT Week, November , 2002 Lassi A. Tuura, Northeastern University Core Framework Infrastructure Lassi A. Tuura Northeastern.
D. Duellmann - IT/DB LCG - POOL Project1 Internal Pool Release V0.2 Dirk Duellmann.
Follow-up to SFT Review (2009/2010) Priorities and Organization for 2011 and 2012.
D. Duellmann, IT-DB POOL Status1 POOL Persistency Framework - Status after a first year of development Dirk Düllmann, IT-DB.
Conditions Database Status and Plans for 2005 Andrea Valassi (CERN IT-ADC) LCG Applications Area Review 31 March 2005.
D. Düllmann - IT/DB LCG - POOL Project1 POOL Release V1.0 Dirk Düllmann LCG Application Area Meeting, 14 th May 2003.
Building Preservation Environments with Data Grid Technology Reagan W. Moore Presenter: Praveen Namburi.
CERN - IT Department CH-1211 Genève 23 Switzerland t Persistency Framework CORAL, POOL, COOL status and plans Andrea Valassi (IT-PSS) On.
The POOL Relational Abstraction Layer 3D Workshop CERN, December 2004 Radovan Chytracek CERN/IT/DB - LCG.
Oracle for Physics Services and Support Levels Maria Girone, IT-ADC 6 April 2005.
POOL Based CMS Framework Bill Tanenbaum US-CMS/Fermilab 04/June/2003.
POOL Historical Notes POOL has been the most advanced and the most used AA project. Currently, excellent teamwork with experiments on new features and.
LCG Application Area Internal Review Persistency framework summary and input for phase-2 planning Dirk Duellmann, CERN IT
Jean-Philippe Baud, IT-GD, CERN November 2007
Database Replication and Monitoring
(on behalf of the POOL team)
ALICE FAIR Meeting KVI, 2010 Kilian Schwarz GSI.
POOL persistency framework for LHC
Dirk Düllmann CERN Openlab storage workshop 17th March 2003
First Internal Pool Release 0.1
Grid Data Integration In the CMS Experiment
Presentation transcript:

POOL Status and Plans Current developments and proposed workplan for 2005 Ioannis Papadopoulos, CERN IT/ADC 31/3/2005 LCG Applications Area Internal Review

People The POOL team (lead by D.D.) –Storage Manager Markus Frank, Giacomo Govi, I.P. –File Catalog Zhen Xie, Maria Girone –Collections Kristo Karr, David Malon, Helmut Schmucker –RAL and RAL-based implementations Radovan Chytracek, Zhen Xie, Giacomo Govi, I.P. –Infrastructure Giacomo Govi, Radovan Chytracek, I.P. Nobody contributing 100% –People moving back to experiment integration and physics services –…faster than originally foreseen The “invisible” support team –Our beta-testers, requirement-providers from the experiments (Vincenzo, Dave, Vakho, Yulia, Sasha, RD, Peter, Nuno, Bill, Mike, Lassi, Jens, Saima,…) –Members of the SPI, SEAL, ROOT and COOL teams (Alberto, Andreas, Eric, Johanne, Manuel, Pere, Rene, Fons, Andrea, Sven, …)

POOL Domain Decomposition POOLCoreAttributeListStorageSvcFileCatalogXMLCatalogMySQLCatalogGrid…CatalogRelationalFileCatalogRootStorageSvcRelationalStorageServicePersistencySvcDataSvcCollectionCollectionBaseRootCollectionMySQLCollectionImplicitCollectionRelationalCollectionRelationalAccessObjectRelationalAccessOracleAccessSQLiteAccessODBCAccessAuthenticationService Uses Implements Interfaces used by application code

The Relational Abstraction Layer (RAL) More details in presentations of AA meetings: – – Requirements collection, design, implementation, testing, deployment, integration and usage in production software of experiments within less than a year. Ongoing functional review of the abstract interfaces AttributeListRelationalAccessOracleAccessSQLiteAccessODBCAccessAuthenticationService

RAL functionality highlights C++ API for SQL-free, technology-neutral access to relational data (inserting, deleting, updating and retrieving rows) SQL-free description and management of existing or new relational schemas Only C++ types in the API Support for bulk operations, client-side caching and SQL variable binding Separated connection and authentication mechanisms: oracle://dbhost[:port/service]/schemaName mysql://dbhost[:port]/dbName sqlite_file:///directoryName/file.db RAL is NOT an interface for issuing arbitrary SQL statements to a database (such as perl or python DBI) RAL enforces “best practices” in database programming Eg. Variable binding at row inserting are done by the plugin. RAL developers interface to the providers of the database services.

RAL backends Oracle – Based on OCI 10g client – Tested against 10g and 9i servers – Implements the full set of the RAL interfaces and their semantics SQLite – A light-weight embeddable SQL database engine – File-based (zero configuration, administration) MySQL – Implementation based on the 4.0.x MyODBC driver – Will soon provide an implementation based on the MySQL C-API Authentication services First implementation XML-based Environment variables Grid-aware implementations will come this year (3D project)

RAL in use and outlook First clients of RAL – Implementation of all the three POOL base components (FileCatalog, Collection, StorageSvc) – Development of COOL (see Andrea’s talk) – ATLAS geometry database – CMS online conditions Outlook – Ongoing internal review to facilitate best the needs of COOL, the 3D project and specific experiment applications – RAL will start being released and packaged independently of the rest of POOL

File Catalog Goals achieved in 2004: – Support of the LCG-2 RLS – Composite Catalogs Single Writer, Many Readers – Definition of new set of interfaces splitting the file lookup and meta-data management functionalities – Automatic recreation of catalogs from a set of files FileCatalogXMLCatalogMySQLCatalogGrid…CatalogRelationalFileCatalog

FileCatalog Implementations RelationalFileCatalog –Based on RAL, first release already with POOL 1.7 –Enhanced functionality wrt native MySQL-based implementation –First performance/scalability tests performed by CMS (using the Oracle backend) with very encouraging results! –Currently working on the adoption of the new interfaces Current implementations EDGCatalog will be frozen with POOL 2.0 (no further development) MySQLCatalog will be gradually replaced with RelationalCatalog (MySQL backend) New grid-aware catalogs –Based on Globus, Glite (Fireman), LFC –Implementations of the new split interfaces –To be released withing the coming weeks (main target for POOL 2.1.0)

Storage Manager Reasons to celebrate during 2004: Migration to ROOT 4 (RootStorageSvc) Support for schema evolution Definition and implementation of a RAL- based implementation of the StorageSvc interfaces Deferred for 2005: On demand dictionary loading Follow the evolution of the SEAL dictionary (Reflex) POOLCoreStorageSvcRootStorageSvcRelationalStorageServicePersistencySvcDataSvcObjectRelationalAccess

RootStorageSvc (developments) Integrated with ROOT 4 (Main target for POOL 2.0) Demonstrated that the schema evolution cases handled by vanilla ROOT can be handled by POOL as well Allows for object storage in branches of ROOT trees Contribution of Bill Tanenbaum, CMS. Currently addressing issues related to cases of data written with ROOT 3 (POOL 1.8) evolved and read using ROOT 4 (POOL 2.0)

RootStorageSvc (outlook) Immediate priority – Migrate to Reflex and Cintex Impact from the SEAL+ROOT merging: – Package is expected to become very thin, a simple adapter of the StorageSvc interfaces

The RelationalStorageService Activity originally launched to serve two main use cases: – Reading existing relational data as objects through the POOL framework – Store condition/configuration/calibration, event meta-data in an RDBMS through the POOL framework Main challenges faced: – Bridge the differences between the object and relational worlds Resolve object identity Map C++ aggregations to relational constraints Need for software to define/store/materialize object/relational mappings. – Allow for user-defined views of the relational data as objects and vice-versa

RelationalStorageService (current capabilities) I/O of most of types of objects: – STL containers (nested containment as well), POOL references, 64- bit integer types – Not yet supported: C-arrays, pointers or C++ references Two types of POOL containers – RDBMS_HOMOGENEOUS (equivalent to ROOTTREE) – RDBMS_POLYMORPHIC (equivalent to ROOTKEY) Consistent concurrent write accesses to the same container – Extensive tests performed using the Oracle RAC setup in IT Implicit handling of several schema evolution cases – Change of type and order of data members, removing data members, changing name of base class(es), moving data members within class hierarchy,… User-customizable object/relational mapping

RelationalStorageService (outlook) Pending items: Populating POOL containers from existing relational data Underlying software infrastructure is ready Requires 2-3 person-weeks to complete Implement the “multicolumn object binding” functionality (StorageSvc semantics) Required for the implementation of the StorageSvc-based collections

Storage Manager: Open items Ref support in vanilla ROOT Work in progress (currently testing Cintex) Expect to benefit from the usage of Reflex/Cintex Frontier integration Prototype exists based on proxy implementation of the IPersistencySvc interface Need to help the Frontier developers implement the relevant StorageSvc interfaces instead

Collections Collection cataloguing Implementation based on the existing FileCatalog interfaces and implementations Support for collection logical identification RelationalCollection RAL-based implementation Enhanced functionality wrt native MySQL implementations Integration testing will start soon Not yet done (for reasons beyond the control of POOL) Collection plug-in for ROOT Integration with ARDA services CollectionCollectionBaseRootCollectionMySQLCollectionImplicitCollectionRelationalCollection

POOL Documentation Identified as POOL’s weakest aspect in the last AA review Component description documents and POOL User Guide automatically built as part of the release procedure from the docbook fragments of the various packages Enforced implicitly simultaneous tagging of source code with up-to-date documentation SCRAM-, CMT-free tutorials from the web There is still room for improvement, especially in the documentation of the developer-level interfaces

Build and release procedures Our build system allows for several platforms to be built and validated (running of the tests) in parallel to each other and the documentation compilation in the release area Regression testing added in the standard validation procedure Real cases from the experiments Support for QmTest added Support GCOV maintained Continue the use of frequent internal releases which allow users to give early feedback (even though there have been some cases of abuse…) Regularly building on eight platforms: rh73_gcc32(_dbg), rh73_gcc323(_dbg), slc3_ia32_gcc323(_dbg), win32_vc71_dbg and osx103_gcc33

Immediate top priorities Release of the FileCatalog with the new “split” interfaces Migration to Reflex of the Storage Manager packages (and to Cintex for RootStorageSvc) Implementation of the reviewed/improved RAL interfaces –Upgrade of the RAL plugins –Migration of the RAL clients to the new interfaces Full adoption of the SEAL component model –Also required for the proper implementation of the StorageSvc- based collections Complete the RelationalStorageService implementation

Summary and Conclusions There has been a year with big development efforts Many thanks to the experiments for providing early feedback There are a few but big developments ahead of us mainly a consequence of the SEAL evolution and its proposed merge with ROOT