More Interfaces Workplan Dirk Düllmann, IT-DB dirk.duellmann@cern.ch LCG Persistency Workshop, June 5th 2002

Conventions for next slides All interface assume usual set of basic fixed length types pool::u32, pool::u64, …. but drop namespace prefix for readability On following slides, I will use open(string &name, …) instead of open(const std::string &name, …) proper use of const, &, virtual is left to the reader Prefix interface by suggested namespace

IFileCatalog Assumptions: “logical file name” will be replaced by a unique file identifier for the scope of the persistency framework This logical file IDs needs to be agreed on at least within LCG if not within a larger scope! I’ll make a proposal after consulting AF people lfid resolveRef(Ref aRef) extract lfid from ref (technology dependent) Proposal: I would move this functionality into the Ref/Token responsibility: eg each Storage Manager token must provide a method to obtain the unique logical file identifier it’s referring to. string pfn findReplica(lfid) perform lfid to pfn lookup assuming EDG RLS will handle lfid rather than lfn lfid createFile(string pfn) allocate unique file ID for a new physical file eg using GUID or from pre-allocated local range or.. removeFile(lfid)

poolImpl::StorageMgr provides pool::ITransactionHandler ISmToken write(ISmToken pl, byte *buf, u32 len); writes an opaque byte sequence into store byte *read(ISmToken obj, u32 &len) reads opaque byte sequence from store ISmToken open(string &psxName, pool::openMode m); opens or creates a file according to given mode return token is usable as placement hint for write close(ISmToken t); closes a file which has been opened with above method

poolImpl::ISmToken Not sure how abstract we can be… class ISmToken { // provides access to file id, eg for catalog lookup/catalog level re-mapping pool::fileUID getFileUID(); // provides access to object id (unique in a file), may be used for object level re-mapping pool::objID getObjectID(); // extenalise/internalise void internalise(const string &exForm); string externalise(); // equality bool operator==(ISmToken rhs); // order / hash int operator>(ISmToken rhs); } Not sure how abstract we can be…

Collections See David Malon’s proposal

Dictionary Interface see Pere’s, Stefan’s and Victor’s slides Propose to accept Gaudi interface as starting point start a quick comparison with cint (C++) and Java reflection interfaces possibly extending it Need agreement about who provides the master dictionary I tend to agree with Pere’s arguments model and prototype for to feeding slaves by September

pool::Ref<T> Proposal: ODMG like interface Ref<T> and RefAny from Espresso CacheMgr interface see next slide ptr(); // return pointer to open object in cache close(); // invalidate real memory pointer

pool::ICacheMgr and pool::Ref<T> addRef(Ref r) // bi-directional link between ref/cm removeRef(Ref r) openRef(Ref r) // state change to data in memory // ref real memory pointer set closeRef(Ref r) // state change to data on disk // ref contains only PmToken On cache manager cleanup it calls closeRef for_each subscribed ref

Proposed Work Packages Refs and Object Lookup Definition of IStorageManager and ISMToken interface Implementation of a RootI/O storage manager File Catalog and Grid Integration MySQL, XML and EDG based implementations of IFileCatalog FileUID generation, local catalog management tools

Proposed Work Packages Collections and Iterators Explicit and implicit collection implementations for RootI/O and RDBMS Conversion and Dictionary Transient and persistent IDictionary implementation Cross population between cint/RootI/O dictionary and Dictionary import/export

Proposed Work Packages Meta Data and Query IAttributeList implementation for RootI/O and RDBMS Query result iterator Integration/Common services/Packaging? RDBMS interface PersistencyManager Project specific development infrastructure

Main question: How to stage required component decoupling? My Proposal : focus first on working Navigation (You all asked for it!) First Stage: IO and streaming still coupled Use RootI/O as storage manager which delivers directly C++ objects instead of byte streams Still using normal Root streaming Decoupled Dictionaries and StreamerSrv components not required yet Can already exercise Refs and navigation with real objects Second Stage: IO and streaming components decoupled (after September release) RootI/O StorageManager now delivers opaque bytes IDictionary and IStreamerSrv moved in place between PersistencyManager and StorageManager

Workplan Proposal Sequence of functional releases which provide an increasing set of interfaces to the user Start integration with most critical functionality Proposed priority order Navigation Collections Meta Data Dictionary Interfaces “September” release (decoupled) Streaming … Does not mean that component WP could/should not start working on implementation, but not yet integration

Workplan Basic Navigation – July IStorageManager, ISmToken provided for RootI/O Ref<T> Basic navigation works (physical OID, not into embedded collections) IFileCatalog basic lookup by fileUID – no property based retrieval yet Two back ends (LCG MySQL + LCG XML) ITransactionHandler - in place for all components working at least for MySQL FileCatalog prototype RootI/O checkpointing -> decision about availability in September release Assume persistency for objects not deriving from TObject as presented by Rene is sufficient Agreed but not yet integrated Interface for persistent and transient dictionaries

Work Plan II Collections - August Collection<T> Ref basic - no property based retrieval Explicit & implicit collection implementation for eg RDBMS Iteration interface & population interface Ref Extension: navigation into embedded collections New IFileCatalog implementation integrated EDG/Globus Proof of concept: Checkpointing of root files all other components as before

Work Plan III Meta Data & Query – September Meta Interface eg IAttributeList in place For file and collection catalogs Same population and query interface for both Registry for descriptions and support for attribute based retrieval (query) IFileProperties + ICollectionProperties Root I/O Checkpointing works all other components as before

Work Plan IV Encapsulated Dictionary – December IDictionary + Meta Class system - Support for dictionary reading via the agreed interface Proof of concept implementation for population of the dictionary from C++ (cint) and at least one external source through new interface all other components as before

Towards the mid’03 release Production LCGRef -> single collection element Production Schema import / export Full population of LCG dictionary from external (non-C++) sources as ATLAS ADL/ LHCb XML Sufficient to use ROOT I/O default converters Proof of concept – Meta Data import/export How to extract all required meta data for a given event collection/run Production checkpoint recovery implementation

Resources / Schedule Very aggressive schedule for “September release” Still large uncertainties about required and available effort how many contributors can be at CERN at least for some time Need to get started to see Expectation Release will aim to be usable for first production tests… … but it’s for sure it won’t pass those tests yet! It will tell us what and how much needs still to be fixed. No way, that a 3 month old implementation can be production quality by then!