G.Govi CERN/IT-DB 1GridPP7 June30 - July 2, 2003 Data Storage with the POOL persistency framework Motivation Strategy Storage model Storage operation Summary Giacomo Govi PPARC-LCG, CERN IT/DB

G.Govi CERN/IT-DB 2GridPP7 June30 - July 2, 2003 Motivation  Provide storage and retrieval of C++ objects No intrusion into experiments data models  Support for various type of data  Event data, Detector data, Analysis data  Different volumes  Different access pattern  Persistency technology may change over time  Different technologies may be used at the same time Avoid to bind to a single choice  Physics software should be independent from the underlying data storage technology

G.Govi CERN/IT-DB 3GridPP7 June30 - July 2, 2003 Strategy  Hide any technology details for the clients Clients deal with objects or object references  Leave Transient data representation free from ‘knowledge’ about persistency Each technology can be handled transparently  Run-time binding of transient data to the underlying technology

G.Govi CERN/IT-DB 4GridPP7 June30 - July 2, 2003 Strategy (cont’d)  Objects maintain their state when made persistent Allow for queries, selections and independent element access  Backend layers built on the technology Use object feature when supported - need to be instructed Split into primitives if no object support – need full access to member data Need for object description: “dictionary”

G.Govi CERN/IT-DB 5GridPP7 June30 - July 2, 2003 Storage scheme Define a model for an object storage system: identifying commonalties among different technologies Adapts to any technology with direct record access Need to know record identifier in advance RDBMS: More or less traditional Primary key must be uniquely determined before writing

G.Govi CERN/IT-DB 6GridPP7 June30 - July 2, 2003 Persistency model Objects & referencesObjects, object IDs, DBs C++ pointer >> object ID Persistent Transient

G.Govi CERN/IT-DB 7GridPP7 June30 - July 2, 2003 Storage functions  Write objects Return a unique identifier of their ‘address’ in the database (Token)  Read back/ modify/ delete stored objects Localize objects in the database using the Tokens  Support of object association Provide a transparent way to navigate into object references  Available: Root I/O backend

G.Govi CERN/IT-DB 8GridPP7 June30 - July 2, 2003 Components breakdown Data Service Ref Cache Data Service Ref Cache ClientClient Persistency Service Storage Service Storage Service CLIENT SIDEPOOL SIDE LCGDictionaryLCGDictionary

G.Govi CERN/IT-DB 9GridPP7 June30 - July 2, 2003 Dictionary generation Technology dependent.h ROOTCINT CINT dictionary code Dictionary Generation CINT dictionary I/O Data I/O GCC-XML LCG dictionary code.xml Code Generator LCG dictionary Gateway Reflection Other Clients

G.Govi CERN/IT-DB 10GridPP7 June30 - July 2, 2003 Data Access through Reference References are implemented as smart pointers  Maintain access to the embedded class members  Provide services to handle persistency  Take care of the memory clean up Reference in the object cache Access to persistency service Ref<T>Ref<T> Pointer to object Dereference

G.Govi CERN/IT-DB 11GridPP7 June30 - July 2, 2003 Cache Access by Smart Pointer Data Service object cache … TokenObject Persistency Service Object type Storage type Persistent Reference T o k e n Cache Ref Data Service Pointer Ref File Catalog

G.Govi CERN/IT-DB 12GridPP7 June30 - July 2, 2003 Data operation: WRITE Storage Service Storage Service Start Transaction Commit Transaction Persistency Service Object Cache ClientClient Data Service cache->transaction().start(...); refA.mark_write(placement);... refC.mark_write(placement); cache->transaction().commit(); Ref mark for write

G.Govi CERN/IT-DB 13GridPP7 June30 - July 2, 2003 Data operation: READ/UPDATE/DELETE Storage Service Storage Service Start Transaction Commit Transaction Persistency Service Object Cache ClientClient Data Service cache->transaction().start(...); refA->myMethod(); refB.mark_update(); refC.mark_delete(); cache->transaction().commit(); Ref. mark for update Ref. mark for delete Ref<A>Ref<A> Tokens

G.Govi CERN/IT-DB 14GridPP7 June30 - July 2, 2003 Link ID Link Info DB/Cont.name, Local lookup table in each file TokenObject Reference Follow Object Associations Entry ID Link ID

G.Govi CERN/IT-DB 15GridPP7 June30 - July 2, 2003 Summary  The POOL framework provides persistency services with a generic store technology  The POOL model can be applied to other technologies based on database files, collections and objects within collections  POOL allows the client to choose technologies according to their needs  Root I/O backend implemented  Proof-of-concept prototype RDBMS backend started