Espresso - a Feasibility Study of a Scalable, Performant ODBMS Dirk Duellmann CERN IT/DB and RD45 n Aim of this Study n Architectural Overview n Espresso Components n Prototype Status & Plans

Overview Why Espresso? n RD45 Risk Analysis Milestone –Understand the effort needed to develop a ODBMS suitable as fallback solution for LHC data stores n Testbed that allows us to test novel solutions for remaining problems –e.g. VLDB issues, asynchronous I/O, user schema & data, modern C++ binding,... n NO plans to stop Objectivity production service!

Overview Could a home grown ODBMS be feasible? n Most Database kernels have been developed in “C” the late 80s and before –Today all main design choices are extensively studied in the computer science literature –C++ Language and Library provide am much better development platform than C n Our specific requirements are better understood –We know much better what we need (and not need). –We could reuse HEP developments in many areas like mass storage interface, security n Building an ODBMS for HEP is an engineering and not a research task –We don’t need to spend O(150) person years which went into the first ODBMS!

Overview System Requirements n Scalability –in data volume and number of client connections n Navigational Access –with performance close network and disk limits n Heterogeneous Access –from multiple platforms and languages n Transactional Safety & Crash Recovery –automatic consistency after soft/hardware failures

Overview A Clean Sheet Approach - What should/could be done differently? n No need for big architectural changes –Objectivity/DB largely fulfils our functional requirements –Migration would be easier if access model is similar (e.g ODMG- like) n Focus on remaining problems –Improved Scalability & Concurrency of the Storage Hierarchy n Larger address space (VLDB) n Segmented and more scalable schema & catalogue –Improved Support for HEP environment n parallel development - concept of user/developer sandbox within the store needed –Simplify Partial Distribution of the Data Store n import export consistent subsets of the store

Overview Flexible Storage Hierarchy n File - Group of physically clustered objects –Smallest possible Espresso store –Contains data and optionally schema –Fast navigation within the file using physical OIDs n Domain - Group of files with tightly coupled objects –Contains domain catalogue, data and additional schema –Navigation between all objects within the domain using physical OIDs n Federation - Group of weakly coupled domains –Domain catalogue (very few updates!) –Shared schema (very few updates!)

User 1 DomainCatalogue SchemamyTrack Tags Histos MyTracks User 1 “sandbox” Period 1 DomainCatalogue P 1 RAW AOD REC Period N DomainCatalogue P n RAW AOD REC Production Server Read-Only Domain (no locking required) CalibDomainCatalogue CalibTPCCalibHCALCalibECAL Calib Server FD FDCatalogue Production ProductionSchema

Overview Espresso OID Layout n Federation –set of weakly coupled domains n Domain#32bit –set of tightly coupled objects –e.g. a run or run period, a end-user workspace n File# 16bit –a single file within a domain n Page# 32bit –a single logical page in the file n Object# 16bit –a single data record on a page e.g. a object or varray Federation Domain File File Page Object

Overview Prototype Implementation n Espresso is implemented in standard C++ –no other dependencies –(for now we use portable network I/O from ObjectSpace) n Expect a full C++ compiler –STL containers n in fact all containers in the current implementation are STL containers –Exceptions n C++ binding uses exceptions to signal error conditions (conforming to ODMG standard) –Namespaces n All of the implementation is contained in namespace “espresso” C++ binding is in namespace "odmg” C++ binding is in namespace "odmg” n Development Platform: RedHat Linux & g++

Overview Component Approach n Espresso is split into a small set of replaceable components with well defined –task –interface –dependency on other components n Common Services n Storage Manager n Schema Manager n Catalogue Manager n Data Server n Lock Server n C++ & Python Binding, (JAVA)

Overview Toplevel Components User API Tool Interface Storage Level Interface OS & Network Abstraction Distribution Net I/OFile I/O StorageMgr Page I/O TransMgrCatalogMgrSchemaMgrC++ BindingJAVA Binding PageServerLocktableLockServer depends on Python Binding

Overview Components: Physical Model n Each top-level component corresponds to one shared library and namespace –shared lib dependencies follow category diagram –components are isolated in their namespace n from other components n from user classes n Each shared lib provides IComponent interface –Factory for main provided interfaces –Version and configuration control on component level n implementation version, date and compiler version n boolean flags for optimised, debug, profiling

Overview Client Side Components n Storage Manager –store and retrieve variable length opaque data objects n maintains OIDs for data objects n implements transactional safety n language and platform independent –current implementation uses “shadow-paging” to implement transactions n Schema Manager –describe the layout of data types n data member position, size and type, byte ordering for primitive types –used for: n Platform Conversion, Generic Browsing, Schema Consistency –current implementation extracts schema from the debug information provided directly by the compiler –no schema pre-processor required

Overview Server Side Components n Data Server –transfer data pages from persistent storage (disk/tape) to memory n file system like interface –trivial implementation for local I/O –multi-threaded server daemon for remote I/O n Lock Server –keep a central table of resource locks n getLock (oid) –implements lock waiting and upgrading –very similar approach to most DBMS n Hash Table of resource locks (resource specified as OID) n Queue of waiters per locked resource –moderate complexity: storage manager implements “real” transaction logic

Overview C++ Language Binding n Support all main language features –Including polymorphic access and templates –No language extensions, No generated code n ODMG 2.0 compliant C++ Binding –Ref templates can be sub-classed to extend their behavior e.g. d_Ref could be extended to monitor object access counts e.g. d_Ref could be extended to monitor object access counts –large fraction of the binding has already been implemented n smart pointers can point to transient objects n persistent capable classes may be embedded into other persistent classes n d_activate and d_deactivate are implemented –design supports multiple DB contexts per process n e.g. for multi-threaded applications and mutiple federations n Work in progress: –B-Tree indices, bi-directional links, installable adapters for persistent objects

Overview First Scalability & Performance Tests n Page Server –up to 70 concurrent clients n Lock Server –up to 150 concurrent clients, up to 3000 locks n Storage Manager –Files up to 2 GB (ext2 file system limit under LINUX) –100 million objects per file n stress tested with “random” bit-patterns –Objects up to 10 MB size –Write Performance: > 40MB/s at 30% CPU n 450MHz dual PIII with 4 stripe RAID 0 on RedHat 6.1 n C++ Binding and Schema Handling –successfully ported several non-trivial applications –HTL histogram examples, simple object browser using python –tagDb and naming examples from HepODBMS

Overview Next Steps n Start detailed requirement discussion with experiments and other interested institutes n Continue Scalability & Performance Test –Storage Manager: larger files (>100GB) –Page Server: connections > 500 –Lock Server: number of locks > 20k –C++ Binding & Schema Manager: port Geant4 persistency examples and Conditions-DB n By summer this year –Written Architectural Overview of the Prototype –Development Plan with detailed estimate of required manpower –Single user toy-system

Overview Summary & Conclusions n We identified solutions for most critical components of a scalable and performant ODBMS –Prototype implementation shows promising performance and scalability –Using a strict component approach allows to split the effort into independently developed, replaceable modules. n The development of an Open Source ODBMS seems possible within the HEP or general science community n A collaborative effort of the order of 15 person years seems sufficient to produce such a system with production quality

The End

Overview Exploit Read-Only Data n Most of our data volume follows the pattern –(private) write-once, –share read-only –e.g. raw data is never updated, reconstructed data is not updated but replaced n Current ODBMS implementations do not really take advantage of this fact –read-only files n no need to obtain any locks for this data n no need to ever update cache content n simple backup strategy n Using the concept of read-only files –e.g. in the catalogue –should significantly reduce the locking overhead and improve the scalability of the system with many concurrent clients

Overview Transactions and Recovery n Shadow Paging –Physical pages on disk are accessed indirectly through a translation table (page map). –Copy-on-Write : page modifications are always written to a new, free physical page –Changed physical pages are made visible to other transactions by updating the page map at commit time Master PageMap 1 Data 2 Data 3 Data 4 Data 5 PageMap

Overview Advantages of this Approach n Single files or complete domains can be used stand-alone without modification –e.g. set of user files containing tags and histograms n Local OIDs could be stored in a more compact form –transparent expansion into a full OID as they are read into memory n “Attaching” or direct sharing of files or complete domains does not need any special treatment –no OID translation needed –read-only files/domains can directly be shared by multiple federations n Domains allow to segment the store into “coherent regions” of associated objects –Efficient distribution, backup and replication of subsets of the data (e.g. a run period, a set of user tracks) –Consistency checks can be constrained to a single domain

Overview Common Services n Services and Interfaces of global visibility –OID, IStorageMgr,IPageServer,ILockServer, ISchemaMgr –Platform & OS abstraction n fixed range types, I/O primitives, process control –component interface n version & configuration control n component factory –extendible diagnostics n named counters, timers to instrument the code n each component may have a sub-tree diagnostic items –error & debug message handler n syslog like: component, level, message –exception base class

Overview Espresso Schema Extraction n Currently implemented oextraction based on the “stabs” standard format for debugging information (used by egcs and Sun CC) obased on GNU “BDF” library and “objdump” source code n Prototype provides full runtime reflection for C++ data odescribes classes and structs with their fields and inheritance osupports namespaces, typedefs and enums and templates olocation and value of virtual function and virtual base class pointers osufficient to allow runtime field by field consistency check against persistent schema o Starting of a modified egcs front-end as schema extractor would be an alternative