New I/O navigation scheme Peter van Gemmeren (ANL)

Outline Current/old, RootTree: I/O navigation infrastructure Recap Disadvantage New: RootTreeIndex: Status, regression Advanced features 6/27/2019 Peter van Gemmeren (ANL): New I/O navigation scheme

Current/old: I/O navigation infrastructure, diagram DataHeader Collection 1 Collection N Collection M Event Loop Enter and iterate over TAG Conditions Token Oid1, Oid2 Oid2 1 2 3 … ##Links Link 1 Link 2 Oid1 Provide key, type, …, persistent address Store Gate 6/27/2019 Peter van Gemmeren (ANL): New I/O navigation scheme

Current/old: I/O navigation infrastructure, RootTreeContainer Object location uses different elements File ID (In ROOT: TFIle), a GUID together with File Catalogs are used to identify the file. Container (In ROOT: TTree, TBranch), OID1, long int is used to give the entry number into (POOL) ##Links container, which stores container name, type id… Object (In ROOT: entry number), OID2, long int is used to give the entry number for the object. 6/27/2019 Peter van Gemmeren (ANL): New I/O navigation scheme

Current/old: I/O navigation infrastructure Has worked solidly for all use cases in Run 1 and 2. But does not support object relocation, e.g. ROOT fast or in memory merging because ROOT TTree entry numbers change. POOL did provide a somewhat cumbersome extension as ##Sections table. When extending ATLAS’ workflows to be more distributed, this has become a bottle neck. Identified as part of I/O review for improvement. 6/27/2019 Peter van Gemmeren (ANL): New I/O navigation scheme

New: I/O navigation infrastructure, diagram DataHeader Collection 1 Collection N Collection M Event Loop Enter and iterate over TAG Conditions Token Oid1, Oid2 ##Links Link 1 Link 2 Provide key, type, …, persistent address Store Gate Index Y 1 Y 2 Y 3 … DataHeader Collection 1 Collection N Collection M Event Loop Enter and iterate over TAG Conditions Token Oid1, Oid2 Oid2 1 2 3 … ##Links Link 1 Link 2 Oid1 Provide key, type, …, persistent address Store Gate 6/27/2019 Peter van Gemmeren (ANL): New I/O navigation scheme

New: I/O navigation infrastructure, RootTreeIndexContainer (203) Object location uses extended elements for container and object Container, OID1, long int is used to give a uid 32 bit & entry number 32 bit, into (POOL) ##Links container, which stores container name, type id… This is stored in the ##Links TTree as an additional branch and indexed by ROOT. Object, OID2, long int is used to give a uid 32 bit & entry number 32 bit for the object. This is stored in the TTree as an additional branch and indexed by ROOT. When reading RootTreeIndexContainer, the OID is looked up in the TIndex column. Enabled in master, w/o apparent problems in common use-cases. Doesn’t quite work yet for merged ##Links, work in progress. 6/27/2019 Peter van Gemmeren (ANL): New I/O navigation scheme

Outlook Implemented new persistent POOL (minor) technology for RootTreeIndexContainer, to add index branch and allow navigation reference to use immutable values rather than entry number. In master, seems to work transparent to existing/old RootTreeContainer. Can be read by release 21 Should (when finished) allow fast and in memory merging features. Some related work minimizing content and complexity (possibly generality) of the current DataHeader into a new persistent version. These change the way our navigational data is written, and therefore we try to get things in early to spot potential problems before real data gets written. 6/27/2019 Peter van Gemmeren (ANL): New I/O navigation scheme