1. New I/O navigation scheme
Peter van Gemmeren (ANL)

2. Outline Current/old, RootTree: I/O navigation infrastructure Recap
Disadvantage
New: RootTreeIndex:
Status, regression
Advanced features
6/27/2019
Peter van Gemmeren (ANL): New I/O navigation scheme

3. Current/old: I/O navigation infrastructure, diagram
DataHeader
Collection 1
Collection N
Collection M
Event Loop
Enter and iterate over
TAG
Conditions
Token
Oid1, Oid2
Oid2 1 2 3 …
##Links
Link 1
Link 2
Oid1
Provide key, type, …, persistent address
Store Gate
6/27/2019
Peter van Gemmeren (ANL): New I/O navigation scheme

4. Current/old: I/O navigation infrastructure, RootTreeContainer
Object location uses different elements
File ID (In ROOT: TFIle), a GUID together with File Catalogs are used to identify the file.
Container (In ROOT: TTree, TBranch), OID1, long int is used to give the entry number into (POOL) ##Links container, which stores container name, type id…
Object (In ROOT: entry number), OID2, long int is used to give the entry number for the object.
6/27/2019
Peter van Gemmeren (ANL): New I/O navigation scheme

5. Current/old: I/O navigation infrastructure
Has worked solidly for all use cases in Run 1 and 2.
But does not support object relocation, e.g. ROOT fast or in memory merging because ROOT TTree entry numbers change.
POOL did provide a somewhat cumbersome extension as ##Sections table.
When extending ATLAS’ workflows to be more distributed, this has become a bottle neck.
Identified as part of I/O review for improvement.
6/27/2019
Peter van Gemmeren (ANL): New I/O navigation scheme

6. New: I/O navigation infrastructure, diagram
DataHeader
Collection 1
Collection N
Collection M
Event Loop
Enter and iterate over
TAG
Conditions
Token
Oid1, Oid2
##Links
Link 1
Link 2
Provide key, type, …, persistent address
Store Gate
Index
Y 1
Y 2
Y 3 …
DataHeader
Collection 1
Collection N
Collection M
Event Loop
Enter and iterate over
TAG
Conditions
Token
Oid1, Oid2
Oid2 1 2 3 …
##Links
Link 1
Link 2
Oid1
Provide key, type, …, persistent address
Store Gate
6/27/2019
Peter van Gemmeren (ANL): New I/O navigation scheme

7. New: I/O navigation infrastructure, RootTreeIndexContainer (203)
Object location uses extended elements for container and object
Container, OID1, long int is used to give a uid 32 bit & entry number 32 bit, into (POOL) ##Links container, which stores container name, type id… This is stored in the ##Links TTree as an additional branch and indexed by ROOT.
Object, OID2, long int is used to give a uid 32 bit & entry number 32 bit for the object. This is stored in the TTree as an additional branch and indexed by ROOT.
When reading RootTreeIndexContainer, the OID is looked up in the TIndex column.
Enabled in master, w/o apparent problems in common use-cases.
Doesn’t quite work yet for merged ##Links, work in progress.
6/27/2019
Peter van Gemmeren (ANL): New I/O navigation scheme

8. Outlook
Implemented new persistent POOL (minor) technology for RootTreeIndexContainer, to add index branch and allow navigation reference to use immutable values rather than entry number.
In master, seems to work transparent to existing/old RootTreeContainer.
Can be read by release 21
Should (when finished) allow fast and in memory merging features.
Some related work minimizing content and complexity (possibly generality) of the current DataHeader into a new persistent version.
These change the way our navigational data is written, and therefore we try to get things in early to spot potential problems before real data gets written.
6/27/2019
Peter van Gemmeren (ANL): New I/O navigation scheme