XRootD & ROOT Considered Root Workshop Saas-Fee September 15-18, 2015 Andrew Hanushevsky, SLAC

September 18, 20152Root Workshop XRootD Brief History of XRootD 1997 – Objectivity, Inc. collaboration Design & Development to scale Objectivity/DB First attempt to use commercial DB for Physics data Successful but problematical 2001 – BaBar decides to use root framework vs Objectivity Collaboration with INFN, Padova & SLAC created Design & develop high performance data access system Work based on what we learned with Objectivity XRootD 2003 – First deployment of XRootD system at SLAC 2015 – Wide deployment with several implementations ALICE, ATLAS, CMS, EXO, Fermi, LSST; among others Protocol available in dCache, DPM, and EOS

September 18, 20153Root Workshop XRootD Why the Name XRootD? Spurred by the pre-packaged rootd daemon ROOT team planned to upgrade it We wanted to avoid duplication Easier to get people to use a familiar name We tried rebranding in 2008 SALL ASCALLA Structured Cluster Architecture for Low Latency Access (SCALLA) Total failure!

September 18, 20154Root Workshop ROOT Related Enhancements Retrospectively, the two important ones… Server-Side Vector Reads Client-Side Cross protocol redirects

September 18, 20155Root Workshop Vector Reads Developed by Rene Brun’s team XRootD Work added to XRootD but with a twist! Can read blocks from multiple files at once Feature never used by ROOT I/O Probably never will be used In some ways we wish we had not added the twist All that said… Many applications still don’t use vector reads! Due to not using TTreeCache And it’s been there for a very long time 

September 18, 20156Root Workshop Vector Reads Enhancements Enhanced over the years Storage system level vector reads added Prior code unrolled vector reads before sending the request to the storage system Developed by Brian Bockelman, UNL Vector read proxy pass-through Proxy server performance improvement

September 18, 20157Root Workshop Cross Protocol Redirects Developed by Lukasz Janyst Allows ROOT I/O to switch protocols Can improve worker node clustered storage I/O E.G. xroot://->file:// Can support multi-protocol federations E.G. xroot:// -> Currently only the xroot client can do this But it’s simple for any protocol to do so We hope all multi-protocol capable plug-ins will add it Available in ROOT 6.04.x

September 18, 20158Root Workshop XRootD ROOT I/O & XRootD I/O XRootD XRootD can only do so much It already efficiently handles most ROOT I/O ROOT I/O adopted many tuning knobs Optimize baskets, autoflush, learning, TTreeCache, streaming, spliting, etc It’s complicated and not easy to optimize Once optimized it’s specific to an analysis mode Change the mode and you may be in trouble!

September 18, 20159Root Workshop Why All The Optimizations? It’s mostly due to spinning disks! File organization & I/O requests are very sensitive to high latency devices It’s also due to network I/O Small reads suffer high latency Especially on the WAN Network issues will be always with us Unless “entangled” networks come to pass

September 18, Root Workshop Are We Too Focused on HDD? Low HDD $/TB prices are still dropping But not in a good way! HDD is becoming more dense, so … Cost per terabyte is vastly decreasing! But HDD’s are not getting any faster They are becoming slower in many respects Shingled drives come to mind We may be too mesmerized by the “deal”

September 18, Root Workshop Need a Change in Hardware!

September 18, Root Workshop The Cost Factor Is Still Large This is what EMC thinks But the ratio is decreasing somewhat faster

September 18, Root Workshop HDD Apples to SSD Oranges! HD’s getting more dense but not faster Approaching archive use utility! power heat HD’s use more power and emit more heat Electricity is not getting cheaper

September 18, Root Workshop The ROOT I/O SSD Challenge If by 2018 SSD’s become active storage Either in a hierarchy or primary storage ROOT I/O may be insufficient Object layout & access algorithms HD-oriented SSD’s have their own peculiarities For example, large page read-out size Time to start rethinking ROOT I/O! How to get the most out of SSD’s

September 18, Root Workshop XRootD XRootD & SSD XRootD XRootD already is SSD ready Already supports tiered storage (i.e. SSD+HDD) Used by SLAC for the ATLAS Tier 2 Reasonable approach until SSD prices drop HD rival is estimated by 2020 Based on improvements to 3D NAND technology So, 2020 may mean primarily SSD access Will ROOT IO be ready?

September 18, Root Workshop An Orthogonal Approach SSI Developing a Scalable Service Interface XRootD Leverage XRootD features for services E.G. Clustered mySQL for LSST queries Testing a 150 node cluster at scale! SSIXRootD SSI runs in parallel with XRootD data service It’s an optional plug-in for increased flexibility What is it? LORB LORB Essentially a LORB (Lightweight Object Request Broker)

September 18, Root Workshop LORB The 10,000 KM LORB View XrdSsiGetClientService() XrdSsiService Provision() XrdSsiSession ProcessRequest() XrdSsiRequest ProcessResponse() Client-Side Processing Flow Three objects (simplicity) Service, Session, and Request objects Client changes object Reflected at the server Server changes object Reflected at the client Object Brokering XRootD Uses XRootD technology Client redirected to the server than can handle the object (provision) SSI objects and interactions have been abstracted to be compatible with most other protocols XrdSsiGetServerService() XrdSsiService Provision() Server-Side Processing Flow XrdSsiResponder SetResponse() XrdSsiSession ProcessRequest()

September 18, Root Workshop What Can You Do With This? Implement arbitrary clustered services Provide ROOT IO flexibility I/O can be optimized out of channel Dependent on what the client is doing Provide PROOF a lot more flexibility Optimized event delivery The possibilities are endless LSST is at the forefront in leveraging SSI Necessary given their real-time requirements

September 18, Root Workshop Conclusion XRootD XRootD is under active development Always looking for new ideas Feel free to suggest them Be a contributor – it’s open source Available on github! XRootD Consider joining the XRootD collaboration It costs no money to join See more at

September 18, Root Workshop Acknowledgements Current Software Contributors ATLAS: Doug Benjamin, Ilija Vukotic CERN: Andreas Peters, Sebastien Ponce, Michal Simon, Elvin Sindrilaru Fermi: Tony Johnson Root: Gerri Ganis SLAC: Andrew Hanushevsky, Wilko Kroeger, Daniel Wang, Wei Yang UCSD: Alja Mrak-Tadel, Matevz Tadel UNL: Brian Bockelman WLCG: Mattias Ellert, Fabrizio Furano, David Smith US Department of Energy Contract DE-AC02-76SF00515 with Stanford University