1. Scalla + Castor2 Andrew Hanushevsky Stanford Linear Accelerator Center Stanford University 27-March-07 Root Workshop http://xrootd.slac.stanford.edu Castor2/xrootd Integration – The Details

2. 14-Decemeber-062: http://xrootd.slac.stanford.edu Outline Introduction Design points Architecture Some Details Data serving and Clustering xrootd and olbd Casor2 Integration Dirty Details Conclusion

3. 14-Decemeber-063: http://xrootd.slac.stanford.edu Scalla What is Scalla? SCA Structured Cluster Architecture for LLA Low Latency Access xrootd Low Latency Access to data via xrootd servers POSIX-style byte-level random access By default, arbitrary data organized as files Hierarchical directory-like name space Does not have full file system semantics Protocol includes high performance features olbd Structured Clustering provided by olbd servers Exponentially scalable and self organizing

4. 14-Decemeber-064: http://xrootd.slac.stanford.edu (the commercial) Scalla Design Points (the commercial) High speed access to experimental data Write once read many times processing mode Small block sparse random access (e.g., root files) High transaction rate with rapid request dispersal (fast opens) Low setup cost High efficiency data server (low CPU/byte overhead, small memory footprint) Very simple configuration requirements No 3 rd party software needed (avoids messy dependencies) Low administration cost Non-Assisted fault-tolerance Self-organizing servers remove need for configuration changes No database requirements (no backup/recovery issues) Wide usability Full POSIX access Server clustering for scalability Castor Plug-in architecture and event notification for applicability (HPSS, Castor, etc)

5. 14-Decemeber-065: http://xrootd.slac.stanford.edu Management (XMI Castor, DPM) lfn2pfn prefix encoding Storage System (oss, drm/srm, etc) authentication (gsi, krb5, etc) Clustering (olbd) authorization (name based) File System (ofs, sfs, alice, etc) Protocol (1 of n) (xrootd) xrootd Plugin Architecture Protocol Driver (Xrd) Many ways to accommodate other systems EventsEvents Events

6. 14-Decemeber-066: http://xrootd.slac.stanford.edu The Upshot Plug-in Architecture Plus Events Easy to integrate other systems Low Latency Data Server Improved performance Depending on integration open latency may be high An actual integration follows How Castor works now How Castor works with xrootd

7. 14-Decemeber-067: http://xrootd.slac.stanford.edu 7 Castor 2 Instance With Rootdclient Castor Disk Server 1: open() ->Stage_open() -> stageGet() 6 Disk Server Oracle 3: Starts rootd via LSF 5: Connects to root and opens file 2: Selects server and stages in file 4: Responds with fileserver and path 6: Reads the file 4 rootd 3 2 1 7: close() -> stageUpdtDone() 5

8. 14-Decemeber-068: http://xrootd.slac.stanford.edu Limitations & Solutions Not Scalable One rootd process per open file Resources are quickly consumed Not Very Good Performance or Robustness rootd is old generation We’ve learned a lot about servers since then What We Really Want A long-lived ultra efficient data server xrootd is a likely candidate Plug-in architecture ideal for integration So surprise here

9. 14-Decemeber-069: http://xrootd.slac.stanford.eduXrdCS2d XMI Plug-in Adding xrootd Accessclient Disk Server Castor olbd xrootd xrootdxrootdOrphanClusterXrdCS2d Oracle

10. 14-Decemeber-0610: http://xrootd.slac.stanford.eduXrdCS2d XMI Plug-in Using Orphan xrootd Accessclient Disk Server Castor olbd xrootd xrootdxrootdOrphanCluster Oracle 4 2 3167 8+11 XrdCS2d 10 5 1: open() 3: Client told to wait for callback 5: Get location and fileid via polled query 2: prepareToGet() 4: Server selected and file staged 6: Client redirected 11: close() -> stageUpdtDone() 7: open() 9: LSF launches XrdCS2e to create LFN to PFN symlink 10: read() 7 8: stage_open() 9 XrdCS2e

11. 14-Decemeber-0611: http://xrootd.slac.stanford.edu Limitations & Solutions Performance only after open() All open requests must go through Castor Good for management but bad for ultimate speed Reliant on Castor availability Single point of failure Mitigate with xrootd based clustering Maneuver around Castor whenever possible Reduce open() latency Reduce single-point dependence

12. 14-Decemeber-0612: http://xrootd.slac.stanford.edu Quick Note on Clustering xrootd servers can be clustered Increase access points and available data Allows for automatic failover Structured point-to-point connections Cluster overhead (human & non-human) scales linearly Cluster size is not limited I/O performance is not affected Always pairs xrootd & olbd servers Symmetric cookie-cutter arrangement Architecture can be used in very novel ways olbd xrootd

13. 14-Decemeber-0613: http://xrootd.slac.stanford.edu File Request Routing ClientManager (Head Node/Redirector) Data Servers open file X A B C go to C open file X Who has file X? I have Cluster Client sees all servers as xrootd data servers 2 nd open X go to C Managers Cache file location No External Database

14. 14-Decemeber-0614: http://xrootd.slac.stanford.eduolbdolbd Adding an xroot Clusterclient Disk Server Castor olbd xrootd xrootd xrootdPeerClusterolbd xrootd xrootCluster serverserver peer OrphanCluster Oracle XrdCS2dXrdCS2d

15. 14-Decemeber-0615: http://xrootd.slac.stanford.eduolbdolbd Offline File Accessclient Disk Server Castor olbd xrootd xrootd xrootdPeerClusterolbd xrootdxrootCluster Have?Have? peer Oracle open() Wait for Callback Redirect to Peer 1 st Time2 nd + Time open()As prevously shown XrdCS2dXrdCS2d

16. 14-Decemeber-0616: http://xrootd.slac.stanford.eduolbdolbd Online File Accessclient Disk Server Castor olbd xrootd xrootd xrootdPeerClusterolbd xrootdxrootCluster peer Oracle open() Redirect to Server 1 st Time2 nd + Time open() read() Have?Have? Yes! XrdCS2dXrdCS2d

17. 14-Decemeber-0617: http://xrootd.slac.stanford.edu Some Unsaid Things File creation and open in write mode Requests are always vectored through Castor Doesn’t address file creation/write scalability Massive first time opens for a particular file Requests will pile-up at the Castor redirector Is not very efficient and has limited scalability Is there another way? or Yes, complicated multicast call backs or Simply additional hardware

18. 14-Decemeber-0618: http://xrootd.slac.stanford.edu Alternative Castor Integrationclient Disk Server Castor olbdxrootdCastorInstance Oracle xroot Cluster olbdxrootdolbdxrootd Independent Systems Complete Fault Toleance

19. 14-Decemeber-0619: http://xrootd.slac.stanford.edu Advantages & Disadvantages Both System Completely Independent High degree of fault tolerance Better Better recovery modes Load isolation Better Better throughput Ability to speed match response time Better Better transaction rate Requires Additional Hardware Servers and disk space one time Approximately one time $50-100K USD (depending on priorities) ongoing However, about 1 to 2 FTE reduction in ongoing effort Difficult to Share Resources Across Experiments May be moot as most experiments are always active and Enables local control Enables local control

20. 14-Decemeber-0620: http://xrootd.slac.stanford.edu Conclusion Scalla can be integrated with a wide variety of systems Castor integration test is starting DPM (G-Lite) integration test is in full swing Likely to be rewritten in the Castor model Alternative integration modes should be considered They may be classic but they have distinct advantages Whatever the outcome it’s the science that matters Use the best model to crank out the analysis as fast as possible

21. 14-Decemeber-0621: http://xrootd.slac.stanford.edu Acknowledgements Castor Integration Olof Barring, Sebastien Ponce, Rosa Maria Garcia Rioja Software Collaborators INFN/Padova: Fabrizio Furano (client-side), Alvise Dorigo Root: Fons Rademakers, Gerri Ganis (security), Bertrand Bellenet (windows) Alice: Derek Feichtinger, Guenter Kickinger STAR/BNL: Pavel Jackl Cornell: Gregory Sharp SLAC: Jacek Becla, Tofigh Azemoon, Wilko Kroeger, Bill Weeks BaBar: Pete Elmer (packaging) Operational collaborators BNL, CNAF, FZK, INFN, IN2P3, RAL, SLAC Funding US Department of Energy Contract DE-AC02-76SF00515 with Stanford University