Potential Data Access Architectures using xrootd OSG All Hands Meeting Harvard University March 7-11, 2011 Andrew Hanushevsky, SLAC

March 7-11, 20112OSG All Hands Meeting Goals xrootd Describe xrootd What it is and what it is not The architecture The clustering model Data access modes xrootd How they relate to the xrootd architecture Conclusion

March 7-11, 20113OSG All Hands Meeting xrootd What Is xrootd ? protocol A file access and data transfer protocol Defines POSIX-style byte-level random access for Arbitrary any Arbitrary data organized as files of any type Identified by a hierarchical directory-like name software A reference software implementation xrootdcmsd Embodied as the xrootd and cmsd daemons xrootd xrootd daemon provides access to data cmsd xrootd cmsd daemon clusters xrootd daemons together Scalla Attempts to brand software as Scalla have failed

March 7-11, 20114OSG All Hands Meeting xrootd What Isnt xrootd ? It is not a POSIX file system FUSE xrootdFS There is a FUSE implementation called xrootdFS xrootdclient An xrootd client simulating a mountable file system It does not provide full POSIX file system semantics It is not an Storage Resource Manager (SRM) BeStMan Provides SRM functionality via BeStMan It is not aware of any file internals (e.g., root files) But is distributed with root and proof frameworks As it provides unique & efficient file access primitives

March 7-11, 20115OSG All Hands Meeting xrootd Primary xrootd Access Modes The root framework Used by most HEP and many Astro experiments (MacOS, Unix and Windows) POSIX preload library Any POSIX compliant application (Unix only, no recompilation needed) FUSE File system in User SpacE xrootdFUSE A mounted xrootd data access system via FUSE (Linux and MacOS only) SRM, globus-url-copy, gridFTP, etc General grid access (Unix only) xrdcp The parallel stream, multi-source copy command (MacOS, Unix and Windows) xrd The command line interface for meta-data operations (MacOS, Unix and Windows)

March 7-11, 20116OSG All Hands Meeting xrootd What Makes xrootd Unusual? A comprehensive plug-in architecture Security, storage back-ends (e.g., tape), proxies, etc Clusters widely disparate file systems Practically any existing file system shared-everythingshared-nothing Distributed (shared-everything) to JBODS (shared-nothing) Unified view at local, regional, and global levels Very low support requirements Hardware and human administration

March 7-11, 20117OSG All Hands Meeting lfn2pfn prefix encoding The Plug-In Architecture Physical Storage System (ufs, hdfs, hpss, etc) Authentication (gsi, krb5, etc) Clustering (cmsd) Authorization (dbms, voms, etc) Logical File System (ofs, sfs, alice, etc) Protocol (1 of n) (xroot, proof, etc) Protocol Driver (XRD) Replaceable plug-ins to accommodate any environment Replaceable plug-ins to accommodate any environment Lets take a closer look at xrootd-style clustering

March 7-11, 20118OSG All Hands Meeting cmsd Clustering xrootd xrootd servers can be clustered Increase access points and reliability Uses highly effective clustering algorithms Cluster overhead (human & non-human) scales linearly Cluster size is not limited I/O performance is not affected xrootdcmsd Always pairs xrootd & cmsd servers Symmetric cookie-cutter arrangement Allows for a single configuration file xrootd

March 7-11, 20119OSG All Hands Meeting A Simple xrootd Cluster /my/file 3: I DO! 1: open(/my/file)4: Try open() at A 5: open(/my/file) Data Servers Manager (a.k.a. Redirector) Client cmsdxrootdcmsdxrootdcmsdxrootdcmsdxrootd Who has /my/file? 2: Who has /my/file? ABC

March 7-11, OSG All Hands Meeting Recapping The Fundamentals xrootdcmsd An xrootd-cmsd pair is the building block xrootd xrootd provides the client interface Handles data and redirections cmsd xrootd cmsd manages xrootds (i.e. forms clusters) Monitors activity and handles file discovery stackable replicable Building blocks are stackable & replicable Can create a wide variety of configurations Much like you would do with LEGO blocks Extensive plug-ins provide adaptability

March 7-11, OSG All Hands MeetingServersServersServers Exploiting Stackability 1: open(/my/file)5: Try open() at ANL Distributed Clusters Meta-Manager (a.k.a. Global Redirector) Client AB C /my/file cmsdxrootdcmsdxrootd cmsdxrootd AB C/my/file cmsdxrootdcmsdxrootd cmsdxrootd AB C /my/file cmsdxrootdcmsdxrootd cmsdxrootd ANLSLACUTA cmsdxrootd A B C Who has /my/file? 3: Who has /my/file? 8: open(/my/file) 4: I DO! cmsdxrootd Manager (a.k.a. Local Redirector) Manager (a.k.a. Local Redirector) Manager (a.k.a. Local Redirector) cmsdxrootd 6: open(/my/file) 7: Try open() at A cmsdxrootd Who has /my/file? 2: Who has /my/file? Data is uniformly available By federating three distinct sites /my/file An exponentially parallel search! (i.e. O(2n)) Federated Distributed Clusters

March 7-11, OSG All Hands Meeting Federated Distributed Clusters Unites multiple site-specific data repositories Each site enforces its own access rules Usable even in the presence of firewalls Scalability increases as more sites join Essentially a real-time bit-torrent social model Federations are fluid and changeable in real time Provide multiple data sources to achieve high transfer rates Increased opportunities for data analysis Based on what is actually available

March 7-11, OSG All Hands Meeting What Federated Clusters Foster Resilient analysis Fetch the last missing file at run-time Copy only when necessary Adaptable analysis Cache files where they are needed Copy whatever analysis demands Storage-starved analysis Real-time access to data across multiple sites Deliver to wherever the compute cycles are Copy Data Access Architecture Direct Data Access Architecture Cached Data Access Architecture

March 7-11, OSG All Hands Meeting Copy Data Access Architecture The built-in File Residency Manager drives Copy On Fault Demand driven (fetch to restore missing file) Copy On Request Pre-driven (fetch files to be used for analysis)ServersServersServers AB C /my/file cmsdxrootdcmsdxrootd cmsdxrootd AB C/my/file cmsdxrootdcmsdxrootd cmsdxrootd AB C /my/file cmsdxrootdcmsdxrootd cmsdxrootd ANLSLACUTA cmsdxrootdcmsdxrootd Manager (a.k.a. Local Redirector) Manager (a.k.a. Local Redirector) Manager (a.k.a. Local Redirector) cmsdxrootd /my/file cmsdxrootd Meta-Manager (a.k.a. Global Redirector) Client xrdcp –x xroot://mm.org//my/file /my open(/my/file) xrdcp copies datausing two sources /my/file

March 7-11, OSG All Hands Meeting Direct Data Access Architecture Use servers as if all of them were local Normal and easiest way of doing this Latency may be an issue (depends on algorithms & CPU-I/O ratio) Requires Cost-Benefit analysis to see if acceptableServersServersServers AB C /my/file cmsdxrootdcmsdxrootd cmsdxrootd AB C/my/file cmsdxrootdcmsdxrootd cmsdxrootd AB C /my/file cmsdxrootdcmsdxrootd cmsdxrootd ANLSLACUTA cmsdxrootdcmsdxrootd Manager (a.k.a. Local Redirector) Manager (a.k.a. Local Redirector) Manager (a.k.a. Local Redirector) cmsdxrootd /my/file cmsdxrootd Meta-Manager (a.k.a. Global Redirector) Client open(/my/file)

March 7-11, OSG All Hands Meeting Cached Data Access Architecture Front servers with a caching proxy server Client access proxy server for all data Server can be central or local to client (i.e. laptop) Data comes from proxys cache or other serversServersServersServers AB C/my/file cmsdxrootdcmsdxrootd cmsdxrootd AB C/my/file cmsdxrootdcmsdxrootd cmsdxrootd AB C/my/file cmsdxrootdcmsdxrootd cmsdxrootd ANLSLACUTA cmsdxrootdcmsdxrootd Manager (a.k.a. Local Redirector) Manager (a.k.a. Local Redirector) Manager (a.k.a. Local Redirector) cmsdxrootd /my/file cmsdxrootd Meta-Manager (a.k.a. Global Redirector) Client open(/my/file) xrootd

March 7-11, OSG All Hands Meeting Conclusion xrootd The xrootd architecture promotes efficiency Can federated almost any file system Gives a uniform view of massive amounts of data Assuming per-experiment common logical namespace Secure and firewall friendly Ideal platform for adaptive caching systems Completely open source under a BSD license See more at

March 7-11, OSG All Hands Meeting Acknowledgements Current Software Contributors ATLAS: Doug Benjamin CERN: Fabrizio Furano, Lukasz Janyst, Andreas Peters, David Smith Fermi/GLAST: Tony Johnson FZK: Artem Trunov BeStMan LBNL: Alex Sim, Junmin Gu, Vijaya Natarajan (BeStMan team) Root: Gerri Ganis, Beterand Bellenet, Fons Rademakers OSG: Tim Cartwright, Tanya Levshina SLAC: Andrew Hanushevsky, Wilko Kroeger, Daniel Wang, Wei Yang UNL: Brian Bockelman UoC: Charles Waldman Operational Collaborators ANL, BNL, CERN, FZK, IN2P3, SLAC, UTA, UoC, UNL, UVIC, UWisc US Department of Energy Contract DE-AC02-76SF00515 with Stanford University