The Next Generation Root File Server Andrew Hanushevsky Stanford Linear Accelerator Center 27-September

27-Sep-042: xrootd What is xrootd? File Server Provides high performance file-based access Scalable, extensible, naively usable Fault tolerant Server failures handled in a natural way Servers may be dynamically added and removed Secure Framework allows use of almost any protocol Rootd Compatible

27-Sep-043: xrootd Goals II Simplicity Can run xrootd out of the box No config file needed for non-complicated/small installations Generality Can configure xrootd for ultimate performance Meant for intermediate to large-scale sites

27-Sep-044: xrootd How Is high performance achieved? Rich but efficient server protocol Combines file serving with P2P elements Allows client hints for improved performance Pre-read, prepare, client access & processing hints, Multiplexed request stream Multiple parallel requests allowed per client An extensible base architecture Heavily multi-threaded Clients are dedicated threads whenever possible Extensive use of OS I/O features Async I/O, device polling, etc. Load adaptive reconfiguration.

27-Sep-045: xrootd xrootd Server Architecture Protocol Layer Filesystem Logical Layer Filesystem Physical Layer Filesystem Implementation Protocol & Thread Manager (included in distribution) p2p heart

27-Sep-046: xrootd Rootd Bilateral Compatibility xrootd rootd xrootd rootd compability TNetFile Application TNetFile TXNetFile Application rootd Client-Side Compatibility Server-Side Compatibility client

27-Sep-047: xrootd How performant is it? Can deliver data at disk speeds (streaming mode) Assuming good network & proper TCP buffer size Low CPU overhead 75% less CPU than NFS for same data load It is memory hungry, however. General requirements Middling speed machine The more CPU’s the better 1-2GB of RAM

27-Sep-048: xrootd How is scalability achieved? Protocol allows server scalability Server directed I/O segmenting Request deferral to pace client load Unsolicited responses for ad hoc client steering P2P elements for lashing servers together Request redirection key element Integrated with a P2P control network olbd servers provide control information

27-Sep-049: xrootd How does it scale? xrootd scales in multiple dimensions Can run multiple load balanced xrootd’s Provides single uniform name and data space Scales from 1 to over 32,000 cooperating data servers Architected as self-configuring structured peer-to-peer (SP 2 ) data servers Servers can be added & removed at any time Client (TXNetFile) understands SP 2 configurations xrootd informs client when running in this mode Client has more recovery options in the event of failure

27-Sep-0410: xrootd Load Balancing Implementation Control Interface (olbd) Load balancing meta operations Find files, change status, forwarded requests Data Interface (xrootd) Data is provided to clients Interfaces to olbd via the ofs layer Separation is important Allows use of any protocol Client need not know the control protocol

27-Sep-0411: xrootd Entities & Relationships data xrootd olbdxrootd olbd Data Clients Redirectors Data Servers M S ctl olbd Control Network Managers & Servers (resource info, file location) xrootd Data Network (redirectors steer clients to data Data servers provide data)

27-Sep-0412: xrootd Typical SP 2 Configuration Dynamic Selection redirector subscribe subscribe

27-Sep-0413: xrootd Example: SLAC Configuration client machines kan01kan02kan03kan04 kanxx bbr-olb03bbr-olb04 kanolb-a

27-Sep-0414: xrootd Why do this? Can transparently & incrementally scale Servers can come and go Load balancing effects recovery New servers can be added at any time Servers may be brought down for maintenance Files can be moved around in real-time Client simply adjust to the new configuration TXNetFile object handles recovery protocol

27-Sep-0415: xrootd What we have seen For a single server: 1,000 simultaneous clients 2,200 simultaneous open files Bottlenecks Disk I/O (memory next behind)

27-Sep-0416: xrootd What Have We Heard The system is too stable Users run extra-long jobs (1-2 weeks) now Error not discovered until weeks later The system is too aggressive New servers are immediately taken over Easy configuration but startling for administrators

27-Sep-0417: xrootd Next: Getting remote data SLAC IN2P3 RAL xrootd’s RAL proxy IN2P3 proxy Firewalls require Proxy servers

27-Sep-0418: xrootd Proxy Service Attempts to address competing goals Security Deal with firewalls Scalability Administrative Configuration Performance Ad hoc forwarding for near-zero wait time Intelligent caching in local domain

27-Sep-0419: xrootd Proxy Implementation Uses capabilities of olbd and xrootd Simply an extension of local load balancing Implemented as a special file system type Interfaces in the ofs layer Functions in the oss layer Primary developer is Heinz Stockinger

27-Sep-0420: xrootd Proxy Interactions client machines red01data02data03proxy01 local olb data01data02data03data04 proxy olb local olb RAL SLAC

27-Sep-0421: xrootd Why This Arrangement? Minimizes cross-domain knowledge Necessary for scalability in all areas Security Configuration Fault tolerance & recovery

27-Sep-0422: xrootd Scalable Proxy Security SLAC PROXY OLBDRAL PROXY OLBD Authenticate & develop session key 2 2 Distribute session key to authenticated subscribers 3 3 Data servers can log into each other using session key Data Servers

27-Sep-0423: xrootd Proxy Performance Introduces minimal latency overhead Virtually undetectably from US/Europe Negligible on faster links 2% slower on fast US/US links 10% slower on LAN Can be further improved Parallel streams Better window size calculation Asynchronous I/O

27-Sep-0424: xrootd Conclusion xrootd provides high performance file access Unique performance, usability, scalability, security, compatibility, and recoverability characteristics Should scale to tens of thousand clients Can support tens of thousand of servers Distributed as part of the CERN root package Open software, supported by SLAC (server) and INFN-Padova (client)