XRootD Release 4 And Beyond GSI Seminar Stanford University/SLAC July15, 2015 Andrew Hanushevsky, SLAC

July 15, 20142GSI Seminar User Visible Changes I New client library: libXrdCl.so First available in now is default Old deprecated client library is libXrdClient.so Copy command: xrdcopy xrdcopy renamed to xrdcp Xrdcopy is now a symlink to xrdcp Original xrdcp renamed toxrdcp-old All have same command line interface!

July 15, 20143GSI Seminar User Visible Changes II New meta command: xrdfs Logical replacement to xrd command Different command line interface More user friendly and intuitive When installing Release 4 RPM XRootD You must de-install previous XRootD release They are incompatible

July 15, 20144GSI Seminar Root Support New client is the default in ROOT Since (Dec 14, 2013) Via TNetXNGFile plug-in Setting envar XNet.UseOldClient to 1 Loads TNetXFile instead (old client) Root 6 does not support new client Due to conflicts with PROOF Expected to be resolved in release 6.1

July 15, 20145GSI Seminar Admin Visible Changes IPv6 support Public Private Networking Disk Caching Proxy HTTP Plug-in Remote Debugging Optional Client Plug-ins Miscellaneous

July 15, 20146GSI Seminar IPv6 Support The new client supports IPv6 It prefers IPv6 whenever possible This corresponds to HEPIX request Connects using mapped IPv4 addr if need be The old client is deprecated It will not support IPv6

July 15, 20147GSI Seminar IPv6 Configuration Frankly none needed Clients detect available network stacks May force it to use various IP modes Via configuration file or envars Server detects interfaces and DNS settings It’s important that DNS is setup correctly May force server to use IPv4 only Via command line option (-I v4)

July 15, 20148GSI Seminar IPv6 Considerations I While servers and redirectors can accept both IPv6 and IPv4 clients... IPv6 clients are assumed dual-stack May be redirected to either IPv6 or IPv4 node Client error recovery will resolve this if unworkable This works if all the servers are dual-stack Generally the preferred migration path Since redirection is via hostname… DNS entry must have no un-routable entries

July 15, 20149GSI Seminar IPv6 Considerations II Client dual-stack assumption will be lifted in release 4.1 (3Q14) Clients will be redirected to compatible nodes If none, the client receives an error Largely driven by public-private networks And interactions with IPv6/4

July 15, GSI Seminar Public-Private Network Support Redirectors are now network cognizant Servers inform redirectors of usable i/f’s Clients always compatibly redirected Private to private and public to public Subject to configured network topology Via new xrd.network directive option Applies to servers and redirectors

July 15, GSI Seminar Public-Private Net Topologies xrd.network routes type [use if1[,if2]] type: local | common | split local (default) No address differentiation (i.e. pre R4 mode) common Private incoming -> private (preferred) or public Public incoming -> only public split Incoming address must match outgoing address If use unspecified addresses come from DNS!

July 15, GSI Seminar Private Addresses & DNS Private addresses should never be in DNS Unless It’s a site local DNS server or The address is zone registered (i.e. only locally available) XRootD This is the assumption used by XRootD To avoid DNS reverse translation timeouts Hence, the use option might be needed If a server connects using a private address

July 15, GSI Seminar Public-Private Considerations The available server interfaces Must be uniform within a cluster Must be compatible with configured topology Restrictions relaxed in R 4.1 Redirectors will match clients & server i/f’s May lead to inaccessible files if no match exists Still working through external access issues May require a separate redirector for external access Due to IPv6/4 and public-private network interactions

July 15, GSI Seminar Networking Is Complex! There are many combinations now Client and server capabilities must now match There are 4 basic combinations PublicPrivate IPv6 IPv4 But things are not that simple! Dual stack clients add another 4 combinations

July 15, GSI Seminar Disk Caching Proxy New proxy server mode Configured via pss.cachelib directive Caches whole files or file segments Mode is configurable Cached content available for future access Until LRU purged (configurable) Many use cases to increase access speed

July 15, GSI Seminar Typical Disk Caching Proxy Uses LocalClients Caching Proxy Remote XRootD Clusters Speed up Remote Access Caching Proxy Speed up HD Access SSD XRootD Cluster FS Caching Proxy Speed up Random Access XRootD Server HDFS

July 15, GSI Seminar HTTP Plug-in Basic http, https, WebDav access Suitable for browsers, curl, wget, & davix Provides another mode of well-known access http is neither low latency nor high-performance Google & Microsoft have proposed improvements Changes submitted to W3C and IETF as http2 Improvements are considered incremental They only address the most vexing problems Configured via xrd.protocol directive And specialized http.xxx directives

July 15, GSI Seminar XRootD XRootD Multi-Protocol Support Always supported multiple protocols Improved architecture makes it much easier XRootD New protocols can leverage XRootD features Security, monitoring, file system plug-ins, etc XRootD Server XRootD Bridge Loadable Protocol Clients Memory Based Protocol Conversion

July 15, GSI Seminar Remote Debugging via DigFS XRootD XRootD provided pseudo file system Provides restricted selectable R/O access to Configuration fileLog files Core files /proc/self (Linux only) Has authentication & authorization options Including access control restrictions View is standardized regardless of location Configured via xrootd.diglib directive

July 15, GSI Seminar core/cmsd core/xrootd The DigFS View /=/ logs/cmsd logs/xrootd proc/cmsd proc/xrootd conf conf/etc (site specific) Virtual exported path

July 15, GSI Seminar DigFS Authorization DigFS consults authorization file Created by the site and specified in config file xrootd.diglib * authfile all [-]conf [-]core [-]logs [-]proc allow gsi host krb5 pwd sss unix g=group h=host n=name o=org r=role ++   

July 15, GSI Seminar XRootD XRootD Client Relationships xrdcopy FTS3 CopyProcessxrdfsXrdClFilesystem PyXRootD Dirac root Gaudi Athena CMSSW PROOF XrdClFileXrdClPostMaster XRootD XRootD - core XRootD XRootD - add on External Package XRootD XRootD – plug-ins Plug-ins replace original implementation All calls may be replaced All layers above benefit with any code changes

July 15, GSI Seminar Client Plug-ins Plug-ins are loaded at run-time Client looks for plug-in configuration files 1 st Locally: ~/.xrootd/client.plugins.d/ 2 nd Globally: /etc/xrootd/client.plugins.d/ Both locations can be over-ridden via envar XRD_PLUGINCONFDIR Plug-ins are strictly version checked Allows for independent development

July 15, GSI Seminar Miscellaneous I Readv proxy pass-through Automatic & greatly improves performance Manual log file rotation (e.g. logrotate) Via extended –k command line option High precision log file timestamps New –z command line option Log timestamp appears in microsecond format

July 15, GSI Seminar Miscellaneous II Special stat() plug-in for odd file systems Configured via the oss.statlib directive Includes plug-in for GPFS with tape backend Cancellable third party copy Used by transfer tools Fast directory listings Stat info can now be included w/ dir entry xrdfs uses this to speed up long listings

July 15, GSI Seminar Miscellaneous III New query config options Use the xrdfs command to display Query cms- shows cmsd status Query role- shows server’s cluster role Query sitename- shows the site’s name Query version- shows server’s version Cluster node blacklisting Via cms.blacklist directive & blacklist file Useful in federated environments

July 15, GSI Seminar Miscellaneous IV New monitoring information User login record now also includes Name of the client’s executable Contents of client’s XRD_MONINFO envar Useful to tie external information to actual data usage E.g. the Panda jobid to cross-reference I/O usage

July 15, GSI Seminar Looking Beyond Release 4 Cross Protocol Redirection Meta-links I/O Throttling Plug-in

July 15, GSI Seminar Cross-Protocol Redirections I XRootD XRootD protocol is capable of redirecting to a protocol other than xroot On file open the server may tell the client that it’s more efficient to try something else E.g. read the file locally from disk Open(xroot://host/filename) -> redirect file://filename

July 15, GSI Seminar Cross-Protocol Redirections II New client already capable of processing out-of protocol redirections Server needs some development to do so Root’s TFile framework needs updating to handle a change in protocols Changes already in development Target for root 6.1 or 6.2

July 15, GSI Seminar Meta-Link Files I XML file that describes one or more data files available for access Meta-link file identified by dot suffix metalink (v3) or meta4 (v4 & incompatible w/ v3) File is read and parsed by the client Client picks one based on certain criteria E.G. priority, location, etc If access fails, client can pick another one

July 15, GSI Seminar Metalink Files II XRootD XRootD client will support meta-link files But the road is not straight-forward Incompatible meta-link formats, which one? Need to avoid encumbering installation I.e. pre-reqs for sites that don’t care about meta-links Targeting availability in 4Q14

July 15, GSI Seminar I/O Throttling Plug-in Allows site to limit I/O access to disk Useful for throttling external access Used in federated environments via proxy server Configured via xrootd.fslib directive And specific throttling directives Plug-in is in code review phase Already used by CMS in production Targeting 3Q14

July 15, GSI Seminar Acknowledgements Current Software Contributors ATLAS: Doug Benjamin, Patrick McGuigan, Ilija Vukotic CERN: Lukasz Janyst, Andreas Peters, Justin Salmon Fermi: Tony Johnson Root: Gerri Ganis, Bertrand Bellenot SLAC: Andrew Hanushevsky, Wilko Kroeger, Daniel Wang, Wei Yang UCSD: Matevz Tadel UNL: Brian Bockelman WLCG: Mattias Ellert, Fabrizio Furano, David Smith US Department of Energy Contract DE-AC02-76SF00515 with Stanford University