Bestman & Xrootd Storage System at SLAC Wei Yang Andy Hanushevsky Alex Sim Junmin Gu

Bestman & Xrootd Components  Bestman gateway  XrootdFS Based on Xrootd Posix library and FUSE  GridFTP for Xrootd Globus GridFTP + Data Storage Interface (DSI) module for Xrootd/Posix  Xrootd Latest development release More functionality

Redirector Xrootd/cmsd Redirector Xrootd/cmsd Name space xrootd Name space xrootd Data server Xrootd/cmsd XrdCnd Data server Xrootd/cmsd XrdCnd BeStMan –G XrootdFS BeStMan –G XrootdFS GridFTP w/ Xrootd DSI or xrootdfs GridFTP w/ Xrootd DSI or xrootdfs Desktop xrootdfs Desktop xrootdfs Remote site Data server Xrootd/cmsd XrdCnd Data server Xrootd/cmsd XrdCnd Data server Xrootd/cmsd XrdCnd Data server Xrootd/cmsd XrdCnd Data server Xrootd/cmsd XrdCnd Data server Xrootd/cmsd XrdCnd SLAC Internet Free Zone SLAC network with Internet World SURL TURL Architecture Xrootd or other POSIX file systems

BeStMan Gateway configuration FactoryID=server # lcg-utils doesn't accept service certificate supportedProtocolList=gsiftp://atl-prod07.slac.stanford.edu;gsiftp://atl-prod08.slac.stanford.edu GridMapFileName=/opt/bestman/conf/grid-mapfile.empty GUMSserviceURL= GUMSCurrHostDN=/DC=org/DC=doegrids/OU=Services/CN=osgserv04.slac.stanford.edu # Do srmLs/Rm/Mkdir/Rmdir/Mv via filesystem checkSizeWithFS=true checkSizeWithGsiftp=false accessFileSysViaSudo=false noSudoOnLs=true # Xrootd only, use xrootdTokenCompName=oss.cgroup to pass space token to GridFTPs xrootdTokenCompName=oss.cgroup staticTokenList=tokenA[desc:tokenA][51200];tokenB[desc:tokenB][12800]

Using Sudo with BeStMan Gateway To use sudo to manage file system (mkdir, rmdir, rm, mv, cp, ls) Assuming user 'daemon' run bestman, need the following in /etc/sudoers Cmnd_Alias SRM_CMD = /bin/rm, /bin/mkdir, /bin/rmdir, /bin/mv, /bin/cp, /bin/ls Runas_Alias SRM_USR = ALL, !root daemonALL=(SRM_USR) NOPASSWD: SRM_CMD Some sudo implementation need "Defaults !requiretty”

Xrootd Export Path, Disk Cache and Space Token  Xrootd Export Path is what user will use to access file all.export = /xrootd => root://host:port//xrootd/file  Xrootd Disk Caches are hard disk partitions storing data files Filesystem Size Used Avail Use% Mounted on /dev/sda6 12G 6.0G 5.0G 55% /xrdcache01 oss.cache public /xrdcache01/public # public is default group Export Path contains directories and symlinks, pointing to data files OSS Cache  To support WLCG static space tokens, add more cache groups oss.cache public /xrdcache01/public xa # “xa”: extend attributes oss.cache tokenA /xrdcache01/tokenA xa User create a file using root://host:port//xrootd/file?oss.cgroup=tokenA

Xrootd Config: Redirector and Data Servers xrootd.fslib /opt/xrootd/lib/libXrdOfs.so all.export /xrootd xrd.port 1094 If xrootd-redirector.slac.stanford.edu all.role manager ofs.forward 3way CNShost:port mv rm rmdir trunc else all.role server ofs.notify closew create mkdir | /path/XrdCnsd -l /tmp/cnsd.log root://CNShost:port # please keep the following two lines unchanged. ofs.notifymsg create $TID create $FMODE $LFN?$CGI ofs.notifymsg closew $TID closew $LFN $FSIZE fi oss.cache public /xrdcache01/public xa oss.cache tokenA /xrdcache01/tokenA xa all.manager xrotod-redirector.slac.stanford.edu 3121 cms.allow host dataserver01.slac.stanford.edu cms.allow host dataserver02.slac.stanford.edu

Xrootd Composite Name Space Config: xrootd.fslib /opt/xrootd/lib/libXrdOfs.so all.export /xrootd xrd.port 1095 oss.cache public /xrdcache01/public xa oss.cache tokenA /xrdcache01/tokenA xa # Xrootd uses: # 1) file xrdcache01/.logs/ /Usage to records space usage. # 2) file /xrdcache01/.quotas to set quota for each space token. For example: # public 51200g (51200GB) # tokena 128m (128MB) # Xrootd doesn’t enforce quota. Only for information. oss.usage log /xrdcache01/.logs quotafile /xrdcache01/.quotas all.adminpath /var/adm/xrootd/cns

How to Get Extended Attributes: $ cat xattr.c #include main(int argc, char* argv[]) { ssize_t rc; char value[1024]; rc = getxattr(argv[1],argv[2],value,1023); if (rc != -1) printf("%s\n",value); } $ export LD_LIBRARY_PATH=/opt/xrootd/lib/libXrdPosixPreload.so $ xattr.exe root://CNShost:port//xrootd\?oss.cgroup=tokenA xroot.space oss.cgroup=tokenA&oss.space= &oss.free= &oss.maxf= &oss. used= &oss.quota= $ xattr.exe root://CNShost:port//xrootd/file1 xroot.xattr xroot.xattross.cgroup=tokenA&oss.type=f&oss.used= &oss.mt= &o ss.ct= &oss.at= &oss.u=*&oss.g=*&oss.fs=w&ofs.ap=a

GridFTP configuration  Globus GridFTP on XrootdFS No additional configuration May have performance penalty  Data Storage Interface (DSI) module for Xrootd/Posix Use along with Xrootd Posix preload library $ cat /opt/vdt/vdt/services/vdt-run-gsiftp.sh #!/bin/sh. /opt/vdt/setup.sh export LD_PRELOAD=/opt/xrootd/lib/libXrdPosixPreload.so export XROOTD_VMP=“xrootd-redirector:port:/xrootd=/xrootd Make sure “libglobus_gridftp_server_posix_gcc32dbg.so” is in LD_LIBRARY_PATH exec /opt/vdt/globus/sbin/globus-gridftp-server -dsi posix How to access: root://xrootd-redirector:port//xrootd = gsiftp://gridftpserver/xrootd

XrootdFS Configuration XrootdFS is a Xrootd client. The following script starts XrootdFS export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/opt/xrootd/lib:/opt/fuse/lib export XROOTDFS_OFSFWD=0 # export XROOTDFS_USER='daemon’ export XROOTDFS_FASTLS="RDR” insmod /lib/modules/`uname -r`/kernel/fs/fuse/fuse.ko 2> /dev/null export XROOTDFS_RDRURL="root://xrootd-redirector:1094//xrootd” export XROOTDFS_CNSURL="root://CNShost:1095//xrootd” MOUNT_POINT="/xrootd” xrootdfsd $MOUNT_POINT -o allow_other,fsname=xrootdfs,max_write=131072,direct_io $ df -h Filesystem Size Used Avail Use% Mounted on xrootdfs 55T 34T 22T 62% /xrootd Use “umount /xrootd” to stop XrootdFS