Andrew McNab - Manchester HEP - 5 March 2002 SlashGrid (“/grid”) Motivation: dynamic-accounts issues Local storage: implementation alternatives Generalisation: remote file access Implementation: Coda, ACLs, plugins Current status Future work a framework for Grid-aware filesystems

Andrew McNab - Manchester HEP - 5 March 2002 Motivation: dynamic accounts For TB1 we provided a patch for Globus gatekeeper, gsi_wuftpd etc to associate Unix UIDs from a pool with the Grid DN identities of incoming requests. This is ok when all jobs do on the machine in question is computation. But (1) any files created by pool UID need to cleaned up before account can be reallocated. But (2) no good for long term storage, since no promise to maintain UID-DN association in long term. But (3) what if malicious user creates a cron entry, writes to some obscure writeable directory we didn’t think of, etc?

Andrew McNab - Manchester HEP - 5 March 2002 Solution: get away from UID filesystems All these problems are fundamentally because files are owned according to UID, but we want UID to have no long term meaning. Obvious solution is to have a filesystem where file ownership depends on Grid DNs not temporary UIDs. –Can then ban user processes from writing anywhere else (straightforward to impose this with a modified ext2 device driver: eg no disk files can be created if UID > 99) UID becomes as transitory as Process Group ID. Problem now becomes: how to implement a DN/Grid aware filesystem?

Andrew McNab - Manchester HEP - 5 March 2002 Implementation alternatives 1) Fake a filesystem by making user process use modified versions of open(), read() etc system calls. –Can do this by relinking, or by an interposition / bypass library that is preloaded before real, shared libc. –But, this cannot enforce access restrictions on files accessible on local disk (since you can use a static binary and ignore permissions) –Need to put filesystem behind a server, accessed via TCP ports, named pipe, or shared memory (all the usual X tricks.) This going to be slow for streaming large files: the very thing we need to be fast. 2) Put filesystem into kernel –Lets kernel enforce access control. Potentially as fast as normal disk. –User space daemon useful to parse proxies, and do any remote IO.

Andrew McNab - Manchester HEP - 5 March 2002 Coda A suitable kernel module already exists for Linux: Coda –introduced into main kernel tree in 1997 (during 2.1) and present in all 2.2 and 2.4 stable kernels. This is part of the Coda project at CMU, an open source fork of AFS2. Very similar architecture to AFS –Kernel module and client side cache daemon (Venus) –Kerberos based Already used “parasitically” by other Linux projects –eg AVFS maps files to virtual filesystems (eg cd into a tar file…) Coda kernel module / Venus also available for *BSD and Windows 98/NT upwards.

Andrew McNab - Manchester HEP - 5 March 2002 Implementation with Coda Coda kernel module talks to client cache daemon by exchanging messages via /dev/cfs0 Since we already have the kernel module, we just need to write a Venus-like daemon: SlashGrid (“/grid”) Coda implementation allows efficient streaming: –open(), close(), stat() handled by calls to Venus/SlashGrid daemon –coda_open call returns the inode of the cached copy to the kernel –subsequent read() and write() operations handled by kernel itself, without daemon being involved. –So streaming a local copy is just as fast as reading/writing a normal disk file. Since SlashGrid called for open()’s etc, can enforce DN based access control at that point.

Andrew McNab - Manchester HEP - 5 March 2002 System calls with SlashGrid kernel a real (ext2) disk open() read() stat() SlashGrid read() write() open() stat() /dev/cfs0/var/spool/slashgrid/fcache ordinary directory/grid/... Standard Unix User process

Andrew McNab - Manchester HEP - 5 March 2002 Remote file access? Another idea that has been around a while: AFS-like system using Grid protocols. All the usual advantages of a global filesystem –Makes a lot of the tedious management of “parameter” files needed by jobs just another operating system service. –Very useful for interactive users: they just see the Grid as one big file system. –Makes all applications (even ls) Grid-enabled immediately. Already using URLs to refer to remote files, so easy to find an appropriate mapping into a filesystem space. So we want to design a system that can be generalised to remote file access too.

Andrew McNab - Manchester HEP - 5 March 2002 ACL format Need to specify permissions in some way. Commonly used compromise between granularity and simplicity is the per-directory ACL (cf AFS) Prototype uses the same format as the GridSite website management system (used for WP6 and GridPP websites): –admin: can modify ACL –write: can write/create files –list: can get a directory listing –read: can read a named file –ACL consists of lines: Currently only implement but in future can add VO groups, CAS authorisation symbols etc (when dust settles?)

Andrew McNab - Manchester HEP - 5 March 2002 ACL implementation Each directory has, or appears to have, a read-only file.grid-acl specifying ACL. Can easily be transferred via existing protocols –eg if cache daemon fetches a file from a remote gsi-ftp server, can fetch the.grid-acl from the same directory without modifying gsi_wuftpd or GridFTP protocol. Modification of ACL done by accessing “virtual files” - these operations are trapped by SlashGrid and ACL updated –cf. Coda’s.CONTROL mechanism –eg remove file.grid-acl-write-%url-encoded-DN% to change the DN’s permission level to write Provide grid-acl command line tool to hide this from users

Andrew McNab - Manchester HEP - 5 March 2002 XML Grid ACL? Current SlashGrid prototype uses same ACL format as GridSite web managament system: –text file with lines of form: LEVEL DN –where LEVEL is admin, write, list, read, none This could be extended to also recognise VO groups, CAS symbols etc as well as DN’s. However, something in XML format would be more easily extensible: /O=Grid/O=UKHEP/OU=hep.man.ac.uk/DN=Andrew McNab admin

Andrew McNab - Manchester HEP - 5 March 2002 Plugin framework Avoid making a monolithic system since: –Lots of interesting filesystems possible: anon ftp, http, https, gsi-ftp, rfio, ldap, SQL databases (cf. Oracle 8i) … –Lots of uncertainty about which caching strategies to use. –Some people will want some but not all of this on their systems. Have /etc/slashgrid.conf that specifies mount points and then which loadable module handles which part of the file system (cf. /etc/fstab) At start time, load dynamic modules which all export a common API. SlashGrid daemon hands each request to the right plugin –user: stat() => coda_getattr => PluginStat() => plugin: stat()

Andrew McNab - Manchester HEP - 5 March 2002 Current status Have implemented SlashGrid daemon and one plugin to provide local file storage with ACLs (certfs.so) SlashGrid obtains DN of a UID from /tmp/x509up_uUID –so you do grid-proxy-init to get started stat / read / creat / mkdir / write / remove / rename / chmod system calls working for files and directories Can already do normal shell commands (ls etc), edit files with emacs, even copy the SlashGrid and certfs sources into the filesystem and build them with make and gcc. Can modify ACL’s with grid-acl tool: –eg grid-acl admin “/O=Grid/CN=J.Smith”

Andrew McNab - Manchester HEP - 5 March 2002 Future work Finish certfs Implement XML ACL’s. Implement an example remote IO plugin –probably read-only, anonymous http, since simplest Encourage other people to write plugins –documentation for Plugin API exists Investigate specialised filesystems for dynamic accounts, automated cleanup, extra logging / auditing,... Look at porting to other OS’s: –Coda kernel module exists for *BSD and Windows already –The Linux Coda module was only 4000 lines of C...

Andrew McNab - Manchester HEP - 5 March 2002 Conclusion Have implemented a read/write filesystem for Linux, based on Grid DNs rather than Unix UIDs. Have done this in an extendable way using plugins for different filesystem types. Should also be straightforward to write a plugin for your favourite remote file access protocol. System is efficient for streaming local copies of files –But can still accommodate many different strategies for fetching, caching and streaming files from remote servers. (Thanks to Anders, Cal and Fabio of Integration Team for useful discussions about all these issues.) has source, notes etc