Andrew File System (AFS) CSS 534 Sourish Chatterjee, Xiaolin Ma

Agenda Overview AFS Design Goals and Choices Architecture Features Venus and Vice Features Authentication Access List Logical Volumes Filename Resolution Cache Management and Concurrency Viewpoints - Past, Now, Future Comparison with other Distributed File System

Overview Developed jointly by Carnegie Mellon University and IBM First wide distributed file system for large number of users 3 major revisions of design and implementation AFS-1 (1984), AFS-2 (1986), AFS-3 (1989). Further development was transferred to Transarc Corporation (Now IBM)

Goals and Area of focus Goal was to build a system which is Highly available Scalable High in performance Fault tolerant Areas of Focus was on Hardware and Operating system Network File System User Interface Message system When designing the Andrew File System, the goal was to support at least 7000 workstations, which means one workstation for each student or staff. The file system is supposed to have high availability, which means users can get access to the files they want anywhere on campus at anytime. And the file system should be scalable. It means it’s easy to expand and friendly to new technologies in the future. A high performance and good fault tolerance are also required, to offer users a fast and reliable experience. Designers mainly focused on these five areas. They are 12345.. So I am going to talk about their designing choices on these five areas

Design Choices Workstations with Unix BSD 4.2 TCP/IP Serving and Updating Whole File Reduce communication overhead Enough granularity Many small servers instead of one large server System control m/c for large AFS 50:1 ( Client : Server) At the very beginning, designers decided to use a high-performance workstation rather than the cheaper machines like PCs to meet the need of some resources consuming tasks. Designers chose Unix, or more specifically, the Unix BSD 4.2, because it is an operating system with good availability and some features like virtual memory and network support. The network is built up gradually by connecting the pre-exist Ethernets of each department. So finally, a hierarchical network with a backbone and large number of local networks was formed. And TCP/IP was selected as the standard protocol. File system( 1 make the file system compatible with UNIX at the system call level similar command 2 whole file strategy 3 )

Architecture N E T W O R K Workstations Servers User Program Venus Vice Unix Kernel Local Cache Unix Kernel File Storage File Storage User Program Venus Unix Kernel Local Cache Vice User Program Unix Kernel Venus File Storage File Storage Unix Kernel Local Cache

Architecture(Contd..) Venus (Workstations) User Program User Program User Program User Program Andrew Cache Manager To Vice File System call File System call File System call File System call File System call diverted to Andrew Cache Mgr Unix Kernel Threads Local File Cache User Program remains suspended until the file fetch/store is completed through AFS

Architecture(Contd..) Vice (File Server) File Manager Auth Manager Update Manager Status Manager From Venus Internal Lightweight Process Threads Unix Kernel Single UNIX process File Storage Authenticates, Check permissions, Transfers Logical Volume, Updates File version, Keeps Workstation interests for callback, Collect other server(s) status

Authentication Manager Features Authentication Workstation File Server Vice Once a user logs into a workstation, the workstation goes through an authentication procedure based on the user's ID and password. The login program sends encrypted message to the authentication manager running on the file server. If the message is valid, a token will be produced and sent back to the workstation. With this token, the cache manager on Venus can establish connections to the file servers freely. The tokens can be passed on to other workstation to support remote login and the tokens expire after 25 hours. Login Program Venus Authentication Manager Credentials Token Unix Kernel Unix Kernel Local File Cache File Storage File Storage

Features Access Lists Inadequate traditional Unix access control mode New access mechanism needed Introduction of Access lists Read the contents of files in the directory Look up(or list the names of ) files in the directory Insert files into the directory Write(or modify) files in the directory Lock files in the directory Delete files from the directory Administer the directory The access control lists work on directory level Andrew file system was designed to support the entire campus. The traditional access control mode of Unix cannot meet the need of a distributed file system, so the designers introduced a new mechanism to supplement the traditional one, which is the access control list mechanism. Permissions to access files or directories are set in the form of access control lists. The control list only work on the directory level. There are seven types of permissions that can be given to users for a directory. They are: 1234567

Admin commands from Update manager Features(Contd..) Logical Volumes Logical aggregation of files. File is the lowest granular unit. Used for fundamental operations like backup, load or space balancing between file servers by admin. Admin commands from Update manager Logical volumes are special aggregates of files. Doesn’t deal with blocks. The smallest unit inside Logical Volume is whole file. Almost exclusive to the administrators and operators, end user seldom use it. They are used for backup, Load Balancing and moving a set of files from one file server to other Actual Files on the Disk Actual Files on the Disk Actual Files on the Disk Logical Volume A Logical Volume B Logical Volume C

Filename Resolution Conventional UNIX 4.2 BSD No inode notion in Vice. Fixed File Name , inode and its path (variable) namei routine - to locate file using iNode and file name. Performance overhead No inode notion in Vice. Hence 96 bit fid was introduced 32 bit Volume # 32 bit VNode # 32 bit Uniquifer Logical Volume ID Index of file info Array unique id for a fid

Cache Management N E T W O R K 1 2 3 Workstation X Vice (W) File A X A Venus Workstation X Vice (W) File A Workstation File Status X A latest version of file Y B Z Invalidated File A Cached File A Venus Workstation Y (W) File B Invalidate File B after 3 File B Cached File B Invalidate callback for Workstation Y Venus Workstation Z (W) File B Cached File B

Concurrency Multiple workstations can perform same operation on a single file in server No implicit locking on files Application programs to cooperate in performing synchronization Empirical study indicate very less percentage of multiple workstation writing a single file

Viewpoints - Past, Now, Future ? Users’ viewpoint: closely similar to the UNIX FS Noticeable wait to fetch a file failure mode access list Operators’ viewpoint: views AFS as a collection of Logical Volumes which is very different from UNIX FS Now AFS is available as a Open source product (OpenAFS) Works on UNIX, Linux, MacOS X and Windows Better user experience because of high network speed. Very small wait time In AFS3 files which are larger than 64KB are send in 64KB chunks Future ? End users’ viewpoint: All the distributed implementation, files transfer or caching mechanism are transparent to end users. From their viewpoints, AFS is very similar to the UNIX, but allows them get a uniform set of files at any workstation on campus. But the users may see two main differences between AFS and UNIX local file system. First is Operators’ viewpoint: unlike the end users, the operators see AFS completely different from the traditional UNIX file system. Usually they see the file system as a collection of logical volumes. Logical volumes are the unit of operations like create, adjust, clone, move, and backup. Future - Looking at future, AFS might not be a great choice for big data. As the future of computing heavily relies on big data there are other file system(s) which are designed to suit those needs. We will end our presentation with some comparison between some other distributed file systems.

Comparison with other File Systems AFS NFS HDFS Target Users Global/Large User set Workgroup Larger than AFS Domain Multi domain support Single domain Multi domain Client Server Trust Kerberos authorization Client identity is trusted by default Client identity is what O/S tells. No Kerberos Auth. Compatibility with O/S Same System Calls as of O/S Different calls. Mainly used for non interactive programs

References http://ra.adm.cs.cmu.edu/anon/anon/usr0/ftp/itc/CMU-ITC-062.pdf, John H. Howard - Carnegie Mellon University http://www.cs.cmu.edu/~coda/docdir/s11.pdf, John H. Howard, Michael L. Kazar, Sherri G. Menees, David A. Nichols, M. Satyanarayanan, Robert N. Sidebotham, Michael J. West - Carnegie Mellon University http://tele.informatik.uni-freiburg.de/lehre/ws01/dsys , Prof Dr. Stefan Leue - University of Freiburg http://www.openafs.org/ http://www.lnse.org/papers/27-D0046.pdf , Monali Mavani, Member, IACSIT

Questions ?