Andrew File System (AFS) Craig Shih & Todor Avramov University of Washington, Bothell CSS534: Parallel Programming December 8, 2016

Outline Overview Consistency Architecture Synchronization and Caching Issues Scalability and Performance AFS vs. NFS References Q&A

Overview Introduced by researchers at Carnegie-Mellon University in the 1980’s Distributed File System with Main goal of Scalability Client-side (Venus), Server-side (Vice) Basic idea of AFS – whole-file caching on local disk of client machine 2 version (AFSv1 and AFSv2) 4 major implementations Transarc(IBM) (deprecated) OpenAFS ARLA Linux Kernel Source Code (early stages of development) AFS v1 – was originally called the ITC distributed file system – had basic design in place, but didn’t scale desirably AFS client-side (venus) Server side – vice AFSv1 – onlyl cached file contents, not directories. When file is called to be open – directory is passed to the server (vice) at which point the entire file is cached onto local disk. When finished (close is called) any changes would be stored back on server. Next time if the same file is called, would be more efficient since it checks if its been changed, if not will just open the local disk’s cached copy. Two main problems with VERSION 1 – Path-traversal costs are too high – had to traverse whole path – too much cpu used The client issues too many TestAuth protocol messages. – too many checks to see if local copy can be used – more often than not, local copy can be used These two main issues limited scalability AFS Version 2 Introduced Callback – to reduce number of client/server interactions. – simply a promise from the server to the client that the server will inform the client when a file that the client is caching has been modified. Also introduced notion of a file –dentifier (FID) – instead of pathnames to specify which file a client was interested – FID would be used FID consists of a Volume ID, a file ID, and a “uniquifier” – essentially using /home/remzi/notes.txt as an example – the first fetch of notes.txt will cache directory home, establish callback for that directory, then do the same with remzi, and then do the same with the actual notes.txt file. This way any changes to the entire path will be marked for callback. Server will notify client if any changes occur. This reduces the need to make multiple hits on the server once initial caching has been established.

AFS Version 2 Introduced Callbacks Reduces number of client/server interactions Is a promise from the server to the client that the server will inform client when a file that the client has cached has been modified. Introduced notion of File Identifier (FID) Replaced whole pathnames as a way to specify file location to server FID consists of a Volume Identifier, File Identifier, and a “uniquifier” ”uniquifier” allows the reuse of Identifiers if a file is deleted. Introduced Callbacks Reduces number of client/server interactions Is a promise from the server to the client that the server will inform client when a file that the client has cached has been modified. Introduced notion of File Identifier (FID) Replaced whole pathnames as a way to specify file location to server FID consists of a Volume Identifier, File Identifier, and a “uniquifier” ”uniquifier” allows the reuse of Identifiers if a file is deleted. Ex. /home/craig/todor/hello.txt Will cache each directory (home, craig, todor) Will then cache the file hello.txt and establish callback for that as well. Server will then notify client if any changes occur. Reduces the need to make multiple hits on the server once initial caching has been established

Callback/ FID Example

Consistency Guarantees Update Visibility When will the server be updated with a new version of the file? Cache Staleness Once the server has a new version, how long before clients see the new version instead of an older cached copy? Two cases to consider: Consistency between processes on different machines Consistency between processes on the same machine AFS Uses Weak (but practical) Consistency Model Guarantees that after the completion of an operation, the next operation performed anywhere in the network will see the updated file system state

Consistency Mechanisms New data written to a file is not stored back at the file server and visible to other clients until the file is closed. Typical UNIX read/write semantics on the same machine – writes to a file are immediately visible to other local processes Open/Close Granularity Client nodes that already have the data cached on their local disks will not have to retrieve from the file server. Minimizes network communication and file server load Whole-file caching on local disk Every time a file is modified in the client side, in addition to storing the file in the local cache, a copy is sent to the file server Minimizes the effects of a client failure Write-through Cache When the server becomes aware that a particular file is modified by a client, the server breaks callbacks (initiates invalidation callbacks) for any clients with local cached copies of that file. Subsequent opens on those clients require a re-fetch of the new version of the file from the server Callbacks

Consistency Timeline

AFS Architecture Vice RPC RPC

System Call Interception in AFS

Vice (File Server)

Synchronization and Caching Issues Callbacks depend on reliable delivery of messages from server to clients Issues: Network is unreliable and may prevent delivery of callback; breaking messages Client may crash File server may crash Fixes After recovery, treat all cache contents as suspect Cache Timeout/Periodic polling (10 minutes) > Synchronization: Update and Callback at the same time for the same file? Locking the cache entries? (Too complicated and may result in deadlock) Add a callback sequence number at the file server? (Best solution) Discard the callback? (Ad hoc solution) What happens when client updates a file and sends it to the fileserver while at the same time the file server calls the client to break a callback promise associated with the same file?

Scalability and Performance Enterprise deployments may exceed 25,000 clients Able to support about 50 clients per server Files commonly accessed locally. File reads usually from local disk cache Client-side performance came close to local performance

AFS vs NFS Performance Assumptions: small and medium files fit into memory of a client – large files fit on a local disk but not in client memory. Access across the network to the remote server for a file block takes Lnet time units. Access to local memory takes Lmem. And access to local disk takes Ldisk. General assumption is that Lnet > Ldisk < Lmem

AFS vs NFS Qualitative AFS NFS Supported Clients Requires Disks Diskless Network Traffic As required (much less) Periodic (high) Server Load Client Response Times Equal OS Support Minimal More Caching Protocol Whole Files Blocks of Files Caching Mechanism Client Disks Client Memory Security Kerberos Weaker more primitive Admin Management Simpler Difficult

References http://pages.cs.wisc.edu/~remzi/OSTEP/dist-afs.pdf http://ra.adm.cs.cmu.edu/anon/usr/ftp/home/ftp/itc/CMU-ITC-063.pdf http://ra.adm.cs.cmu.edu/anon/anon/usr0/ftp/itc/CMU-ITC-062.pdf http://tele.informatik.uni-freiburg.de/lehre/ws01/dsys/Lectures/Lecture19-1.pdf https://en.wikipedia.org/wiki/Andrew_File_System Distributed Systems: Concepts and Design, Edition 4, Addison-Wesley Copyright © George Coulouris, Jean Dollimore, Tim Kindberg, Pearson Education 2005, Chapter 8.4 Andrew File System

Questions