CODA/AFS (Constant Data Availability) --Jay N. Vyas & Naveen Mohan--

Background CODA is what? – A Distributed File System (DFS) Developed by? – Mahadev Satyanarayanan Where & Why? – At Carnegie Mellon University as a Research Project in 1987

CODA/AFS - Introduction A decentralized distributed file system (DFS) to be accessed from autonomous workstations. Derived from AFS (Andrew File System) and inherits many of the core features. AFS was a very successful DFS for a campus-sized user community. Was planned to be extended at a national level but WWW took over instead. Key ideas of AFS include: Close-to-open consistency Callbacks

Design Motto CODA had three fundamental objectives: Scalability - to build a system that can easily cope with addition of users and sites Fault-Tolerance - to remain usable in the advent of server failures, communication failures and voluntary disconnections Unix Emulation – to make Coda appear as a giant failure-proof shared Unix file system

Key Features How is CODA different from AFS then? - Fault Tolerance! Provides “constant availability” through Replication Introduces a “disconnected mode” for portable computers (Mobile Computing) Supplementary Features: Is freely available under a Liberal License (Source Code is accessible) High Performance through Client side persistent caching Uninterrupted service during partial network failures Highly Scalable Provides Unified file sharing using well defined semantics even in the event of network failures

History: AFS-1 semantics First version of AFS: Revalidated cached file on each open Propagated modified files when they were closed If two users on two different workstations modify the same file at the same time, the users closing the file last will overwrite the changes made by the other user A successful open implied that the resulting copy of the file was the latest in the system

Open-to-Close Semantics Example: Time F F’ F” First client Second client F” overwrites F’ First client F’ F” overwrites F’ F’ F” F Time Second client

AFS-2 semantics AFS-1 required each client to call the server every time it was opening an AFS file Most of these calls were unnecessary as user files are rarely shared AFS-2 introduces the callback mechanism Do not call the server, it will call you! When a client opens an AFS file for the first time, server promises to notify it whenever it receives a new version of the file from any other client Promise is called a callback Relieves the server from having to answer a call from the client every time the file is opened Significant reduction of server workload

AFS-2 semantics (Contd..) Callbacks can be lost! Client will call the server every tau minutes to check whether it received all callbacks it should have received Cached copy is only guaranteed to reflect the state of the server copy up to tau minutes before the time the client opened the file for the last time

Coda semantics Client keeps track of subset s of servers it was able to connect the last time it tried Updates s at least every tau seconds At open time, client checks it has the most recent copy of file among all servers in s Cached copy can be up to tau minutes behind the server copy Conflicts between cached copy and most recent copy in s can happen Coda guarantees they will always be detected At close time, client propagates the new version of the file to all servers in s If s is empty. client marks the file for later propagation When s is empty, it is said to operate in disconnected mode. - A successful close indicates that the file has been propagated to the set of currently accessible servers, or that no server is available and the file has been marked for propagation at the earliest opportunity.

Server Replication Data structures: Volume A set of files and directories located on one server (shared) Each file and directory in Coda has a unique low-level file identifier (FID). All replicas of an object has the same FID. VSG (Volume Storage Group): Set of servers storing replicas of a volume. AVSG (Accessible Volume Storage Group): Venus (client cache manager) keeps track of the subset of servers accessible to the client for every volume the client has cached. Preferred Server: One of the AVSG servers of the client, that it accesses during a cache miss. Chosen based on physical proximity, server load or server CPU power. Holds all callbacks for client & answers all the read – write requests from client.

Server Replication(contd..) Strategy Read – any , Write – all approach Client still checks with other servers (in AVSG) to find which one has the latest version/copy of the data. When a file is closed after modification by a client it is transferred in parallel to all members of the AVSG. Minimizes Server CPU load Improved scalability, since Server is the bottleneck in many distributed file systems. Cache Coherence – Client identifies 3 types of events no later than Τ seconds, Enlargement of AVSG – Accessibility of a previously inaccessible server. Shrinking of AVSG – Inaccessibility of a previously accessible server. Lost Callback event Coda Version Vector ( CVV ) A volume CVV is similar to a file or directory CVV, but summarizes update information on the entire volume. - Server CPU load is minimized because the burden of data propagation is on the client rather than the server. At present, a server performs no explicit remote actions upon recovery from a crash. Rather, it depends upon clients to notify it of stale or conflicting data. - CVV : It is updated as a side-effect of every operation that modifies the volume. A mismatch in the volume CVV’s indicates that some AVSG members missed an update

Parallel Communication Replication requires multiple sites to be contacted. If serialized, latency would be degraded intolerably. MultiRPC - parallel remote procedure call mechanism which were previously used was extended to use hardware multicast to reduce the latency and network load.

Server Replication (Conflict Scenario) Unit of replication: volume Volume Storage Group (VSG): set of servers that have a copy of a volume Accessible Volume Storage Group (AVSG): set of servers in VSG that the client can contact Use vector versioning One entry for each server in VSG When file updated, corresponding version in AVSG is updated Versioning vector (Coda Version Vector) when partition happens: [1,1,1] Client A updates file  versioning vector in its partition: [2,2,1] Client B updates file  versioning vector in its partition: [1,1,2] Partition repaired  compare versioning vectors: conflict!

Conflict Resolution Conflict Detection Coda does not prevent inconsistent updates but guarantees that inconsistent updates will always be detected Automated Resolution All directory conflicts are automatically resolved by a compensating sequence of create or delete operations, except Update/update conflict Remove/update conflict Name/name conflict Damage Containment Marks all accessible replicas of the objects inconsistent. (if can’t be automatically resolved). Repair Tool (Manual) Special Interface to Venus (Client Cache Manger) Enables the tool to overwrite inconsistent files and to perform directory operations on inconsistent directories, subject to normal access restrictions The Coda repair tool allows users to manually resolve conflicts. It uses a special interface to Venus so that file requests from the tool are distinguishable from normal file requests.

Server Replication - Performance

Introducing Mounting Operation The Andrew file system, Coda's predecessor, pioneered the idea and stored all files under /afs. A client (user) connects to “Coda” and not to individual servers; but the latter come into play invisibly. All the files, which any of the servers may provide to the client, are available under the /coda directory, and all clients see the same name space. Why is Single mount point (name space) advantageous: Users will see the same file tree Auto-discovery of new servers and shares by the Client machines in /coda tree

CODA Operation – Disconnection Mode How does it provide services when the Network crashes? Scenario: User wants to open a file “foo” – cat /coda/tmp/foo A System call is generated to the kernel and it looks for inode of the file which returns the file handle associated with that program (what is inode? RECALL!) Open call proceeds to the Virtual File System (VFS) (What is VFS?) which is handed to the Coda File system present on the Kernel CFS in turn “may” forward the request to “Venus” (What is Venus? RECALL!) Venus checks first in the client cache for /tmp/foo, if cache-miss, then contacts the Vice servers. After successful file location, Venus responds back to the kernel with info which returns to the calling program.

Fig. The Architecture

When does Disconnection Mode crank up? Possibilities: When there is no network connection to any server which has the files. Happens for laptops when taken off the network, or during network failures. Or if the servers fail.

And how does it work? (Caching) Entire file fetched from the servers and stored as a container file in the local cache area by Venus on receiving the first kernel request (currently/usr/coda/venus.cache/). If the file is opened a second time or R/W operations performed, local copy is only used. If it’s modified and closed, then Venus updates the servers by sending the new file. Operations like making directories, removing files or directories and creating or removing (symbolic) links are propagated to the servers also.

From Caching to Disconnection Servers synchronously updated with modified files in normal connection mode Time-out occurs if Server unavailable or Network failure The Ball Begins to Roll!!! (Activate Disconnect Mode) Instead of instantly updating the user with time-out, Venus logs all updates in CML (Client Modification Log); frequently flushed to the disk Upon reconnection of the network, CML file is reintegrated with the server– Run Updates Another Fault Tolerance feature achieved with crisp Transparency…

Issues with disconnected operation Hoarding Files Unable to serve cache-misses during network anomaly Hence Stack up updates to the important files on local cache (client side) Known as “Hoard Walk” Client/Server Conflict Multiple clients may have modified the file during disconnect mode to CML. Needs Repair – either Automatically (Resolvers) with rare chances of Human intervention

CONCLUSION & Paper Highlights Coda project had an unusually long life First work to introduce disconnected mode Server Replication resulting in increased System resiliency Coda/AFS introduced many new ideas Close-to-open consistency Optimistic concurrency control Callbacks (partially superseded by leases)

References Coda File System: http://www.coda.cs.cmu.edu "The Coda Distributed File System." The Coda Distributed File System. N.p., n.d. Web. 11 Mar. 2015. URL: http://www.coda.cs.cmu.edu/ljpaper/lj.html Mahadev Satyanarayanan; Kistler, J.J.; Kumar, P.; Okasaki, M.E.; Siegel, E.H.; Steere, D.C., "Coda: a highly available file system for a distributed workstation environment," Computers, IEEE Transactions on, vol.39, no.4, pp.447,459, Apr 1990 doi: 10.1109/12.54838 URL: http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=54838&isnumber=1979