2. Distributed File Systems

3. File  A file is a collection of data with a user view (file structure) and a physical view (blocks).  Files contain both data and attributes File = Name + Attributes + Data  A directory is a file that provides a mapping from text names to internal file identifiers.

4. File Systems File system is the interface to disk storage File systems implement file management: – Naming and locating a file – Accessing a file – create, delete, open, close, read, write – Physical allocation of a file.

5. 4 File system modules What is a file system? (a typical module structure for implementation of non-DFS) File s Device Blocks Directories

6. Distributed File System A distributed file system (DFS) is a file system with distributed storage and users. Files may be located remotely on servers. SUN Network File System NFS ver3& ver4 Andrew File System AFS

7. A case for DFS Introduction I want to store my thesis on the server! I need to have my book always available.. I need to store my analysis and reports safely… I need storage for my reports My boss wants… Server A Uhm… perhaps time has come to buy a rack of servers….

8. Introduction A Case for DS A Server A Server B Server C Wow… now I can store a lot more documents… Hey… but where did I put my docs? Same here… I don’t remember.. I am not sure whether server A, or B, or C… Uhm… … maybe we need a DFS?... Well after the paper and a nap…

9. A Case for DFS Introduction Server C Server B Server A Good… I can access my folders from anywhere.. Wow! I do not have to remember which server I stored the data into… Nice… my boss will promote me! It is reliable, fault tolerant, highly available, location transparent…. I hope I can finish my newspaper now… Distributed File System

10. Cluster-Based Distributed File Systems (1) The difference between (a) distributing whole files across several servers and (b) striping files for parallel access.

11. File service architecture Client computerServer computer Application program Application program Client module Flat file service Directory service

12. Distributed File System (DFS) Requirements Transparency - server-side changes should be invisible to the client-side. – Access transparency: A single set of operations is provided for access to local/remote files. – Location Transparency: All client processes see a uniform file name space. – Migration Transparency: When files are moved from one server to another, users should not see it – Performance Transparency – Scaling Transparency File Replication – A file may be represented by several copies for service efficiency and fault tolerance. Concurrent File Updates – Changes to a file by one client should not interfere with the operation of other clients simultaneously accessing the same file.

13. 12 DFS: Case Studies NFS (Network File System) –Developed by Sun Microsystems (in 1985) –Most popular, open, and widely used. –NFS protocol standardised through IETF (RFC 1813) AFS (Andrew File System) –Developed by Carnegie Mellon University as part of Andrew distributed computing environments (in 1986) –A research project to create campus wide file system. –Public domain implementation is available on Linux (LinuxAFS) –It was adopted as a basis for the DCE/DFS file system in the Open Software Foundation (OSF, www.opengroup.org) DEC (Distributed Computing Environment)

14. NFS It used originally for Unix-based workstations In NFS, each file server provides a view for its local file system NFS can use both TCP and UDP

15. NFS - History  1985: Original Version (in-house use)  1989: NFSv2 (RFC 1094) –Operated entirely over UDP –Stateless protocol (the core) –Support for 2GB files  1995: NFSv3 (RFC 1813) –Support for 64 bit (> 2GB files) –Support for asynchronous writes –Support for TCP –Support for additional attributes –Other improvements  2000-2003: NFSv4 (RFC 3010, RFC 3530) –Collaboration with IETF –Sun hands over the development of NFS  2010: NFSv4.1 –Adds Parallel NFS (pNFS) for parallel data access

16. NFS NFS come with communication protocol allows clients with different OS and machines to access files on server is called Open Network Computing Protocol ONC Client with windows system can access UNIX file server because of NFS protocol independent of OS

17. NFS Access We have two types of access models: Remote Access model and download/upload model The remote access model is that client request to file client unaware of actual location of file Upload/download model is that client download file from server and access it locally.

18. NFS Access The remote access model. The upload/download model

19. NFS architecture UNIX kernel protocol Client computerServer computer system calls LocalRemote UNIX file system NFS client NFS server UNIX file system Application program Application program NFS UNIX UNIX kernel Virtual file system Other file system

20. NFS Architecture The basic NFS architecture for UNIX systems.

21. NFS Communication a)Reading data from a file in NFS version 3. b)Reading data using a compound procedure in version 4.

22. Naming in NFS Mounting (part of) a remote file system in NFS.

23. Naming in NFS Mounting nested directories from multiple servers in NFS.

24. Automounting

25. Dropbox Folder Automatic synchronization

26. File Handles In Unix, file has a node number or file handle which is reference to a file in NFS independent of the name of file. It is created by server when file is created It is 128 byte in ver. 4 of NFS

27. NFS v3 (1995) ASYN_WRITE: asynchronous write –Allow server-side caching COMMIT: –Flush server’s write buffers READPLUSDIR: –Obtain directory’s file names, handles & attributes 64 bits for file size/offset –NFS v2 allows only 32 bits

28. NFS v4 (2000) Based on SUN’s WebNFS Stateful protocol: –Open/close requests –Integrates mount & locking protocols –Lease-based locking –COMPOUND request: group of multiple operations

29. NFS operations Operationv3v4Description CreateYesNoCreate a regular file CreateNoYesCreate a nonregular file LinkYes Create a hard link to a file SymlinkYesNoCreate a symbolic link to a file MkdirYesNoCreate a subdirectory in a given directory MknodYesNoCreate a special file RenameYes Change the name of a file RmdirYesNoRemove an empty subdirectory from a directory OpenNoYesOpen a file CloseNoYesClose a file LookupYes Look up a file by means of a file name ReaddirYes Read the entries in a directory ReadlinkYes Read the path name stored in a symbolic link GetattrYes Read the attribute values for a file SetattrYes Set one or more attribute values for a file ReadYes Read the data contained in a file WriteYes Write data to a file

30. 29 read(fh, offset, count) -> attr, data write(fh, offset, count, data) -> attr create(dirfh, name, attr) -> newfh, attr remove(dirfh, name) status getattr(fh) -> attr setattr(fh, attr) -> attr lookup(dirfh, name) -> fh, attr rename(dirfh, name, todirfh, toname) link(newdirfh, newname, dirfh, name) readdir(dirfh, cookie, count) -> entries symlink(newdirfh, newname, string) -> status readlink(fh) -> string mkdir(dirfh, name, attr) -> newfh, attr rmdir(dirfh, name) -> status statfs(fh) -> fsstats NFS server operations (simplified) fh = file handle: Filesystem identifieri-node numberi-node generation * Model flat file service Read(FileId, i, n) -> Data Write(FileId, i, Data) Create() -> FileId Delete(FileId) GetAttributes(FileId) -> Attr SetAttributes(FileId, Attr) Model directory service Lookup(Dir, Name) -> FileId AddName(Dir, Name, File) UnName(Dir, Name) GetNames(Dir, Pattern) ->NameSeq

31. File attribute record structure File length Creation timestamp Read timestamp Write timestamp Attribute timestamp Reference count Owner File type Access control list

32. File Attributes (1) Some general mandatory file attributes in NFS. AttributeDescription TYPEThe type of the file (regular, directory, symbolic link) SIZEThe length of the file in bytes CHANGE Indicator for a client to see if and/or when the file has changed FSIDServer-unique identifier of the file's file system

33. File Attributes (2) Some general recommended file attributes. AttributeDescription ACLan access control list associated with the file FILEHANDLEThe server-provided file handle of this file FILEIDA file-system unique identifier for this file FS_LOCATIONSLocations in the network where this file system may be found OWNERThe character-string name of the file's owner TIME_ACCESSTime when the file data were last accessed TIME_MODIFYTime when the file data were last modified TIME_CREATETime when the file was created

34. File Sharing (1) a)On a single processor, when a read follows a write, the value returned by the read is the value just written. b)In a distributed system with caching, obsolete values may be returned.

35. Semantics of File Sharing MethodComment UNIX semanticsEvery operation on a file is instantly visible to all processes Session semanticsNo changes are visible to other processes until the file is closed Immutable filesNo updates are possible; simplifies sharing and replication TransactionAll changes occur atomically

36. Server Caching In delayed-write, when a page has been altered, its new contents are written back to the disk only when the buffer page is required for another page. –Unix sync operation writes pages to disk every 30 seconds In write-through, data in write operations is stored in the memory cache at the server and written to disk before a reply is sent to the client.

37. Failures & Retransmission Three situations for handling retransmissions. a)The request is still in progress b)The reply has just been returned c)The reply has been some time ago, but was lost.

38. Andrew File System (AFS) Developed at CMU (~1985), commercial product by Transarc (part of OSF/DCE) Segment network into clusters, with one file server per cluster Dynamic reconfigurations to balance load Stateful protocol + aggressive caching –Servers participate in client cache management Entire files are cached Session semantics –Weaker than UNIX semantics –See new data on next open() –Immediate updates of metadata

39. Interception and Caching fd = open(pathname) –Files in local file system are opened as normal. –Cached files in shared file system are opened locally. –Other shared files are copied to cache. All read/write directed to cached copy. close(fd) –Local or cached copy is closed. –If shared file is modified it is copied back to server.

40. Vice File Service Flat file service with volumes 96 bit file identifier Volume –Group of related files (e.g. one user’s files) –Used for location and management –Server is custodian of volume Volume location database replicated at each server volume numberfile handleuniquifier

41. Vice Interface Fetch(fid) -> attr, dataReturns the attributes (status) and, optionally, the contents of file identified by the fid and records a callback promise on it. Store(fid, attr, data)Updates the attributes and (optionally) the contents of a specified file. Create() -> fidCreates a new file and records a callback promise on it. Remove(fid)Deletes the specified file. SetLock(fid, mode)Sets a lock on the specified file or directory. The mode of the lock may be shared or exclusive. Locks that are not removed expire after 30 minutes. ReleaseLock(fid)Unlocks the specified file or directory. RemoveCallback(fid)Informs server that a Venus process has flushed a file from its cache. BreakCallback(fid)This call is made by a Vice server to a Venus process. It cancels the callback promise on the relevant file.

42. AFS properties Advantages –Only contact server on open/close –Usually single user at one workstation –Most files are read in entirety –Cache copies valid for long periods –Disk based cache survives reboot –Simplified caching scheme Disadvantages –Files larger than cache can’t be opened –Workstations require disk –Not appropriate for database support

43. The Coda File System Type of userDescription OwnerThe owner of a file GroupThe group of users associated with a file EveryoneAny user of a process InteractiveAny process accessing the file from an interactive terminal NetworkAny process accessing the file via the network Dialup Any process accessing the file through a dialup connection to the server BatchAny process accessing the file as part of a batch job AnonymousAnyone accessing the file without authentication AuthenticatedAny authenticated user of a process ServiceAny system-defined service process

44. Overview of Coda (I)

45. Overview of Coda (II)

46. Communication (I) Side effects in Coda's RPC2 system.

47. Communication (II) a)Sending an invalidation message one at a time. b)Sending invalidation messages in parallel.

48. Naming Clients in Coda have access to a single shared name space.

49. File Identifiers Replicated Volume ID (32-bits) vnode

50. xFS: A “Serverless” File System Distribute file server processing across a set of available hosts, at the granularity of individual files –Separate file management & file storage/access –Dynamically assign files to hosts Software RAID storage system –Striping file data across disks –Log-structured organization Manager Map –Replicated at all hosts

51. Overview of xFS A typical distribution of xFS processes across multiple machines.

52. Processes (I) The principle of log-based striping in xFS.

53. Processes (II) Reading a block of data in xFS.

54. Naming Main data structures used in xFS. Data structureDescription Manager mapMaps file ID to manager ImapMaps file ID to log address of file's inode InodeMaps block number (i.e., offset) to log address of block File identifierReference used to index into manager map File directoryMaps a file name to a file identifier Log addressesTriplet of stripe group, ID, segment ID, and segment offset Stripe group mapMaps stripe group ID to list of storage servers