Network File Systems Victoria Krafft CS /4/05

General Idea People move around Machines may want to share data Want a system with: No new interface for applications No need to copy all the data No space consuming version control

Network File Systems Diagram from

A Brief History Network File System (NFS) developed in 1984 Simple client-server model Some problems Andrew File System (AFS) developed in 1985 Better performance More caching client-side SFS 1999 NFS can be run on untrusted networks

Lingering Issues Central server is a major bottleneck All choices still require lots of bandwidth LANs getting faster & lower latency Remote memory faster than local disk ATM faster with more nodes sending data

Cooperative Caching Michael D. Dahlin, Randolph Y. Wang, Thomas E. Anderson, and David A. Patterson in 1994 ATM, Myrinet provide faster, low-latency network This makes remote memory 10-20x faster than disk Want to get data from memory of other clients rather than server disk

Cooperative Caching Data can be found in: 1. Local memory 2. Server memory 3. Other client memory 4. Server disk How should we distribute cache data?

Design Decisions Private/Global Coop. Cache? Coordinated Cache Entries? Static/Dynamic Partition? Block Location? Weighted LRU Hash N-Chance Direct Client Cooperation Greedy Forwarding Cent. Coord Private Any Client DynamicStatic Coordination Global No Coordination Fixed

Direct Client Cooperation Active clients use idle client memory as a backing store Simple Don’t get info from other active clients

Greedy Forwarding Each client manages its local cache greedily Server stores contents of client caches Still potentially large amounts of data duplication No major cost for performance improvements

Centrally Coordinated Caching Client cache split into two parts – local and global

N-Chance Forwarding Clients prefer to cache singlets, blocks stored in only one client cache. Instead of discarding a singlet, set recirculation count to n and pass on to a random other client.

Sensitivity Variation in Response Time with Client Cache Size Variation in Response Time with Network Latency

Simulation results Average read response timeSever load

Simulation results Slowdown

Results N-Chance forwarding close to best possible performance Requires clients to trust each other Requires fast network

Serverless NFS Thomas E. Anderson, Michael D. Dahlin, Jeanna M. Neefe, David A. Patterson, Drew S. Roselli, and Randolph Y. Wang in 1995 Eliminates central server Takes advantage of ATM and Myrinet

Starting points RAID: Redundancy if nodes leave or fail LFS: Recovery when system fails Zebra: Combines LFS and RAID for distributed systems Multiprocessor Cache Consistency: Invalidating stale cache info

To Eliminate Central Servers Scaleable distributed metadata, which can be reconfigured after a failure Scalable division into groups for efficient storage Scalable log cleaning

How it works Each machine has one or more roles: 1. Client 2. Storage Server 3. Manager 4. Cleaner Management split among metadata managers Disks clustered into stripe groups for scalability Cooperative caching among clients

xFS xFS is a prototype of the serverless network file system Lacks a couple features: Recovery not completed Doesn’t calculate or distribute new manager or stripe group maps No distributed cleaner

File Read

File Write Buffered into segments in local memory Client commits to storage Client notifies managers of modified blocks Managers update index nodes & imaps Periodically, managers log changes to stable storage

Distributing File Management First Writer – management goes to whoever created the file *does not include all local hits

Cleaning Segment utilization maintained by segment writer Segment utilization stored in s-files Cleaning controlled by stripe group leader Optimistic Concurrency control resolves cleaning / writing conflicts

Recovery Several steps are O(N 2 ), but can be run in parallel Steps For Recovery

xFS Performance Aggregate Bandwidth Writing 10MB files Aggregate Bandwidth Reading 10MB files NFS max with 2 clients AFS max with 32 clients AFS max with 12 clients NFS max with 2 clients

xFS Performance Average time to complete the Andrew benchmark, varying the number of simultaneous clients

System Variables Aggregate Large-Write Bandwidth with Different Storage Server Configurations Variation in Average Small File Creation Speed with more Managers

Possible Problems System relies on secure network between machines, and trusted kernels on distributed nodes Testing done on Myrinet

Low-Bandwidth NFS Want efficient remote access over slow or wide area networks File systems better than CVS, copying all data over Want close-to-open consistency

LBFS Large client cache containing user’s working set of files Don’t send all the data – reconstitute files from previous data, and only send changes

File indexing Non-overlapping chunks between 2K and 64K Broken up using 48 byte Rabin fingerprints Identified by SHA-1 hash, indexing on first 64 bits Stored in database, recomputed before use to avoid synchronization issues

Protocol Based on NFS, added GETHASH, MKTMPFILE, TMPWRITE, CONDWRITE, COMMITTMP Security infrastructure from SFS Whole file caching Retrieve from server on read unless valid copy in cache Write back to server when file closed

File Reads

File Writes

Implementation LBFS server accesses file system as an NFS client Server creates trash directory for temporary files Server inefficient when files overwritten or truncated, which could be fixed by lower- level access. Client uses xfs driver

Evaluation

Bandwidth consumption Much higher bandwidth for first build

Application Performance

Bandwidth and Round Trip Time

Conclusions New technologies open up new possibilities for network file systems Cost of increased traffic over Ethernet may cause problems for xFS, cooperative caching.