Download presentation
Presentation is loading. Please wait.
Published byAmbrose Stephens Modified over 9 years ago
1
Almaden Rice University Nache: Design and Implementation of a Caching Proxy for NFSv4 Ajay Gulati, Rice University Manoj Naik, IBM Almaden Renu Tewari, IBM Almaden
2
Almaden © 2003 IBM Corporat2006n Rice Talk Outline Federated File System NFSv4: Overview and New Features Nache Architecture Nache Implementation Experimental Results Conclusions and Discussion
3
Almaden © 2003 IBM Corporat2006n Rice Federated File System File system federation with unified namespace Flexible data management –Replication, migration and read/write caching across distributed file servers What are the options? –AFS, DCE/DFS provide distributed FS over WAN, but… difficult to deploy, not widely used –GPFS, Lustre are cluster filesystems, but… good for controlled environments, no federation –NFS is a standard, but… v2/v3 have been designed mainly for LAN, are chatty and not suited for WAN, firewall issues –NFSv4?
4
Almaden © 2003 IBM Corporat2006n Rice Why NFSv4? Open industry standard Optimizations for WAN –Compound operations A single RPC request contains multiple ops, reduces round-trips – Read-Write delegations Efficient cache consistency, no need to contact server once a delegation is obtained Delegations can be recalled on conflicting access Federation support –Client redirection Server returns NFS4ERR_MOVED to redirect clients Client requests fs_locations attribute Client connects to new location
5
Almaden © 2003 IBM Corporat2006n Rice AlmadenHouston New York Admin Clients Filesystem Namespace Unification Separate namespace across servers Unified namespace for clients
6
Almaden © 2003 IBM Corporat2006n Rice Clients (San Jose) Server (San Jose) Server (New York) /project 1. OPEN /project/p1 2. NFS4ERR_MOVED 3. GETATTR 4. fs_locations Data Access using NFSv4 Redirection Limitations High network latency when clients access remote servers Delegation are unsuited for data sharing, fewer awarded and frequently broken No unified caching, every client has to access remote server 6. /project/p1 7. DELEGRETURN 5.Client connects to New York server 8. Server responds
7
Almaden © 2003 IBM Corporat2006n Rice What is a Proxy Cache (Nache)? An NFSv4 server acts as a proxy for another remote NFS server –caches data for reads and writes using delegations –becomes the temporary “owner” of the data servicing opens, lock requests and reads and writes Multiple clients can share a proxy cache using a single server-to-proxy delegation Extends beyond web-proxy caching
8
Almaden © 2003 IBM Corporat2006n Rice Data Access using a Proxy Cache Clients (San Jose) Proxy Cache (San Jose) Server (New York) /project 3./project/p1 1.OPEN /project/p1 Benefits with NFSv4 Proxy Reduced network traffic and delay – Clients access files through local proxy cache – Clients can share delegations – fewer conflicts/recalls Improved performance for cached files – Clients can collectively take advantage of aggressive readahead, write back and local locking at the proxy – Both data and delegations are shared 2. Delegate /project/p1 to proxy
9
Almaden © 2003 IBM Corporat2006n Rice Nache Architecture Nache ServerNache Client VFS Layer Buffer Cache cacheFS Local Client (export to) Remote Server (mount from) (Linux nfsd module)(Linux nfs module) Nache
10
Almaden © 2003 IBM Corporat2006n Rice Nache Implementation Cascaded mounts NFS operation forwarding Sub-operation RPC call handling
11
Almaden © 2003 IBM Corporat2006n Rice Cascaded Mounts Proxy exports a NFS mounted file system Added export functionality for NFS – export_ops added, not available in vanilla NFS Nache Server /watson/almaden mount server:/ /almaden ls /net at client shows contents of /watson at server Client mount nache:/ /net /net /export
12
Almaden © 2003 IBM Corporat2006n Rice NFS Operation Forwarding NFS ops are redirected from nfsd to nfs using vfs layer Issues: certain operations are stateful –OPEN, CREATE, LOCK, ULOCK, CLOSE ClientNache Server VFS Nache Client NFSD VFS NFS ClientNache Server Application
13
Almaden © 2003 IBM Corporat2006n Rice Sub-Operation RPC calls Each FS operation at Nache server translates to a separate RPC call from Nache client Solution: –Remove such redundant checks from code path Nache ServerNache Client VFS Layer Local Client Remote Server (Linux nfsd module)(Linux nfs module) 1.OPEN 2.LOOKUP 3.ACCESS 4.OPEN
14
Almaden © 2003 IBM Corporat2006n Rice Experimental Methodology Setup –2-6 local NFS clients (San Jose) –One local proxy with modified kernel modules (San Jose) –One local NFS server (San Jose) –One remote NFS server (New York) Benchmarks –Filebench (developed by Sun) –Compilation of various packages – Linux kernel, emacs, gdb –Our micro-benchmarks
15
Almaden © 2003 IBM Corporat2006n Rice Experiment Categories Delegation Benefits Nache Benefits Nache Overhead
16
Almaden © 2003 IBM Corporat2006n Rice Read Delegation Benefits Client repeats open-read-close on 100 files Server ops reduced by 50% due to delegations
17
Almaden © 2003 IBM Corporat2006n Rice Write Delegation Benefits A client repeats open-read/write-close on 100 files Server ops reduced by 4 times
18
Almaden © 2003 IBM Corporat2006n Rice Delegation Benefits (Server load) Server ops reduced due to delegations
19
Almaden © 2003 IBM Corporat2006n Rice Delegation Benefits (Client performance) Time taken is lower and ops/sec is higher due to delegations
20
Almaden © 2003 IBM Corporat2006n Rice Experiment Categories Delegation Benefits Nache Benefits Nache Overhead
21
Almaden © 2003 IBM Corporat2006n Rice Nache Benefits - Filebench (webserver) Ops at server reduces by 38% with 4 clients
22
Almaden © 2003 IBM Corporat2006n Rice Nache Benefits – Kernel compile Ops at server unaffected by increase in number of clients
23
Almaden © 2003 IBM Corporat2006n Rice Nache Benefits – Compile Time Compile time unaffected by increase in number of clients
24
Almaden © 2003 IBM Corporat2006n Rice Experiment Categories Delegation Benefits Nache Benefits Nache Overhead
25
Almaden © 2003 IBM Corporat2006n Rice Nache Overhead over WAN (Throughput) Micro benchmark Ops/s (Nache) Ops/s (no Nache) Overhead(%) Create-Files91643 Random-Reads7748-60 Random-Writes8.61014 Random-Append2.22.48 Nache induces an overhead of 8-43% Code can be further optimized to reduce overheads
26
Almaden © 2003 IBM Corporat2006n Rice Nache Overhead over WAN (Latency) Micro benchmark Latency (Nache) Latency (no Nache) Overhead (%) Create-Files5217300942 Random-Reads12.920.7-37 Random-Writes116.39818 Random-Append883.48218 Proxy induces an overhead of 8-42% Code can be further optimized to reduce overheads
27
Almaden © 2003 IBM Corporat2006n Rice Result Summary Delegations provide better throughput, lower latency and reduced load at server Nache improves performance in presence of sharing among clients Overhead of Nache in absence of sharing is not too high except for few cases and can be further reduced
28
Almaden © 2003 IBM Corporat2006n Rice Conclusions NFSv4 is a good alternative for building federated file systems Delegations help in reducing the number of server operations and provide better caching Nache can be integrated with a federated file system architecture and improve performance in presence of sharing
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.