Graybox NFS Caching Proxy By: Paul Cychosz and Garrett Kolpin

First we need to know a little about how the “graybox” works. Inference by observation No server or client-side modifications necessary What are Graybox Techniques?

Figure out what the client is caching Maintain low overlap between client and proxy caches Our Problem

Treat the client as a graybox Monitor NFS traffic Infer from NFS traffic the contents of client cache Our Solution

Cache overlap decreased by 5% to 50% when using graybox techniques Conclusions

Approach Results Conclusions Outline

Approach NFS specifics:  Implemented in kernel as another FS type  Replaces “file_operations” with its own functions for read, write, open, etc.  Uses VFS abstraction VFS NFS kernel code RPC / XDR TCP / UDP retrans., data wrapping

Approach Client cache / Replacement Caches on page level Not much NFS specific code – Uses global Linux page cache Not a static size, no upper-bound Timestamps: “jiffies”, per page.

Approach Reads typically > ~85% of traffic NFS / RPC “read” structure RPC Packets sent out: access to dir, getattr to file, access to file, then read, read, read, … Client use of getattr to revalidate cached copies  Exploit this, proxy explicitly look for these to determine client cache contents

Approach Proxy Sees:  File handle  Offset  size --becomes our “cache element”  Store in hash table  Cache on reads, run our cache element replacement policy on getattr (proof of cached client copy)

Approach Read system call Cache lookup Revalidate data Read setup OR Replacement (possible force-out) Lookup Return dataAdd OR NFS ClientProxy hit miss hit miss getattr read Data Forward req. to server Typical read()

Disable swap space Use mincore() to view pages that client has cached Use “balloon” program to keep client cache contents manageable Viewing Client Cache Contents Test Environment:

Approach  Logistics: Initial setup, client/server/proxy all on same PC – doesn’t work  Client cache size changes -solved: compare percentages, normalize  Repeated RPC calls not idempotent Difficulties  Not a significant problem, proxy efficiency degrades linearly with NFS performance if network traffic is bad

Approach  Ambiguities:  Different FS activities DON’T generate unique RPC patterns  getattr used for a few things besides reads -- Null RPC packets Cannot tell exact blocks client caching, only which files Need second, local replacement policy on proxy! Difficulties

Approach Global Policies LRU MRU Local Policies Forcing entire file MRU Cache Replacement

Approach Results Conclusions Outline

Control Experiments –Global LRU, no local force-out –Global MRU, no local force-out Combinations –Global LRU, whole file force-out –Global LRU, block-level local MRU –Global MRU, whole file force-out –Global MRU, block-level local MRU What We Tested

Used Postmark benchmark 12 files with sizes randomly selected between 4 and 50 kilobytes Gives us more of a “real-world” workload How We Tested

Results

Approach Results Conclusions Outline

Decrease in overlap by between 5 and 50 percent as compared to control tests MRU global file replacement is good by itself MRU global policy with whole-file local policy gives highest cache fill and lowest overlap Conclusions

Questions? Feedback? ? ? ?