Wide-area cooperative storage with CFS

Overview CFS = Cooperative File System Peer-to-peer read-only file system Distributed hash table for block storage Lookup performed by Chord

Design Overview CFS clients contain 3 layers: A file system client Dhash storage layer Chord lookup layer CFS servers contain 2 layers:

Overview con’t Disk blocks:DHash blocks::Disk Addresses:Block Identifiers CFS file systems are read-only as far as clients are concerned can be modified by its publisher

File System Layout Insert file system blocks into the CFS system using a content hash as its identifier Then signs the root block with its private key Inserts the root block into CFS using the corresponding public key as its identifier

Publisher Updates Updating the file system’s root block to point to the new data Authentication by checking to make sure that the same key signed both old and new block Timestamps prevent replays of old data File systems are updated without changing the root blocks identifier

CFS properties Decentralized control Scalability Availability Load balance Persistence Quotas Efficiency

Chord Layer Same Chord protocol as mentioned earlier with a few modifications Server Selection Node ID authentication

Quick Chord Overview Consistent Hashing Successor lists finger tables Node joins/leaves Successor lists O(N) finger tables O(log N)

Server Selection Chooses next node to contact from finger table Gets the closest node to the destination What about network latency? Introduced measuring and storing latency in the finger tables Calculated when acquiring finger table entries Reasoning: RPCs to different nodes will incur varying latency so you want to choose on that minimizes this

Node ID authentication Idea: network security All chord IDs must be in the form h(x) H = SHA-1 has function x = the nodes IP + virtual node index When a new node joins the system Existing node will send a message to the new node The ID must match the claimed IP + virtual node index hash to be accepted

DHash Layer Handles Uses Chord to locate blocks Storing and retrieving blocks Distribution Replication Caching of blocks Uses Chord to locate blocks Key CFS design: split each file system into blocks and distribute those across many servers

Replication DHash replicates each block on k servers immediately after the blocks sucessor Why? …even if the block’s successor fails, the block is still available Server independence guaranteed because location on the ring is determined by hash of IP not by physical location

Replication con’t Could save space by storing coded pieces of blocks but….storage space is not expected to be a highly-constrained resource Placement of replicas allows a client to select the replica with the fastest download Result from Chord lookup will be the immediate predecessor to the node with X This node’s successor table contains entries for the latencies of the nearest Y nodes

Caching Caching blocks prevents overloading servers with popular data Using Chord… Clients contact server closer and closer to the desired location Once source is found or an intermediate cached copy is found all servers just contacted receive file to cache Replaces cached blocks in least-recently-used order

Load Balance Virtual servers…1 real server acting as several “virtual” servers Administrator can configure the number of based on server’s storage and network capabilities Possible Side-effect: creating more hops in Chord algorithm More nodes = more hops Solution: allow virtual server’s to look at each other’s tables

Quotas Control amount of data a publisher can inject Based on reliable ID of publishers won’t work because it requires centralized administration CFS uses quotas based on IP address of publishers Each server imposes a 0.1% limit…so as the capacity grows the total data amount grows Not easy to subvert this system because publishers must respond to initial confirmation requests

Updates and Deletion Only allows the publisher to modify data 2 conditions for acceptance Marked as a content-hash block  supplied key = SHA-1 hash of the blocks content Marked as a signed block  signed by public key = SHA-1 hash is the block’s CFS key No explicit delete Publishers must refresh blocks if they want them stored CFS deletes blocks that have not been refreshed recently

Experimental Results – Real Life 12 machines over the internet – US, the Netherlands, Sweden and South Korea

Lookup Range of servers 10,000 blocks 10,000 lookups for random blocks Distribution is roughly linear on log plot …so O(log N)

Load Balance Theoretical: Actual: 64 physical servers 1,6 and 24 virtual servers each Actual: 10,000 blocks 64 actual 6 virtual each

Caching Single block (1) 1,000 server system Average without:5.7 (10 look-ups)

Storage Space Control Varying number of virtual servers 7 physical servers 1, 2, 4, 8, 16, 32, 64 and 128 virtual 10,000 blocks

Effect of Failure 1,000 blocks 1,000 server system Each block has 6 replicas Fraction of servers fail before the stabilization algorithm is run

Effect of Failure con’t Same set-up as before X% of servers fail

Conclusion Highly scalable, available, secure read-only file system Uses peer-to-peer Chord protocol for lookup Uses replication and caching to achieve availability and load balance Simple but effective protection against inserting large amount of malicious data