Naming and Integrity: Self-Verifying Data in Peer-to-Peer Systems Hakim Weatherspoon, Chris Wells, John Kubiatowicz University of California, Berkeley Future Directions in Distributed Computing. Thursday, June 6, 2002

FuDiCo 2002©2002 Hakim Weatherspoon/UC BerkeleyNaming and Integrity:2 OceanStore Context: Ubiquitous Computing Computing everywhere: –Desktop, Laptop, Palmtop. –Cars, Cellphones. –Shoes? Clothing? Walls? Connectivity everywhere: –Rapid growth of bandwidth in the interior of the net. –Broadband to the home and office. –Wireless technologies such as CDMA, Satellite, laser.

FuDiCo 2002©2002 Hakim Weatherspoon/UC BerkeleyNaming and Integrity:3 Archival Storage Where is persistent information stored? –Want: Geographic independence for availability, durability, and freedom to adapt to circumstances How is it protected? –Want: Encryption for privacy, secure naming and signatures for authenticity, and Byzantine commitment for integrity Is it Available/Durable? –Want: Redundancy with continuous repair and redistribution for long-term durability

FuDiCo 2002©2002 Hakim Weatherspoon/UC BerkeleyNaming and Integrity:4 Path of an Update

FuDiCo 2002©2002 Hakim Weatherspoon/UC BerkeleyNaming and Integrity:5 Questions about Data? How to use redundancy to protect against data being lost? How to verify data?

FuDiCo 2002©2002 Hakim Weatherspoon/UC BerkeleyNaming and Integrity:6 Archival Dissemination Built into Update Erasure codes –redundancy without overhead of strict replication –produce n fragments, where any m is sufficient to reconstruct data. m < n. rate r = m/n. Storage overhead is 1/r.

FuDiCo 2002©2002 Hakim Weatherspoon/UC BerkeleyNaming and Integrity:7 Durability Fraction of Blocks Lost Per Year (FBLPY)* –r = ¼, erasure-encoded block. (e.g. m = 16, n = 64) –Increasing number of fragments, increases durability of block Same storage cost and repair time. –n = 4 fragment case is equivalent to replication on four servers. * Erasure Coding vs. Replication, H. Weatherspoon and J. Kubiatowicz, In Proc. of IPTPS 2002.

FuDiCo 2002©2002 Hakim Weatherspoon/UC BerkeleyNaming and Integrity:8 Naming and Verification Algorithm Use cryptographically secure hash algorithm to detect corrupted fragments. Verification Tree: –n is the number of fragments. –store log(n) + 1 hashes with each fragment. –Total of n.(log(n) + 1) hashes. Top hash is a block GUID (B-GUID). –Fragments and blocks are self-verifying Fragment 3: Fragment 4: Data: Fragment 1: Fragment 2: H2H34HdF1 - fragment dataH14 data H1H34HdF2 - fragment dataH4H12HdF3 - fragment dataH3H12HdF4 - fragment dataF1F2F3F4 H1H2H3H4 H12H34 H14 B-GUID Hd Data Encoded Fragments F1 H2 H34 Hd Fragment 1: H2H34HdF1 - fragment data

FuDiCo 2002©2002 Hakim Weatherspoon/UC BerkeleyNaming and Integrity:9 Enabling Technology GUID Fragments

FuDiCo 2002©2002 Hakim Weatherspoon/UC BerkeleyNaming and Integrity:10 Complex Objects I Unit of Coding data Verification Tree GUID of d Encoded Fragments: Unit of Archival Storage

FuDiCo 2002©2002 Hakim Weatherspoon/UC BerkeleyNaming and Integrity:11 Complex Objects II GUID of d 1 d1d1 Unit of Coding Encoded Fragments: Unit of Archival Storage Verification Tree Blocks Data VGUID d2d2 d4d4 d3d3 d8d8 d7d7 d6d6 d5d5 d9d9 Data B -Tree Indirect Blocks M

FuDiCo 2002©2002 Hakim Weatherspoon/UC BerkeleyNaming and Integrity:12 Complex Objects III Data Blocks VGUID i VGUID i + 1 d2d2 d4d4 d3d3 d8d8 d7d7 d6d6 d5d5 d9d9 d1d1 Data B -Tree Indirect Blocks M d' 8 d' 9 M backpointer copy on write AGUID = hash{name+keys}

FuDiCo 2002©2002 Hakim Weatherspoon/UC BerkeleyNaming and Integrity:13 Mutable Data Need mutable data for real system. –Entity in network. –A-GUID to V-GUID mapping. –Byzantine Commitment for Integrity –Verifies client privileges. –Creates a serial order. –Atomically applies update. Versioning system –Each version is inherently read-only.

FuDiCo 2002©2002 Hakim Weatherspoon/UC BerkeleyNaming and Integrity:14 Archiver I Archiver Server Architecture –Requests to archive objects recv’d thru network layers. –Consistency mechanisms decides to archive obj. Asynchronous DiskAsynchronous Network Network Operating System Java Virtual Machine Thread Scheduler X Y Consistency Location & Routing Archiver Introspection Modules DispatchDispatch

FuDiCo 2002©2002 Hakim Weatherspoon/UC BerkeleyNaming and Integrity:15 Archiver Control Flow GenerateChkpt Stage GenerateFrags Stage Disseminator Stage Consistency Stage(s) GenerateFragsChkptReq DisseminateFragsReq GenerateFragsReq GenerateFragsResp GenerateFragsChkptResp Send Frags to Storage Servers DisseminateFragsResp Req

FuDiCo 2002©2002 Hakim Weatherspoon/UC BerkeleyNaming and Integrity:16 Performance Performance of the Archival Layer –Performance of an OceanStore server in archiving a objects. –analyze operations of archiving data (this includes signing updates in a BFT protocol). No archiving Inlined archiving (synchronous) (m = 16, n = 32) Delayed archiving (asynchronous) (m = 16, n = 32) Experiment Environment –OceanStore servers were analyzed on a 42-node cluster. –Each machine in the cluster is a IBM xSeries 330 1U rackmount PC with two 1.0 GHz Pentium III CPUs 1.5 GB ECC PC133 SDRAM two 36 GB IBM UltraStar 36LZX hard drives. The machines use a single Intel PRO/1000 XF gigabit Ethernet adaptor to connect to a Packet Engines Linux SMP kernel.

FuDiCo 2002©2002 Hakim Weatherspoon/UC BerkeleyNaming and Integrity:17 Performance: Throughput Data Throughput –No archive 5MB/s. –Delayed 3MB/s. –Inlined 2.5MB/s.

FuDiCo 2002©2002 Hakim Weatherspoon/UC BerkeleyNaming and Integrity:18 Performance: Latency Latency –Archive only Y-intercept 3ms, slope 0.3s/MB. –Inlined Archive = No archive + only archive.

FuDiCo 2002©2002 Hakim Weatherspoon/UC BerkeleyNaming and Integrity:19 Future Directions Caching for performance –Automatic Replica Placement Automatic Repair

FuDiCo 2002©2002 Hakim Weatherspoon/UC BerkeleyNaming and Integrity:20 Caching Automatic Replica Placement –Replicas are soft-state. –Can be constructed and destroyed as necessary. Prefetching –Reconstruct replicas from fragments in advance of use

FuDiCo 2002©2002 Hakim Weatherspoon/UC BerkeleyNaming and Integrity:21 Efficient Repair Global. –Global Sweep and repair not efficient. –Want detection/notification of node removal in system. –Not as affective as distributed mechanisms. Distributed. –Exploit DOLR’s distributed information and locality properties. –Efficient detection and then reconstruction of fragments AE L2 L1 L2 L3 L2 L1 L2 L3 L2 Ring of L1 Heartbeats

FuDiCo 2002©2002 Hakim Weatherspoon/UC BerkeleyNaming and Integrity:22 Conclusion Storage efficient, self-verifying mechanism. –Erasure codes are good. Self-verifying data assist in –Secure read-only data –Secure caching infrastructures –Continuous adaptation and repair For more information: