1. Pond The OceanStore Prototype

2. Introduction Problem: Rising cost of storage management Observations: Universal connectivity via Internet $100 terabyte storage within three years Solution: OceanStore

3. OceanStore Internet-scale Cooperative file system High durability Universal availability Two-tier storage system Upper tier: powerful servers Lower tier: less powerful hosts

4. OceanStore

5. More on OceanStore Unit of storage: data object Applications: email, UNIX file system Requirements for the object interface Information universally accessible Balance between privacy and sharing Simple and usable consistency model Data integrity

6. OceanStore Assumptions Infrastructure untrusted except in aggregate Most nodes are not faulty and malicious Infrastructure constantly changing Resources enter and exit the network without prior warning Self-organizing, self-repairing, self-tuning

7. OceanStore Challenges Expressive storage interface High durability on untrusted and changing base

8. Data Model The view of the system that is presented to client applications

9. Storage Organization OceanStore data object ~= file Ordered sequence of read-only versions Every version of every object kept forever Can be used as backup An object contains metadata, data, and references to previous versions

10. Storage Organization A stream of objects identified by AGUID Active globally-unique identifier Cryptographically-secure hash of an application-specific name and the owner’s public key Prevents namespace collisions

11. Storage Organization Each version of data object stored in a B-tree like data structure Each block has a BGUID Cryptographically-secure hash of the block content Each version has a VGUID Two versions may share blocks

12. Storage Organization

13. Application-Specific Consistency An update is the operation of adding a new version to the head of a version stream Updates are applied atomically Represented as an array of potential actions Each guarded by a predicate

14. Application-Specific Consistency Example actions Replacing some bytes Appending new data to an object Truncating an object Example predicates Check for the latest version number Compare bytes

15. Application-Specific Consistency To implement ACID semantic Check for readers If none, update Append to a mailbox No checking No explicit locks or leases

16. Application-Specific Consistency Predicate for reads Examples Can’t read something older than 30 seconds Only can read data from a specific time frame

17. System Architecture Unit of synchronization: data object Changes to different objects are independent

18. Virtualization through Tapestry Resources are virtual and not tied to particular hardware A virtual resource has a GUID, globally unique identifier Use Tapestry, a decentralized object location and routing system Scalable overlay network, built on TCP/IP

19. Virtualization through Tapestry Use GUIDs to address hosts and resources Hosts publish the GUIDs of their resources in Tapestry Hosts also can unpublish GUIDs and leave the network

20. Replication and Consistency A data object is a sequence of read-only versions, consisting of read-only blocks, named by BGUIDs No issues for replication The mapping from AGUID to the latest VGUID may change Use primary-copy replication

21. Replication and Consistency The primary copy Enforces access control Serializes concurrent updates

22. Archival Storage Replication: 2x storage to tolerate one failure Erasure code is much better A block is divided into m fragments m fragments encoded into n > m fragments Any m fragments can restore the original object

23. Caching of Data Objects Reconstructing a block from erasure code is an expensive process Need to locate m fragments from m machines Use whole-block caching for frequently- read objects

24. Caching of Data Objects To read a block, look for the block first If not available Find block fragments Decode fragments Publish that the host now caches the block Amortize the cost of erasure encoding/decoding

25. Caching of Data Objects Updates are pushed to secondary replicas via application-level multicast tree

26. The Full Update Path Serialized updates are disseminated via the multicast tree for an object At the same time, updates are encoded and fragmented for long-term storage

27. The Full Update Path

28. The Primary Replica Primary servers run Byzantine agreement protocol Need more than 2/3 nonfaulty participants Messages required grow quadratic in the number of participants

29. Public-Key Cryptography Too expensive Use symmetric-key message authentication codes (MACs) Two to three orders of magnitude faster Downside: can’t prove the authenticity of a message to the third party Used only for the inner ring Public-key cryptography for outer ring

30. Proactive Threshold Signatures Byzantine agreement guarantees correctness if not more than 1/3 servers fail during the life of the system Not practical for a long-lived system Need to reboot servers at regular intervals Key holders are fixed

31. Proactive Threshold Signatures Proactive threshold signatures More flexibility in choosing the membership of the inner ring A public key is paired with a number of private keys Each server uses its key to generate a signature share

32. Proactive Threshold Signatures Any k shares may be combined to produce a full signature To change membership of an inner ring Regenerate signature shares No need to change the public key Transparent to secondary hosts

33. The Responsible Party Who chooses the inner ring? Responsible party: A server that publishes sets of failure- independent nodes Through offline measurement and analysis

34. Software Architecture Java atop the Staged Event Driven Architecture (SEDA) Each subsystem is implemented as a stage With each own state and thread pool Stages communicate through events 50,000 semicolons by five graduate students and many undergrad interns

35. Software Architecture

36. Language Choice Java: speed of development Strongly typed Garbage collected Reduced debugging time Support for events Easy to port multithreaded code in Java Ported to Windows 2000 in one week

37. Language Choice Problems with Java: Unpredictability introduced by garbage collection Every thread in the system is halted while the garbage collector runs Any on-going process stalls for ~100 milliseconds May add several seconds to requests travel cross machines

38. Experimental Setup Two test beds Local cluster of 42 machines at Berkeley Each with 2 1.0 GHz Pentium III 1.5GB PC133 SDRAM 2 36GB hard drives, RAID 0 Gigabit Ethernet adaptor Linux 2.4.18 SMP

39. Experimental Setup PlanetLab, ~100 nodes across ~40 sites 1.2 GHz Pentium III, 1GB RAM ~1000 virtual nodes

40. Storage Overhead For 32 choose 16 erasure encoding 2.7x for data > 8KB For 64 choose 16 erasure encoding 4.8x for data > 8KB

41. The Latency Benchmark A single client submits updates of various sizes to a four-node inner ring Metric: Time from before the request is signed to the signature over the result is checked Update 40 MB of data over 1000 updates, with 100ms between updates

42. The Latency Benchmark Update Latency (ms) Key Size Update Size 5% Time Median Time 95% Time 512b 4kB394041 2MB103710861348 1024b 4kB9899100 2MB109811501448 Latency Breakdown PhaseTime (ms) Check0.3 Serialize6.1 Apply1.5 Archive4.5 Sign77.8

43. The Throughput Microbenchmark A number of clients submit updates of various sizes to disjoint objects, to a four-node inner ring The clients Create their objects Synchronize themselves Update the object as many time as possible for 100 seconds

44. The Throughput Microbenchmark

45. Archive Retrieval Performance Populate the archive by submitting updates of various sizes to a four-node inner ring Delete all copies of the data in its reconstructed form A single client submits reads

46. Archive Retrieval Performance Throughput: 1.19 MB/s (Planetlab) 2.59 MB/s (local cluster) Latency ~30-70 milliseconds

47. The Stream Benchmark Ran 500 virtual nodes on PlanetLab Inner Ring in SF Bay Area Replicas clustered in 7 largest P-Lab sites Streams updates to all replicas One writer - content creator – repeatedly appends to data object Others read new versions as they arrive Measure network resource consumption

48. The Stream Benchmark

49. The Tag Benchmark Measures the latency of token passing OceanStore 2.2 times slower than TCP/IP

50. The Andrew Benchmark File system benchmark 4.6x than NFS in read-intensive phases 7.3x slower in write-intensive phases