1. CS 443 Advanced OS Fabián E. Bustamante, Spring 2005 Glacier: Highly Durable, Decentralized Storage Despite Massive Correlated Failures Andreas Haeberlen, Alan Mislove, and Peter Druschel Presenter: Yi Qiao

2. 2 Outline Introduction Related Work Assumption and Targeted Environment Glacier Object Aggregation Security Evaluation Conclusion

3. 3 Introduction How to achieve high availability in decentralized storage systems? Replication Problems –Failure is not independent –Worms make the problem worse – Catastrophic effects of losing some data Glacier –A distributed storage system that is robust to large-scale correlated failures Highly durable, decentralized storage –Trades efficiency of storage for durability –No any assumptions about the nature and correlation of failures –Aggregation of small objects and a fragment maintenance protocol to reduce the message overhead

4. 4 Related Works OceanStore and Phoenix –Apply introspection to defend against correlated failures Difficult to capture all correlations Introspection itself can make the system vulnerable to attacks –Glacier relies on minimal assumptions about the nature of failures by using larger storage overhead TotalRecall –Optimizes availability under churn, no worst-case guarantees PAST, Farsite –Replication against data loss Weatherspoon et al. –Erasure codes can achieve better MTTF than plain replication

5. 5 Assumptions and Intended Environment Intended to be used in an environment consisting of desktop computers within an organizational intranet –Some fraction of nodes can be home desktops connected via DSL or wireless LAN –Modest amount of churn and good network connectivity –Used on combination with conventional decentralized replication storage layer Lifetime – hundreds of days; session time – days or hours Three operation model –Normal operation –Large-scale failure – up to a fraction of fmax nodes failures Protecting data stored on non-faulty nodes –Recovery mode reconstitutes aggregates and restores missing fragments

6. 6 Glacier Participating storage nodes form an overlay network –The set of keys forms a circular space –Each node stores objects with keys in their own key segment –Uses underlying DHT layer for secure routing and communication Operates along with a primary store with full replicas Aggregation of small objects Erasure coding of aggregation Fragments placement at random selected nodes

7. 7 Glacier Durability guarantee –f<=fmax –Each object survives with probability p>=pmin Application Interface –put(i,v,o,l) –get(i,v) o –refresh(i,v,l) –No primitives for deletion or overwriting Leases are used, can be renewed when necessary

8. 8 Glacier Fragments and manifests –Erasure code – reduce storage overhead Object O of size (O) is stored as n fragments F1, F2,.. Fn of size (O)/r, any r of which are sufficient for object restore Object key k, Fragment Fi key (k,i,v) –Object authenticator and Manifest AO=(H(O), H(F1), H(F2), …, H(Fn), v, l) Corrupted fragments can be detected and removed Key ownership –Keys are assigned by consistent hashing over the set of nodes that are either on-line or were on-line within a period Tmax

9. 9 Glacier Fragment placement –Fragments of the same object placed on different random chosen nodes –Fragments of objects with similar keys should be grouped together –Placement function should be stable –P(k,i,v)=k+i/(n+1)+H(v) Primary replica – position k n fragments – n+1 equidistant points in the circular space H(v) – prevents load imbalance –When inserting new object (k,v), if owner of P(k,i,v) is offline, discard the fragment and restore later

10. 10 Glacier Fragment maintenance –Fragment insertion misses, key space ownership change, failures may cause fragments lost –A simple protocol The node compiles a list of all keys (k,v) in its local fragment store and sends the list to some of its peers Each peer replies with a list of manifests for missing object The node requires k fragments from its peers, validates them, and computes the fragment to store locally

11. 11 Glacier Recovery –No need for Glacier to explicitly detect failure Compromised nodes –Fail permanently, other nodes take over the key segments –Repaired and rejoin the system with an empty fragment store –Limits the number of simultaneous fragment reconstructions for a fixed number to avoid congestive collapse Garbage Collection –Happens when lease expires –Can be carried out independently by each storage node –Grace period TG for maximal clock difference

12. 12 Glacier Configuration –An object can be reconstructed if r out of N fragment can be obtained –Change N and r so that P meets desired durability –Still offers protection even when fmax is chosen too low –Lease time must be larger than maximal duration of a large- scale failure – order of months

13. 13 Glacier – Object Aggregation Massive redundancy – substantially large number of internal objects than application objects Aggregation of small application objects to reduce the cost of fragments creation and maintenance – tuples (oi, ki, vi) Aggregation is performed on a per-user basis –Simple, but loses the opportunity of bundle objects from different users Local aggregate directory –Aggregates link – forming a directed acyclic graph

14. 14 Glacier – Object Aggregation Recovery –Both primary store data and the aggregated directory could be lost after a correlated failure –Recover aggregated directory by walking the DAG Consolidation –Periodically check the aggregate directory for aggregates whose leases will expire soon Not renew the lease if many objects have expired leases –Non-expired objects are consolidated with new objects to generate a new aggregate –Particularly effective when object lifetimes are bimodal Consolidated aggregate contains mostly long-lived objects

15. 15 Glacier - Security Potential attacks against either durability or the integrity of data stored in Glacier –Attacks on integrity –Attacks on durability –Attacks on the time source –Space-filling attacks –Attacks on Glacier itself –Haystack-needle attacks

16. 16 Evaluation Glacier prototype –On top of FreePastry implementation of the Pastry structured overlay –Uses PAST as its primary store Two sets of experiments –ePost A cooperative, server-less email system for a small groups of users Glacier used as the storage layer –Trace-driven simulations A much larger workload with 147 users and up to 1,000 nodes

17. 17 Evaluation ePost experiments –20 to 30 nodes, mostly desktop PCs running linux –8 passive users and 9 active users –Uses Glacier to store email and corresponding metadata N=48, r=5, fmax=60%, pmin=0.999999 Experiment too small to guarantee uncorrelated fragment losses –Glacier was able to handle all the failures with the development and test of ePost

18. 18 Evaluation ePost Workload –Cumulative size of inserted objects over time Live – objects not expired yet –Histogram of object sizes Bimodal –Large number of objects between 1-10 KB »Justified aggregation –A low number of large objects »Usually attachments

19. 19 Evaluation ePost storage –Amount of storage required by Glacier for the workload Grows slowly as new emails entering the system XML data structure creates an additional 32% overhead –On-disk data structures VS actual email payload Storage overload close to 9.6 * 1.32

20. 20 Evaluation ePost traffic –Five categories Insertion, Refresh, maintenance, handoff, lookup –In times with low failures, traffic dominated by insertions and refreshes –During unstable period, handoff and maintenance traffic increases

21. 21 Evaluation ePOST aggregation –Compare the number of objects with the number of aggregates in the system Aggregation reduces the number of keys by over one order of magnitude Low number of expired objects – effective aggregate consolidation

22. 22 Evaluation Simulation Study – Diurnal Behavior –Glacier and the aggregation layer implemented –Trace from department email server –Diurnal behavior affects message overhead Higher churn –Less insertion traffic –More maintenance message for lost fragments recovery

23. 23 Evaluation Simulation Study – Load –See how load influences the message overhead Under light load, message overhead remains about constant –Aggregates are performed periodically by every node Higher load makes overhead increases about linearly

24. 24 Evaluation Simulation Study – Scalability –Increase overlay size, study per-node traffic overhead Remains approximately constant Grows slowly since the messages are routed using Pastry

25. 25 Conclusions Ensures durability of unrecoverable data in a cooperative, decentralized storage system Robust for large-scale, correlated, Byzantine failures of storage nodes No introspection Massive redundancy to mask the effects of correlated failures Erasure codes and garbage collection to reduce storage cost Aggregation and fragment maintenance protocol to reduce message costs