1. On Object Maintenance in Peer-to-Peer Systems IPTPS 2006 Kiran Tati and Geoffrey M. Voelker UC San Diego

2. June 16, 2015IPTPS 20062 Object Maintenance l Storage is an essential service provided by p2p systems l Challenge is maintaining object availability and reliability in face of node churn u Node unavailability and failures impact object availability l Various object maintenance strategies have been developed u Farsite, DHash, PAST, OceanStore, TotalRecall, etc. l Minimize maintenance bandwidth overhead using u Placement strategies (successors, leaf sets, random indirect) u Redundancy mechanisms (replication, erasure coding) u Repair strategies (react immediately, lazily to failures)

3. June 16, 2015IPTPS 20063 Goals l Revisit how churn impacts maintenance overheads u Provide insight into issues and approaches l Highlight differences in system environments u Separate impact of temporary vs. permanent churn l Implications on object maintenance overhead u What should strategies focus on optimizing? l Try to provide insight independent of a particular strategy

4. June 16, 2015IPTPS 20064 Churn Review l Temporary Churn u Node leaves but eventually comes back u Object data is not lost l Permanent Churn u Node departs and never shows up again u Data on these nodes lost forever l Churn varies depending on system environment u Wide-area platforms: PlanetLab u Corporate: FarSite u File sharing: Overnet

5. June 16, 2015IPTPS 20065 Churn Spectrum Temporary Churn LowHigh Low High Permanent Churn PlanetLab Corporate File Sharing

6. June 16, 2015IPTPS 20066 Temporary Churn l Let’s look at temporary churn effects u No permanent churn (no data loss) u No significant change in node availabilities u Idealistic, but provides useful intuition l Object maintenance strategies place redundant data on fixed set of nodes u Object availability depends on node availability u As nodes come and go, object availability can change l Let’s track the available nodes from a group of nodes that are online at some point in time

7. June 16, 2015IPTPS 20067 Node Churn Initially number of available nodes in a group decreases as nodes leave the system After reaching a minimum value, it again increases as nodes rejoin the system This process continues till the object removed If we have enough redundancy to survive the minimum point, we mask the temporary churn

8. June 16, 2015IPTPS 20068 Implications l Can “mask” temporary churn u Redundancy inversely proportional to minimum point in graph l Want to create object with this degree of redundancy u Proactive: Estimate using node availability history u Reactive: Extend redundancy “on-demand” u Once sufficient redundancy reached, maintenance overhead very low for this object due to temporary churn »React only to shifts in node availabilities, etc. »If not reached, overhead extremely high l Environments with low permanent churn (PlanetLab) u Redundancy to mask temporary churn dominates object maintenance overhead u Tuning such redundancy has biggest impact on overhead

9. June 16, 2015IPTPS 20069 Permanent Churn l Permanent churn induces repairs u Repairs replenish redundancy lost due to permanent failures u In practice, eventually every node fails l Repair frequency depends on u Amount of permanent churn u Redundancy used to create/repair an object

10. June 16, 2015IPTPS 200610 Repairs l At each repair, a strategy can choose l Large redundancy factor u Fewer repairs u Extra storage cost l Small redundancy factor u Extra bandwidth for each repair u Reduces storage cost l What should an object maintenance strategy choose?

11. June 16, 2015IPTPS 200611 Analysis l Consider repairing an object of size f u Nodes fail permanently at rate d (“half death time”) l Repair triggered when availability threatened u Threshold x is redundancy needed for temporary churn »Below this threshold, object not available l Restore redundancy onto N new nodes using chunks of size c u Repair cost is (f + Nc) * (x + N) / 2 N d l Minimize for N, # of additional nodes to place data u N ~ sqrt(x) u Optimum depends on amount of redundancy needed to mask temporary churn – not the degree of permanent churn

12. June 16, 2015IPTPS 200612 Incremental Repairs

13. June 16, 2015IPTPS 200613 Implications l Repairs dominate object maintenance overhead in environments with high permanent churn u Tuning repair strategy has biggest impact on overhead u PlanetLab: Low permanent churn, not the critical problem u Also depends on object lifetime  longer lived, more repairs l Interestingly, choosing amount of redundancy to repair depends on temporary churn (not permanent) u Repairs triggered when object availability threatened l Better to make small repairs frequently u Also reduces storage

14. June 16, 2015IPTPS 200614 Capacity and Maintenance l Storage capacity also impacts object maintenance overhead l When a strategy creates/repairs, it stores redundant data on randomly chosen online nodes u Highly available node gets picked more often l Favoring highly available nodes reduces the redundancy required to mask temporary churn u Also less redundancy to cope with permanent churn l As storage utilization increases u More redundancy required to deal with churn u Higher per-object maintenance overheads as system fills

15. June 16, 2015IPTPS 200615 Implications

16. June 16, 2015IPTPS 200616 Conclusions l System environment matters u Different degrees of both temporary and permanent churn l Temporary churn u Can mask with sufficient redundancy u Dominant overhead is creating objects when low perm churn l Permanent churn u As permanent churn increases, repairs dominate overhead u Repair amount depends on degree of temporary churn l Storage u More redundancy, maintenance overhead as systems fills up