1. 1 stdchk : A Checkpoint Storage System for Desktop Grid Computing Matei Ripeanu – UBC Sudharshan S. Vazhkudai – ORNL Abdullah Gharaibeh – UBC The University of British Columbia Oak Ridge National Laboratory Samer Al-Kiswany – UBC

2. 2 Checkpointing Introduction Checkpointing uses: fault tolerance, debugging, or migration. Typically, an application running for days on hundreds of nodes (e.g. a desktop gird ) saves checkpoint images periodically. ICDCS ‘08... C C C C

3. 3 Deployment Scenario ICDCS ‘08

4. 4 The Challenge Although checkpointing is necessary:  It is a pure overhead from the performance point of view. Requirement: High performance, scalable, and reliable storage system optimized for checkpointing applications.  Generates a high load on the storage system Most of the time spent writing to the storage system. ICDCS ‘08 Challenge: Low cost, transparent support for checkpointing at file- system level.

5. 5  Write intensive application ( bursty ). e.g., a job running on hundreds of nodes.  periodically checkpoints 100s of GB of data. Checkpointing Workload Characteristics  Write once, rarely read during application execution.  Potentially high similarity between consecutive checkpoints.  Applications specific checkpoint image life span. When it is safe to delete the image ? ICDCS ‘08

6. 6 Why Checkpointing-Optimized Storage System?  Optimizing for checkpointing workload can bring valuable benefits:  High throughput through specialization.  Considerable storage space and network effort saving. through transparent support for incremental checkpointing.  Simplified data management by exploiting the particulaities of checkpoint usage scenarios.  Reduce the load on a share file-system  Can be built atop scavenged resources – low cost. ICDCS ‘08

7. 7 stdchk A checkpointing optimized storage system built using scavenged resources. ICDCS ‘08

8. 8 Outline  stdchk architecture  stdchk features  stdchk system evaluation ICDCS ‘08

9. 9 stdchk Architecture Manager (Metadata management) Benefactors (Storage nodes) Client (FS interface) ICDCS ‘08

10. 10  Simplified data management stdchk Features  POSIX file system API – as a result using stdchk does not require modifications to the application.  High-throughput for write operations  Support for transparent incremental checkpointing ICDCS ‘08  High reliability through replication

11. 11 Optimized Write Operation Alternatives Write procedure alternatives:  Complete local write  Incremental write  Sliding window write ICDCS ‘08

12. 12 Optimized Write Operation Alternatives Application stdchk FS Interface stdchk Disk ICDCS ‘08 Compute Node Write procedure alternatives:  Complete local write  Incremental write  Sliding window write

13. 13 ICDCS ‘08 Optimized Write Operation Alternatives stdchk Compute Node Application stdchk FS Interface Disk Write procedure alternatives:  Complete local write  Incremental write  Sliding window write

14. 14 Memory Disk ICDCS ‘08 Compute Node stdchk Application stdchk FS Interface Write procedure alternatives:  Complete local write  Incremental write  Sliding window write Optimized Write Operation Alternatives

15. 15 Write Operation Evaluation ICDCS ‘08 Testbed: 28 machines Each machine has : two 3.0GHz Xeon processors, 1 GB RAM, two 36.5GB SCSI disks.

16. 16 Achieved Storage Bandwidth The average ASB over a 1 Gbps testbed. Sliding Window write achieves high bandwidth (110 MBps)  Saturates the 1 Gbps link ICDCS ‘08

17. 17  Checkpointing optimized data management stdchk Features  POSIX file system interface – no required modification to the application  High throughput write operation  Transparent incremental checkpointing ICDCS ‘08

18. 18 Transparent Incremental Checkpointing But : How much similarity is there between consecutive checkpoints ? How can we detect similarities between checkpoints? Is this fast enough? Incremental checkpointing may bring valuable benefits:  Lower network effort.  Less storage space used. ICDCS ‘08

19. 19 Checkpoint T0 X Y Z Hashing X Y Z T0 ICDCS ‘08 Similarity Detection Mechanism – Compare-by-Hash

20. 20 W Y Z X Y Z T0 W T1 Hashing Checkpoint T1 Will store T1 ICDCS ‘08 Similarity Detection Mechanism – Compare-by-Hash

21. 21  How to divide the file into blocks?  Fixed-size blocks + compare-by-Hash (FsCH)  Content-based blocks + compare-by-Hash (CbCH) ICDCS ‘08 Similarity Detection Mechanism

22. 22 FsCH Insertion Problem Checkpoint i Checkpoint i+1 Result: Lower similarity detection ratio. B1 B2 B3 B4 B5 B1 B2 B3 B4 B5 B6 ICDCS ‘08

23. 23 Content-based Compare-by-Hash (CbCH) ICDCS ‘08 Checkpoint i Hashing B1 B2 B3 B4 HashValue K = 0 ? m bytes k bits offset

24. 24 Result: Higher similarity detection ratio. Content-based Compare-by-Hash (CbCH) Checkpoint i Checkpoint i+1 But: Computationally intensive. B1 BX B3 B4 ICDCS ‘08 B1 B2 B3 B4

25. 25 Evaluating Similarity Between Consecutive Checkpoints Avg. Checkpoint size Number of checkpoints Type 2.4 MB100Application level 450 MB  1200 System level - BLCR 1 GB  400 Virtual machine level - Xen The Applications : BMS* and BLAST Checkpointing interval: 1, 5 and 15 minutes ICDCS ‘08 * Checkpoints by Pratul Agarwal (ORNL)

26. 26 Similarity Ratio and Detection Throughput The table presents the average rate of detected similarity and the throughput in MB/s (in brackets) for each heuristic. ICDCS ‘08 But: Using the GPU, CbCH achieves over 190 MBps throughput !! - StoreGPU: Exploiting Graphics Processing Units to Accelerate Distributed Storage Systems, S. Al-Kiswany, A. Gharaibeh, E. Santos- Neto, G. Yuan, M. Ripeanu, HPDC, 2008. BLASTBMS Technique  XenBLCRApp 5 or 15 min15 min5 min1 min Interval  0.0% [110]6.3% [113]23.4% [109]0.0% [108]1MBFsCH 0.0% [28.4]70% [26.4]82% [26.6]0.0% [28.4] no- overlap m=20B, k=14b CbCH BLASTBMS Technique  XenBLCRApp 5 or 15 min15 min5 min1 min Interval  0.0%6.3%23.4%0.0%1MBFsCH 0.0%70%82%0.0% no- overlap m=20B, k=14b CbCH

27. 27 Compare-by-Hash Results Achieved Storage Bandwidth ICDCS ‘08 FsCH slightly degrades achieved bandwidth. But reduces the storage space used and network effort by 24%

28. 28 Outline  stdchk architecture  stdchk features  stdchk overall system evaluation ICDCS ‘08

29. 29 stdchk Scalability 7 clients: Each client writes 100 files (100MB each). Total of 70GB. stdchk pool of 20 benefactor nodes. ICDCS ‘08 Nodes Join Nodes Leave Steady stdchk sustains high loads :  Number of nodes  Workload

30. 30 Experiment with Real Application Improvement stdchk Local disk 27.0%16,49722,733Checkpointing time (s) 69.0%1.143.55Data size (TB) 1.3%455,894462,141Total execution time (s) Application : BLAST Execution time: > 5 days Checkpointing interval : 30s Stripe width : 4 benefactors Client machine: two 3.0GHz Xeon processors, SCSI disks. ICDCS ‘08

31. 31 Summary stdchk : A checkpointing optimized storage system built using scavenged resources. stdchk features:  High throughput write operation  Saves considerable disk space and network effort.  Checkpointing optimized data management  Easy to adopt – implements a POSIX file system interface  Inexpensive - built atop scavenged resources Consequently, stdchk :  Offloads the checkpointing workload from the shared FS.  Speeds up the checkpointing operations (reduces checkpointing overhead) ICDCS ‘08

32. 32 Thank you netsyslab.ece.ubc.ca ICDCS ‘08