1 stdchk : A Checkpoint Storage System for Desktop Grid Computing Matei Ripeanu – UBC Sudharshan S. Vazhkudai – ORNL Abdullah Gharaibeh – UBC The University of British Columbia Oak Ridge National Laboratory Samer Al-Kiswany – UBC

2 Checkpointing Introduction Checkpointing uses: fault tolerance, debugging, or migration. Typically, an application running for days on hundreds of nodes (e.g. a desktop gird ) saves checkpoint images periodically. ICDCS ‘08... C C C C

3 Deployment Scenario ICDCS ‘08

4 The Challenge Although checkpointing is necessary:  It is a pure overhead from the performance point of view. Requirement: High performance, scalable, and reliable storage system optimized for checkpointing applications.  Generates a high load on the storage system Most of the time spent writing to the storage system. ICDCS ‘08 Challenge: Low cost, transparent support for checkpointing at file- system level.

5  Write intensive application ( bursty ). e.g., a job running on hundreds of nodes.  periodically checkpoints 100s of GB of data. Checkpointing Workload Characteristics  Write once, rarely read during application execution.  Potentially high similarity between consecutive checkpoints.  Applications specific checkpoint image life span. When it is safe to delete the image ? ICDCS ‘08

6 Why Checkpointing-Optimized Storage System?  Optimizing for checkpointing workload can bring valuable benefits:  High throughput through specialization.  Considerable storage space and network effort saving. through transparent support for incremental checkpointing.  Simplified data management by exploiting the particulaities of checkpoint usage scenarios.  Reduce the load on a share file-system  Can be built atop scavenged resources – low cost. ICDCS ‘08

7 stdchk A checkpointing optimized storage system built using scavenged resources. ICDCS ‘08

8 Outline  stdchk architecture  stdchk features  stdchk system evaluation ICDCS ‘08

9 stdchk Architecture Manager (Metadata management) Benefactors (Storage nodes) Client (FS interface) ICDCS ‘08

10  Simplified data management stdchk Features  POSIX file system API – as a result using stdchk does not require modifications to the application.  High-throughput for write operations  Support for transparent incremental checkpointing ICDCS ‘08  High reliability through replication

11 Optimized Write Operation Alternatives Write procedure alternatives:  Complete local write  Incremental write  Sliding window write ICDCS ‘08

12 Optimized Write Operation Alternatives Application stdchk FS Interface stdchk Disk ICDCS ‘08 Compute Node Write procedure alternatives:  Complete local write  Incremental write  Sliding window write

13 ICDCS ‘08 Optimized Write Operation Alternatives stdchk Compute Node Application stdchk FS Interface Disk Write procedure alternatives:  Complete local write  Incremental write  Sliding window write

14 Memory Disk ICDCS ‘08 Compute Node stdchk Application stdchk FS Interface Write procedure alternatives:  Complete local write  Incremental write  Sliding window write Optimized Write Operation Alternatives

15 Write Operation Evaluation ICDCS ‘08 Testbed: 28 machines Each machine has : two 3.0GHz Xeon processors, 1 GB RAM, two 36.5GB SCSI disks.

16 Achieved Storage Bandwidth The average ASB over a 1 Gbps testbed. Sliding Window write achieves high bandwidth (110 MBps)  Saturates the 1 Gbps link ICDCS ‘08

17  Checkpointing optimized data management stdchk Features  POSIX file system interface – no required modification to the application  High throughput write operation  Transparent incremental checkpointing ICDCS ‘08

18 Transparent Incremental Checkpointing But : How much similarity is there between consecutive checkpoints ? How can we detect similarities between checkpoints? Is this fast enough? Incremental checkpointing may bring valuable benefits:  Lower network effort.  Less storage space used. ICDCS ‘08

19 Checkpoint T0 X Y Z Hashing X Y Z T0 ICDCS ‘08 Similarity Detection Mechanism – Compare-by-Hash

20 W Y Z X Y Z T0 W T1 Hashing Checkpoint T1 Will store T1 ICDCS ‘08 Similarity Detection Mechanism – Compare-by-Hash

21  How to divide the file into blocks?  Fixed-size blocks + compare-by-Hash (FsCH)  Content-based blocks + compare-by-Hash (CbCH) ICDCS ‘08 Similarity Detection Mechanism

22 FsCH Insertion Problem Checkpoint i Checkpoint i+1 Result: Lower similarity detection ratio. B1 B2 B3 B4 B5 B1 B2 B3 B4 B5 B6 ICDCS ‘08

23 Content-based Compare-by-Hash (CbCH) ICDCS ‘08 Checkpoint i Hashing B1 B2 B3 B4 HashValue K = 0 ? m bytes k bits offset

24 Result: Higher similarity detection ratio. Content-based Compare-by-Hash (CbCH) Checkpoint i Checkpoint i+1 But: Computationally intensive. B1 BX B3 B4 ICDCS ‘08 B1 B2 B3 B4

25 Evaluating Similarity Between Consecutive Checkpoints Avg. Checkpoint size Number of checkpoints Type 2.4 MB100Application level 450 MB  1200 System level - BLCR 1 GB  400 Virtual machine level - Xen The Applications : BMS* and BLAST Checkpointing interval: 1, 5 and 15 minutes ICDCS ‘08 * Checkpoints by Pratul Agarwal (ORNL)

26 Similarity Ratio and Detection Throughput The table presents the average rate of detected similarity and the throughput in MB/s (in brackets) for each heuristic. ICDCS ‘08 But: Using the GPU, CbCH achieves over 190 MBps throughput !! - StoreGPU: Exploiting Graphics Processing Units to Accelerate Distributed Storage Systems, S. Al-Kiswany, A. Gharaibeh, E. Santos- Neto, G. Yuan, M. Ripeanu, HPDC, BLASTBMS Technique  XenBLCRApp 5 or 15 min15 min5 min1 min Interval  0.0% [110]6.3% [113]23.4% [109]0.0% [108]1MBFsCH 0.0% [28.4]70% [26.4]82% [26.6]0.0% [28.4] no- overlap m=20B, k=14b CbCH BLASTBMS Technique  XenBLCRApp 5 or 15 min15 min5 min1 min Interval  0.0%6.3%23.4%0.0%1MBFsCH 0.0%70%82%0.0% no- overlap m=20B, k=14b CbCH

27 Compare-by-Hash Results Achieved Storage Bandwidth ICDCS ‘08 FsCH slightly degrades achieved bandwidth. But reduces the storage space used and network effort by 24%

28 Outline  stdchk architecture  stdchk features  stdchk overall system evaluation ICDCS ‘08

29 stdchk Scalability 7 clients: Each client writes 100 files (100MB each). Total of 70GB. stdchk pool of 20 benefactor nodes. ICDCS ‘08 Nodes Join Nodes Leave Steady stdchk sustains high loads :  Number of nodes  Workload

30 Experiment with Real Application Improvement stdchk Local disk 27.0%16,49722,733Checkpointing time (s) 69.0% Data size (TB) 1.3%455,894462,141Total execution time (s) Application : BLAST Execution time: > 5 days Checkpointing interval : 30s Stripe width : 4 benefactors Client machine: two 3.0GHz Xeon processors, SCSI disks. ICDCS ‘08

31 Summary stdchk : A checkpointing optimized storage system built using scavenged resources. stdchk features:  High throughput write operation  Saves considerable disk space and network effort.  Checkpointing optimized data management  Easy to adopt – implements a POSIX file system interface  Inexpensive - built atop scavenged resources Consequently, stdchk :  Offloads the checkpointing workload from the shared FS.  Speeds up the checkpointing operations (reduces checkpointing overhead) ICDCS ‘08

32 Thank you netsyslab.ece.ubc.ca ICDCS ‘08