Rio de Janeiro, October, 2005 SBAC Portable Checkpointing for BSP Applications on Grid Environments Raphael Y. de Camargo Fabio Kon Alfredo Goldman Department of Computer Science IME / USP

Rio de Janeiro, October, 2005 SBAC INTRODUCTION ● Computational Grids: ubiquitous access and coordinated usage of distributed resources ● Opportunistic Grids: usage of idle time of non- dedicated resources (desktop PCs) – Resources are heterogeneous (Mac, Windows, Linux) – Failure rate is higher than dedicated resources – Fails on a daily basis

Rio de Janeiro, October, 2005 SBAC INTEGRADE ● Grid middleware: usage of idle computing power from personal computers ● Federation of clusters – Clusters composed of a collection of resource providing nodes ● Sequential, parameter sweeping, and BSP applications

Rio de Janeiro, October, 2005 SBAC MOTIVATION ● Fault-tolerance is essential, specially when running parallel applications – Failure of a single node require restarting the application from the beginning – Checkpointing can be used as a fault- tolerance mechanism ● Mechanisms supporting heterogeneity improve resource utilization – Portable checkpointing mechanism allows reinitialization on machines of different architecture

Rio de Janeiro, October, 2005 SBAC OUR APPROACH ● Source code instrumentation – Perform additional tasks – Logging, profiling, persistence ● BSP application on heterogeneous nodes ● Portable checkpointing of applications ● Pre-compiler based on OpenC++ – Open-source tool for compile time reflection

Rio de Janeiro, October, 2005 SBAC BSP MODEL ● Bridging model – Link architecture to software ● Execution performed in supersteps – Computation and synchronization phases ● Two communication mechanisms: – Direct Remote Memory Access (DRMA) – Bulk Synchronous Message Passing (BSMP) ● Existing implementations: – Oxford BSPLib, PUB, BSP-G ● Work only on homogeneous clusters

Rio de Janeiro, October, 2005 SBAC HETEROGENEOUS NODES ● Extended BSPLib API – Some mehods receive extra parameter describing data type information → used to convert data – Pointer data types are defined by their declaration – Arbitrary data casts are not allowed ● Reasonable requirement for portability ● Pre-compiler automatically modifies application source code to use the extended API – Not need for manual modifications

Rio de Janeiro, October, 2005 SBAC CHECKPOINTING APPROACHES ● System-level checkpoint – Data is copied in chekpoints directly from application address space ● Application-level checkpoint – Instrument application source code to save its state – Semantic information about data-types is available ● Allows generation of portable checkpoints ● Drawbacks – Need to modify application source-code – Checkpoints at certain points in the application

Rio de Janeiro, October, 2005 SBAC CHECKPOINTING LIBRARY ● Pre-compiler instruments application source code – No manual instrumentation of source code – Necessary access to source code ● Checkpointing Library – Timer with a minimum checkpoint interval – Saving performed by a separate thread – Checkpoint can be stored in filesystem (NFS) or remote checkpoint repository (TCP/IP) ● Execution Manager – Coordinates checkpointing of BSP parallel applications

Rio de Janeiro, October, 2005 SBAC SAVING EXECUTION DATA Execution stack –Information from active function calls Local variables, function parameters, return address, and control information –Dependent on architecture and OS Heap area –Memory chunks allocated by application Execution Stack: control information local variables function parameters control information local variables function parameters Necessary to save – Execution stack + global variables – Data in heap area – Other information

Rio de Janeiro, October, 2005 SBAC ● Save only data necessary for reconstruction – List of function calls – Value of parameters and local variables ● Data added to an auxiliary stack during execution ● Recovery – Data read from checkpoint – Functions called – Local variables and parameter values assigned – Data conversion is performed if necessary EXECUTION STACK

Rio de Janeiro, October, 2005 SBAC POINTERS AND HEAP MEMORY ● Memory addresses – Specific for an execution – Architecture dependent ● Checkpoint generation – Data from heap area is copied to checkpoint – Memory addresses → offsets in checkpoint ● Recovery – Memory areas are allocated – Data is copied to these memory areas

Rio de Janeiro, October, 2005 SBAC EXPERIMENTS ● Parallel BSP applications – Similarity between large sequences of characters – Matrix multiplication ● Testbed: – Machines in two labs: ● 11 AthlonXP 1700+, 512MB ● 1 Power PC G4, 512MB ● 2 Athlon , 512MB ● 100Mbps Ethernet in 2 connected LANs

Rio de Janeiro, October, 2005 SBAC CHECKPOINTING OVERHEAD ● Simulation parameters – Matrix multiplication application using 9 nodes – Matrix size: 450x450 and 1800x1800 – Checkpoint sizes: 2.3MB and 37.1MB – Checkpointing intervals: 10, 30, and 60s

Rio de Janeiro, October, 2005 SBAC CHECKPOINTING OVERHEAD ● Storage on local machine or remote repository is faster than with NFS ● When using a remote repository, the overhead was consistently below 10%, even with a 10s interval

Rio de Janeiro, October, 2005 SBAC DYNAMIC GRID SIMULATION ● We simulated a dynamic environment where machine can fail unexpectedly ● Sequence similarity application using 10 nodes – Machine fails according to an exponential distribution ● MTBF (1/ λ) = 600s and 1800s ● Smaller checkpointing intervals → smaller execution times ckp interval 1/λt total 1/λt total 10s600s517.4s 1800s 490.3s 30s600s571.4s 1800s 519.3s 60s600s699.4s 1800s 534.5s

Rio de Janeiro, October, 2005 SBAC HETEROGENEOUS NODES Matrix multiplication on 4 heterogeneous nodes –3 AtlhonXP (x86) + 1 PowerPC G4 (ppc) Elements of type long double ● Time spent on data conversion is small compared to total execution time Matrix sizet exec t x86 t ppc 500x s0.042s0.217s 1000x s0.066s0.348s 2000x s0.078s0.430s

Rio de Janeiro, October, 2005 SBAC RESTART AN APPLICATION ● Time to recover from a checkpoint saved on different architectures ● Application that generates a graph of structures containing 20K nodes ● When recovering on an x86 machine – From x86: 0.179s – From x86-64:0.186s → 3.9% slower than x86 – From PPC: 0.192s → 7.2% slower than x86 ● Overhead when reading checkpoint data

Rio de Janeiro, October, 2005 SBAC CONCLUSIONS ● Overhead of portability is small, and can lead to better resource utilization ● Possible to execute BSP applications on heterogeneous nodes ● Ongoing work – Distributed checkpoint repository ● Scalability and Fault-tolerance – Simulations in large scale and wide area Grids – Support for multithreaded C++ applications

Rio de Janeiro, October, 2005 SBAC QUESTIONS For more information, please visit the poject page: