A Parallel Computational Model for Heterogeneous Clusters Jose Luis Bosque, Luis Pastor, IEEE TRASACTION ON PARALLEL AND DISTRIBUTED SYSTEM, VOL. 17, NO. 12, DECEMBER 2006 Presented by 張肇烜

Outline Introduction Heterogeneous LogGP HLogGP Validation Experimental Results Conclusions

Introduction During the last decade, Beowulf clusters have had tremendous dissemination and acceptance. However the design and implementation of efficient parallel algorithms for clusters is still a problematic issue.

Introduction (cont.) In this paper, a new heterogeneous parallel computational model based on the LogGP model is proposed.

Heterogeneous LogGP Reasons for selecting LogGP model  The architecture is very similar to a cluster.  LogGP removes the synchronization points needed in other models.  LogGP allows considering both short and long messages.

Heterogeneous LogGP (cont.)  LogGP assumes finite network capacity, avoiding situation where the network becomes a bottleneck.  This model encourages techniques that yield good results in practice, such as designing algorithms with balanced communication patterns.

Heterogeneous LogGP (cont.) HLogGP Definition:  Latency, L: Communication latency depends on both network technology and topology.  The Latency Matrix of a heterogeneous cluster can be defined as a square matrix L={l 1,1, …, l m,m }.

Heterogeneous LogGP (cont.)  Overhead, o: the time needed by a processor to send or receive a message is referred to as overhead.  Sender overhead vector, O s ={os 1,…,os m }.  Receiver overhead vector, O r ={or 1,…,or m }.  Gap between message, g: this parameter reflects each node’s proficiency at sending consecutive short messages.  A gap vector g={g 1,…,g m }.

Heterogeneous LogGP (cont.)  Gap per byte, G: The Gap per byte depends on network technology.  In a heterogeneous network, a message can cross different switches with different bandwidths.  Gap matrix G={G 1,1,…,G m,m }.

Heterogeneous LogGP (cont.)  Computational power, P i : The number of nodes cannot be used in a heterogeneous model for measuring the system’s computational power.  A computational power vector P={P 1,…,P m }.

HLogGP Validation Cluster Description: 100Mbps 10Mbps (slow, S) (fast, F)

HLogGP Validation (cont.) Benchmark 1:

HLogGP Validation (cont.) Benchmark 2: Source code of the benchmark for measuring the gap between messages.

HLogGP Validation (cont.) Overhead:

HLogGP Validation (cont.) Overhead:

HLogGP Validation (cont.) Latency: Switch-switch hub-hub Switch-hub

HLogGP Validation (cont.) Gap between messages:

HLogGP Validation (cont.) Gap per Byte:

HLogGP Validation (cont.) Computational power:

Experimental Results Three objectives were pursued in the tests presented here.  To verify HLogGP is accurate enough to predict the response time of a parallel program.  To verify that heterogeneity has a strong impact on system performance.  To show how the cluster parametrization may be used for determining the performance of a parallel program on a real application environment.

Experimental Results (cont.) A volumetric magnetic resonance image compression application was selected. The sequential process may be divided into the following stages.  Data acquisition.  Data read and memory allocation.  Computation of the 3D Harr wavelet transform.

Experimental Results (cont.)  Thresholding.  Encoding of the subbands using the run- length encoding compression algorithm.  Write back of the compressed image.

Experimental Results (cont.) A theoretical analysis of the application’s response time is presented.  First stage: The master distributes the raw data among the slave processors. The number of total slices. The cluster’s total computational power. Slices of each slave i will receive

Experimental Results (cont.)  The total time for this stage is : Cycles of sending overhead to get the first byte into the network Subsequent bytes take G cycles to be sent Each byte travels through the network for cycles The receiving processor spends in receiving overhead

Experimental Results (cont.)  Second stage: In this case, the response time is the time spent by the last slave to finish its work.  The total response time for the second stage is estimated as the response time of a generic slave processor:

Experimental Results (cont.)  Third stage: The master process has to first gather the partial results produced by all of the slave processes.  The total response time of the third phase is calculated as a sumatory:

Experimental Results (cont.)  Fourth stage: The master process has to send an image subband to each of the slave processes.  The total time for this stage is:

Experimental Results (cont.)  Fifth stage: This stage is similar to the second, the amount of work is not distributed according to the nodes’ computational power.  This time could be given approximately by the following expression :

Experimental Results (cont.)  Sixth stage: This stage is similar to the third stage, the message’s sizes cannot be determined a priori. K is determined by the subband size

Experimental Results (cont.) Execution Results

Experimental Results (cont.) Execution Results

Conclusion In this paper, HlogGP model for heterogeneous clusters has been proposed and validated. The model can be applied to heterogeneous clusters where either the nodes, the interconnection network, or both are heterogeneous.