High Performance Cluster Computing Architectures and Systems Hai Jin Internet and Cluster Computing Center

2 Multiple Path Communication Introduction Heterogeneity in Networks and Applications Multiple Path Communication Case Study Summary and Conclusion

3 Introduction (1) Communication is major factor of performance in network-based computing Utilize all available network resources within a system Substantial benefits in simultaneously exploiting multiple independent communication paths between processors Heterogeneous network environment Single application execution Different network - different performance characteristic

4 Introduction (2) Performance-based path determination (PBPD) Exploiting multiple physical communication paths & multiple communication protocols within a single parallel application program to reduce communication overhead Different physical networks & protocols have different performance characteristics Different types of communication within an application might be best suited to one of several different communication paths bulk data transfer (image data): – high-bandwidth communication short control messages (synch, acknowledgement): - low-latency communication

5 Introduction (3) Performance-based path determination (PBPD) Performance-based path selection (PBPS) Dynamically select the best communication path among several in a given system Performance-based path aggregation (PBPA) Aggregate multiple communication paths into a single virtual path for sending an individual message in an application Each independent path simultaneously carries a fraction of a single message Useful in bandwidth-limited situations Control is based on message size, message type, or network traffic load

6 Heterogeneity in Network and Applications Heterogeneity inherent interprocessor communication Different types of messages latency-limited bandwidth-limited

7 Varieties of Communication Networks WAN (1 Mbps) MAN (100 Mbps) SAN (more than 1 Gbps) Ethernet (10 Mbps – 1000 Mbps) ATM (155 Mbps) FDDI (100 Mbps) HiPPI (800 Mbps – 1.6 Gbps) Infinitband (10Gbps)

8 Exploiting Multiple Communication Paths Messages size communication groups priority level Different type of network services to support different types of message communication Network services connection oriented connectionless Both are logically sufficient for the implementation of any communication pattern Each offers performance & programming benefits for some classes of applications

9 Benefits to Support Multiple Communication Paths Efficient network utilization Most appropriate communication paths to be used Alternative network protocols Connection oriented / connectionless Robust communication Multiple networks provides extra reliability Network load balancing Quality of service Multiple paths can support diverse QoS requirements

10 Multiple Path Communication A single application is likely to require several different types of messages for communication Different types of messages may be better suited to a different type of communication mechanism

11 Performance-Based Path Selection (PBPS) Useful when one provides better performance in one situation while the other is better in another situation Possible to dynamically select the appropriate communication path for a given communication event f1(m1) = t1 f2(m2) = t2 f PBPS (m) = Best[f i (m)], where (i= 1.. N)

12 When sending a message of size m i, performance- based path selection (PBPS) uses the lower latency curve among f 1 (m i ) and f 2 (m i )

13 Performance-Based Path Aggregation (PBPA) Can be applied when different paths show similar characteristics 2 nearly identical networks aggregate 2 networks into a single virtual network bandwidth will be nearly twice Divide – transmit - aggregate Important consideration determine the size of the submessages f PBPA (m) = f i (m i ), where (i= 1.. N), (m =  m i )

14 When using performance-based path aggregation (PBPA) with two networks, a message of size m 1 + m 2 is split into two submessages such that messages of size m 1 and m 2 are sent over networks f 1 (m i ) and f 2 (m i ) simultaneously

15 PBPD Library Custom library whose main feature is the support of multiple communication paths in a single application program Based on common TCP layer Add integer field ‘ length ’ tells message size Used by PBPA if a message is too small to segment Handle the multiplexing of different TCP connections Use UNIX select system call with appropriate table lookups

16 Implementation and Protocol Hierarchy of the PBPD Communication Routines

17 Case Study: Multiple Path Characteristic Communication type in parallel application program point-to-point, collective 4 Silicon Graphics Challenge L shared-memory multiprocessors. 4 or 8 R10000 processors at 196MHz per node 10Mbps Ethernet and 266Mbps Fiber Channel

18 Multiple Heterogeneous Network Configuration used in the Experiments

19 Point-to-point and Broadcast Characteristics of Ethernet using the TCP and UDP Communication Protocols

20 Example of a Broadcast Operation of a Separate Addressing Method using the PBPA Technique

21 Point-to-point Characteristics of Ethernet and Fibre Channel using the TCP Protocol

22 Broadcast Characteristics of Ethernet and Fibre Channel using the TCP Protocol

23 Case Study: Communication Patterns of Parallel Applications Performance of PBPD at application-level depends on the communication patterns of the specific application being executed

24 Parallel Benchmark Programs Tested ProgramsDescription CGConjugate gradient MGMultigrid ISInteger sort FilterSmoothing (averaging) filter GaussGaussian elimination HoughLine recognition algorithm KirschImage processing TRFDTwo-electron integral transformation WarpSpatial domain image restoration BTSimulated CFD application using Block tridiagonal solver LUSimulated CFD application using LU solver SPSimulated CFD application using Pentadiagonal solver MICOMMiami isopycnic coordinate ocean model

25 Case Study: Computation Model Data parallelization Medium to coarse-grained parallelism Similar communication pattern for every node except when starting application Each processor tends to alternate computation and communication at the same time communication congestion is inevitable

26 Relative Times of Communication Events for the IS Benchmark

27 Relative Times of Communication Events for the MG Benchmark

28 Case Study: Message Size and Destination Distributions Distribution of message destinations Uniformly distributed among all nodes Biased to some destinations for each node Distribution of message sizes

29 Overall Message Destination Distribution for All of the Test Programs

30 The Message Destination Distribution for Each Processor in the CG Benchmark

31 The Cumulative Distribution of Message Sizes in the Test Programs

32 Experiments Results Parameterize communication pattern Point-to-point communication Small message mean size l1, probability to appear is b Large message mean size l2, probability to appear is 1 - (b+c) Broadcast communication Message mean size l3, probability to appear is c Assumption 1 master, p-1 slave processors Message size for each communication follow Poison distribution with 3 different mean value l1, l2, l3

33 Parameter Values Used in the Synthetic Benchmark Application type A B C pt2pt (small) mean l 1 pt2pt (large) mean l 2 broadcast mean l 3 b1-(b+c)c 45% 10% 25% 50% 5% 90%

34 Speedup using PBPS with the TCP and UDP Protocols over Ethernet. The speedups are normalized to the case when using the TCP protocol alone

35 Speedup using PBPA Technique with Ethernet and Fiber Channel using the TCP Protocols. The speedups are normalized to the case when using the Ethernet alone

36 Summary and Conclusion Communication overhead can be reduced by exploiting heterogeneity in both communication path and application Reduce communication overhead PBPD technique can achieve performance improvement based on message type