SCTP versus TCP for MPI Brad Penoff, Humaira Kamal, Alan Wagner Department of Computer Science University of British Columbia Distributed Research Group SC-2005 Nov 16

What is SCTP? Stream Control Transmission Protocol (SCTP) General purpose unicast transport protocol for IP network data communications Recently standardized by IETF Can be used anywhere TCP is used

What is SCTP? Stream Control Transmission Protocol (SCTP) General purpose unicast transport protocol for IP network data communications Recently standardized by IETF Can be used anywhere TCP is used Question Can we take advantage of SCTP features to better support parallel applications using MPI?

Communicating MPI Processes TCP is often used as transport protocol for MPI SCTP

Overview of SCTP

SCTP Key Features Reliable in-order delivery, flow control, full duplex transfer. TCP-like congestion control Selective ACK is built-in the protocol

SCTP Key Features Message oriented Use of associations Multihoming Multiple streams within an association

Associations and Multihoming Endpoint X NIC1 2 Endpoint Y NIC3 4 Network x.x Network x.x IP= IP= IP= IP= Association

Logical View of Multiple Streams in an Association

Partially Ordered User Messages Sent on Different Streams

Can be received in the same order as it was sent (required in TCP).

Partially Ordered User Messages Sent on Different Streams

Delivery constraints: A must be before C and C must be before D

MPI Point-to-Point Overview

MPI Point-to-Point Message matching is done based on Tag, Rank and Context (TRC). Combinations such as blocking, non-blocking, synchronous, asynchronous, buffered, unbuffered. Use of wildcards for receive MPI_Send(msg,count,type,dest-rank,tag,context ) MPI_Recv(msg,count,type,source-rank,tag,context )

MPI Messages Using Same Context, Two Processes

Out of order messages with same tags violate MPI semantics

Using SCTP for MPI Striking similarities between SCTP and MPI

SCTP-based MPI

MPI over SCTP : Design and Implementation LAM (Local Area Multi-computer) is an open source implementation of MPI library. We redesigned LAM TCP RPI module to use SCTP. RPI module is responsible maintaining state information of all requests.

Implementation Issues Maintaining State Information Maintain state appropriately for each request function to work with the one-to-many style. Message Demultiplexing Extend RPI initialization to map associations to rank. Demultiplexing of each incoming message to direct it to the proper receive function. Concurrency and SCTP Streams Consistently map MPI tag-rank-context to SCTP streams, maintaining proper MPI semantics. Resource Management Make RPI more message-driven. Eliminate the use of the select() system call, making the implementation more scalable. Eliminating the need to maintain a large number of socket descriptors.

Implementation Issues Eliminating Race Conditions Finding solutions for race conditions due to added concurrency. Use of barrier after association setup phase. Reliability Modify out-of-band daemons and request progression interface (RPI) to use a common transport layer protocol to allow for all components of LAM to multihome successfully. Support for large messages Devised a long-message protocol to handle messages larger than socket send buffer. Experiments with different SCTP stacks

Features of Design Head-of-Line Blocking Avoidance Scalability, 1 socket per process Multihoming Added Security

Head-of-Line Blocking

Performance

SCTP Performance SCTP stack is in early stages and will improve over time Performance is stack dependant (Linux lksctp stack << FreeBSD KAME stack) - SCTP bundles messages together so it might not always be able to pack a full MTU - Comprehensive CRC32c checksum – offload to NIC not yet commonly available

Experiments MPBench Ping-pong comparison NAS Parallel benchmarks Task Farm Program 8 nodes, Dummynet, fair comparison: Same socket buffer sizes, Nagle disabled, SACK ON, No multihoming, CRC32c OFF

Experiments: Ping-pong MPBench Ping Pong Test under No Loss

Experiments: NAS

Experiments: Task Farm Non-blocking communication Overlap computation with communication Use of multiple tags

Task Farm - Short Messages

Task Farm - Head-of-line blocking

Conclusions SCTP is a better match for MPI Avoids unnecessary head-of-line blocking due to use of streams Increased fault tolerance in presence of multihomed hosts Built-in security features Improved congestion control SCTP may enable more MPI programs to execute in LAN and WAN environments.

Future Work Release our LAM SCTP RPI module Modify real applications to use tags as streams Continue to look for opportunities to take advantage of standard IP transport protocols for MPI

More information about our work is at: Thank you! Or Google “sctp mpi”

Extra Slides

Associations and Multihoming Endpoint X NIC1 2 Endpoint Y NIC3 4 Network x.x Network x.x IP= IP= IP= IP=

MPI over SCTP : Design and Implementation Challenges: Lack of documentation Code examination Our document is linked-off LAM/MPI website Extensive instrumentation Diagnostic traces Identification of problems in SCTP protocol

MPI API Implementation Request Progression Layer Short Messages vs. Long Messages

Partially Ordered User Messages Sent on Different Streams

Added Security User data can be piggy-backed on third and fourth leg SCTP’s Use of Signed Cookie

Added Security 32 bit Verification Tag – reset attack Autoclose feature No half-closed state

NAS Benchmarks The NAS benchmarks approximate real world parallel scientific applications We experimented with a suite of 7 benchmarks, 4 data set sizes SCTP performance comparable to TCP for large datasets.

Farm Program - Long Messages

Head-of-line blocking – Long messages

Experiments: Benchmarks SCTP outperformed TCP under loss for ping pong test.

Experiments: Benchmarks SCTP outperformed TCP under loss for ping pong test.

Experiments: Benchmarks SCTP outperformed TCP under loss for ping pong test.