1 Optiputer High Performance Transport Protocols & Implementation Plans Ryan X. Wu, Andrew A. Chien, Eric Weigle, Nut Taesombut CSAG, CSE, UCSD 01/25/2005

2 High Performance Transport Problem OptIPuter is Bridging the Gap Between High Speed Link Technologies and Growing Demands of Advanced Applications Transport Protocols Are the Weak Link –TCP Has Well-Documented Problems That Militate Against its Achieving High Speeds –OptIPuter Pursuing Range of Transport Protocols –Shared, Routed Infrastructure: –XCP [USC/ISI], –UDT [Gu&Grossman, UIC] –Provisioned Lambda: –RBUDP/LambdaStream [He&Xiong&Leigh, EVL, UIC] –GTP [Wu&Chien, UCSD] –Cluster-based: –CEP [Weigle&Chien, UCSD] –How to make applications utilize these protocols? OptIPuter Transport Protocol Implementation/Integration –Goal: Flexibility of Protocol Choices under Single Application Implementation –Solution: Globus/XIO

3 Optiputer High Performance Transport Protocols

4 RBUDP/LambdaStream Suitable for continuous data intensive applications. Quickly grasp the underlying bandwidth and maintain very low jitter. More in Eric He’s talk [He&Xiong&Leigh, EVL/UIC]

5 XCP for High BDP Networks – Shared, Routed Environment Systematic Implementation and Evaluation –Build/Test FreeBSD Kernel implementation, Performance Evaluation –Full protocol specification and mature the protocol –Work with the community (researchers, applications developers, users, vendors, operators, IETF), To Develop Deployment plan –Initial Results are Promising: Match Simulations XCP MeasuredTCP Measured [Bannister&Falk, USC-ISI]

6 UDT: UDP-based Data Transport Protocol UDP-based Data Transfer Protocol -- Successor of SABUL Congestion/Flow Control -- For shared networks -- Configurable congestion control Implementation -- User space: no need for root privilege or kernel recompilation -- Socket API (will also provide XIO API soon) -- Open source: udt.sf.netudt.sf.net [GU&Grossman, UIC]

7 GTP: Group Transport Protocol: Receiver-based Congestion Management Request-response for Reliable Data Transfer Single Flow Adaptation and Capacity Estimation Receiver-based Flow Scheduling for Fairness and Low Loss Rate –Balance Concurrent Data Fetching from Multiple Sources –Fair across Varied Sender RTTs –Efficient Transitions under Rapid Changes R1 R2 Multipoint-to-point contention at receivers …... Single Flow Control and Monitoring Centralized Rate Allocation UDP (data flow) / TCP (control flow) IP Applications GTP GTP Receiver Architecture [Wu & Chien, UCSD]

8 CEP: Composite Endpoint Protocol We need a High-performance, scalable, robust, flexible, adaptive, easy-to-use mechanism to terminate disproportionately large data transfers. User controls transfer through simple sockets- like interface, and transfer constraint specification. –Can emulate GridFTP, MPI-IO, and parallel filesystem interfaces easily. CEP handles actual transfers subject to user constraints via scheduling mechanism –Uses multiple cooperating nodes. –Heterogeneous hosts may take part unequally in a connection. –Scheduler handles failures, fairness, changing resources over time. –Does the “Heavy Lifting” –TCP, UDP, FAST, HSTCP,... used as underlying protocol. [Weigle & Chien, UCSD]

9 OptIPuter Transport Protocol Implementation/Integration

10 Transport Protocol Deployment Problem Issues: –How to make applications utilize OptIPuter transport protocols? –What is the relation between transport protocols and backbone management and configuration? Each transport protocol has its own API and semantics –Different targeted network environments –Shared, routed network –Private, or QoS-provisioned network –Different semantics –Blocking/non-blocking –Point-to-point, multipoint-to-point Applications need to be modified to use different protocols. –Changes may be complex –Need to maintain multiple versions with different transport protocols Provide the same API for all OptIPuter transport protocols –Allow convenient integration with applications –Support further testing/evaluating different transport protocols –Solution: Globus XIO: eXtensive Input Output library

11 Globus/XIO Globus XIO user API –Single API/Single Set of semantics. –Simple open/close/read/write API Driver Abstraction –Hides protocol details –Allows for extensibility –Drivers can be selected at runtime XIO interface for OptIPuter transport protocols –A simple XIO wrapper (~1000 lines of code) on top of each transport protocol to provide XIO API –Almost no changes needed to the original protocol implementation User API Framework Driver Stack Transform Transport XIO Framework

12 XIO Example Init the protocol stack –globus_xio_stack_init(&stack, NULL); Load drivers –globus_xio_driver_load(“TCP", &txdriver); –or ‘UDP’, “UDT’’, “GTP”… –globus_xio_stack_push_driver(stack, txdriver); Init and set protocol attributes –globus_xio_attr_init(&attr); –globus_xio_attr_cntl(attr, txdriver, GLOBUS_XIO_SET_LISTENING_PORT, hosts, numhosts); Create handle –globus_xio_handle_create(&handle, stack); Send/receive data –globus_xio_open(&handle, NULL, target); –globus_xio_write(handle, "hello world\n", 12, 1, &nbytes, NULL); –globus_xio_close(handle, NULL);

13 OptIPuter Communication Software Architecture Standard and Custom Communication Protocols are Accessible through a Unique API DVC Descriptor Provides Information about Resource Configuration, Naming, and Other Properties GTPXCP UDT Lambda- Stream DVC Descriptor High-level IPC Library (e.g., MPI, PVM) DVC. APIs Socket Wrapper (C and Java) XIO Interface DVC Config. Distributed Applications/ Web Services/Viz. Apps CEP

14 Implementation Plan Current Implementation Status –The TCP, UDP and simple UDT XIO drivers are coming with the Globus/XIO –GTP XIO driver is available Implementation and Integration Plan –XIO driver for UDT 2.0 will be available early next month –XIO drivers for LambdaStream and CEP are in progress –Demonstrate applications running with different protocols via XIO interface –Further testing and large-scale experiments on different transport protocols

15 For More Information A. Falk, T. Faber, J. Bannister, A. Chien, R. Grossman, J. Leigh, Transport protocols for high performance, Communications of the ACM, Volume 46, Number 11, November 2003, pp X. Wu and A. Chien, GTP: Group Transport Protocol for Lambda Grids, IEEE Symposium on Cluster Computing and the Grid (CCGrid), April Y. Gu, X. Hong, and R. Grossman, Experiences in Design and Implementation of a High Performance Transport Protocol, (submitted for publication). Eric Weigle and Andrew A. Chien, The Composite Endpoint Protocol (CEP): Scalable Endpoints for Terabit Flows Submitted to CCGrid LambdaStream Globus XIO: