UDT: UDP based Data Transfer Yunhong Gu & Robert Grossman Laboratory for Advanced Computing University of Illinois at Chicago

02/17/2004 PFLDnet Outline Background UDT Protocol UDT Congestion Control Implementation/Simulation Results Summary

02/17/2004 PFLDnet Background Distributed data intensive applications over wide area optical networks:  Grid computing, access of bulk scientific data, data mining, high resolution video, etc. Transport protocol support:  Efficient and fair bandwidth unitization TCP does not work!

02/17/2004 PFLDnet Trans-Atlantic TCP Performance Chicago -> Amsterdam, 1Gbps link capacity, 110ms RTT  TCP: default setting (64KB buffer)  TCP: 12MB buffer (=1Gbps*110ms)  Parallel TCP: 64 TCP concurrent flows, with each having 1MB buffer Two concurrent TCP flows, 1 from Chicago to Amsterdam, 1 within Chicago local networks:  2Mps vs. 940Mbps!

02/17/2004 PFLDnet Why TCP Fails Discover/recover slow on high BDP links  Increase 1 byte per RTT Drastic decrease in sending rate Fairness bias on longer RTT links More prone to link error in high BDP links B: throughout in packets per second, p: loss rate

02/17/2004 PFLDnet Requirements to the New Protocol FAST  High utilization of the abundant bandwidth either with single or multiplexed connections FAIR  Intra-protocol fairness, independent of RTT FRIENDLY  TCP compatibility

02/17/2004 PFLDnet Use Scenarios Small number of sources shares abundant bandwidth Bulk data transfer  Most of the packets can be packed in maximum segment size (MSS) in a UDT session  MSS can be set up by applications and the optimal value is the path MTU

02/17/2004 PFLDnet UDT: UDP based Data Transfer  Reliable, application level, duplex, transport protocol, over UDP with congestion control  Implementation: Open source C++ library Two orthogonal parts  The UDT protocol framework that can be implemented above UDP, with any suitable congestion control algorithms  The UDT congestion control algorithm, which can be implemented in any transport protocols such as TCP What’s UDT?

02/17/2004 PFLDnet Packet Structure Data Packet:  Header: 1bit flag + 31bit sequence number Control Packet:  Header: 1bit flag + 3bit type + 12bit reserved + 16bit ACK seq. no. + (0 - 32n)bit control info  Type: ACK, ACK2, NAK, Handshake, Keep-alive, and Shutdown Actual size of a UDT packet can be ascertained from UDP header

02/17/2004 PFLDnet Data Packet 0Packet Sequence Number User Data Payload  Flag Bit: 0  UDT uses 31-bit packet based sequence number, ranging from 0 and ( )  Sequence number may be wrapped if it exceeds the maximum available number

02/17/2004 PFLDnet Control Packet 1typereservedACK Seq. No. Control Information Field  Flag Bit: 1  type: 3-bit  handshake (000), shutdown (101), keep-alive (001)  ACK (010), ACK2 (110), NAK (011)  UDT uses sub-sequencing: each ACK and related ACK2 are assigned a 16-bit unique ACK sequence number

02/17/2004 PFLDnet Acknowledgements Selective acknowledgement (ACK)  Generated at every constant interval to send back largest continuously received sequence number of data packets.  The sender sends back an ACK2 to the receiver for each ACK (sub-sequencing).  Also carries RTT, packet arrival speed, and estimated link capacity. Explicit negative acknowledgement (NAK)  Generated as soon as loss is detected.  Loss information may be resent if receiver has not received the retransmission after an increasing interval.  Loss information is compressed in NAK.

02/17/2004 PFLDnet Timing Packet Scheduling Timer  Tuned by Rate Control  High precision in CPU clock cycles Rate Control Timer: trigger rate control  RCTP = 0.01 seconds ACK Timer: trigger acknowledgement  ATP = RCTP

02/17/2004 PFLDnet Timing (cont.) NAK Timer: trigger negative acknowledgement  NTP = RTT Retransmission Timer: trigger retransmission based on time-out and maintain connection status  RTP = (exp-count + 1) * RTT + ATP where exp-count is the number of continuous time-out

02/17/2004 PFLDnet UDT Architecture DATA ACK ACK2 NAK Sender Recver Sender Recver  Pkt. Scheduling Timer  ACK Timer  NAK Timer  Retransmission Timer  Rate Control Timer Sender

02/17/2004 PFLDnet Congestion Control Rate based congestion control (Rate Control)  RC tunes the packet sending period.  RC is triggered periodically at the sender side.  RC period is constant of 0.01 seconds. Window based flow control (Flow Control)  FC limits the number of unacknowledged packets.  FC is triggered on each received ACK at the sender side.

02/17/2004 PFLDnet Rate Control AIMD: Increase parameter is related to link capacity and current sending rate; Decrease factor is 1/9, but not decrease for all loss events. Link capacity is probed by packet pair, which is sampled UDT data packets.  Every 16th data packet and it successor packet are sent back to back to form a packet pair.  The receiver uses a median filter on the interval between the arrival times of each packet pair to estimate link capacity. ……

02/17/2004 PFLDnet Rate Control (cont.) 1. If loss rate is greater than 1%, do not increase; 2. Number of packets to be increased in next RCTP time is: where B is estimated link capacity, C is current sending rate. Both are in packets or packets per second. MSS is the packet size in bytes. β = 1.5 * Recalculate packet sending period (STP).

02/17/2004 PFLDnet Rate Control (cont.) C (Mbps)B - C (Mbps)Increase Param. (Pkts) [0, 9000)(1000, 10000]10 [9000, 9900)(100, 1000]1 [9900, 9990)(10, 100]0.1 [9990, 9999)(1, 10]0.01 [9999, )(0.1, 1] < B = 10Gbps, MSS = 1500 bytes

02/17/2004 PFLDnet Rate Control (cont.) Decrease sending rate by 1/9, (or equivalently, increase packet sending period by 1.125), only if 1. Received an NAK, whose last lost sequence number is greater than the largest sequence number when last decrease occurred; or 2. The number of loss events since last decrease has exceeded a threshold, which increases exponentially and is reset when condition 1 is satisfied. No data will be sent out for the next RCTP time if a decrease occurs.  Help to clear congestion.

02/17/2004 PFLDnet BDP W = W* AS*(RTT+ATP)*0.125 AS is the packets arrival speed at receiver side.  The receiver records the packet arrival intervals. AS is calculated from the average of latest 16 intervals after a median filter.  It is carried back within ACK. Flow Control

02/17/2004 PFLDnet Slow Start Flow window starts at 2 and increases to the number of acknowledged packets, until the sender receives an NAK or reaches the maximum window size, when slow start ends. Packet sending period is 0 during slow start phase and set to the packet arrival interval at the end of the phase. Slow start only occurs at the beginning of a UDT session.

02/17/2004 PFLDnet Implementation: Performance

02/17/2004 PFLDnet Implementation: Intra-protocol Fairness

02/17/2004 PFLDnet Implementation: TCP Friendliness

02/17/2004 PFLDnet Implementation: TCP Friendliness (cont.)

02/17/2004 PFLDnet Implementation: File Transfer ToStarLightCanarieSARA From StarLight Canarie SARA CanarieStarLightSARA 1Gbps/15.9ms1Gbps/110ms Disk R: 800Mbps W: 550Mbps Disk R: 800Mbps W: 500Mbps Disk R: 1300Mbps W: 900Mbps

02/17/2004 PFLDnet Simulation: UDT Throughput at Different Bandwidth and RTT

02/17/2004 PFLDnet Simulation: Performance of Concurrent UDT Flows

02/17/2004 PFLDnet Simulation: Intra-protocol Fairness

02/17/2004 PFLDnet Simulation: RTT Independence

02/17/2004 PFLDnet Simulation: TCP Friendliness

02/17/2004 PFLDnet Simulation: Convergence/Stability

02/17/2004 PFLDnet Simulation: Complex Scenario 100  5010    Flow ID Throughput (Mbps) Link capacity Mbps Flow and its ID Node DropTail

02/17/2004 PFLDnet Simulation: Multi-bottleneck A x 200 B C X AB AC X AB AC

02/17/2004 PFLDnet Summary UDT Protocol  Application level upon UDP  Selective acknowledgement / explicit negative acknowledgement UDT Congestion Control  Rate Control Bandwidth estimation for fast probing available bandwidth and fast recovery AIMD for fairness Constant rate control interval  Flow Control Dynamic flow window according to packet receiving speed

02/17/2004 PFLDnet UDT Characters Good use of available bandwidth Application level - no changes in router and operating system No manual tuning Fair and Friendly: intra-protocol fairness, TCP friendliness, and RTT independence. Open source

Thank You! LAC: UDT: sourceforge.net/projects/dataspacesourceforge.net/projects/dataspace Internet Draft: draft-gg-udt-01.txt