1. UDT: UDP based Data Transfer Yunhong Gu & Robert Grossman Laboratory for Advanced Computing University of Illinois at Chicago Németh Felicián, Tarján Péter The work is supported by the 2/032/2004 ELTE-BUTE-Ericsson NKFP project on Research and Developments of Tools Supporting Optimal Usage of Heterogen Communication Networks

2. 02/17/2004 PFLDnet 2004 2 Outline Background UDT Protocol UDT Congestion Control Implementation/Simulation Results Summary

3. 02/17/2004 PFLDnet 2004 3 Background Distributed data intensive applications over wide area optical networks:  Grid computing, access of bulk scientific data, data mining, high resolution video, etc. Transport protocol support:  Efficient and fair bandwidth unitization TCP does not work!

4. 02/17/2004 PFLDnet 2004 4 Trans-Atlantic TCP Performance Chicago -> Amsterdam, 1Gbps link capacity, 110ms RTT  TCP: 5Mbps @ default setting (64KB buffer)  TCP: 100Mbps @ 12MB buffer (=1Gbps*110ms)  Parallel TCP: 800Mbps @ 64 TCP concurrent flows, with each having 1MB buffer Two concurrent TCP flows, 1 from Chicago to Amsterdam, 1 within Chicago local networks:  2Mps vs. 940Mbps!

5. 02/17/2004 PFLDnet 2004 5 Why TCP Fails Discover/recover slow on high BDP links  Increase 1 byte per RTT Drastic decrease in sending rate Fairness bias on longer RTT links More prone to link error in high BDP links B: throughout in packets per second, p: loss rate

6. 02/17/2004 PFLDnet 2004 6 Requirements to the New Protocol FAST  High utilization of the abundant bandwidth either with single or multiplexed connections FAIR  Intra-protocol fairness, independent of RTT FRIENDLY  TCP compatibility

7. 02/17/2004 PFLDnet 2004 7 Use Scenarios Small number of sources shares abundant bandwidth Bulk data transfer  Most of the packets can be packed in maximum segment size (MSS) in a UDT session  MSS can be set up by applications and the optimal value is the path MTU

8. 02/17/2004 PFLDnet 2004 8 UDT History 2000: SABUL Concept 2001: SABUL version 1.0 2002: dSABUL 2002: SABUL version 2.0, 2.1, 2.2, 2.3, 2003: UDT 1.0 2004: UDT 1.1 & 1.2 2004: UDT 2.0 2005: UDT 3.0

9. 02/17/2004 PFLDnet 2004 9 Papers Supporting Configurable Congestion Control in Data Transport Services Yunhong Gu and Robert L. Grossman UIC/LAC Technical Report, in submission Experiences in Design and Implementation of a High Performance Transport Protocol Yunhong Gu, Xinwei Hong, and Robert L. Grossman SC 2004, Nov 6 - 12, Pittsburgh, PA, USA. An Analysis of AIMD Algorithms with Decreasing Increases Yunhong Gu, Xinwei Hong and Robert L. Grossman First Workshop on Networks for Grid Applications (Gridnets 2004), Oct. 29, San Jose, CA, USA. Optimizing UDP-based Protocol Implementation Yunhong Gu and Robert L. Grossman PFLDNet 2005, Lyon, France, Feb. 2005 SABUL: A Transport Protocol for Grid Computing Yunhong Gu and Robert L. Grossman, Journal of Grid Computing, 2003, Volume 1, Issue 4, pp. 377-386

10. 02/17/2004 PFLDnet 2004 10 UDT: UDP based Data Transfer  Reliable, application level, duplex, transport protocol, over UDP with congestion control  Implementation: Open source C++ library Two orthogonal parts  The UDT protocol framework that can be implemented above UDP, with any suitable congestion control algorithms  The UDT congestion control algorithm, which can be implemented in any transport protocols such as TCP What’s UDT?

11. 02/17/2004 PFLDnet 2004 11 Congestion Control Schemes Window control sends data in bursts  TCP pacing (decreases throughput) Rate control  can lead to continuous loss  Rate control + Supportive window control

12. 02/17/2004 PFLDnet 2004 12 Packet Structure Data Packet:  Header: 1bit flag + 31bit sequence number Control Packet:  Header: 1bit flag + 3bit type + 12bit reserved + 16bit ACK seq. no. + (0 - 32n)bit control info  Type: ACK, ACK2, NAK, Handshake, Keep-alive, and Shutdown Actual size of a UDT packet can be ascertained from UDP header

13. 02/17/2004 PFLDnet 2004 13 Data Packet 0Packet Sequence Number User Data Payload  Flag Bit: 0  UDT uses 31-bit packet based sequence number, ranging from 0 and (2 31 - 1)  Sequence number may be wrapped if it exceeds the maximum available number

14. 02/17/2004 PFLDnet 2004 14 Control Packet 1typereservedACK Seq. No. Control Information Field  Flag Bit: 1  type: 3-bit  handshake (000), shutdown (101), keep-alive (001)  ACK (010), ACK2 (110), NAK (011)  UDT uses sub-sequencing: each ACK and related ACK2 are assigned a 16-bit unique ACK sequence number

15. 02/17/2004 PFLDnet 2004 15 Acknowledgements Selective acknowledgement (ACK)  Generated at every constant interval to send back largest continuously received sequence number of data packets.  The sender sends back an ACK2 to the receiver for each ACK (sub-sequencing).  Also carries RTT, packet arrival speed, and estimated link capacity. Explicit negative acknowledgement (NAK)  Generated as soon as loss is detected.  Loss information may be resent if receiver has not received the retransmission after an increasing interval.  Loss information is compressed in NAK.

16. 02/17/2004 PFLDnet 2004 16 Timing Packet Scheduling Timer  Tuned by Rate Control  High precision in CPU clock cycles  Implementation depends on self-clocking, packet sending burst Rate Control Timer: trigger rate control  RCTP = 0.01 seconds ACK Timer: trigger acknowledgement  ATP = RCTP

17. 02/17/2004 PFLDnet 2004 17 Timing (cont.) NAK Timer: trigger negative acknowledgement  NTP = RTT Retransmission Timer: trigger retransmission based on time-out and maintain connection status  RTP = (exp-count + 1) * RTT + ATP where exp-count is the number of continuous time-out

18. 02/17/2004 PFLDnet 2004 18 UDT Architecture DATA ACK ACK2 NAK Sender Recver Sender Recver  Pkt. Scheduling Timer  ACK Timer  NAK Timer  Retransmission Timer  Rate Control Timer Sender

19. 02/17/2004 PFLDnet 2004 19 Congestion Control Rate based congestion control (Rate Control)  RC tunes the packet sending period.  RC is triggered periodically at the sender side.  RC period is constant of 0.01 seconds. Window based flow control (Flow Control)  FC limits the number of unacknowledged packets.  FC is triggered on each received ACK at the sender side.

20. 02/17/2004 PFLDnet 2004 20 Rate Control AIMD: Increase parameter is related to link capacity and current sending rate; Decrease factor is 1/9, but not decrease for all loss events. Link capacity is probed by packet pair, which is sampled UDT data packets.  Every 16th data packet and it successor packet are sent back to back to form a packet pair.  The receiver uses a median filter on the interval between the arrival times of each packet pair to estimate link capacity. ……

21. 02/17/2004 PFLDnet 2004 21 Receiver Based Packet Pair (RBPP) OK in high-speed optical links Not working well in certain cases  multi-channel links Underestimation  UDT become conservative Overestimation  UDT become more aggressive.  Finally, UDT turns into window-based cong. control V. Paxson, End-to-End Internet Packet Dynamics, IEEE/ACM Transactions on Networking, Vol.7, No.3, pp. 277-292, June 1999.

22. 02/17/2004 PFLDnet 2004 22 Rate Control (cont.) 1. If loss rate is greater than 1%, do not increase; 2. Number of packets to be increased in next RCTP time is: where B is estimated link capacity, C is current sending rate. Both are in packets or packets per second. MSS is the packet size in bytes. β = 1.5 * 10 -6. 3. Recalculate packet sending period (STP). An Analysis of AIMD Algorithms with Decreasing Increases Yunhong Gu, Xinwei Hong and Robert L. Grossman First Workshop on Networks for Grid Applications (Gridnets 2004), Oct. 29, San Jose, CA, USA.

23. 02/17/2004 PFLDnet 2004 23 Rate Control (cont.) C (Mbps)B - C (Mbps)Increase Param. (Pkts) [0, 9000)(1000, 10000]10 [9000, 9900)(100, 1000]1 [9900, 9990)(10, 100]0.1 [9990, 9999)(1, 10]0.01 [9999, 9999.9)(0.1, 1]0.001 9999.9+<0.10.00067 B = 10Gbps, MSS = 1500 bytes

24. 02/17/2004 PFLDnet 2004 24 UDT Algorithm

25. 02/17/2004 PFLDnet 2004 25 Rate Control (cont.) Decrease sending rate by 1/9, (or equivalently, increase packet sending period by 1.125), only if 1. Received an NAK, whose last lost sequence number is greater than the largest sequence number when last decrease occurred; or 2. The number of loss events since last decrease has exceeded a threshold, which increases exponentially and is reset when condition 1 is satisfied. No data will be sent out for the next RCTP time if a decrease occurs.  Help to clear congestion. In a short period, loss rate due to congestion is larger than loss rate due to physical link error

26. 02/17/2004 PFLDnet 2004 26 BDP W = W*0.875 + AS*(RTT+ATP)*0.125 AS is the packets arrival speed at receiver side.  The receiver records the packet arrival intervals. AS is calculated from the average of latest 16 intervals after a median filter.  It is carried back within ACK. Flow Control

27. 02/17/2004 PFLDnet 2004 27 Slow Start Flow window starts at 2 and increases to the number of acknowledged packets, until the sender receives an NAK or reaches the maximum window size, when slow start ends. Packet sending period is 0 during slow start phase and set to the packet arrival interval at the end of the phase. Slow start only occurs at the beginning of a UDT session.

28. 02/17/2004 PFLDnet 2004 28 Implementation: Performance

29. 02/17/2004 PFLDnet 2004 29 Implementation: Intra-protocol Fairness

30. 02/17/2004 PFLDnet 2004 30 Implementation: TCP Friendliness

31. 02/17/2004 PFLDnet 2004 31 Implementation: TCP Friendliness (cont.)

32. 02/17/2004 PFLDnet 2004 32 Implementation: File Transfer ToStarLightCanarieSARA From StarLight460505560 Canarie440502- SARA441-660 CanarieStarLightSARA 1Gbps/15.9ms1Gbps/110ms Disk R: 800Mbps W: 550Mbps Disk R: 800Mbps W: 500Mbps Disk R: 1300Mbps W: 900Mbps

33. 02/17/2004 PFLDnet 2004 33 Simulation: UDT Throughput at Different Bandwidth and RTT

34. 02/17/2004 PFLDnet 2004 34 Simulation: Performance of Concurrent UDT Flows

35. 02/17/2004 PFLDnet 2004 35 Simulation: Intra-protocol Fairness

36. 02/17/2004 PFLDnet 2004 36 Simulation: RTT Independence

37. 02/17/2004 PFLDnet 2004 37 Simulation: TCP Friendliness

38. 02/17/2004 PFLDnet 2004 38 Simulation: Convergence/Stability

39. 02/17/2004 PFLDnet 2004 39 Simulation: Complex Scenario 100  5010    Flow ID123456 Throughput (Mbps) 89.390.05.1841.750.84.78 Link capacity Mbps Flow and its ID Node DropTail

40. 02/17/2004 PFLDnet 2004 40 Simulation: Multi-bottleneck A x 200 B C X0.1110204060 AB198.8189.2180.1170.9152.5137.6 AC0.0980.9799.95519.8839.4657.70 X80100120140160180 AB108.4104.6100.8101.3100.7100.3 AC73.4992.4298.4798.0498.6599.00

41. 02/17/2004 PFLDnet 2004 41 Summary UDT Protocol  Application level upon UDP  Selective acknowledgement / explicit negative acknowledgement UDT Congestion Control  Rate Control Bandwidth estimation for fast probing available bandwidth and fast recovery AIMD for fairness Constant rate control interval  Flow Control Dynamic flow window according to packet receiving speed

42. 02/17/2004 PFLDnet 2004 42 UDT Characters Good use of available bandwidth Application level - no changes in router and operating system No manual tuning Fair and Friendly: intra-protocol fairness, TCP friendliness, and RTT independence. Open source

43. Thank You! LAC: www.lac.uic.eduwww.lac.uic.edu UDT: sourceforge.net/projects/dataspacesourceforge.net/projects/dataspace Internet Draft: draft-gg-udt-01.txt

44. Q&A

45. 02/17/2004 PFLDnet 2004 45 Comparison: mostly “Related Works” x: packet sending rate

46. 02/17/2004 PFLDnet 2004 46 SYS interval: 10 ms Acceptable trade-off between efficiency and fairness  Intra-protocol and TCP Larger values  Less responsive to network change  Slower in loss recovery  More stable  Friendlier to TCP Smaller: the opposite

47. 02/17/2004 PFLDnet 2004 47 Target environment Large BDP networks, Bulk data transfer Speed of the control flow: ~4.5 kBps Who use it?  Using Teraflows to Transport Sloan Digital Sky Survey (SDSS) Data http://sdss.ncdm.uic.edu:8080/sdss/  e-VLBI compares Tsunami, UDT, FAST TCP http://apan.net/meetings/honolulu2004/materials/engineering/8-APAN2.ppt