Measurement-Based Performance Profiles and Dynamics of UDT Over Dedicated Connections Qiang Liu and Nagi Rao Oak Ridge National Laboratory Chase Wu and Daqing Yun New Jersey Institute of Technology Raj Kettimuthu and Ian Foster Argonne National Laboratory International Conference on Network Protocols November 8-11, 2016, Singapore Sponsored by U.S. Department of Energy

Outline Introduction UDT Protocol Review UDT Measurements Throughput Profile Time Traces Simple Transport Model Poincare and Lyapunov Analysis Summary

Dedicated Network Connections Increasing deployments and availability DOE OSCARS provides ESnet WAN connections Google B4 SDN dedicated networks Provide desirable features Dedicated capacity - no competing traffic Low loss rates – no externally induced losses from “other” traffic Expectations for transport methods Simple and predictable flow dynamics Peak throughput should be easier to achieve using “simple” flow control Easier to tune parameters due to predictable, simple dynamics Surprisingly, our measurements show much more complex dynamics

Transport Protocols TCP UDP Provides congestion control to share network with other flows Widely deployed, including over dedicated connections CUBIC, Hamilton, HighSpeed, Reno, Scalable, Laminar, BBR, and others UDP Dedicated connections with no other traffic – simple flow control Loss recovery and flow control at application level for transport Open/open-source: UDT, SABUL, Hurricane, Tsunami, PCC, others Commercial: ASPERA In this paper: UDT – UDP-Based Data Transfer Gu and Grossman (2004) Implements flow control and loss recovery over UDP Application-level implementation – suited for dedicated connections Expected dynamics: smoothly reach peak throughput AND stabilize

Our Overall Contributions Experimental study of UDT throughput and dynamics Collected extensive throughput measurements Computed throughput profiles with respect to RTT Applied Poincare-map methods to analyze dynamics Reveal more complex dynamics than previously known/expected Simple Generic Transport Model Abstracts ramp-up and sustainment phases Explains overall qualitative behavior – applied to UDT monotonicity – throughput decrease with RTT concavity – highly desirable throughput profile property connection between stability and throughput: stability implies concavity In summary, shows that stability is a highly desirable protocol property

UDT: Gu and Grossman (2004) Basic Transport Mechanism: Rate control: Decreasing additive increase, multiplicative decrease (DAIMD) method Flow control: maintain a number of unacknowledged packets within a window Expected performance as indicated by protocol Ramp-up followed by stable throughput performance Flat throughput profile Our experiments show vastly rich dynamics Proposed by Gu and Grossman Last two major upgrades 2008 (called udt1 in the paper) and 2011 (called udt2) Udt2 has more aggressive minimum rate increase compared to udt1 Earlier studies showed performance over 1Gbps links Our experiements

UDT Rate Control Gu and Grossman (2007) Positive feedback unit: packets/second L: link capacity S: Packet size (e.g., 1,500 as default) SYN: fixed control interval (e.g., 10ms) C = x * S * 8 – throughput Ʈ = 9 – protocol parameter As x , C(x) , α(x) - “D-AI” Negative feedback With loss, the decrease factor is a constant β = 1/9 – “MD” Step-wise increase once every SYN = 10ms

UDT Flow Control Gu and Grossman (2007) UDT rate control “accounts” for RTT using SYN period Window control Limits the number of unacknowledged packets Done at the receiver side Window size is sent back in acknowledgments Window size is updated every SYN time as w: unit - packet/sec AS: packet arrival rate since last SYN (or last w update) λ: factor for the moving average Flow control is used to limit the total number of unacknowledged packets Implementation details can be found in the udt source file and documentation Note the (estimated) RTT appears in the window size update equation

UDT Stability Gu and Grossman (2007) UDT reaches asymptotic stable throughput proceeds in stages stabilizes at flow rate loss rate: p minimum step size to achieve stability is where e is integer satisfying Overall, UDT should reach throughput and stabilize there. We collect various measurements to gain insights into UDT dynamics Flow control is used to limit the total number of unacknowledged packets Implementation details can be found in the udt source file and documentation Note the (estimated) RTT appears in the window size update equation

ORNL Network Testbed (since 2004) 10GE-OC192 ANUE OC192 e300 10GE-OC192 emulated connections: RTT: 0-800ms: 9.6 Gbps HP 32/48-core 2/4-socket 6.8/7.2 Linux HP 32/48-core 2/4-socket 6.8/7.2 Linux ANUE 10GigE 10GigE emulated connections: rtt: 0-800ms Cisco Nexus 7k Ciena CN4500 Ciena CN4500 Cisco Nexus 7k ORNL-ATL-ORNL connection : rtt: 11.6 ms: 100Gbps RTTs used in measurements: cross-country (0-100ms) across continents (100-200ms) across globe (366ms) host host dedicated connection rtt: 0-366ms 10/9.6 Gbps UDT client UDT server

HP Proliant 48-core Linux workstation: 2.23GHz AMD Opteron processors NICs Four sockets 8 Dies 48 cores Packets traverse complex data and execution paths

UDT Measurements: Memory-to-Memory Throughput throughput profile: function of RTT throughput trace: time-series varies with RTT varies with time This is with the latest UDT build and jumbogram of 9KB More results can be found in the paper Measured characteristics: profile: overall decrease with RTT time-trace: significant variations: same RTT repeated – significant drops Analytical model expects: flat profile: independent of RTT trace: smooth rise and constant

UDT Time Traces 10GigE – No loss Select time traces Low RTT: Fast ramp-up followed by drops in sustainment phase (often does not recover to level prior to the drop) High RTT: relative slow ramp-up followed by more steady sustainment phase However, no loss as indicated by zero NACKs from traces Large variations across repeated measurements during sustainment Sustainment relatively stable but variable ramp-up time

UDT Time Traces SONET – With loss (shown with NACKs) For SONET, rather significant number of losses in low RTT regime Frequent sharp drop coincides with high NACKs Throughput rate can be highly oscillatory Certain low RTT traces demonstrate far fewer losses, but as RTT increases, definite reduction in losses On the left: 11.6ms; right, 22.6ms Significant (but inconsistent) loss at lower RTT Losses disappear with increasing RTT

UDT Throughput Profiles Datagram size 1.5k bytes (default) 9k jumbogram UDT2 jumbogram UDT1: more convex and overall lower throughput at higher RTT UDT2: 9k jumbogram: more concave profile (w/ larger variations) configuration Hosts Connection UDT build Datagram size UDT1-10GigE 48-core, 4 socket 10GigE 2008 9k UDT2-10GigE 2011 UDT1-SONET 32-core, 2 sockets OC192 UDT2-SONET UDT2-10GigE-1500 1.5k UDT2-SONET-1500

TCP Profiles: memory to memory transfer 10Gbps dedicated connections: bohr03 and bohr04 CUBIC congestion control module- default under Linux TCP buffers tuned for 200ms rtt: 1-10 parallel streams highly-desirable concave region 8 streams single stream country globe RTT: cross-country (0-100ms), cross-continents (100-200ms), across globe(366ms)

Desired Features of Concave Region Concave regions is very desirable throughput does not decay as fast throughput is above linear interpolation Function is concave iff derivative is non-increasing not satisfied by Mathis model: Measurements: throughput profiles for rtt: 0-366ms Concavity: small rtt region Only some TCP versions: CUBIC, Hamilton TCP Scalable TCP Not for some TCP versions: Reno These are loadable linux modules

UDT Throughput Profiles SONET: monotonic increase before 60ms, then decrease; concave to convex transition point: 70-80ms 10GigE: Relatively stable mean throughput (but with large variations) through 80ms, then decreases; concave to convex transition point: 100-110ms SONET vs. 10GigE Overall: concave followed by convex but different spans

Generic Transport Model Ramp-up Phase throughput increases from initial value to around capacity e.g. TCP slow start, UDT ramp-up time trace of throughput during ramp-up Sustainment Phase Throughput is maintained around a peak value TCP congestion avoidance, UDT stable peak time trace of throughput during sustainment peak connection capacity buffer limit throughput ramp-up sustainment time t

Boundary Case: Average Throughput: Monotonicity link capacity reached sustainment ramp up peak is reached and sustained time Average Throughput: Monotonicity increases with decreases with

Monotonicity Conditions Average Throughput: Not always decreasing in Effective sustained phase: monotonically decreases in Ineffective sustained phase: may lead to: lower throughput convex region may increase in if decreases “faster” than Some UDT measurements show this behavior

Concavity: Exponential ramp-up and Sustained throughput Faster than Slow Start: Illustrative case More increases than slow start: data sent: Average Throughput: decreasing function of implies concavity of

Poincare Map Well-Known tool for analyzing time series – used in chaos theory Poincare map time series: generated as UDT Poincare map provides important qualitative information: range specifies achievable throughput stabilization at 10Gpbs Ideal Poincare map from the analytic equations Actual poincare map reflects the rich dynamics seen in time traces Ideal: simulated using model - smooth monotone Estimated from measurements Indicative of complex dynamics

Lyapunov Exponent: Stability and Concavity Log derivative of Poincare map Provides critical insights into dynamics Stable trajectories: Chaotic trajectories: indicate exponentially diverging trajectories with small state variations larger exponents indicate large deviations protocols are operating at peak at RTT stability implies average close to peak - implies concavity positive exponents imply lowered throughput – trajectories can only go down then, weak sustainment implies convexity Figure to top right shows the correlation between max L_m and average send rate Can be shown analytically (as in Sec. IV-C)

Throughput vs. Lyapunov Exponent We now look at another important factor that contributes to the rich dynamics and complex profiles of udt – rtt estimates For sonet, most estimates are close to the actual value; but in 10GigE, the rtt estimates are much higher (for low rtt regimes)  correspond to the drops in throughput time traces Rtt estimate is obtained with a two-way acknowledgment mechanism whose performance worsens when send rate is high On the other hand, a different physical connection mode for sonet/oc192 results in severe losses when send rate is high, which combined with multiplicative decrease, result in rapid oscillations in traces overall throughput decrease with larger Lyapunov exponent

Instability leads to lower throughput profiles For protocols operating at peak throughput - instability implies lower average throughput Informally, unstable trajectory at peak can only “go down” to lower throughput throughput traces of N repeated executions trajectory envelope – ideally, constant at connection capacity C For protocols operating at peak, envelop is invariant with RTT where profile is estimated using collected at different RTTs For protocols operating at peak, instability implies several below implies lower its time integral in turn, implies lower For fixed using shows that larger Lyapunov exponent leads to lower throughput profiles Figure to top right shows the correlation between max L_m and average send rate Can be shown analytically (as in Sec. IV-C)

Instability also shrinks concave region Two protocols and with Lyapunov exponents and Consider Trajectories of deviate faster than those of both operating at peak which implies For fixed we have since , concavity of is equivalent to condition for a fixed configuration, the condition leads to which implies has larger concave region In general, stable throughput dynamics are highly desirable for achieving (a) peak throughput, and (b) concave throughput profiles Figure to top right shows the correlation between max L_m and average send rate Can be shown analytically (as in Sec. IV-C)

RTT Estimates SONET: 22.6ms 10GigE: 22.6ms 22.6ms At lower RTT, rtt estimates are dominated by complex host packet dynamics – inter-packet times have higher variations in 10GigE 10GigE: 366ms 10GigE: 91.6ms 366ms We now look at another important factor that contributes to the rich dynamics and complex profiles of udt – rtt estimates For sonet, most estimates are close to the actual value; but in 10GigE, the rtt estimates are much higher (for low rtt regimes)  correspond to the drops in throughput time traces Rtt estimate is obtained with a two-way acknowledgment mechanism whose performance worsens when send rate is high On the other hand, a different physical connection mode for sonet/oc192 results in severe losses when send rate is high, which combined with multiplicative decrease, result in rapid oscillations in traces 91ms At higher RTT, rate rtt estimates are accurate and stable Need further investigation: host dynamics dominate at lower RTT?

Summary Contributions Future directions Extensive UDT memory-to-memory measurements over dedicated physical and emulated links New insights: Rich dynamics, monotonicity and concavity profiles Simple transport model to explain qualitative properties of profiles Poincare map and Lyapunov exponents analysis: connect rich dynamics with overall qualitative profiles Overall: Establishes that stability is highly desirable for protocols operating at peak: (a) higher throughput, and (b) concave profile Future directions Detailed study and analysis of UDT: RTT estimation and smoothening host packet dynamics and complex dynamics randomness and stochastic approximation for stabilization More detailed quantitative analytical models to fit measurements Analysis of performance bounds on simple transport model parameters Automatic parameter optimization of protocols based on dynamics

Thank you! Questions?

UDT Stability Proof: Gu and Grossman (2007) UDT DAIMD algorithm Probability that I packets are lost during (t,t+1) where small p(t) Flow equations Utility function: implies stability Flow control is used to limit the total number of unacknowledged packets Implementation details can be found in the udt source file and documentation Note the (estimated) RTT appears in the window size update equation positive and non-decreasing in x concave and strictly decreasing in x