1. Measurement-Based Performance Profiles and Dynamics of UDT Over Dedicated Connections
Qiang Liu and Nagi Rao
Oak Ridge National Laboratory
Chase Wu and Daqing Yun
New Jersey Institute of Technology
Raj Kettimuthu and Ian Foster
Argonne National Laboratory
International Conference on Network Protocols
November 8-11, 2016, Singapore
Sponsored by
U.S. Department of Energy

2. Outline Introduction UDT Protocol Review UDT Measurements
Throughput Profile
Time Traces
Simple Transport Model
Poincare and Lyapunov Analysis
Summary

3. Dedicated Network Connections
Increasing deployments and availability
DOE OSCARS provides ESnet WAN connections
Google B4 SDN dedicated networks
Provide desirable features
Dedicated capacity - no competing traffic
Low loss rates – no externally induced losses from “other” traffic
Expectations for transport methods
Simple and predictable flow dynamics
Peak throughput should be easier to achieve using “simple” flow control
Easier to tune parameters due to predictable, simple dynamics
Surprisingly, our measurements show much more complex dynamics

4. Transport Protocols TCP UDP
Provides congestion control to share network with other flows
Widely deployed, including over dedicated connections
CUBIC, Hamilton, HighSpeed, Reno, Scalable, Laminar, BBR, and others
UDP
Dedicated connections with no other traffic – simple flow control
Loss recovery and flow control at application level for transport
Open/open-source: UDT, SABUL, Hurricane, Tsunami, PCC, others
Commercial: ASPERA
In this paper: UDT – UDP-Based Data Transfer
Gu and Grossman (2004)
Implements flow control and loss recovery over UDP
Application-level implementation – suited for dedicated connections
Expected dynamics: smoothly reach peak throughput AND stabilize

5. Our Overall Contributions
Experimental study of UDT throughput and dynamics
Collected extensive throughput measurements
Computed throughput profiles with respect to RTT
Applied Poincare-map methods to analyze dynamics
Reveal more complex dynamics than previously known/expected
Simple Generic Transport Model
Abstracts ramp-up and sustainment phases
Explains overall qualitative behavior – applied to UDT
monotonicity – throughput decrease with RTT
concavity – highly desirable throughput profile property
connection between stability and throughput: stability implies concavity
In summary, shows that stability is a highly desirable protocol property

6. UDT: Gu and Grossman (2004) Basic Transport Mechanism:
Rate control: Decreasing additive increase, multiplicative decrease (DAIMD) method
Flow control: maintain a number of unacknowledged packets within a window
Expected performance
as indicated by protocol
Ramp-up followed by stable
throughput performance
Flat throughput profile
Our experiments show
vastly rich dynamics
Proposed by Gu and Grossman
Last two major upgrades 2008 (called udt1 in the paper) and 2011 (called udt2)
Udt2 has more aggressive minimum rate increase compared to udt1
Earlier studies showed performance over 1Gbps links
Our experiements

7. UDT Rate Control Gu and Grossman (2007)
Positive feedback
unit: packets/second
L: link capacity
S: Packet size (e.g., 1,500 as default)
SYN: fixed control interval (e.g., 10ms)
C = x * S * 8 – throughput
Ʈ = 9 – protocol parameter
As x , C(x) , α(x) “D-AI”
Negative feedback
With loss, the decrease factor is a constant β = 1/9 – “MD”
Step-wise increase once every SYN = 10ms

8. UDT Flow Control Gu and Grossman (2007)
UDT rate control “accounts” for RTT using SYN period
Window control
Limits the number of unacknowledged packets
Done at the receiver side
Window size is sent back in acknowledgments
Window size is updated every SYN time as
w: unit - packet/sec
AS: packet arrival rate since last SYN (or last w update)
λ: factor for the moving average
Flow control is used to limit the total number of unacknowledged packets
Implementation details can be found in the udt source file and documentation
Note the (estimated) RTT appears in the window size update equation

9. UDT Stability Gu and Grossman (2007)
UDT reaches asymptotic stable throughput
proceeds in stages
stabilizes at flow rate
loss rate: p
minimum step size to achieve stability is
where
e is integer satisfying
Overall, UDT should reach throughput and stabilize there.
We collect various measurements to gain insights into UDT dynamics
Flow control is used to limit the total number of unacknowledged packets
Implementation details can be found in the udt source file and documentation
Note the (estimated) RTT appears in the window size update equation

10. ORNL Network Testbed (since 2004)
10GE-OC192
ANUE
OC192
e300
10GE-OC192
emulated connections: RTT: ms: 9.6 Gbps
HP 32/48-core
2/4-socket
6.8/7.2 Linux
HP 32/48-core
2/4-socket
6.8/7.2 Linux
ANUE
10GigE
10GigE emulated connections: rtt: ms
Cisco
Nexus 7k
Ciena
CN4500
Ciena
CN4500
Cisco
Nexus 7k
ORNL-ATL-ORNL connection : rtt: 11.6 ms: 100Gbps
RTTs used in measurements:
cross-country (0-100ms)
across continents ( ms)
across globe (366ms)
host
host
dedicated
connection
rtt: 0-366ms
10/9.6 Gbps
UDT
client
UDT
server

11. HP Proliant 48-core Linux workstation: 2.23GHz AMD Opteron processors
NICs
Four sockets
8 Dies
48 cores
Packets traverse complex data and execution paths

12. UDT Measurements: Memory-to-Memory Throughput
throughput profile: function of RTT
throughput trace: time-series
varies with RTT
varies with time
This is with the latest UDT build and jumbogram of 9KB
More results can be found in the paper
Measured characteristics:
profile: overall decrease with RTT
time-trace:
significant variations: same RTT
repeated – significant drops
Analytical model expects:
flat profile: independent of RTT
trace: smooth rise and constant

13. UDT Time Traces 10GigE – No loss
Select time traces
Low RTT: Fast ramp-up followed by drops in sustainment phase (often does not recover to level prior to the drop)
High RTT: relative slow ramp-up followed by more steady sustainment phase
However, no loss as indicated by zero NACKs from traces
Large variations across repeated measurements during sustainment
Sustainment relatively stable but variable ramp-up time

14. UDT Time Traces SONET – With loss (shown with NACKs)
For SONET, rather significant number of losses in low RTT regime
Frequent sharp drop coincides with high NACKs
Throughput rate can be highly oscillatory
Certain low RTT traces demonstrate far fewer losses, but as RTT increases, definite reduction in losses
On the left: 11.6ms; right, 22.6ms
Significant (but inconsistent) loss at lower RTT
Losses disappear with increasing RTT

15. UDT Throughput Profiles
Datagram size
1.5k bytes (default)
9k jumbogram
UDT2 jumbogram
UDT1: more convex and overall lower throughput at higher RTT
UDT2: 9k jumbogram: more concave profile (w/ larger variations)
configuration
Hosts
Connection
UDT build
Datagram size
UDT1-10GigE
48-core, 4 socket
10GigE
2008 9k
UDT2-10GigE
2011
UDT1-SONET
32-core, 2 sockets
OC192
UDT2-SONET
UDT2-10GigE-1500
1.5k
UDT2-SONET-1500

16. TCP Profiles: memory to memory transfer
10Gbps dedicated connections: bohr03 and bohr04
CUBIC congestion control module- default under Linux
TCP buffers tuned for 200ms rtt: 1-10 parallel streams
highly-desirable concave region
8 streams
single
stream
country
globe
RTT: cross-country (0-100ms), cross-continents ( ms), across globe(366ms)

17. Desired Features of Concave Region
Concave regions is very desirable
throughput does not decay as fast
throughput is above linear interpolation
Function is concave iff derivative is non-increasing
not satisfied by Mathis model:
Measurements: throughput profiles for rtt: 0-366ms
Concavity: small rtt region
Only some TCP versions:
CUBIC,
Hamilton TCP
Scalable TCP
Not for some TCP versions:
Reno
These are loadable linux modules

18. UDT Throughput Profiles
SONET: monotonic increase before 60ms, then decrease; concave to convex transition point: 70-80ms
10GigE: Relatively stable mean throughput (but with large variations) through 80ms, then decreases; concave to convex transition point: ms
SONET vs. 10GigE
Overall: concave followed by convex but different spans

19. Generic Transport Model
Ramp-up Phase
throughput increases from initial value to around capacity e.g. TCP slow start, UDT ramp-up
time trace of throughput during ramp-up
Sustainment Phase
Throughput is maintained around a peak value
TCP congestion avoidance, UDT stable peak
time trace of throughput during sustainment
peak
connection capacity
buffer limit
throughput
ramp-up
sustainment
time t

20. Boundary Case: Average Throughput: Monotonicity link capacity reached
sustainment
ramp up
peak is reached and sustained
time
Average Throughput: Monotonicity
increases with
decreases with

21. Monotonicity Conditions
Average Throughput: Not always decreasing in
Effective sustained phase:
monotonically decreases in
Ineffective sustained phase:
may lead to:
lower throughput
convex region
may increase in
if decreases “faster” than
Some UDT measurements show this behavior

22. Concavity: Exponential ramp-up and Sustained throughput
Faster than Slow Start: Illustrative case
More increases than slow start:
data sent:
Average Throughput:
decreasing function of
implies concavity of

23. Poincare Map
Well-Known tool for analyzing time series – used in chaos theory
Poincare map
time series:
generated as
UDT Poincare map provides important qualitative information:
range specifies achievable throughput
stabilization
at 10Gpbs
Ideal Poincare map from the analytic equations
Actual poincare map reflects the rich dynamics seen in time traces
Ideal: simulated using model
- smooth monotone
Estimated from measurements
Indicative of complex dynamics

24. Lyapunov Exponent: Stability and Concavity
Log derivative of Poincare map
Provides critical insights into dynamics
Stable trajectories:
Chaotic trajectories:
indicate exponentially diverging trajectories
with small state variations
larger exponents indicate large deviations
protocols are operating at peak at RTT
stability implies average close to peak - implies concavity
positive exponents imply lowered throughput – trajectories can only go down
then, weak sustainment implies convexity
Figure to top right shows the correlation between max L_m and average send rate
Can be shown analytically (as in Sec. IV-C)

25. Throughput vs. Lyapunov Exponent
We now look at another important factor that contributes to the rich dynamics and complex profiles of udt – rtt estimates
For sonet, most estimates are close to the actual value; but in 10GigE, the rtt estimates are much higher (for low rtt regimes)  correspond to the drops in throughput time traces
Rtt estimate is obtained with a two-way acknowledgment mechanism whose performance worsens when send rate is high
On the other hand, a different physical connection mode for sonet/oc192 results in severe losses when send rate is high, which combined with multiplicative decrease, result in rapid oscillations in traces
overall throughput decrease with larger Lyapunov exponent

26. Instability leads to lower throughput profiles
For protocols operating at peak throughput - instability implies lower average throughput
Informally, unstable trajectory at peak can only “go down” to lower throughput
throughput traces of N repeated executions
trajectory envelope – ideally, constant at connection capacity C
For protocols operating at peak,
envelop is invariant with RTT
where profile is estimated using collected at different RTTs
For protocols operating at peak, instability implies several below
implies lower its time integral
in turn, implies lower
For fixed using
shows that larger Lyapunov exponent leads to lower throughput profiles
Figure to top right shows the correlation between max L_m and average send rate
Can be shown analytically (as in Sec. IV-C)

27. Instability also shrinks concave region
Two protocols and with Lyapunov exponents and
Consider
Trajectories of deviate faster than those of both operating at peak
which implies
For fixed
we have
since , concavity of is equivalent to condition
for a fixed configuration, the condition leads to
which implies has larger concave region
In general, stable throughput dynamics are highly desirable for achieving
(a) peak throughput, and (b) concave throughput profiles
Figure to top right shows the correlation between max L_m and average send rate
Can be shown analytically (as in Sec. IV-C)

28. RTT Estimates SONET: 22.6ms 10GigE: 22.6ms 22.6ms
At lower RTT, rtt estimates are dominated by complex host packet dynamics – inter-packet times have higher variations in 10GigE
10GigE: 366ms
10GigE: 91.6ms
366ms
We now look at another important factor that contributes to the rich dynamics and complex profiles of udt – rtt estimates
For sonet, most estimates are close to the actual value; but in 10GigE, the rtt estimates are much higher (for low rtt regimes)  correspond to the drops in throughput time traces
Rtt estimate is obtained with a two-way acknowledgment mechanism whose performance worsens when send rate is high
On the other hand, a different physical connection mode for sonet/oc192 results in severe losses when send rate is high, which combined with multiplicative decrease, result in rapid oscillations in traces
91ms
At higher RTT, rate rtt estimates are accurate and stable
Need further investigation: host dynamics dominate at lower RTT?

29. Summary Contributions Future directions
Extensive UDT memory-to-memory measurements over dedicated physical and emulated links
New insights: Rich dynamics, monotonicity and concavity profiles
Simple transport model to explain qualitative properties of profiles
Poincare map and Lyapunov exponents analysis: connect rich dynamics with overall qualitative profiles
Overall: Establishes that stability is highly desirable for protocols operating at peak: (a) higher throughput, and (b) concave profile
Future directions
Detailed study and analysis of UDT:
RTT estimation and smoothening
host packet dynamics and complex dynamics
randomness and stochastic approximation for stabilization
More detailed quantitative analytical models to fit measurements
Analysis of performance bounds on simple transport model parameters
Automatic parameter optimization of protocols based on dynamics

30. Thank you!
Questions?

31. UDT Stability Proof: Gu and Grossman (2007)
UDT DAIMD algorithm
Probability that I packets are lost during (t,t+1)
where small p(t)
Flow equations
Utility function: implies stability
Flow control is used to limit the total number of unacknowledged packets
Implementation details can be found in the udt source file and documentation
Note the (estimated) RTT appears in the window size update equation
positive and non-decreasing in x
concave and strictly decreasing in x