1. Aditya Akella Thesis Oral June 22, 2005
End Point-Based Routing Strategies for Improving Internet Performance and Resilience
Aditya Akella
Thesis Oral
June 22, 2005

2. Internet Access Speeds are Improving
>100 Mbps
Few kbps (years ago)
1.5 Mbps
10 Mbps
45 Mbps
>
Well-connected 

3. Higher Access Speed  Better Internet Experience?
Download times
Download times
Internet
1 Tbps
1Gb
1Tb
1Gb
1Tb
100Mb
100Mb
Well-connected 

4. Performance Bottlenecks in the Internet
Constrained network links limit performance
Response times, transfer speeds, resilience
Wide-Area Bottlenecks: little spare capacity; Inside or between ISPs
Verio
ATT
Cogent
MCI
Sprint
Verizon

5. Wide-area Bottlenecks are Prevalent
Quite prevalent, very congested [Akella03]
50% paths have bottlenecks; spare capacity < 10Mbps
No hope for good performance?
Wide-area bottleneck

6. Internet Routing is Rigid
Must enable extra flexibility for end-points
Extra flexibility  More routes, informed choice
Special mechanisms that work over Internet routing
Internet’s rich topology  many alternate paths!
But Internet routing is rigid, not performance-aware
Provides one path per destination
Wide-area bottleneck

7. Central Questions How much extra flexibility do end-points need?
What mechanisms could they employ?
Past research advocates arbitrary flexibility
Internet-wide special-purpose infrastructures
My Thesis: Arbitrary flexibility not necessary
Sufficient to intelligently choose from 2 or 3 routes
No special infrastructure  End-point based

8. Multihoming Route Control
Up to 30% better Internet performance (RTT, throughput, availability)
Performance comparable with enabling arbitrary flexibility
Can realize benefits via simple techniques, e.g., ISP probing 
Verizon
ATT
Cogent 
MCI
Sprint
Verio 
Use multiple ISP connections in a smart manner

9. Dissertation Road-Map
Where do performance problems lie?
What mechanisms help overcome the problems?
(IMC 03)
(SIGCOMM 03, SIGCOMM 04)
Thesis proposal
Practical Multihoming Route Control System
This talk
(SIGCOMM 03, USENIX 04)
Will such systems be effective in the future?
(PODC 03)

10. Outline of the Talk Wide-Area Bottlenecks
Past work on Improving Internet Performance
Benefits Multihoming Route Control
Route Control implementation
Performance Scaling in the Internet
Summary and Open Issues

11. Measuring Wide-Area Bottlenecks
Wide-area bottleneck  where an unconstrained TCP flow sees delays and losses
78 ISPs probed in all
From 26 sources (PlanetLab)
A new TCP-like tool to identify bottlenecks: BFind
tier-3
tier-3
tier-3
tier-3
tier-4
tier-4
tier-2
tier-2
tier-2
tier-3
tier-2
tier-1
tier-4
tier-4
tier-1
tier-3
tier-3
tier-1
tier-2
tier-4
tier-4
tier-4
tier-4
tier-4
tier-2
tier-3
tier-4
tier-4

12. %bottlenecks %all links %bottlenecks %all links
Measurement Results
Intra-ISP links
Inter-ISP links
Found bottlenecks in 900 paths (out of 2028)
~45% of all paths
55% had >40Mbps available capacity
%bottlenecks %all links
%bottlenecks %all links
Tiers 3, 4
25% 9%
Tier 1
15%
51%
Low-latency
38%
57%
Tiers (3,4) – Tiers (3,4)
17% 4%
High-latency
19%
Tier 1 – Tier 1 2% 6%
Available bandwidth at bottlenecks
Tier-1 ISPs bottlenecks  highest available b/w
Tier-2 and tier-3 bottlenecks identical

13. Outline of the Talk Wide-Area Bottlenecks
Past work on Improving Internet Performance
Benefits Multihoming Route Control
Route Control implementation
Performance Scaling in the Internet
Summary and Open Issues

14. Performance-Aware Internet Routing
Intra-domain traffic engineering [Shaikh99]
Inter-domain traffic engineering [Feamster03]
ATT A B C D B A A -
1.0
0.6
0.1 B
0.2 -
0.3
1.0 C C
0.8
0.4 -
0.2 D
Sprint D’ C’ A’ B’ D
0.5
0.7
0.5 -
Estimate traffic, assign routing weights such that no link is over-loaded
Problems:
Limited to 1-2 ISPs; not end-to-end
Coarse time-scales

15. Overlay Routing for Better End-to-End Performance
Overlay network
Significantly improve Internet performance [Savage99, Andersen01]
Compose Internet routes on the fly 
n! route choices;
Very high flexibility
Overlay nodes
Problems:
Poor interaction with ISP policies  Expensive
Is arbitrary flexibility essential?
Download cnn.com over Internet2

16. Outline of the Talk Wide-Area Bottlenecks
Past work on Improving Internet Performance
Benefits Multihoming Route Control
Route Control implementation
Performance Scaling in the Internet
Summary and Open Issues

17. Multihoming Route Control
Multiple ISPs  multiple BGP paths
ISPs differ in connectivity and performance
Also, ISP paths non-overlapping [Akella03, Teixeira03]
Cleverly exploit variation, independence: Multihoming Route Control
Moderately improved routing flexibility; End-only mechanism
Better at night
Better in mornings
Verio
Sprint
ATT
Clever use of multiple ISP routes

18. Multihoming Route Control Challenges
Oracle tells which ISP is best
Potential benefits?
How many ISPs?
Which ISPs?
Verio
Benefits in practice?
Deployment issues?
Pricing, Global effects, routing tables
Sprint
ATT
Later in the talk…

19. Multihoming emulation
Measurement Testbed
Multihoming emulation
Potential Benefits?
How many ISPs?
Which ISPs?
Concurrent measurements over 1 week  offline analysis
Multihoming set up
Control traffic scheduling
Multiple destinations
Many cities
Atlanta 4
Boston 3
Chicago 8
Dallas
Los Angeles 6
New York
San Francisco 10
Seattle
Washington DC
Others 13
Akamai: 68 nodes, 17 US cities
Singly-homed to distinct ISPs
Attached to ISPs of different “sizes”

20. Potential Benefits of Multihoming to k ISPs
Sprint
Verio
k = 2
Tot ISPs in city: 3
Emulated Multihoming
ATT
 Potential benefits?
 How many ISPs?
Which ISPs?
Pick k ISPs from all serving the city
Compute ratio; average across destinations, time
Perf from k ISPs
Perf when using all ISPs
Best set of k ISPs  k-Multihoming
Best when using Sprint & Verio
Best when using Sprint, ATT & Verio
Compare

21. Round-Trip Time (RTT) Improvement
k-Multihoming RTT
30% better
All-Multihoming RTT
NYC: 2-multihoming relative to best ISP
Median RTT
Improves by 8ms
90th percentile RTT
Improves by 40ms
Averaged across destinations
Negligible
Coast-to-coast: 50ms Significant for Web transfers (~10 RTTs)
Up to 30% average improvement relative to single ISP
Diminishing returns beyond 3 ISPs
Throughput (1MB transfers) and availability (pings, traceroutes) benefits are similar (e.g., 20% for throughput)

22. Choosing Your ISPs: A Case Study in SF
 Potential Benefits? ~30% improvement
 How many ISPs? No benefit beyond 3
 Which ISPs? Good combination
 Potential Benefits? ~30% improvement
 How many ISPs? No benefit beyond 3
Which ISPs?
Ranks
Performance
Optimal
Greedy & Smart
1, 2, 5
1.06
Ranks
Performance
Moderate improvement in flexibility helps a lot!
Moderately improved end-only flexibility vs. arbitrary flexibility from Overlay routing? 1
1.38
1, 2, 3
1.14
Naive ?
Ranks
Performance
10, 9, 8
1.25

23. Comparing Against Overlay Routing
“k-Overlay routing”
is strictly better than
k-Multihoming
k.n! paths
k paths
By what extent is Overlay routing better?
Use testbed machines as intermediate overlay nodes

24. k-Overlays vs. k-Multihoming
k-Multihoming RTT
k-Overlay RTT
3-Overlays relative to 3-Multihoming
Across city- destination
pairs
1-Overlays
Median RTT difference
85% are less than 5ms
90th percentile RTT difference
85% are less than 10ms
k-Multihoming
1-Overlays vs 3-Multihoming
Multihoming ~2% better in some cities, identical in others
Multihoming essential to overcome serious first hop ISP problems
3-Overlay routing RTT 6% better on average than 3-Multihoming (Throughput difference less than 3%)

25. Benefits of Multihoming Route Control
Multihoming: compliant with Internet routing
 Good Internet performance achievable with Internet routing
End-to-end performance today: There is still some hope in Internet Routing!

26. Outline of the Talk Wide-Area Bottlenecks
Past work on Improving Internet Performance
Benefits Multihoming Route Control
Route Control implementation
Performance Scaling in the Internet
Summary and Open Issues

27. Ideal Multihoming Three assumptions: Perfect information No overhead
Traffic control
Best at 10:00am
Best at 8:00am
Best at 8:30am
Best at 9:00am
Practical implementation must address these
Best
ISP

28. Keeping Up-to-date Information
Passive probing  use existing transfers
Active probing  send out-of-band probes
How many destinations? Top few by request volume
Current sample good estimate of future ISP performance  no history!
Verio
Sprint
ATT
Regularly monitor
performance
over ISP links

29. Directing Traffic on Optimal Links
Response sent to
Outbound control easy
Inbound  hard!
Internet routing destination address based
Routing table impact?
Use provider-assigned addresses
Verio
Sprint
ATT
/18
Owns: /16
/18
Split net block into 3 parts
/18
CNN.com
x.x.x.x

30. Performance Evaluation
Route control implementation over a Linux-based Web proxy
Trace-based evaluation of a 3-multihomed network using DummyNet
Web performance < 10% away from optimal (i.e., oracle)
About 30% better than single ISP
Sampling every 60s  good response times, resilience
Performance penalty due to probing, NAT < 2%

31. Related Work on Route Control
Commercial route control products for data centers, e.g., Internap, netVMG
Multihomed Overlays for reliable Web access [Andersen05]
Multihomed load balancing across DSL links [Guo04]
Path switching for VoIP applications [Tao04]
Follow-up study [Qiu04]:
Cost
Dedicated links: Not an issue
Usage-based charging: Online algorithms with cost and performance 10% away from optimal offline cases
Global effects: Simulation of equilibria
Many multihomed users  Avg. latency worsens by < 1ms
Other singly-homed users  Negligibly worse latencies
Good news for multihoming!

32. Route Control Contributions
End-only flexibility of Multihoming Route Control: 30% better Internet performance from 3 ISPs
Using 3 ISPs Comparable with overlay routing
NAT, active/passive measurement can help realize benefits in practice
Verizon 
MCI
Cogent
Sprint
ATT
Verio  
Use multiple ISP connections in a smart manner

33. Outline of the Talk Wide-Area Bottlenecks
Past work on Improving Internet Performance
Benefits Multihoming Route Control
Route Control implementation
Performance Scaling in the Internet
Summary and Open Issues

34. Performance Scaling in the Internet
Network growth
Maintains “power-law”
Theorem: The expected maximum load on edges in the ISP-level graph is W(n1.8) (n nodes; unit traffic; routed along shortest paths)
Real-life example: Japan [Fukuda05]
Traffic volumes increase, usage patterns may change
But link capacities will also improve (Moore’s law)
However, routing, structure may impose restrictions
Some portions carry lot more traffic  Speeds need to scale faster
Poor scaling properties?
Routing-based schemes ineffective at ensuring good performance
Straw man approach: alter the structure
Add parallel links between adjacent ISPs
Function of the “degrees” of ISPs
Adding parallel links in proportion to minimum degree achieves linear scaling
Good future performance  Basic changes to Topology and/or Routing

35. Outline of the Talk Wide-Area Bottlenecks
Past work on Improving Internet Performance
Benefits Multihoming Route Control
Route Control implementation
Performance Scaling in the Internet
Summary and Open Issues

36. Summary of Contributions
“An integrated approach to optimizing Internet performance”
In-depth analysis of current performance bottlenecks
Analysis of methods to circumvent bottlenecks
Study of performance scaling in the future Internet
Several important lessons for the design of Internet’s routing protocol and infrastructure!
The most significant contribution, though:
Do end-networks today require special support from protocols or infrastructure? NO

37. Open Issues Longer-term measurement analysis Why diminishing returns?
Bottlenecks: persistence
Multihoming: ISP choice
Why diminishing returns?
A result of Internet’s ISP hierarchy?
Global effect of route control?
Dynamics of interactions, pricing, traffic engineering
Better models for congestion scaling
ISP-level graph may not stay “power-law”? What then?
Analysis of the router-level structure