1. Turning Heterogeneity into an Advantage in Overlay Routing
Zhichen Xu, Mallik Mahalingam, Magnus Karlsson
Internet Systems and Storage Lab
Hewlett-Packard Company
INFOCOM 2003

2. IEEE INFOCOM 2003 Zhichen Xu
9/20/2018
Motivation
Distributed hash table (DHT) based overlay networks provide a simple abstraction that maps “keys” to “values”
They are scalable, fault-tolerant, self-organizing and have low maintenance cost
They can be used in many important applications,
E.g., distributed storage, DNS, media streaming, web caching, content-based searching, distributed firewalls, etc.
As a result, these applications can benefit from the above properties
Several proposals: Pastry, Tapestry, Chord, CAN, eCAN, SkipNet ...
Provide a homogeneous abstraction to the applications, but vary in their logical structures and flexibility
Routing is logical and at the application level
9/20/2018
IEEE INFOCOM Zhichen Xu
HP template

3. Each logical hop can correspond to multiple physical hops1 1 2 3 2 3
It is important that the structure of the overlay efficiently uses the underlying physical network!
Images downloaded from
9/20/2018
IEEE INFOCOM Zhichen Xu

4. IEEE INFOCOM 2003 Zhichen Xu
9/20/2018
Related work
Within the overlay [Castro et al]
Proximity routing, e.g., Chord [Stoica et al]
Choices limited
Geographic layout, e.g., Topologically-aware CAN [Ratnasamy et al]
uneven distribution of the nodes and
chance of overloading nodes
Proximity-neighbor selection, e.g., Pastry [Rowstron et al], eCAN [Xu and Zhang]
Routing table entries selected according to proximity metric among nodes that satisfy the constraint
Performance constrained by the logical structure of the default overlay!
Auxiliary networks, e.g. Brocade
Constructing a secondary overlay network, however current proposal
Still use logical routing in the secondary network
Pushes the problem to an auxiliary network of a smaller size
Dilemma in picking the size of the secondary network
Still logical routing!
9/20/2018
IEEE INFOCOM Zhichen Xu
HP template

5. IEEE INFOCOM 2003 Zhichen Xu
9/20/2018
Our contributions
Decouple the homogeneous overlay abstraction from routing
Constructing unconstrained auxiliary routing network using
AS-level topology derived from BGP reports
Landmark-numbering scheme
Route advertisement using a distance vector algorithm with route summarization to reduce state
Work with all currently existing overlays
Simulation results show that our approach can achieve close to optimal routing performance
1.04 to 1.12 times optimal routing for an Internet-like topology
Previous approaches 2.5 to 5 times optimal for the same topology
9/20/2018
IEEE INFOCOM Zhichen Xu
HP template

6. Expressway definitions2 4
Expressway
High speed connections 1 3
An ordinary node establishes connection with the expressway node that is closest to it 1 2
Nodes with good connectivity and availability elect themselves as expressway nodes 3 4
Default overlay, CAN as an example
9/20/2018
IEEE INFOCOM Zhichen Xu

7. Expressway challenges
How does a node (ordinary or expressway) find the close-by expressway nodes?
How routes are propagated and how do we control the routing state?
What can the expressway be used for?
9/20/2018
IEEE INFOCOM Zhichen Xu

8. IEEE INFOCOM 2003 Zhichen Xu
9/20/2018
Landmark clustering
Landmark3
Landmark space
di: distance to landmark I
<d1, d2, d3>
Landmark1
Landmark vector
Nodes with similar distances to landmarks likely close to each other
Landmark2
Related work
Landmark ordering [Ratnasamy et al 2002]:
Coordinate-based [Ng and Zhang 2001]:
9/20/2018
IEEE INFOCOM Zhichen Xu
HP template

9. Locating close-by expressway node
9/20/2018
Locating close-by expressway node
Landmark3
DHT a a b b c
Landmark1 c
Landmark2
Landmark vector as key to store information of the expressway nodes on the DHT such that distances in the “landmark space” are preserved
A node uses its landmark vector to search the DHT to find close-by nodes
Expressway nodes finds and connects to physically close-by expressway nodes to form the expressway network
9/20/2018
IEEE INFOCOM Zhichen Xu
HP template

10. Dimensionality mismatch problem
9/20/2018
Dimensionality mismatch problem
Landmark3
DHT
Dimension
reduction a a b b c
Landmark1 c
Landmark2
But, the dimensionality of the landmark space and that of the DHT is usually different
9/20/2018
IEEE INFOCOM Zhichen Xu
HP template

11. Space Filling Curves : Hilbert Curve2 3 8 7 1 4 5 6
Points close to each other in n-d space mapped to points close to each other in 1-d space, and vice versa
9/20/2018
IEEE INFOCOM Zhichen Xu

12. IEEE INFOCOM 2003 Zhichen Xu
9/20/2018
Proximity-preserving dimension reduction of landmark vectors : landmark numbering
Landmark space
Default overlay CAN 5 6 2 3 7 8 4 3 1 4 5 6 7 1 2
Landmark number
(a)
(b)
9/20/2018
IEEE INFOCOM Zhichen Xu
HP template

13. Route advertisement with summarization
9/20/2018
Route advertisement with summarization
An expressway node advertises all ordinary nodes that are in its physical proximity to neighboring expressway nodes
Given a destination, an expressway node returns the next hop expressway node on the shortest path
Uses a distance vector algorithm, except
advertise summarization of multiple nodes, and transport address of one representative node
Please read the paper for more detail
only expressway nodes participate in route advertisement
Route advertisement messages are controlled with a time-to-live (TTL) expressed as the number of expressway hops
9/20/2018
IEEE INFOCOM Zhichen Xu
HP template

14. IEEE INFOCOM 2003 Zhichen Xu
Expressway Usages
Direct route:
Requires slightly more storage space to keep the route summary and relies on IP routing
Expressway–node forwarding:
If a node leaves the system, it is less expensive to repair
May deliver routing performance better than default IP routing [RON 2001, Detour 1999]
Nature for multicast
Ordinary nodes cache addresses of nodes associated with the same expressway node
source 1 2 3 4
Direct route
dest
9/20/2018
IEEE INFOCOM Zhichen Xu

15. Experimental evaluation : 2-d eCAN as default overlay
9/20/2018
Experimental evaluation : 2-d eCAN as default overlay
Compare against
eCAN with roughly the same amount of state (50-75% better than basic CAN with similar state)
Logical auxiliary: a Brocade-like system, but perf. >>that Of Brocade
AS topology:
1000 AS from a total of 13,000 active AS
Assume 100 ms inter-AS delay and 10 ms intra-AS delay
Transit-stub graph using GT-ITM:
10,000 nodes, 228 transit domains
100ms for cross transit links, 20 ms for links inside a transit, 5 ms for links connecting a transit and stub node, and 2 ms for links inside a stub
9/20/2018
IEEE INFOCOM Zhichen Xu
HP template

16. eCAN represents state-of-the-art
High-order routing tables are soft-state, therefore it has a lower maintenance cost than that of CAN of a high-dimension
Allows for proximity-neighbor selection
Neighbor selection based on landmark clustering & controlled data placement
1-d eCAN is topology-aware Chord
The notion of “high order” zones allows for controlled server and data placement for locality preservation
9/20/2018
IEEE INFOCOM Zhichen Xu

17. IEEE INFOCOM 2003 Zhichen Xu
9/20/2018
Parameters used
# of nodes: K (4K as default)
Fraction of nodes that are expressway nodes: 1/1-1/64 (1/10 as default)
stretch = routing delay / shortest-path delay
9/20/2018
IEEE INFOCOM Zhichen Xu
HP template

18. Comparison of various approaches
9/20/2018
Comparison of various approaches
AS topology
Transit-stub graph
Our approach: 1.02 to 1.5 times of optimal
Other approaches: 2.5 to 6.6 times of optimal
9/20/2018
IEEE INFOCOM Zhichen Xu
HP template

19. Effect of varying the ratio of expressway nodes in the system
9/20/2018
Effect of varying the ratio of expressway nodes in the system
As the percentage of expressway nodes increases, expressway better approximates the underlying physical network
Whereas a “logical auxiliary” cannot take advantage of this
9/20/2018
IEEE INFOCOM Zhichen Xu
HP template

20. IEEE INFOCOM 2003 Zhichen Xu
9/20/2018
Conclusions
Propose generic techniques to construct an auxiliary routing network for DHT-based overlays
Decouples routing from DHT abstraction to take advantage of the heterogeneity that exists in the system
Achieves routing performance close to optimal
The expressway nodes need to be relatively stable
9/20/2018
IEEE INFOCOM Zhichen Xu
HP template

21. Other projects using a DHT
9/20/2018
Other projects using a DHT
eCAN, a hierarchical version of CAN
Content-based search on DHT [HotNets’02]
pFilter: global data filtering and dissemination [FTDCS’03]
Scalable multicast trees [NOSDAV’03]
Sedar: semantic, deep archival system [FTDCS’03]
9/20/2018
IEEE INFOCOM Zhichen Xu
HP template