Turning Heterogeneity into an Advantage in Overlay Routing Zhichen Xu, Mallik Mahalingam, Magnus Karlsson Internet Systems and Storage Lab Hewlett-Packard Company INFOCOM 2003

IEEE INFOCOM 2003 Zhichen Xu 9/20/2018 Motivation Distributed hash table (DHT) based overlay networks provide a simple abstraction that maps “keys” to “values” They are scalable, fault-tolerant, self-organizing and have low maintenance cost They can be used in many important applications, E.g., distributed storage, DNS, media streaming, web caching, content-based searching, distributed firewalls, etc. As a result, these applications can benefit from the above properties Several proposals: Pastry, Tapestry, Chord, CAN, eCAN, SkipNet ... Provide a homogeneous abstraction to the applications, but vary in their logical structures and flexibility Routing is logical and at the application level 9/20/2018 IEEE INFOCOM 2003 Zhichen Xu HP template

Each logical hop can correspond to multiple physical hops 1 1 2 3 2 3 It is important that the structure of the overlay efficiently uses the underlying physical network! Images downloaded from http://www.mapresources.com/photoshop_maps/ 9/20/2018 IEEE INFOCOM 2003 Zhichen Xu

IEEE INFOCOM 2003 Zhichen Xu 9/20/2018 Related work Within the overlay [Castro et al] Proximity routing, e.g., Chord [Stoica et al] Choices limited Geographic layout, e.g., Topologically-aware CAN [Ratnasamy et al] uneven distribution of the nodes and chance of overloading nodes Proximity-neighbor selection, e.g., Pastry [Rowstron et al], eCAN [Xu and Zhang] Routing table entries selected according to proximity metric among nodes that satisfy the constraint Performance constrained by the logical structure of the default overlay! Auxiliary networks, e.g. Brocade Constructing a secondary overlay network, however current proposal Still use logical routing in the secondary network Pushes the problem to an auxiliary network of a smaller size Dilemma in picking the size of the secondary network Still logical routing! 9/20/2018 IEEE INFOCOM 2003 Zhichen Xu HP template

IEEE INFOCOM 2003 Zhichen Xu 9/20/2018 Our contributions Decouple the homogeneous overlay abstraction from routing Constructing unconstrained auxiliary routing network using AS-level topology derived from BGP reports Landmark-numbering scheme Route advertisement using a distance vector algorithm with route summarization to reduce state Work with all currently existing overlays Simulation results show that our approach can achieve close to optimal routing performance 1.04 to 1.12 times optimal routing for an Internet-like topology Previous approaches 2.5 to 5 times optimal for the same topology 9/20/2018 IEEE INFOCOM 2003 Zhichen Xu HP template

Expressway definitions 2 4 Expressway High speed connections 1 3 An ordinary node establishes connection with the expressway node that is closest to it 1 2 Nodes with good connectivity and availability elect themselves as expressway nodes 3 4 Default overlay, CAN as an example 9/20/2018 IEEE INFOCOM 2003 Zhichen Xu

Expressway challenges How does a node (ordinary or expressway) find the close-by expressway nodes? How routes are propagated and how do we control the routing state? What can the expressway be used for? 9/20/2018 IEEE INFOCOM 2003 Zhichen Xu

IEEE INFOCOM 2003 Zhichen Xu 9/20/2018 Landmark clustering Landmark3 Landmark space di: distance to landmark I <d1, d2, d3> Landmark1 Landmark vector Nodes with similar distances to landmarks likely close to each other Landmark2 Related work Landmark ordering [Ratnasamy et al 2002]: Coordinate-based [Ng and Zhang 2001]: 9/20/2018 IEEE INFOCOM 2003 Zhichen Xu HP template

Locating close-by expressway node 9/20/2018 Locating close-by expressway node Landmark3 DHT a a b b c Landmark1 c Landmark2 Landmark vector as key to store information of the expressway nodes on the DHT such that distances in the “landmark space” are preserved A node uses its landmark vector to search the DHT to find close-by nodes Expressway nodes finds and connects to physically close-by expressway nodes to form the expressway network 9/20/2018 IEEE INFOCOM 2003 Zhichen Xu HP template

Dimensionality mismatch problem 9/20/2018 Dimensionality mismatch problem Landmark3 DHT Dimension reduction a a b b c Landmark1 c Landmark2 But, the dimensionality of the landmark space and that of the DHT is usually different 9/20/2018 IEEE INFOCOM 2003 Zhichen Xu HP template

Space Filling Curves : Hilbert Curve 2 3 8 7 1 4 5 6 Points close to each other in n-d space mapped to points close to each other in 1-d space, and vice versa 9/20/2018 IEEE INFOCOM 2003 Zhichen Xu

IEEE INFOCOM 2003 Zhichen Xu 9/20/2018 Proximity-preserving dimension reduction of landmark vectors : landmark numbering Landmark space Default overlay CAN 5 6 2 3 7 8 4 3 1 4 5 6 7 1 2 Landmark number (a) (b) 9/20/2018 IEEE INFOCOM 2003 Zhichen Xu HP template

Route advertisement with summarization 9/20/2018 Route advertisement with summarization An expressway node advertises all ordinary nodes that are in its physical proximity to neighboring expressway nodes Given a destination, an expressway node returns the next hop expressway node on the shortest path Uses a distance vector algorithm, except advertise summarization of multiple nodes, and transport address of one representative node Please read the paper for more detail only expressway nodes participate in route advertisement Route advertisement messages are controlled with a time-to-live (TTL) expressed as the number of expressway hops 9/20/2018 IEEE INFOCOM 2003 Zhichen Xu HP template

IEEE INFOCOM 2003 Zhichen Xu Expressway Usages Direct route: Requires slightly more storage space to keep the route summary and relies on IP routing Expressway–node forwarding: If a node leaves the system, it is less expensive to repair May deliver routing performance better than default IP routing [RON 2001, Detour 1999] Nature for multicast Ordinary nodes cache addresses of nodes associated with the same expressway node source 1 2 3 4 Direct route dest 9/20/2018 IEEE INFOCOM 2003 Zhichen Xu

Experimental evaluation : 2-d eCAN as default overlay 9/20/2018 Experimental evaluation : 2-d eCAN as default overlay Compare against eCAN with roughly the same amount of state (50-75% better than basic CAN with similar state) Logical auxiliary: a Brocade-like system, but perf. >>that Of Brocade AS topology: 1000 AS from a total of 13,000 active AS Assume 100 ms inter-AS delay and 10 ms intra-AS delay Transit-stub graph using GT-ITM: 10,000 nodes, 228 transit domains 100ms for cross transit links, 20 ms for links inside a transit, 5 ms for links connecting a transit and stub node, and 2 ms for links inside a stub 9/20/2018 IEEE INFOCOM 2003 Zhichen Xu HP template

eCAN represents state-of-the-art High-order routing tables are soft-state, therefore it has a lower maintenance cost than that of CAN of a high-dimension Allows for proximity-neighbor selection Neighbor selection based on landmark clustering & controlled data placement 1-d eCAN is topology-aware Chord The notion of “high order” zones allows for controlled server and data placement for locality preservation 9/20/2018 IEEE INFOCOM 2003 Zhichen Xu

IEEE INFOCOM 2003 Zhichen Xu 9/20/2018 Parameters used # of nodes: 512-8K (4K as default) Fraction of nodes that are expressway nodes: 1/1-1/64 (1/10 as default) stretch = routing delay / shortest-path delay 9/20/2018 IEEE INFOCOM 2003 Zhichen Xu HP template

Comparison of various approaches 9/20/2018 Comparison of various approaches AS topology Transit-stub graph Our approach: 1.02 to 1.5 times of optimal Other approaches: 2.5 to 6.6 times of optimal 9/20/2018 IEEE INFOCOM 2003 Zhichen Xu HP template

Effect of varying the ratio of expressway nodes in the system 9/20/2018 Effect of varying the ratio of expressway nodes in the system As the percentage of expressway nodes increases, expressway better approximates the underlying physical network Whereas a “logical auxiliary” cannot take advantage of this 9/20/2018 IEEE INFOCOM 2003 Zhichen Xu HP template

IEEE INFOCOM 2003 Zhichen Xu 9/20/2018 Conclusions Propose generic techniques to construct an auxiliary routing network for DHT-based overlays Decouples routing from DHT abstraction to take advantage of the heterogeneity that exists in the system Achieves routing performance close to optimal The expressway nodes need to be relatively stable 9/20/2018 IEEE INFOCOM 2003 Zhichen Xu HP template

Other projects using a DHT 9/20/2018 Other projects using a DHT eCAN, a hierarchical version of CAN Content-based search on DHT [HotNets’02] pFilter: global data filtering and dissemination [FTDCS’03] Scalable multicast trees [NOSDAV’03] Sedar: semantic, deep archival system [FTDCS’03] 9/20/2018 IEEE INFOCOM 2003 Zhichen Xu HP template