Presentation is loading. Please wait.

Presentation is loading. Please wait.

Implementation and Deployment of a Large-scale Network Infrastructure Ben Y. Zhao L. Huang, S. Rhea, J. Stribling, A. D. Joseph, J. D. Kubiatowicz EECS,

Published byHugo Huxford Modified over 9 years ago

Similar presentations

Presentation on theme: "Implementation and Deployment of a Large-scale Network Infrastructure Ben Y. Zhao L. Huang, S. Rhea, J. Stribling, A. D. Joseph, J. D. Kubiatowicz EECS,"— Presentation transcript:

1 Implementation and Deployment of a Large-scale Network Infrastructure Ben Y. Zhao L. Huang, S. Rhea, J. Stribling, A. D. Joseph, J. D. Kubiatowicz EECS, UC Berkeley HP Labs, November 2002

2 HP Labs, 11/26/02 © ravenben@eecs.berkeley.edu2 Next-generation Network Applications Scalability in applications Process/threads on single node server Cluster (LAN): fast, reliable, unlimited comm. Next step: scaling to the wide-area Complexities of global deployment Network unreliability BGP slow convergence, redundancy unexploited Lack of administrative control over components Constrains protocol deployment: multicast, congestion ctrl. Management of large scale resources / components Locate, utilize resources despite failures

3 HP Labs, 11/26/02 © ravenben@eecs.berkeley.edu3 Enabling Technology: DOLR (Decentralized Object Location and Routing) GUID1 DOLR GUID1 GUID2

4 HP Labs, 11/26/02 © ravenben@eecs.berkeley.edu4 A Solution Decentralized Object Location and Routing (DOLR) wide-area overlay application infrastructure Self-organizing, scalable Fault-tolerant routing and object location Efficient (b/w, latency) data delivery Extensible, supports application-specific protocols Recent work Tapestry, Chord, CAN, Pastry Kademlia, Viceroy, …

5 HP Labs, 11/26/02 © ravenben@eecs.berkeley.edu5 What is Tapestry? DOLR driving OceanStore global storage (Zhao, Kubiatowicz, Joseph et al. 2000) Network structure Nodes assigned bit sequence nodeIds namespace: 0-2 160, based on some radix (e.g. 16) keys from same namespace Keys dynamically map to 1 unique live node: root Base API Publish / Unpublish (Object ID) RouteToNode (NodeId) RouteToObject (Object ID)

6 HP Labs, 11/26/02 © ravenben@eecs.berkeley.edu6 4 2 3 3 3 2 2 1 2 4 1 2 3 3 1 3 4 1 1 43 2 4 Tapestry Mesh Incremental prefix-based routing NodeID 0x43FE NodeID 0xEF31 NodeID 0xEFBA NodeID 0x0921 NodeID 0xE932 NodeID 0xEF37 NodeID 0xE324 NodeID 0xEF97 NodeID 0xEF32 NodeID 0xFF37 NodeID 0xE555 NodeID 0xE530 NodeID 0xEF44 NodeID 0x0999 NodeID 0x099F NodeID 0xE399 NodeID 0xEF40 NodeID 0xEF34

7 HP Labs, 11/26/02 © ravenben@eecs.berkeley.edu7 Routing Mesh Routing via local routing tables Based on incremental prefix matching Example: 5324 routes to 0629 via 5324  0231  0667  0625  0629 At n th hop, local node matches destination at least n digits (if any such node exists) i th entry in n th level routing table points to nearest node matching: prefix(local_ID, n)+i Properties At most log(N) # of overlay hops between nodes Routing table size: b  log(N) Actual entries have c-1 backups, total size: c  b  log(N)

8 HP Labs, 11/26/02 © ravenben@eecs.berkeley.edu8 Object Location Randomization and Locality

9 HP Labs, 11/26/02 © ravenben@eecs.berkeley.edu9 Object Location Distribute replicates of object references Only references, not the data itself (level of indirection) Place more of them closer to object itself Publication Place object location pointers into network Store hops between object and “root” node Location Route message towards root from client Redirect to object when location pointer found

10 HP Labs, 11/26/02 © ravenben@eecs.berkeley.edu10 Node Insertion Inserting new node N Notify need-to-know nodes of N, N fills null entries in their routing tables Move locally rooted object references to N Construct locally optimal routing table for N Notify nearby nodes to N for optimization Two phase node insertion Acknowledged multicast Nearest neighbor approximation

11 HP Labs, 11/26/02 © ravenben@eecs.berkeley.edu11 Acknowledged Multicast Reach need-to-know nodes of N (e.g. 3111) Add to routing table Move root object references G N 3211 3229 3013 3222 3205 3023 3022 3021

12 HP Labs, 11/26/02 © ravenben@eecs.berkeley.edu12 Nearest Neighbor N iterates: list = need-to-know nodes, L = prefix (N, S) Measure distances of List, use to fill routing table, level L Trim to k closest nodes, list = backpointers from k set, L-- Repeat until L == 0 N Need-to-know nodes

13 HP Labs, 11/26/02 © ravenben@eecs.berkeley.edu13 Talk Outline Algorithms Architecture Architectural components Extensibility API Evaluation Ongoing Projects Conclusion

14 HP Labs, 11/26/02 © ravenben@eecs.berkeley.edu14 Single Tapestry Node Transport Protocols Network Link Management Application Interface / Upcall API Decentralized File Systems Application-Level Multicast Approximate Text Matching Router Routing Table & Object Pointer DB Dynamic Node Management

15 HP Labs, 11/26/02 © ravenben@eecs.berkeley.edu15 Single Node Implementation Application Programming Interface Applications Dynamic TapestryCore Router Patchwork Network StageDistance Map SEDA Event-driven Framework Java Virtual Machine Enter/leave Tapestry State Maint. Node Ins/del Routing Link Maintenance Node Ins/del Messages UDP Pings route to node / obj API calls Upcalls fault detect heartbeat msgs

16 HP Labs, 11/26/02 © ravenben@eecs.berkeley.edu16 Message Routing Router: fast routing to nodes / objects Receive new location msg Forward to nextHop(h+1,G) Forward to nextHop(0,obj) Signal App Upcall Handler Upcall? yes no yes Have Obj Ptrs Receive new route msg Signal App Upcall Handler Upcall? no yes Forward to nextHop(h+1,G)

17 HP Labs, 11/26/02 © ravenben@eecs.berkeley.edu17 Extensibility API deliver (G, A id, Msg) Invoked at message destination Asynchronous, returns immediately forward (G, A id, Msg) Invoked at intermediate hop in route No action taken by default, application calls route() route (G, A id, Msg, NextHopNodeSet) Called by application to request message be routed to set of NextHopNodes

18 HP Labs, 11/26/02 © ravenben@eecs.berkeley.edu18 Local Operations Accessibility to Tapestry maintained state nextHopSet = Llookup(G, Num) Accesses routing table Returns up to num candidates for next hop towards G objReferenceSet = Lsearch(G, num) Searches object references for G Returns up to num references for object, sorted by increasing network distance

19 HP Labs, 11/26/02 © ravenben@eecs.berkeley.edu19 Deployment Status C simulator Packet level simulation Scales up to 10,000 nodes Java implementation 50000 semicolons of Java, 270 class files Deployed on local area cluster (40 nodes) Deployed on Planet Lab global network (~100 distributed nodes)

20 HP Labs, 11/26/02 © ravenben@eecs.berkeley.edu20 Talk Outline Algorithms Architecture Evaluation Micro-benchmarks Stable network performance Single and parallel node insertion Ongoing Projects Conclusion

21 HP Labs, 11/26/02 © ravenben@eecs.berkeley.edu21 Micro-benchmark Methodology Experiment run in LAN, GBit Ethernet Sender sends 60001 messages at full speed Measure inter-arrival time for last 50000 msgs 10000 msgs: remove cold-start effects 50000 msgs: remove network jitter effects Sender Control Receiver Control Tapestry LAN Link

22 HP Labs, 11/26/02 © ravenben@eecs.berkeley.edu22 Micro-benchmark Results Constant processing overhead ~ 50  s Latency dominated by byte copying For 5K messages, throughput = ~10,000 msgs/sec

23 HP Labs, 11/26/02 © ravenben@eecs.berkeley.edu23 Large Scale Methodology Planet Lab global network 98 machines at 42 institutions, in North America, Europe, Australia (~ 60 machines utilized) 1.26Ghz PIII (1GB RAM), 1.8Ghz PIV (2GB RAM) North American machines (2/3) on Internet2 Tapestry Java deployment 6-7 nodes on each physical machine IBM Java JDK 1.30 Node virtualization inside JVM and SEDA Scheduling between virtual nodes increases latency

24 HP Labs, 11/26/02 © ravenben@eecs.berkeley.edu24 Node to Node Routing Ratio of end-to-end routing latency to shortest ping distance between nodes All node pairs measured, placed into buckets Median=31.5, 90 th percentile=135

25 HP Labs, 11/26/02 © ravenben@eecs.berkeley.edu25 Object Location Ratio of end-to-end latency for object location, to shortest ping distance between client and object location Each node publishes 10,000 objects, lookup on all objects 90 th percentile=158

26 HP Labs, 11/26/02 © ravenben@eecs.berkeley.edu26 Latency to Insert Node Latency to dynamically insert a node into an existing Tapestry, as function of size of existing Tapestry Humps due to expected filling of each routing level

27 HP Labs, 11/26/02 © ravenben@eecs.berkeley.edu27 Bandwidth to Insert Node Cost in bandwidth of dynamically inserting a node into the Tapestry, amortized for each node in network Per node bandwidth decreases with size of network

28 HP Labs, 11/26/02 © ravenben@eecs.berkeley.edu28 Parallel Insertion Latency Latency to dynamically insert nodes in unison into an existing Tapestry of 200 Shown as function of insertion group size / network size 90 th percentile=55042

29 HP Labs, 11/26/02 © ravenben@eecs.berkeley.edu29 Results Summary Lessons Learned Node virtualization: resource contention Accurate network distances hard to measure Efficiency verified Msg processing = 50  s, Tput ~ 10,000msg/s Route to node/object small factor over optimal Algorithmic scalability Single node latency/bw scale sublinear to network size Parallel insertion scales linearly with group size

30 HP Labs, 11/26/02 © ravenben@eecs.berkeley.edu30 Talk Outline Algorithms Architecture Evaluation Ongoing Projects P2P landmark routing: Brocade Applications: Shuttle, Interweave, ATA Conclusion

31 HP Labs, 11/26/02 © ravenben@eecs.berkeley.edu31 State of the Art Routing High dimensionality and coordinate-based P2P routing Tapestry, Pastry, Chord, CAN, etc… Sub-linear storage and # of overlay hops per route Properties dependent on random name distribution Optimized for uniform mesh style networks

32 HP Labs, 11/26/02 © ravenben@eecs.berkeley.edu32 Reality AS-2 P2P Overlay Network AS-1 AS-3 SR Transit-stub topology, disparate resources per node Result: Inefficient inter-domain routing (b/w, latency)

33 HP Labs, 11/26/02 © ravenben@eecs.berkeley.edu33 Landmark Routing on P2P Brocade Exploit non-uniformity Minimize wide-area routing hops / bandwidth Secondary overlay on top of Tapestry Select super-nodes by admin. domain Divide network into cover sets Super-nodes form secondary Tapestry Advertise cover set as local objects brocade routes directly into destination’s local network, then resumes p2p routing

34 HP Labs, 11/26/02 © ravenben@eecs.berkeley.edu34 AS-2 P2P Network AS-1 AS-3 Brocade Layer SD Original Route Brocade Route Brocade Routing

35 HP Labs, 11/26/02 © ravenben@eecs.berkeley.edu35 Applications under Development OceanStore: global resilient file store Shuttle Decentralized P2P chat service Leverages Tapestry for fault-tolerant routing Interweave Keyword searchable file sharing utility Fully decentralized, exploits network locality Approximate Text Addressing Uses text fingerprinting to map similar documents to single IDs Killer app: decentralized spam mail filter

36 HP Labs, 11/26/02 © ravenben@eecs.berkeley.edu36 For More Information Tapestry and related projects (and these slides): http://www.cs.berkeley.edu/~ravenben/tapestry OceanStore: http://oceanstore.cs.berkeley.edu Related papers: http://oceanstore.cs.berkeley.edu/publications http://www.cs.berkeley.edu/~ravenben/publications ravenben@eecs.berkeley.edu

Download ppt "Implementation and Deployment of a Large-scale Network Infrastructure Ben Y. Zhao L. Huang, S. Rhea, J. Stribling, A. D. Joseph, J. D. Kubiatowicz EECS,"

Similar presentations

Brocade: Landmark Routing on Peer to Peer Networks Ben Y. Zhao Yitao Duan, Ling Huang, Anthony Joseph, John Kubiatowicz IPTPS, March 2002.

Brocade: Landmark Routing on Peer to Peer Networks Ben Y. Zhao Yitao Duan, Ling Huang, Anthony Joseph, John Kubiatowicz IPTPS, March 2002.
Perspective on Overlay Networks Panel: Challenges of Computing on a Massive Scale Ben Y. Zhao FuDiCo 2002.

Perspective on Overlay Networks Panel: Challenges of Computing on a Massive Scale Ben Y. Zhao FuDiCo 2002.
Tapestry: Scalable and Fault-tolerant Routing and Location Stanford Networking Seminar October 2001 Ben Y. Zhao

Tapestry: Scalable and Fault-tolerant Routing and Location Stanford Networking Seminar October 2001 Ben Y. Zhao
Tapestry: Decentralized Routing and Location SPAM Summer 2001 Ben Y. Zhao CS Division, U. C. Berkeley.

Tapestry: Decentralized Routing and Location SPAM Summer 2001 Ben Y. Zhao CS Division, U. C. Berkeley.
Pastry Peter Druschel, Rice University Antony Rowstron, Microsoft Research UK Some slides are borrowed from the original presentation by the authors.

Pastry Peter Druschel, Rice University Antony Rowstron, Microsoft Research UK Some slides are borrowed from the original presentation by the authors.
Peter Druschel, Rice University Antony Rowstron, Microsoft Research UK

Peter Druschel, Rice University Antony Rowstron, Microsoft Research UK
Exploiting Route Redundancy via Structured Peer to Peer Overlays Ben Y. Zhao, Ling Huang, Jeremy Stribling, Anthony D. Joseph, and John D. Kubiatowicz.

Exploiting Route Redundancy via Structured Peer to Peer Overlays Ben Y. Zhao, Ling Huang, Jeremy Stribling, Anthony D. Joseph, and John D. Kubiatowicz.
Chord: A scalable peer-to- peer lookup service for Internet applications Ion Stoica, Robert Morris, David Karger, M. Frans Kaashock, Hari Balakrishnan.

Chord: A scalable peer-to- peer lookup service for Internet applications Ion Stoica, Robert Morris, David Karger, M. Frans Kaashock, Hari Balakrishnan.
*Towards A Common API for Structured Peer-to-Peer Overlays Frank Dabek, Ben Y. Zhao, Peter Druschel, John Kubiatowicz, Ion Stoica MIT, U. C. Berkeley,

*Towards A Common API for Structured Peer-to-Peer Overlays Frank Dabek, Ben Y. Zhao, Peter Druschel, John Kubiatowicz, Ion Stoica MIT, U. C. Berkeley,
Approximate Object Location and Spam Filtering on Peer-to-Peer Systems Feng Zhou, Li Zhuang, Ben Y. Zhao, Ling Huang, Anthony D. Joseph and John D. Kubiatowicz.

Approximate Object Location and Spam Filtering on Peer-to-Peer Systems Feng Zhou, Li Zhuang, Ben Y. Zhao, Ling Huang, Anthony D. Joseph and John D. Kubiatowicz.
Pastry Peter Druschel, Rice University Antony Rowstron, Microsoft Research UK Some slides are borrowed from the original presentation by the authors.

Pastry Peter Druschel, Rice University Antony Rowstron, Microsoft Research UK Some slides are borrowed from the original presentation by the authors.
1 Accessing nearby copies of replicated objects Greg Plaxton, Rajmohan Rajaraman, Andrea Richa SPAA 1997.

1 Accessing nearby copies of replicated objects Greg Plaxton, Rajmohan Rajaraman, Andrea Richa SPAA 1997.
Rapid Mobility via Type Indirection Ben Y. Zhao, Ling Huang, Anthony D. Joseph, John D. Kubiatowicz Computer Science Division, UC Berkeley IPTPS 2004.

Rapid Mobility via Type Indirection Ben Y. Zhao, Ling Huang, Anthony D. Joseph, John D. Kubiatowicz Computer Science Division, UC Berkeley IPTPS 2004.
Common approach 1. Define space: assign random ID (160-bit) to each node and key 2. Define a metric topology in this space,  that is, the space of keys.

Common approach 1. Define space: assign random ID (160-bit) to each node and key 2. Define a metric topology in this space,  that is, the space of keys.
Outline for today Structured overlay as infrastructures Survey of design solutions Analysis of designs.

Outline for today Structured overlay as infrastructures Survey of design solutions Analysis of designs.
The Oceanstore Regenerative Wide-area Location Mechanism Ben Zhao John Kubiatowicz Anthony Joseph Endeavor Retreat, June 2000.

The Oceanstore Regenerative Wide-area Location Mechanism Ben Zhao John Kubiatowicz Anthony Joseph Endeavor Retreat, June 2000.
1 Towards a Common API for Structured Peer-to-Peer Overlays Frank Dabek, Ben Zhao, Peter Druschel, John Kubiatowicz, Ion Stoica Presented for Cs294-4 by.

1 Towards a Common API for Structured Peer-to-Peer Overlays Frank Dabek, Ben Zhao, Peter Druschel, John Kubiatowicz, Ion Stoica Presented for Cs294-4 by.
Scribe: A Large-Scale and Decentralized Application-Level Multicast Infrastructure Miguel Castro, Peter Druschel, Anne-Marie Kermarrec, and Antony L. T.

Scribe: A Large-Scale and Decentralized Application-Level Multicast Infrastructure Miguel Castro, Peter Druschel, Anne-Marie Kermarrec, and Antony L. T.
Brocade Landmark Routing on Structured P2P Overlays Ben Zhao, Yitao Duan, Ling Huang Anthony Joseph and John Kubiatowicz (IPTPS 2002) Goals Improve routing.

Brocade Landmark Routing on Structured P2P Overlays Ben Zhao, Yitao Duan, Ling Huang Anthony Joseph and John Kubiatowicz (IPTPS 2002) Goals Improve routing.
Spring 2003CS 4611 Peer-to-Peer Networks Outline Survey Self-organizing overlay network File system on top of P2P network Contributions from Peter Druschel.

Spring 2003CS 4611 Peer-to-Peer Networks Outline Survey Self-organizing overlay network File system on top of P2P network Contributions from Peter Druschel.

Similar presentations

Ads by Google