Deterministic Distributed Resource Discovery Shay Kutten Technion David Peleg Weizmann Inst. Uzi Vishkin Univ. of Maryland & Technion

Distributed Resource Discovery [HLL99] pointer The corresponding directed graph (on nodes A,B,C): (assumption: weakly connected) Problem: compute the connected component router A C B A BC

Distributed Resource Discovery- [Harcol-Balter, Leighton, Lewin 1999] (“Names Dropper” protocol) Licensed to AKAMAI, a leading Web caching provider The resource discovery task: the set of servers changes  Find the current set (towards deciding which Akamai server to assign to each user)

Akamai servers need to learn how to reach each other via the Internet ( towards deciding which Akamai server is “ close ” to the user). Akamai Server Internet user CNN

Think Distributed AB a pointer used by the algorithm (one per node, initially pointing at the node itself) AAddress “known” to Node A (“knowledge graph”) Mr. and Ms. B IP street Interland a pointer used by the algorithm A?

When the Algorithm is started A C D B Weakly connected directed graph a pointer used by the algorithm (one per node, initially pointing at the node itself) AAddress “known” to Node A (“knowledge graph”)

Algorithm Actions A C D B C (1)Node A learns C’s addr. From B, thus, directed edge (A  C) is added to knowledge graph.

Algorithm Actions A C D B C (1)Node A learns C’s addr. From B, thus, directed edge (A  C) is added to knowledge graph. (2) Node D who knows of C, change PTR to point at C, thus, pointer graph changes. D

Algorithm Actions A C D B C (1)Node A learns C’s addr. From B, thus, directed edge (A  C) is added to knowledge graph. (2) Node D who knows of C, change PTR to point at C, thus, pointer graph changes. D change

Algorithm’s result pointers form star, knowledge graph is strongly connected, Plus, possibly, additional links (possibly clique).

Model [HLL99] - s ynchronous (not necessary for our alg) - s imultaneously start (for our alg: just “ close ” ) - complete comm. Graph Complexity: #Msgs ( connections [HLL99] ), #bits (#pointers*log n ), time (#rounds)

Motivation (2) Gnutella Napster server (mp3 music files) I have “ Love Song ” I want “ Love Song ” I have “ Madonna ”

Motivation (2) Gnutella Napster server I have Love Song I want Love Song A Ask A

Motivation (2) Gnutella Napster server I want Love Song A Ask A

Motivation (2) Gnutella Napster server A Ask A Love Song

Motivation (2) Gnutella vs. Napster Napster server If the court closes The service collapsed

Motivation (2) Gnutella vs. Napster Napster serverIf the court closes The service collapsed

Motivation (2) Gnutella vs. Napster Napster server If the court closes the service collapses

Motivation (2) Gnutella vs. Napster Napster server If the court closes the service collapses Gnutella: attempts to solve by having every client=server Same problem: user knows some others  find more increase connectivity to withstand disconnections

Motivation(2) Gnutella Users get disconnected often Bob Carol Initially Alice knows only Bob & Carol (personally, or from ICQ, or from Google…) and can connect via them To others they know. Alice David

Motivation(2) gnutella Users get disconnected often Bob Carol Initially Alice knows only Bob & Carol (personally, or from ICQ, or from Google…) Alice learned from Carol how to reach David, so that when Bob & Carol are not online Alice is still connected. Alice David

Motivation: “ Peer to Peer ” Applications (P2P) (1) Akamai (2) Gnutella (originally Nullsoft (Winamp)) 2000 (3) Gpulp: general Purpose Location Protocol (4) Genny (5) Freenet (6) JXTA (Sun Microsystems ) April 2001 (7) Retsina: Discovery of Infrastructure in Multi- Agent Systems (CMU)

This paper [HLL99] deterministic randomized termination detection no detection O(log n) rounds O(log n) O(n log n) messages O(n log n) O(E log n) bits O(n log n)

Algorithm strategies D E F G H I (1) Shrink tall trees to make them stars [SV82] 12 5 (2) Merge stars to get one tree (changed from [SV82]) (3) Carefully connect weakly connected (explained later; one of the differences from [Shiloach, Vishkin 82])

Example of a technique taken from [SV82] (Handling tall trees in “pointer graph”) D E F G H I C B A Shortcuts [SV82]

D E F G H I C B A Handing tall trees in “pointer graph” (cont.) Shortcuts ([SV82] Each child learns the pointer to its grandparent

D E F G H I C B A Shortcuts Handing tall trees in “pointer graph” (cont.)

Algorithm: An Example of a Technique different than [SV82] (weakly connected directed graph) Without an additional technique, may take  (n) time: A Worst case (only A is active, its star grows by 1 per phase): Phase 1

(weakly connected diagraph) Without an additional technique, may take  (n) time: A Worst case (only A is active, its star grows by 1 per phase): phase 2

(weakly connected diagraph) Without an additional technique, may take  (n) time: A Worst case (only A is active, its star grows by 1 per phase): phase n-1

(weakly connected diagraph) Idea: star root A picks 2 neighbors (not just 1) in phase i. (In phase i there are only n/2 stars) i Prevent  (n) time, and still do not send too many messages: Example: i=2 2 =4 A i i

(weakly connected diagraph) Idea: A picks 2 neighbors (not just 1) in phase i. i Prevent  (n) time: Some delicate points: resolving collisions, mixed topologies prove convergence in O(log n) phases A

Correctness Lemma: No cycles are created (hook on smaller + non-star does not connect) Assured by a new technique: 3  7 Possible because of Weak connectivity!! No connection  No cycle Lemma: always some active star

Complexity Two kinds of progress in each phase: (1) A tall tree gets shallower by a constant factor. Obstacles: - # of star trees may grow because of (1) - Trees may get taller because of (2) But (with the right combination) there is a progress in the combination. (2) Active star roots merge.

Conclusion - improved complexity in all measures (verifies [HLL99] conjecture, forecasting a simple algorithm with these complexities). - Deterministic. - Terminates (answers an open problem of Lipton). Further research - Adaptive algorithms (vs. “ one shot ” ) - Lower bounds? (in [CGK95] O(n) msgs for undirected case).