Koorde: A Simple Degree Optimal DHT Adopted from Frans Kaashoek and David Karger MIT The blue lines are addred by Guihai Chen

DHT Routing Distributed hash tables Implement hash table interface Map any ID to the machine responsible for that ID (in a consistent fashion) Standard primitive for P2P Machines not all aware of each other Each tracks small set of “neighbors” Route to responsible node via sequence of “hops” to neighbors

Performance Measures Degree Hop count Fault tolerance How many neighbors nodes have Hop count How long to reach any destination node Fault tolerance How many nodes can fail Maintenance overhead E.g., making sure neighbors are up Load balance How evenly keys distribute among nodes

Tradeoffs With larger degree, hope to achieve Smaller hop count Better fault tolerance But higher degree implies More routing table state per node Higher maintenance overhead to keep routing tables up to date Load balance “orthogonal issue”

Current Systems Chord, Kademlia, Pastry, Tapestry O(log n) degree O(log n) hop count O(log n) ratio load balance Chord: O(1) load balance with O(log n) “virtual nodes” per real node Multiplies degree to O(log2 n)

Outliers CAN Viceroy Degree d O(dn1/d) hops O(log n) hop count Constant average degree But some nodes have degree log n Outliers:局外人

Lower Bounds to Shoot For Theorem: if max degree is d, then hop count is at least logd n Proof: < dh nodes at distance h Allows degree O(1) and O(log n) hops Or deg. O(log n) and O(log n / loglog n) hops Theorem: to tolerate half nodes failing, (e.g. net partition) need degree W(log n) Pf: if less, some node loses all neighbors Might as well take O(log n / loglog n) hops! Something wrong:if max degree is d, then hop count is at least logd nif max degree is d, then hop count is at least FLOOR(logd n) otherwise, there are 2 counter examples: 3-node ring and 5-node ring.

Koorde New routing protocol Shares almost all aspects with Chord But, meets (to within constant factor) all lower bounds just mentioned: Degree 2 and O(log n) hops Or degree log n and O(log n / loglog n) hops and fault tolerant Like Chord, O(log n) load balance or constant with O(log n) times degree

Chord Review Chord consists of Consistent hashing to assign IDs to nodes Good load balance Efficient routing protocol to find right node Fast join/leave protocol Few data items shifted Fault tolerance to half of nodes failing Efficient maintenance over time ■ Koorde routing protocol to find right node

Consistent Hashing Assign ID to “successor” node on ring 6 60 51 13 6 60 51 13 Assign doc with hash 49 to node 51 Assign ID to “successor” node on ring 49 18 47 22 42 36 31

Chord Routing Each node keeps successor pointer Also keeps power-of-two “fingers” neighbors providing shortcuts So log n fingers 60 6 51 13 18 47 22 42 36 31

Chord Lookups 60 6 51 13 18 47 22 42 36 31

Koorde Idea Chord acts like a hypercube Koorde uses a deBruijn network Fingers flip one bit Degree log n (log n different flips) Diameter log n Koorde uses a deBruijn network Fingers shift in one bit Degree 2 (2 possible bits to shift in)

De Bruijn Graph Nodes are b-bit integers (b = log n) Node u has 2 neighbors (bit shifts): 2u mod 2b and 2u+1 mod 2b Or Node u=ubub-1…u1 has 2 left shift neighbors u°0 = ub-1ub-2…u10 and u°1 =ub-1ub-2…u11 100 110 1 1 000 010 101 111 1 1 1 1 1 001 011 1

De Bruijn Routing Shift in destination bits one by one b hops complete route Route from 000 to 110: 100 110 1 1 010 101 111 000 1 1 1 1 1 001 011 1

Routing Code Procedure u.LOOKUP(k, toShift) /* u is machine, k is target key toShift is target bits not yet shifted in */ if k = u then Return u /* as owner for k */ else /* do de Bruijn hop */ t = u °topBit(toShift) Return t.lookup(k, toshift áá 1) Initially call self.LOOKUP(k,k) toshift <<1 : left shift by 1 bit, topBit(toShift):the current bit of destination which should be shifted in.

Summary Each node has 2 outgoing neighbors Also two incoming Can show good routing load balance Need b = log n bits for n distinct nodes So log n hops to route

Problems to Solve Want b-bit ring, b >> log n, to avoid colliding identifiers as nodes join Implies use b >> log n hops Worse, most nodes not present to route! Solutions Imaginary routing: present nodes simulate routing actions of absent nodes Short cuts: use gaps to start route with most of destination bits already shifted in Read the paper to understand b >> log n

Imaginary routing Node u holds two pointers Successor on ring One finger: predecessor of 2u (mod 2b) On sparse ring, is also predecessor of 2u+1 So handles both de Bruijn edges Node u “owns” all imaginary nodes between self and (real) successor Simulates de Bruijn routing from those imaginary nodes to others by forwarding to the others’ real owners

Code Procedure u.LOOKUP(k, toShift, i) if k Î (u,u.successor] then return u.successor /* as bucket for k */ else if i Î (u,u.successor] then /* i belongs to u; do de Bruijn hop */ return u.finger.LOOKUP(k, toshift áá 1, i °topBit(toShift)) else /* i doesn’t belong to u; forward it */ return u.successor.LOOKUP(k, toShift, i) Initially call self.LOOKUP(k,k,self) i represents the next routing node of full de Bruijn graph. (u,u.successor] :including u, but not u.successor. Argument i always represents the next station of full De Bruijn graph.

True route tracks imaginary start finger (< double) imaginary(double) target successor

Correctness Once b de Bruijn steps happen, done At this point, i = k Will follow successors to bucket for k Successor steps delay de Bruijn steps, but not forever After finite number of successor steps, reach predecessor of i Conclude: all necessary de Bruijn steps happen in finite time. So correct.

How long? Only b de Bruijn steps Just bound (expected) number of successor steps per de Bruijn step Nodes randomly distributed on ring So node expects to own size 1/n interval So distance to imaginary node on de Bruijn step is 1/n De Bruijn step doubles everything, makes distance 2/n Expect 2 nodes in interval of that size

Few Successor Steps start 1/n target < 2/n

Summary Each de Bruijn hop followed by 2 successor hops (in expectation) b de Bruijn hops Conclude 2b successor hops so 3b hops in total Expectation argument extends to “with high probability” argument (same bounds) Remaining problem: b>>log n, too big

Exploit Address Blocks Only n real nodes Each owns ~1/n “block” of keyspace Within that block, only top log n bits “significant”; low bits arbitrary So set low bits to high bits of target Then just have to shift out log n most significant bits So log n de Bruijn hops, So O(log n) hops in total

Example Start at u = 001011011… Successor 001110101…. u “owns” imaginary 00101****** Target 1101011…. Set imaginary start 001011101011… Only need to shift out 00101 5 hops, independent of b

Summary Koorde uses 2 neighbors per node (one successor, one finger) And requires O(log n) routing hops with high probability

Variant: Koorde-K We used a binary de Bruijn Network Generalizes to other base K: 021 022 020 110 100 102 002 111 101 2 010 012 011 000 112 1 001 120 121 122

Analysis To represent n distinct node ids need logK n base-K digits Suggests logK n hops to route Same problem as Koorde: b >> logK n Same solution: imaginary routing Node u points at predecessor(Ku) Same analysis: K de Bruijn hops interspersed with successor hops

Successor Hops Now de Bruijn hop multiplies ids by K So expect K nodes between finger and next imaginary node Implies K successor hops per de Bruijn hop Gives K logK n hops---no good To avoid successor hops, u fingers predecessor(Ku) and following K nodes Allows K successor hops by one finger Gives O(logK n) hops as desired

Summary Using K fingers per node, can achieve O(logK n) = O(log n / log K) routing hops As discussed earlier, degree log n is necessary (and sufficient) for fault tolerance (and is degree of most previous systems) So, O(log n / log log n ) hops

Summary: What do we Gain? Lower degree for same number of hops Storage isn’t really an issue But lower degree should translate into lower maintenance traffic Lower hop count for same degree And tunable Other systems also have tunable hop count But at low hop counts (high degree) their extra log factor in degree does matter

What do we lose? Chord is “self stabilizing” Koorde is not From successors, can build entire routing system quickly by “pointer jumping” to find fingers Koorde is not Given only successor pointers, no clear fast way to find fingers Not a problem for joins, because joiner can use lookup to find its finger But could be a problem if massive changes

More Info http://www.pdos.lcs.mit.edu/chord/