Copyright notice © 2008 Raul Jimenez - rauljc@kth.se - http://tslab.ssvl.kth.se/raul/ This work is licensed under the Creative Commons Attribution 2.5 Sweden License. To view a copy of this license, visit http://creativecommons.org/licenses/by/2.5/se/deed.en or send a letter to Creative Commons, 171 Second Street, Suite 300, San Francisco, California, 94105, USA. Copy, distribution, and reuse of this work is permitted, so long as the redistributed copy displays a link to http://tslab.ssvl.kth.se/raul/. The logos displayed on the title page belong to KTH and the P2P-Next consortium, respectively. The snapshot displayed on slide 4 is part of the animation movie “elephants dream” (http://orange.blender.org/media-gallery) The figure displayed on slide 21 is part of the technical report referenced on slide 22. Although not mandatory, the author would be glad to hear from those who find this material useful. Of course, any comment which could complement and/or improve this work is very welcome.

Decentralized peer discovery performance in swarm-protocols Raul Jimenez KTH, Sweden rauljc@kth.se tslab.ssvl.kth.se/raul

Overview BitTorrent Content Discovery Data Exchange Peer Discovery Centralized Tracker Decentralized Tracker (DHT) Gossiping (PEX) DHT Issues NAT effects Experiments

Go to a search engine (e.g. legaltorrents.org) BitTorrent CD PD DEX Tracker DHT PEX DHT Issues NAT Experiments BitTorrent Content Discovery Data Exchange Peer Discovery blender_foundation_-_ elephants_dream.torrent info_hash: c70bcf4657cac1253d12 acda95b04d1e4dcd23f4 Tracker (updated list of peers): http://tracker001.legaltorrents.com: 7070/announce 123.11.22.33 2223 234.22.11.22 3227 178.98.237.83 19033 231.187.99.123 6227 Go to a search engine (e.g. legaltorrents.org) Type “elephants dream” Download .torrent file

Decentralized Tracker (DHT) Gossiping (PEX) BitTorrent BitTorrent CD PD DEX Tracker DHT PEX DHT Issues NAT Experiments Content Discovery Data Exchange Peer Discovery Centralized Tracker Decentralized Tracker (DHT) Gossiping (PEX) A tracker keeps track of every peer participating in the swarm Gossiping protocol Disconnected graphs Messages between peers Needs bootstrap Complements tracker

Centralized Tracker 1 P1 announces Tracker tracks P1 BitTorrent Centralized Tracker CD PD DEX Tracker DHT PEX DHT Issues NAT Experiments 1 tracker.kth.se peers: 1 P1 announces Tracker tracks P1 Tracker returns list

Centralized Tracker 1 2 P2 announces Tracker tracks P2 BitTorrent Centralized Tracker CD PD DEX Tracker DHT PEX DHT Issues NAT Experiments 1 tracker.kth.se 2 peers: 1 2 P2 announces Tracker tracks P2 Tracker returns list P2 contacts peers

Centralized Tracker 1 2 P3 announces Tracker tracks P3 3 BitTorrent Centralized Tracker CD PD DEX Tracker DHT PEX DHT Issues NAT Experiments 1 tracker.kth.se 2 peers: 1 2 3 P3 announces Tracker tracks P3 3 Tracker returns list P3 contacts peers

Decentralized Tracker (DHT) Gossiping (PEX) BitTorrent BitTorrent CD PD DEX Tracker DHT PEX DHT Issues NAT Experiments Content Discovery Data Exchange Peer Discovery Centralized Tracker Decentralized Tracker (DHT) Gossiping (PEX) DHT Issues NAT effects Experiments Lookup times Dead-end lookups NAT behavior NAT types Effects on the DHT Measuring reachability with different methods

Distributed Tracker Distributed Hash Table BitTorrent Distributed Tracker Distributed Hash Table CD PD DEX Tracker DHT PEX DHT Issues NAT Experiments Kademlia DHT (Mainline implementation) Totally decentralized - bootstrap? O(Log(n)) routing table size O(Log(n)) hops to find the list of peers Iterative routing algorithm Tracks every peer in the BitTorrent swarm

Distributed Hash Table (DHT) BitTorrent Distributed Hash Table (DHT) CD PD DEX Tracker DHT PEX DHT Issues NAT Experiments Simple demo 4-bits identifiers (24 = 16 identifiers) 1 node per bucket 10 nodes Real world 160-bit identifiers 8 nodes per bucket 1 million nodes ?

DHT: Routing Table Routing table Level 0 Level 1 Level 2 Level 3 BitTorrent DHT: Routing Table CD PD DEX Tracker DHT PEX DHT Issues NAT Experiments Routing table Level 0 Level 1 Level 2 Level 3 MyNode 1001

DHT: Routing Table 1xxx 0xxx Routing table Level 0 0010 Level 1 BitTorrent DHT: Routing Table CD PD DEX Tracker DHT PEX DHT Issues NAT Experiments 1xxx 0xxx Routing table Level 0 0010 Level 1 Level 2 Level 3 MyNode 1001

DHT: Routing Table 11xx 00xx Routing table 10xx 01xx Level 0 0010 BitTorrent DHT: Routing Table CD PD DEX Tracker DHT PEX DHT Issues NAT Experiments 11xx 00xx Routing table 10xx 01xx Level 0 0010 Level 1 1111 Level 2 Level 3 MyNode 1001

DHT: Routing Table 111x 000x 110x 001x 101x 010x Routing table Level 0 BitTorrent DHT: Routing Table CD PD DEX Tracker DHT PEX DHT Issues NAT Experiments 111x 000x 110x 001x 101x 010x Routing table Level 0 0010 Level 1 1111 100x 011x Level 2 1011 Level 3 MyNode 1001

DHT: Routing Table 111x 000x 110x 001x 101x 010x Routing table Level 0 BitTorrent DHT: Routing Table CD PD DEX Tracker DHT PEX DHT Issues NAT Experiments 111x 000x 110x 001x 101x 010x Routing table Level 0 0010 Level 1 1111 1001 011x Level 2 1011 1000 Level 3 ------ MyNode 1001

DHT: Find Peers 1xxx 0xxx Level 0 0010 0101 torrentID Level 1 1111 BitTorrent DHT: Find Peers CD PD DEX Tracker DHT PEX DHT Issues NAT Experiments 1xxx 0xxx Level 0 0010 0101 torrentID Level 1 1111 Level 2 1011 Level 3 MyNode 1001

DHT: Find Peers Level 0 1001 Level 1 0110 Level 2 0000 11xx 00xx BitTorrent DHT: Find Peers CD PD DEX Tracker DHT PEX DHT Issues NAT Experiments Level 0 1001 Level 1 0110 Level 2 0000 11xx 00xx Level 3 0011 10xx 01xx 0101 torrentID MyNode 1001

DHT: Find Peers 111x 000x 110x 001x 101x 010x 0101 torrentID 100x 011x BitTorrent DHT: Find Peers CD PD DEX Tracker DHT PEX DHT Issues NAT Experiments 111x 000x 110x 001x 101x 010x 0101 torrentID 100x 011x Level 0 1101 Level 1 0000 MyNode 1001 Level 2 0100 Level 3

DHT: Find Peers 111x 000x 110x 001x Level 0 1100 101x 010x Level 1 BitTorrent DHT: Find Peers CD PD DEX Tracker DHT PEX DHT Issues NAT Experiments 111x 000x 110x 001x Level 0 1100 101x 010x Level 1 0011 Level 2 0110 Level 3 0101 0101 torrentID 100x 011x MyNode 1001

DHT Issues Long lookup times (around a minute) BitTorrent DHT Issues CD PD DEX Tracker DHT PEX DHT Issues NAT Experiments Long lookup times (around a minute) Especially long at the end of the lookup

Reported causes 20% dead nodes in 95% of the lookups BitTorrent Reported causes CD PD DEX Tracker DHT PEX DHT Issues NAT Experiments 20% dead nodes in 95% of the lookups Too many dead nodes in the routing table: Levels 0-15: 15% dead nodes Levels 15-18: 65% dead nodes 57% of the nodes suffer “infant mortality” Reference: Scott A Crosby and Dan S Wallach An Analysis of BitTorrent's Two Kademlia-Based DHTs Technical Report TR-07-04, Department of Computer Science, Rice University, June 2007

Are reachability issues (most likely caused by NAT boxes) BitTorrent CD PD DEX Tracker DHT PEX DHT Issues NAT Experiments Are reachability issues (most likely caused by NAT boxes) the main cause of BitTorrent's poor DHT performance?

Network Address Translation (NAT) BitTorrent Network Address Translation (NAT) CD PD DEX Tracker DHT PEX DHT Issues NAT Experiments NAT open partially open NAT temporarily open NAT timeout, or full table, or ...

NAT types Match IP and port (symmetric) IP2:2222 IP1:1111 IP1:9999 BitTorrent NAT types CD PD DEX Tracker DHT PEX DHT Issues NAT Experiments Match IP and port (symmetric) IP2:2222 IP1:1111 IP1:9999 IP3:3333

NAT types Match IP and port (symmetric) BitTorrent NAT types CD PD DEX Tracker DHT PEX DHT Issues NAT Experiments Match IP and port (symmetric) Match only IP (restricted cone) IP2:2222 IP1:1111 IP1:9999 IP3:3333

NAT types Match IP and port (symmetric) BitTorrent NAT types CD PD DEX Tracker DHT PEX DHT Issues NAT Experiments Match IP and port (symmetric) Match only IP (restricted cone) Match any connection (full cone) IP2:2222 IP1:1111 IP1:9999 IP3:3333 Reference: RFC3489 STUN - Simple Traversal of User Datagram Protocol (UDP) Through Network Address Translators (NATs)

? NAT types Match IP and port Match only IP Match any connection ... BitTorrent NAT types CD PD DEX Tracker DHT PEX DHT Issues NAT Experiments Match IP and port Match only IP Match any connection ... IP2:2222 IP1:1111 ? IP1:9999 IP3:3333

BitTorrent 1 Peer discovery No peers to contact tracker.kth.se peers: CD PD DEX Tracker DHT PEX DHT Issues NAT Experiments 1 tracker.kth.se peers: 1 Peer discovery No peers to contact

BitTorrent 1 2 Peer discovery P2 contacts P1 tracker.kth.se peers: 1 2 CD PD DEX Tracker DHT PEX DHT Issues NAT Experiments 1 tracker.kth.se 2 peers: 1 2 Peer discovery P2 contacts P1

BitTorrent 1 2 Peer discovery P3 contacts P1 3 P3 contacts P2 CD PD DEX Tracker DHT PEX DHT Issues NAT Experiments 1 tracker.kth.se 2 peers: 1 2 3 Peer discovery P3 contacts P1 3 P3 contacts P2

BitTorrent 1 4 2 Peer discovery P4 contacts P1 3 P4 contacts P2 CD PD DEX Tracker DHT PEX DHT Issues NAT Experiments 1 4 tracker.kth.se 2 peers: 1 2 3 4 Peer discovery P4 contacts P1 3 P4 contacts P2 P4 contacts P3

Temporal and partial reachability in DHT BitTorrent Temporal and partial reachability in DHT CD PD DEX Tracker DHT PEX DHT Issues NAT Experiments Higher (apparent) churn Broken routes

Higher (apparent) churn BitTorrent Higher (apparent) churn CD PD DEX Tracker DHT PEX DHT Issues NAT Experiments Churn rate: number of nodes joining and leaving the DHT during a given period. A new node joins Check reachability Level x N2 Add to the routing table 1 2 Dead node (remove) A new node joins ...

Broken routes Node 1 finds node 2 N1 checks N2's reachability BitTorrent Broken routes CD PD DEX Tracker DHT PEX DHT Issues NAT Experiments Node 1 finds node 2 Level x N2 N1 checks N2's reachability N1 adds N2 to routing table 1 2 N3 sends message to N1 N2 is the next hop N2 is unreachable from N3 3

Experiments 3 PCs running DHT and local 1 PC running remote 129 496 nodes checked DHT local remote

Experiments Check candidate node's reachability BitTorrent Experiments CD PD DEX Tracker DHT PEX DHT Issues NAT Experiments Check candidate node's reachability from the DHT (same IP, same port) opening a new connection (same IP, different port) from another host (different IP) 33% U 61% U 63% U

What to do? Prevent my routing table from pollution Check for global reachability before adding a node Don't pollute others' routing tables When I'm aware of being partially reachable, tell others: “don't add me to your routing table” Free riders?

Thank you!