1. By Behzad Akbari Spring 2011
P2P Overlay Networks By
Behzad Akbari
Spring 2011
These slidesin some parts are based on the slides of J. Kurose (UMASS) and Sanjay Rao ()

2. Outline Overview of P2P overlay networks
Applications of overlay networks
Unstructured overlay networks
Structured overlay networks
Overlay multicast networks
P2P media streaming networks 2

3. Overview of P2P overlay networks
What is a P2P system?
P2P refers to applications that take advantage of resources (storage, cycles, content, bandwidth, human presence) available at the end systems of the internet.
What is an overlay network?
Overlay networks refer to networks that are constructed on top of another network (e.g. IP).
What is a P2P overlay network?
Any overlay network that is constructed by the Internet peers in the application layer on top of the IP network. 3

4. Overview of P2P overlay networks
P2P overlay network properties
Efficient use of resources
Self-organizing
All peers organize themselves into an application layer network on top of IP.
Scalability
Consumers of resources also donate resources
Aggregate resources grow naturally with utilization
Reliability
No single point of failure
Redundant overlay links between the peers
Redundant data source
Ease of deployment and administration
The nodes are self-organized
No need to deploy servers to satisfy demand.
Built-in fault tolerance, replication, and load balancing
No need any change in underlay IP networks 4

5. Applications of P2P overlay networks
P2P file sharing
Napster, Gnutella, Kaza, Emule, Edonkey, Bittorent, etc.
Application layer multicasting
P2P media streaming
Content distribution
Distributed caching
Distributed storage
Distributed backup systems
Grid computing 5

6. Classification of P2P overlay networks
Unstructured overlay networks
The overlay networks organize peers in a random graph in flat or hierarchical manners.
Structured overlay networks
Are based on Distributed Hash Tables (DHT)
the overlay network assigns keys to data items and organizes its peers into a graph that maps each data key to a peer
Overlay multicast networks
The peers organize themselves into an overlay tree for multicasting.
P2P media streaming networks
Used for large scale media streaming applications (e.g IPTV) over the Internet. 6

7. Unstructured P2P File Sharing Networks
Centralized Directory based P2P systems
Pure P2P systems
Hybrid P2P systems 7

8. Unstructured P2P File Sharing Networks (…)
Centralized Directory based P2P systems
All peers are connected to central entity
Peers establish connections between each other on demand to exchange user data (e.g. mp3 compressed data)
Central entity is necessary to provide the service
Central entity is some kind of index/group database
Central entity is lookup/routing table
Examples: Napster, Bittorent 8

9. Unstructured P2P File Sharing Networks(…)
Pure P2P systems
Any terminal entity can be removed without loss of functionality
No central entities employed in the overlay
Peers establish connections between each other randomly
To route request and response messages
To insert request messages into the overlay
Examples: Gnutella, FreeNet 9

10. Unstructured P2P File Sharing Networks(…)
Hybrid P2P systems
Main characteristic, compared to pure P2P: Introduction of another dynamic hierarchical layer
Election process to select an assign Super peers
Super peers: high degree (degree>>20, depending on network size)
Leaf nodes: connected to one or more Super peers (degree<7)
Example: KaZaA
Superpeer
leafnode 10

11. Centralized Directory based P2P systems : Napster
Napster history: the rise
January 1999: Napster version 1.0
May 1999: company founded
December 1999: first lawsuits
2000: 80 million users
Napster history: the fall
Mid 2001: out of business due to lawsuits
Mid 2001: dozens of P2P alternatives that were harder to touch, though these have gradually been constrained

12. Napster Technology: Directory Service
centralized
directory server
peers
Alice
Bob 1 2 3
User installing the software
Download the client program
Register name, password, local directory, etc.
Client contacts Napster (via TCP)
Provides a list of music files it will share
… and Napster’s central server updates the directory
Client searches on a title or performer
Napster identifies online clients with the file
… and provides IP addresses
Client requests the file from the chosen supplier
Supplier transmits the file to the client
Both client and supplier report status to Napster

13. Napster Technology: Properties
Server’s directory continually updated
Always know what music is currently available
Point of vulnerability for legal action
Peer-to-peer file transfer
No load on the server
Plausible deniability for legal action (but not enough)
Proprietary protocol
Login, search, upload, download, and status operations
No security: clear text passwords and other vulnerability
Bandwidth issues
Suppliers ranked by apparent bandwidth & response time

14. Napster: Limitations of Central Directory
Single point of failure
Performance bottleneck
Copyright infringement
So, later P2P systems were more distributed
Gnutella went to the other extreme…
File transfer is decentralized, but locating content is highly centralized

15. Pure P2P system: Gnutella
Query flooding
Join: contact a few nodes to become neighbors
Publish: no need!
Search: ask neighbors, who ask their neighbors
Fetch: get file directly from another node
Gnutella history
2000: J. Frankel & T. Pepper released Gnutella
Soon after: many other clients (e.g., Morpheus, Limewire, Bearshare)
2001: protocol enhancements, e.g., “ultrapeers”

16. Gnutella: Query Flooding
Fully distributed
No central server
Public domain protocol
Many Gnutella clients implementing protocol
Overlay network: graph
Edge between peer X and Y if there’s a TCP connection
All active peers and edges is overlay net
Given peer will typically be connected with < 10 overlay neighbors

17. Gnutella: Protocol Query message sent over existing TCP connections
File transfer:
HTTP
Query message sent over existing TCP connections
Peers forward Query message
QueryHit sent over reverse path
Query
QueryHit
Scalability:
limited scope flooding

18. Gnutella: Peer Joining
Joining peer X must find some other peers
Start with a list of candidate peers
X sequentially attempts TCP connections with peers on list until connection setup with Y
X sends Ping message to Y
Y forwards Ping message.
All peers receiving Ping message respond with Pong message
X receives many Pong messages
X can then set up additional TCP connections

19. Gnutella: Pros and Cons
Advantages
Fully decentralized
Search cost distributed
Processing per node permits powerful search semantics
Disadvantages
Search scope may be quite large
Search time may be quite long
High overhead, and nodes come and go often

20. Hybrid P2P system: KaAzA
KaZaA history
2001: created by Dutch company (Kazaa BV)
Smart query flooding
Join: on start, the client contacts a super-node (and may later become one)
Publish: client sends list of files to its super-node
Search: send query to super-node, and the super-nodes flood queries among themselves
Fetch: get file directly from peer(s); can fetch from multiple peers at once

21. KaZaA: Exploiting Heterogeneity
Each peer is either a group leader or assigned to a group leader
TCP connection between peer and its group leader
TCP connections between some pairs of group leaders
Group leader tracks the content in all its children

22. KaZaA: Motivation for Super-Nodes
Query consolidation
Many connected nodes may have only a few files
Propagating query to a sub-node may take more time than for the super-node to answer itself
Stability
Super-node selection favors nodes with high up-time
How long you’ve been on is a good predictor of how long you’ll be around in the future

23. P2P Case study: Skype inherently P2P: pairs of users communicate.
Skype clients (SC)
inherently P2P: pairs of users communicate.
proprietary application-layer protocol (inferred via reverse engineering)
hierarchical overlay with SNs
Index maps usernames to IP addresses; distributed over SNs
Skype
login server
Supernode
(SN)

24. Peers as relays Problem when both Alice and Bob are behind “NATs”.
NAT prevents an outside peer from initiating a call to insider peer
Solution:
Using Alice’s and Bob’s SNs, Relay is chosen
Each peer initiates session with relay.
Peers can now communicate through NATs via relay

25. BitTorrent BitTorrent history and motivation
2002: B. Cohen debuted BitTorrent
Key motivation: popular content
Popularity exhibits temporal locality (Flash Crowds)
Focused on efficient fetching, not searching
Distribute same file to many peers
Single publisher, many downloaders
Preventing free-loading

26. BitTorrent: Simultaneous Downloading
Divide large file into many pieces
Replicate different pieces on different peers
A peer with a complete piece can trade with other peers
Peer can (hopefully) assemble the entire file
Allows simultaneous downloading
Retrieving different parts of the file from different peers at the same time
And uploading parts of the file to peers
Important for very large files

27. BitTorrent: Tracker Infrastructure node
Keeps track of peers participating in the torrent
Peers register with the tracker
Peer registers when it arrives
Peer periodically informs tracker it is still there
Tracker selects peers for downloading
Returns a random set of peers
Including their IP addresses
So the new peer knows who to contact for data

28. BitTorrent: Chunks Large file divided into smaller pieces
Fixed-sized chunks
Typical chunk size of 256 Kbytes
Allows simultaneous transfers
Downloading chunks from different neighbors
Uploading chunks to other neighbors
Learning what chunks your neighbors have
Periodically asking them for a list
File done when all chunks are downloaded

29. BitTorrent: Overall Architecture
Web page
with link
to .torrent A B C
Peer
[Leech]
Downloader
“US”
[Seed]
Tracker
Web Server
.torrent

30. BitTorrent: Overall Architecture
Web page
with link
to .torrent A B C
Peer
[Leech]
Downloader
“US”
[Seed]
Tracker
Get-announce
Web Server

31. BitTorrent: Overall Architecture
Web page
with link
to .torrent A B C
Peer
[Leech]
Downloader
“US”
[Seed]
Tracker
Response-peer list
Web Server

32. BitTorrent: Overall Architecture
Web page
with link
to .torrent A B C
Peer
[Leech]
Downloader
“US”
[Seed]
Tracker
Shake-hand
Web Server

33. BitTorrent: Overall Architecture
Web page
with link
to .torrent A B C
Peer
[Leech]
Downloader
“US”
[Seed]
Tracker
pieces
Web Server

34. BitTorrent: Overall Architecture
Web page
with link
to .torrent A B C
Peer
[Leech]
Downloader
“US”
[Seed]
Tracker
pieces
Web Server

35. BitTorrent: Overall Architecture
Web page
with link
to .torrent A B C
Peer
[Leech]
Downloader
“US”
[Seed]
Tracker
Get-announce
Response-peer list
pieces
Web Server

36. BitTorrent: Chunk Request Order
Which chunks to request?
Could download in order
Like an HTTP client does
Problem: many peers have the early chunks
Peers have little to share with each other
Limiting the scalability of the system
Problem: eventually nobody has rare chunks
E.g., the chunks need the end of the file
Limiting the ability to complete a download
Solutions: random selection and rarest first

37. BitTorrent: Rarest Chunk First
Which chunks to request first?
The chunk with the fewest available copies
I.e., the rarest chunk first
Benefits to the peer
Avoid starvation when some peers depart
Benefits to the system
Avoid starvation across all peers wanting a file
Balance load by equalizing # of copies of chunks

38. Free-Riding Problem in P2P Networks
Vast majority of users are free-riders
Most share no files and answer no queries
Others limit # of connections or upload speed
A few “peers” essentially act as servers
A few individuals contributing to the public good
Making them hubs that basically act as a server
BitTorrent prevent free riding
Allow the fastest peers to download from you
Occasionally let some free loaders download

39. Bit-Torrent: Preventing Free-Riding
Peer has limited upload bandwidth
And must share it among multiple peers
Prioritizing the upload bandwidth: tit for tat
Favor neighbors that are uploading at highest rate
Rewarding the top four neighbors
Measure download bit rates from each neighbor
Reciprocates by sending to the top four peers
Recompute and reallocate every 10 seconds
Optimistic unchoking
Randomly try a new neighbor every 30 seconds
So new neighbor has a chance to be a better partner

40. BitTorrent Today Significant fraction of Internet traffic
Estimated at 30%
Though this is hard to measure
Problem of incomplete downloads
Peers leave the system when done
Many file downloads never complete
Especially a problem for less popular content
Still lots of legal questions remains
Further need for incentives

41. Structured overlay networks
Overlay topology construction is based on NodeID’s that are generated by using Distributed Hash Tables (DHT).
In this category, the overlay network assigns keys to data items and organizes its peers into a graph that maps each data key to a peer.
This structured graph enables efficient discovery of data items using the given keys.
It Guarantees object detection in O(log n) hops.
Examples: Content Addressable Network (CAN), Chord, Pastry. 41

42. Structured-DHT-based P2P overlay networks

43. CAN: Content Addressable Network
Each key maps to one point in the d-dimensional space
Each node responsible for all the keys in its zone.
Divide the space into zones. C D E A B

44. CAN(…) Fault tolerant:
know your neighbor’s neighbors. When node fails, one of the neighbor takes over.
If adjacent nodes fail at the same time, use flooding to discover topology
Node removal:
Use heartbeat
recover structure and repair routing (background zone reassignment)
expanding ring search to build neighbor information before repairing simultaneous failures
When joining, use the neighbor which is least loaded

45. Pastry
Prefix-based
Route to node with shared prefix (with the key) of ID at least one digit more than this node.
Neighbor set, leaf set and routing table.
d471f1
d467c4
d46a1c
d462ba
d4213f
Route(d46a1c)
d13da3
65a1fc

46. Pastry (…)
Routing table, leaf set and neighbor set example in a Pastry node
b=2 and l=8

47. Overlay Multicasting

48. The key decision behind IP Multicast
Gatech
Stanford
Berkeley
Per-group Router State
“Smart Network”

49. Internet Design Philosophy
Dumb routers , smart end systems ....
Minimal functionality in routers
Push functionality to end systems as much as possible
Key reason attributed to success and scaling of Internet IP
Application/
Transport
Internet architecture
Physical/
Link layer
“Thin Waist”

50. Significance of IP Multicast
First major functionality added to routers since original design
Per-group state implies:
Routing tables with potentially millions of entries!
Today’s routers have only tens of thousands of entries in routing tables though there are millions of end systems!
Potential scaling concerns
Per-group (per-flow) based solutions typically find slow acceptance in the Internet

51. Other concerns with IP Multicast
Slow deployment
Very difficult to change network infrastructure and routers
Difficult to support higher layer functionality
IP Multicast: Best effort delivery
Model complicates support for reliability and congestion control

52. Congestion control with IP Multicast
Stanford, LAN
Stanford, modem
Network with
IP Multicast
Purdue
Berkeley1
China
What rate must the source send at ?
London

53. Back to the drawing board…
Which is the right layer to implement multicast functionality?
IP Multicast: IP layer, efficient
Naïve Unicast: Application layer, inefficient
Can we do it efficiently at the application level ? IP
Application
Internet architecture
Physical/
Link ?

54. Overlay Multicast Dumb Network Overlay Tree Gatech Stan1 Stanford
Berk1
Berkeley
Overlay Tree
Berk2
Stan1
Gatech
Stan2
Purdue
Berk1
Berk2

55. Overlay Multicast: Benefits
No per-group state in routers
Easy to deploy: No change to network infrastructure
Can simplify support for congestion control etc.
Leverage computation and storage
(e.g. Transcoding)
Purdue
Stan-LAN
Berk1
Unicast congestion
control
Gatech
Stan-Modem
Berk2

56. Overlay Performance Dumb Network
Even a well-designed overlay cannot be as efficient as IP Mulitcast
But performance penalty can be kept low
Trade-off some performance for other benefits
Dumb Network
Gatech
Duplicate Packets:
Bandwidth Wastage
Stanford
Berkeley
Increased Delay

57. Architectural Spectrum
Purely application
end-point
Infrastructure-Centric
+Instantaneous Deployment
+Low setup overhead, low cost
+Uses bandwidth resources at end systems
+Can scale to support millions of groups
+ Security
+ Robustness

58. Router-Based
(IP Multicast)
No Router Support
Infrastructure-Centric
(CDNs, e.g. Akamai)
Application End-points
or end-systems with
infrastructure support
Application End-points Only,
End-System Only
End-System, Application-Level, Overlay
or Peer-to-Peer Multicast

59. Applications File Download Vs. Video Conferencing/Broadcasting
Bandwidth demanding : hundreds or thousands of Kbps
Real-time constraints: Continuous streaming delivery
Broadcasting Vs. Conferencing
Conferencing: smaller scale, anyone can be source.
Broadcasting: single source, tens or hundreds of thousands of simultaneous participants

60. Design Issues Construct efficient overlays Low Overheads
High bandwidth, low latency to receivers
Low Overheads
Fault tolerant
Self-organizing, Self-Improving
Honor per-node access bandwidth constraints
System Issues: NATs etc.

61. What overlay to construct?
Stan-LAN
Stan-LAN NJ
Purdue
Berk2
Stan-Modem
Berk1
Stan-LAN
Stan-Modem
Purdue
Berk1
Berk2
No formal structure
Each data pkt send using
epidemic algorithms
Overlay with a structure
Tree
Multi-Tree
Mesh

62. Inefficient Overlay Trees
Purdue
Purdue
High latency
Berk2
Gatech
Stan2
Stan1-Modem
Berk1
Stan2
Stan1-Modem
Berk1
Gatech
Berk2
-Poor network usage
-Potential congestion near Purdue
Gatech
Purdue
Berk2
Stan-Modem
Stan-LAN
Berk1
Poor bandwidth
to members

63. Efficient overlay trees
Purdue
Stan-Modem
Berk1
Stan-LAN
Gatech
Purdue
Berk2
Stan-LAN
Stan-Modem
Berk1
Berk2
Gatech

64. Self-Organizing Protocols
Construct efficient overlays in a distributed fashion
Members may have limited knowledge of the Internet
Adapt to dynamic join/leave/death of members
Adapt to network dynamics and congestion

65. Example
Purdue
Berk2
Berk1
Stan-Modem
Stan-Lan joins

66. Example
Purdue
Berk2
Berk1
Stan-Modem
Stan-Lan

67. Example
Purdue
Berk2
Berk1
Stan-Modem
Bad Bw Perf!
Switch!
Stan-Lan

68. Example
Purdue
Berk2
Berk1
Stan-Modem
Stan-Lan

69. Example More “clustering” if Stan-Modem moves to Stan-Lan Purdue Berk2

70. Example
Purdue
Berk2
Berk1
Stan-Modem
Stan-Lan

71. Stan-Modem is disconnected:
Example
Purdue
Berk2
Berk1
Stan-Modem
Stan-Modem is disconnected:
Back to Berk1

72. Key Components of Protocol
Overlay Management:
How many people does a member know?
How is this knowledge maintained?
Overlay Optimization:
Constructing efficient overlay among members
Distributed heuristics
No entity with global knowledge of hosts and network

73. Group Management
Build separate control structure decoupled from tree
Each member knows small random subset of group members
Knowledge maintained using gossip-like algorithm
Members also maintain path from source
Other design alternatives possible:
Example: More hierarchical structure
No clear winner between design alternatives S A

74. Bootstrap Node that joins:
Asia US2
Euro2
Euro3
Source (US)
Asia
US1
US2
Euro1
Modem1 US
US4, …
Euro2, Euro3, ...
Node that joins:
Gets a subset of group membership from source
Finds parent using parent selection algorithm

75. Other causes for parent switch
Member Leave/Death
Congestion/ poor bandwidth
Euro2, Euro3, ...
Modem1
Source (US)
US1
Asia
Euro1
US2
US4, …
Source (US)
Asia
US1
US2
Euro2
Modem1
US4, …
Euro3, ...
Source (US)
Asia
US1
Better Clustering
US2
Euro2
Modem1
US4, …
Euro3, ...
Euro4

76. Parent Selection X sends PROBE_REQ to subset of members it knows
Source (US)
Asia
US1
US2
X sends PROBE_REQ to subset of members it knows
Waits for 1 second for the responses
Evaluates remote nodes and chooses a candidate parent
Euro1
Modem1 X
US4, …
Euro2, Euro3, ...

77. Factors in Parent Selection
Filter out P if it is a descendant of X
Performance of P
Application throughput P received over last few seconds
Delay of path from S to P
Saturation level of P
Performance of link P-X
Delay of link P-X
TCP bandwidth of link P-X S P X

78. Multiple Overlay Trees
With support from MDC, split into T-equally sized stripes
T trees, each distributes a single stripe of size S/T
Overall quality depends on the number of stripes received
Number of trees node i is entitled to =
S Kbps
Peer A
Peer C
Source
S/3
S/3
S/3
Tree 1
Tree 2
Tree 3

79. Mesh-Based Approach Tree-based: Mesh-based: Involves “push”
Well-known structure
Mesh-based:
Each node maintains a set of partners
Exchange data availability info with partners
Pull data from partner if do not possess it already

81. Mesh-Based Streaming Vs. BitTorrent
Clever Scheduling Algorithm regarding which segments to fetch

82. Deployments Research Industrial (last 2-3 yrs)
ESM Broadcasting system (2002-)
CoolStreaming (2004-)
Report peak sizes >25000 users
Industrial (last 2-3 yrs)
PPLive
Reports even larger peak sizes
Zattoo
GridMedia

83. Research Challenges/Open Issues
Tree Vs. Mesh Approaches, Hybrids
Access-bandwidth scarce regimes
Incentives and fairness
Extreme peer dynamics, flash crowd
Support for heterogeneous receivers
System Issues
NATs/Firewalls
Transport Protocol
Start-up delay/buffer interaction
Security Challenges

84. Snapshot from Sigcomm Broadcast
U.S. East Coast
U.S. Central
U.S. West Coast
Europe
Asia
Unknown
Source