1. Small-Scale Peer-to-Peer Publish/Subscribe
Vinod Muthusamy, Hans-Arno Jacobsen
University of Toronto
July 17, 2005
P2PKM 2005 San Diego, CA

2. Small-Scale P2P Publish/Subscribe
Motivation
Many distributed hash table (DHT)-based peer-to-peer applications
Storage (OceanStore, Past)
Database (PIER)
Naming service (Chord-DNS)
Publish/subscribe (Scribe, P2P-ToPSS, Meghdoot)
Typically designed for and evaluated on a large-scale
Small-scale performance is important
Even large systems start small
Benefits even on small scale
Small-scale case study of a DHT-based pub/sub protocol
July 17, 2005 (P2PKM ’05)
Small-Scale P2P Publish/Subscribe

3. Small-Scale P2P Publish/Subscribe
Agenda
Background
Publish/subscribe model
Distributed hash tables
Small-scale benefits
Publish/subscribe on DHT
Evaluation
Conclusions
July 17, 2005 (P2PKM ’05)
Small-Scale P2P Publish/Subscribe

4. Publish/subscribe model
TSX
Stock markets
NASDAQ
NYSE
Simple interface
Decoupled interaction
Network efficiency
Publisher
Publisher
AMGN=58
Publications
IBM=84
ORCL=12
JNJ=58
HON=24
INTC=19
MSFT=27
Broker Network
Notification
Subscriptions
Subscriber
Subscriber
Subscriptions:
IBM > 85
ORCL < 10
JNJ > 60
July 17, 2005 (P2PKM ’05)
Small-Scale P2P Publish/Subscribe

5. Distributed hash tables
Example: Pastry
Distributed version of a hash table data structure
Store (key, value) pairs
Each node receives at most K/N keys
Each node has about O(logN) routing state
Lookups resolved with O(logN) hops
July 17, 2005 (P2PKM ’05)
Small-Scale P2P Publish/Subscribe

6. Small-Scale P2P Publish/Subscribe
Small-scale benefits
Distributed pub/sub
DHT
Large-scale benefits (traditional focus)
Multicast
Distributed matching
Organic (infrastructureless) scaling
Self-organizing
Small-scale benefits (untapped)
(also large-scale benefits)
Decoupled interaction
Declarative bindings
Incremental service deployment
Late binding
Effective use of commodity components (fault tolerance, load balancing)
Incremental scalability
July 17, 2005 (P2PKM ’05)
Small-Scale P2P Publish/Subscribe

7. Service oriented architecture (SOA)
An illustration of small-scale benefits
SOA: a distributed architecture
Loosely coupled reusable components
Enterprise service bus (ESB) comm. fabric
Can be realized as a pub/sub broker network
Pub/sub support in IBM Websphere, Sonic ESBs
Example
Three-tier Web server
Communication between components through pubs and subs
Decoupled interaction
Declarative bindings
Simple to add a Monitor component
Incremental service deployment
Late binding
Application
Server
Enterprise Service Bus
User
Products
Monitor
Database
Database
July 17, 2005 (P2PKM ’05)
Small-Scale P2P Publish/Subscribe

8. Small-Scale P2P Publish/Subscribe
Pub/sub case study
P2P-ToPSS
A pub/sub system built over DHT
Large-scale performance previously studied [under submission]
Evaluate small-scale benefits (of the same algorithm) now
Subscription load balance
July 17, 2005 (P2PKM ’05)
Small-Scale P2P Publish/Subscribe

9. Distributed multidimensional matching (DMM)
1) Pub/sub domain:
3) Network domain:
price < 50
weight > 50
Subscription
price = 10
weight = 70
Publication - 1
2) Spatial domain:
100 00 01 10 11 01 11 1
weight 00 10
DHT network used to map from tree nodes to peers
price
100
July 17, 2005 (P2PKM ’05)
Small-Scale P2P Publish/Subscribe

10. Small-Scale P2P Publish/Subscribe
Distributing the tree
Let region r have z-code z, and tree node n(r)
Store n(r) at peer p(r), where p(r) is where DHT would store key z
Each peer performs mapping locally
Hash function ensures even distribution
Network
price
weight
Region
Subscription
1101
z-code
price = …
weight = …
p/w
p_w_1101
name/code
AF3483BQ1
peer-id
hash
July 17, 2005 (P2PKM ’05)
Small-Scale P2P Publish/Subscribe

11. Small-Scale P2P Publish/Subscribe
Load balancing
Peers can be overloaded due to
Excessive subscriptions storage
Excessive publication traffic
Overloaded peer delegates subscriptions to children in the DMM tree
A local decision
Network 00 01
p/w
July 17, 2005 (P2PKM ’05)
Small-Scale P2P Publish/Subscribe

12. Small-Scale P2P Publish/Subscribe
Evaluation
SimPastry simulator (Microsoft Research)
Pastry DHT
Simulate network latencies, message costs
Nodes connected in a LAN
1ms latency between every pair of nodes
Subscriptions
Attribute name + range
E.g.: “20 < weight < 40”
Metric
Storage load of subscriptions
July 17, 2005 (P2PKM ’05)
Small-Scale P2P Publish/Subscribe

13. Small-Scale P2P Publish/Subscribe
DHT performance
1000 random keys
DHT successfully balances load
July 17, 2005 (P2PKM ’05)
Small-Scale P2P Publish/Subscribe

14. Small-Scale P2P Publish/Subscribe
Pub/sub performance
weight
price
1000 fine-grained subscriptions
Load balancing distributes storage load
July 17, 2005 (P2PKM ’05)
Small-Scale P2P Publish/Subscribe

15. Small-Scale P2P Publish/Subscribe
Pub/sub performance
weight
price
1000 coarse-grained subscriptions
Load balancing distributes but also increases storage load
July 17, 2005 (P2PKM ’05)
Small-Scale P2P Publish/Subscribe

16. Small-Scale P2P Publish/Subscribe
Observation
Load balancing technique assumes available resources somewhere in the network
Leads to runaway subscription delegation
Common assumption in DHT applications
CAN DHT’s active caching and replication
Meghdoot’s hotspot circumvention
In DHT, no knowledge of aggregate available resources
Fundamental assumption of DHT applications not valid in small-scale networks
July 17, 2005 (P2PKM ’05)
Small-Scale P2P Publish/Subscribe

17. Small-Scale P2P Publish/Subscribe
Conclusions
Pub/sub and DHTs
Proven benefits in large-scale network
Untapped benefits in small-scale networks
Common assumption on DHT applications
Sufficient available resources in the network
Small-scale networks break this assumption
Future work
Implement algorithms to estimate global resource availability
Curb runaway subscription delegation with this information
Evaluate other DHT applications in small-scale networks
July 17, 2005 (P2PKM ’05)
Small-Scale P2P Publish/Subscribe

18. Small-Scale Peer-to-Peer Publish/Subscribe
Q&A
Small-Scale Peer-to-Peer Publish/Subscribe
July 17, 2005 (P2PKM ’05)
Small-Scale P2P Publish/Subscribe