1. A Scalable Content-Addressable Network
University of Athens
Department of Informatics and Telecommunications
Postgraduate Program
Course: Distributed Systems
Supervisor: Professor Alex Delis
A Scalable Content-Addressable Network
In Proceedings of ACM SIGCOMM 2001
S. Ratnasamy, P. Francis, M. Handley, R. Karp, S. Shenker
Zacharias Boufidis
June 3rd, 2002

2. Scope CAN: Content-Addressable Network:
A distributed infrastructure for providing hash
table-like functionality on Internet-like scales
A distributed indexing scheme for mapping file names to location
Content-Addressable Network:
storing content at a “point” / routing between points
Department of Informatics and Telecommunications - University of Athens

3. Outline Motivation / Objectives
CAN Design: insertion, retrieval, deletion
Design Optimizations
Performance Evaluation
Related Work
Discussion
Department of Informatics and Telecommunications - University of Athens

4. Motivation - Applicability
P2P file sharing systems
- challenge: huge amount of resources
- drawback: centralized indexing scheme
(Napster: single point of failure/control, Gnutella: flooding => scalabilitty)
Large-scale storage management systems (e.g OceanStore)
- Efficient insertion and retrieval of content
Location-independent naming schemes (vs. DNS)
- Naming scheme decoupled from the name resolution process
Application-layer multicasting
- No need for distribution trees
- Scalability (group sizes larger than 1000 nodes, no single source
service model)

5. Objectives Indexing mechanism: totally distributed
scalable (per-node storage and communication costs grow slowly as the network size grows)
fault-tolerant
self-organizing
query speed (low-latency - efficient routing)
locality (queries for nearby nodes stay local)
guarantees for content retrieval
Department of Informatics and Telecommunications - University of Athens

6. Design Overview
Basic idea: a virtual d-dimensional Cartesian coordinate space dynamically partioned => zone per node
Data stored as <key,value> pairs
Insertion: Key k uniform hash function point P in a zone
Retrieval: Key k uniform hash function point P in a zone or
routing the request until it reaches the node in which
zone P lies

7. Assumptions
Associated DNS domain name - external mechanism upon joining
A priori knowledge of the deterministic hash function
Department of Informatics and Telecommunications - University of Athens

8. Design - CAN Routing
Routing table: IP address & virtual coordinate zone of neighbours (coordinate zones adjoining)
Greedy forwarding (recursively) of
msg[P(k), dst_coordinates]
to neighbour with coordinates closest to P
Department of Informatics and Telecommunications - University of Athens

9. Design - CAN Construction (1)
Node Insertion (of N1)
1. N1: find_anyCANnode(N2)
2. N1: choose_point(rand, P)
3. Route to N3 owning P
4. N1: JOIN request to N3
5. N3: zone_splitting(),
send(<key,value> from half_zone) to N1,
send(neighbout_info (IPaddr)) to N1,
inform_neighrbours
Pros: only neighbours affected
=> scalability

10. Design - CAN Construction (2)
Node departure
1. Explicit handover of a zone and <k,v> pairs
=> possible merge of zones
2. Node failure:
Robustness: immediate takeover based on metrics (smallest current zone, least-loaded node, connectivity, etc.)
(-) loss of <k,v> pairs
solution: periodic update messages
Expanding ring seach first to avoid inconsistencies
Background zone-reassignment
=> 1 zone/node

11. Design Considerations
Low per-node state [O(d), independent of n]
vs.
short path lengths [O(d (n^1/d)) hops]
d: dimensions, n: nodes
Design Goal:
lookup latency = (Avg. # CAN hops) x (Avg. per hop latency)
comparable to IP latency
=> techniques for reducing the CAN routing latency
=> (+) robustness (routing, data availability)
(-) per-node state
(-) system complexity

12. Design Improvements (1)
Multi-dimensioned coordinate spaces
- More dimensions => more neighbours per node
Goal: reduction of routing path length (# hops) => reduction of path latency
(+) routing fault tolerance
(-) more state per node (routing table)
Multiple realities
- More independent coordinate spaces => allocation of multiple
different zones per node (replication of hash tables on every reality)
=> selection of min CAN hop route => reduction of Avg. path length =>path latency
(+) data availability
(+) enhanced routing fault tolerance
(-) more state per node
The first solution outperforms in terms of routing efficiency

13. Design Improvements (2)
Refinement of CAN routing metrics
RTT-weighted routing:
- Goal: reduction of per-hop latency
metric: progress(Cartesian distance) / RTT
- Simulation results: 24-40% improvement
Department of Informatics and Telecommunications - University of Athens

14. Design Improvements (3)
Overloading coordinate zones
- When joining, zone sharing (if possible) instead of splitting
- State info: neighbour list + peer list
Selection of a peer’s neighbour based on measured lowest RTT
- Hash tables: replication vs. Partitioning
tradeoffs: availability, size of data stores, consistency
(+) Reduced # hops
(+) Reduced per hop latency
(+) Improved fault tolerance
(-) System complexity
(-) Additional control traffic
reduced path length
- Simulation results: 45% improvement of per-hop latency

15. Design Improvements (4)
Multiple hash functions
- Mapping a single key to multiple nodes (replication)
=> parallel queries
(+) Reduced query latency
(+) Increased data availability
(-) Increased size of the <key, value> database
(-) Increased query traffic
Topologically-sensitive CAN construction
- Node insertion based on RTT from landmarks (instead of
random insertion)
metric: latency stretch = CAN latency / network level latency
(+) Reduced path latency
(-) Uneven load distribution => need for load balancing

16. Design Improvements (5)
Uniform partitioning
- Volume-based zone splitting
(+) Achieves some form of load balancing
(-) «Hot spot» problem: some <key, value> pairs are more popular
Caching & Replication for «hot spot» management
- Caching of recently accessed keys (belonging to other nodes)
- Replication: active pushing out of popular keys to neighbours
within a region
(+) Increased data availability
(+) Reduced query latency
(+) Load balancing
Department of Informatics and Telecommunications - University of Athens

17. Performance Evaluation (1)
Critical factors:
1. increase in # of dimensions d
=> reduction of path length
2. Use of RTT-weighted routing
=> optimization of next-hop forwarding
=> reduction of path latency
Department of Informatics and Telecommunications - University of Athens

18. Performance Evaluation (2)
Effect of link delay distribution on latency stretch:
Remarks:
1. Increase in # of nodes
=> slow increase of latency stretch
(i.e., slow increase of total
path latency)
2. Random delay: the largest latency
stretch
3. Larger backbone
=> reduced density of CAN nodes => less effect of RTT-weighted
routing
=> degraded gains
Department of Informatics and Telecommunications - University of Athens

19. Related Work LS and DV routing Hierarchical routing (BGP)
(-) Widespread, frequent (in case of topology changes) dissemination of
topology info
Hierarchical routing (BGP)
(-) Fault tolerance
CAN: Self-configured routing
Plaxton’s Algorithm (~prefix-based routing)
- Similar routing hops & routing table size
(-) Neighbour discovery (global knowledge of topology)
Geographic routing
(-) Geographic forwarding needs real-world location service - difficulty in
implementing neighbour relationships (no radio coverage feature
available in CANs)

20. Conclusion Content-Addressable Networks - Added Value:
Scalable routing and efficient indexing
Completely distributed, self-organizing, fault tolerant
Department of Informatics and Telecommunications - University of Athens

21. Discussion (1) Paper Evaluation Plus: - Novelty
- Comparison to best rival scheme & other competing indexing schemes
Minus:
- Limited evaluation criteria
- Drawbacks of the proposed solution?
-“Chalk and Cheese”: indexing techniques & routing algorithms
Department of Informatics and Telecommunications - University of Athens

22. Discussion (2) Evaluation of indexing technique - Open issues
- Index length?
- Index construction time? (Convergence speed)
- Index construction = f(size of data store)?
- Index update cost?
- Extensibility? (types of queries supported, apart from keyword search)
- Memory/processing requirements?
- Effect of caches compared to “cold start”?
- Stability (concurrent joins) &Consistencies:
a) intermediate consistencies b) simultaneous updates?
- Ease of implementation?

23. Discussion (3) Evaluation of routing algorithm - Open issues
- Routing loops?
- Control of routing paths? (recursive routing)
- Routing of query responses?
Department of Informatics and Telecommunications - University of Athens

24. Discussion (4) Other Issues - Security (DoS attacks, AAA)
- Parameter tuning needed to achieve scalability (Cannot vary d as n
increases - n not known by any node)
- CAN maintenance protocol overhead? (Cost of update operation)
- Accommodation of administrative boundaries? (handling of key
value pairs?)
- Initial knowledge of the deterministic hash function?
Ways to be changed dynamically? Implications? (total
reconstruction of the CAN?)
- Specification of inter-update times, caching TTL values, etc.

25. Discussion (5) Enhancements
- Initialization of alternate paths from intermediate nodes
- Employment of hierarchical model or islands of CANs:
- The uniform manipulation fo CAN nodes does not represent the state of the
Internet - even P2P systems need not be purel P2P - some degree of loose
organization might be required
- Dynamic parameter tuning to achieve scalability & efficient routing
- Landmark hierarchal routing or GLS spatial hierarchy?
- Stronger coupling of routing algorithms to underlying topology
and node capability (some knowledge about the network could really help)
- Derivation of appropriate network-layer models
- QoS-routing based on CANs
(idea: hashing IPaddr & ToS field, use one reality per QoS metric, etc.)

26. Thank You!
Department of Informatics and Telecommunications - University of Athens