1. T-110.455 Network Application Frameworks and XML Distributed Hash Tables (DHTs) and multi-homing 16.2.2005 Sasu Tarkoma Based on slides by Pekka Nikander

2. Contents n Introduction u Layered model, routing table vs. address n Multi-homing n Overlay networks n Summary

3. Introduction n A name determines the node / service n An address gives its location u May change in mobility n A path determines how to get there n Routing determines the path from an address n Directories convert names to addresses n You can think of routing tables and directories as distributed databases

4. Layered-model revisited Object API Firewall bypass End-to-end Routing Congestion control Presentation DNS names IP addresses Routing paths XML presentation Finding, meta-data, invoking, syncing, mobility. Web Services

5. Topology in address vs. routing table Reactive AD HOC (MANET) routing Pure source routing ATM PNNI CIDR Original IP routing Proactive ad hoc (MANET) routing Host-based hop-by-hop

6. Identity/Locator split Process Transport ID Layer IP Layer Link Layer identifier locator n New name space for IDs u Maybe based on DNS u Maybe a separate namespace u Maybe IP addresses are used for location u Good for hiding IP versions n Communication end- points (sockets) bound to identifiers

7. Multi-addressing n Multi-homing u Server has multiple addresses on different networks for increased reliability u Client has multiple addresses n Multi-homing requires support for address change n Topology change can cause renumbering u From old prefix to new prefix u Changing the IP host addresses of each device within the network u Related with multi-homing and must be supported by mobility protocols

8. Multi- homing Mobility One host Single subnet Parts of topology All hosts End-host multihoming SoHo site multihoming enterprise multihoming Moving multi-homed networks ad hoc networks End-host mobility Moving networks (NEMO) reliability, load- sharing HIP, SCTP Mobile IP, HIP, SCTP Site multihoming challenge: prefixes create additional state MULTI6 in IETF for IPv6 multihoming

9. ISP1 ISP2 Multi-homed site NAT1 NAT2 Site multi-homing Multi-homing Examples Wireless Host Internet WLAN GPRS End-host multi-homing

10. Multi-homing n End-host multihoming u Transport-level (SCTP) F Good when there is a connection F Needs mobility management for boot-strapping u Locator / Identity split F Host Identity Protocol F Natural way to decouple topological location/locations and identity F Needs directories to solve bootstrapping problems n We need directories for coping with mobility and multi-homing u Scalability, low latency updates

11. Overlays n Motivation n Terminology n Overlays u Clusters and wide-area n Applications for DHTs u Internet Indirection Infrastructure (i3) u Delegation Oriented Architecture (DOA)

12. Motivation for Overlays n Directories are needed u Name resolution u Mobility support F Fast updates n Required properties u Fast updates u Scalability n DNS has limitations u Update latency u Administration n One solution u Overlay networks

13. Terminology n Peer-to-peer (P2P) u Different from client-server model u Each peer has both client/server features n Overlay networks u Routing systems that run on top of another network, such as the Internet. n Distributed Hash Tables (DHT) u An algorithm for creating efficient distributed hash tables (lookup structures) u Used to implement overlay networks n Typical features of P2P / overlays u Scalability, resilience, high availability, and they tolerate frequent peer connections and disconnections

14. Peer-to-peer in more detail n A P2P system is distributed u No centralized control u Nodes are symmetric in functionality n Large faction of nodes are unreliable u Nodes come and go n P2P enabled by evolution in data communications and technology n Current challenges: u Security, IPR issues n P2P systems are decentralized overlays

15. Evolution of P2P systems n Started from centralized servers u Napster F Centralized directory F Single point of failure n Second generation used flooding u Gnutella F Local directory for each peer F High cost, worst-case O(N) messages for lookup n Research systems use DHTs u Chord, Tapestry, CAN,.. u Decentralization, scalability

16. Challenges n Challenges in the wide-area u Scalability u Increasing number of users, requests, files, traffic u Resilience: more components -> more failures u Management: intermittent resource availability -> complex management

17. Overlay Networks n Origin in Peer-to-Peer (P2P) n Builds upon Distributed Hash Tables (DHTs) n Easy to deploy u No changes to routers or TCP/IP stack u Typically on application layer n Overlay properties u Resilience u Fault-tolerance u Scalability

18. Upper layers Overlay Congestion End-to-end Routing DNS names, custom identifiers Overlay addresses IP addresses Routing paths

19. Cluster vs. Wide-area n Clusters are u single, secure, controlled, administrative domains u engineered to avoid network partitions u low-latency, high-throughput SANs u predictable behaviour, controlled environment n Wide-area u Heterogeneous networks u Unpredictable delays and packet drops u Multiple administrative domains u Network partitions possible

20. Wide-area requirements n Easy deployment n Scalability to millions of nodes and billions of data items n Availability u Copes with routine faults n Self-configuring, adaptive to network changes n Takes locality into account

21. Distributed Data Structures (DSS) n DHTs are an example of DSS n DHT algorithms are available for clusters and wide-area environments u They are different! n Cluster-based solutions u Ninja u LH* and variants n Wide-area solutions u Chord, Tapestry,.. u Flat DHTs, peers are equal u Maintain a subset of peers in a routing table

22. Distributed Data Structures (DSS) n Ninja project (UCB) u New storage layer for cluster services u Partition conventional data structure across nodes in a cluster u Replicate partitions with replica groups in cluster F Availability u Sync replicas to disk (durability) n Other DSS for data / clusters u LH* Linear Hashing for Distributed Files u Redundant versions for high-availability

23. Cluster-based Distributed Hash Tables (DHT) n Directory for non-hierarchical data n Several different ways to implement n Each “brick” maintains a partial map n Overlay addresses used to direct to the right “brick” n Resilience through parallel, unrelated mappings

24. client service DSS lib service DSS lib storage “brick” storage “brick” storage “brick” storage “brick” storage “brick” storage “brick” SAN Service interacts with DSS lib Hash table API Redundant, low latency, high throughput network Brick = single- node, durable hash table, replicated clients interact with any service “front-end”

25. DHT interfaces n DHTs offer typically two functions u put(key, value) u get(key) --> value u delete(key) n Supports wide range of applications u Similar interface to UDP/IP F Send(IP address, data) F Receive(IP address) --> data n No restrictions are imposed on the semantics of values and keys n An arbitrary data blob can be hashed to a key n Key/value pairs are persistent and global

26. Distributed applications Distributed Hash Table (DHT) Node put(key, value)get(key) value DHT balances keys and data across nodes

27. Some DHT applications n File sharing n Web caching n Censor-resistant data storage n Event notification n Naming systems n Query and indexing n Communication primitives n Backup storage n Web archive

28. Chord n Chord is an overlay algorithm from MIT u Stoica et. al., SIGCOMM 2001 n Chord is a lookup structure (a directory) u Resembles binary search n Uses consistent hashing to map keys to nodes u Keys are hashed to m-bit identifiers u Nodes have m-bit identifiers F IP-address is hashed u SHA-1 is used as the baseline algorithm n Support for rapid joins and leaves u Churn u Maintains routing tables

29. Chord routing I n A node has a well determined place within the ring n Identifiers are ordered on an identifier circle modulo 2 m u The Chord ring n A node has a predecessor and a successor n A node stores the keys between its predecessor and itself u The (key, value) is stored on the successor node of key n A routing table (finger table) keeps track of other nodes

30. Finger Table n Each node maintains a routing table with at most m entries n The i:th entry of the table at node n contains the identity of the first node, s, that succeeds n by at least 2 i-1 on the identifier circle n s = successor(n + 2 i-1 ) u The i:th finger of node n

31. N1 N8 N14 N21 N32 N38 N42 N51 N56 2 m -1 0 +1 +2 +4 +8 +16 +32 FingerMaps to Real node 1,2,3 4 5 6 x+1,x+2,x+4 x+8 x+16 x+32 N14 N21 N32 N42 m=6 for j=1,...,m the fingers of p+2 j-1 Predecessor node

32. Chord routing II n Routing steps u check whether k is found between n and the successor of n u if not, forward the request to the closest finger preceding k n Each know knows a lot about nearby nodes and less about nodes farther away n The target node will be eventually found

33. Chord lookup N1 N8 N14 N21 N32 N38 N42 N51 N56 2 m -1 0 m=6

34. Chord Properties n Each node is responsible for K/N keys (K is the number of keys, N is the number of nodes) n When a node joins or leaves the network only O(K/N) keys will be relocated n Lookups take O(log N) messages n To re-establish routing invariants after join/leave O(log 2 N) messages are needed

35. Tapestry n DHT developed at UCB u Zhao et. al., UC Berkeley TR 2001 n Used in OceanStore u Secure, wide-area storage service n Tree-like geometry n Suffix - based hypercube u 160 bits identifiers n Suffix routing from A to B u hop(h) shares suffix with B of length digits u 1245 routes to 5432 via F 5432 -> 3235 -> 1245 -> 6245 -> 1245 n Tapestry Core API: u publishObject(ObjectID,[serverID]) u routeMsgToObject(ObjectID) u routeMsgToNode(NodeID)

36. Pastry I n A DHT based on a circular flat identifier space n Prefix-routing u Message is sent towards a node which is numerically closest to the target node u Procedure is repeated until the node is found u Prefix match: number of identical digits before the first differing digit u Prefix match increases by every hop n Similar performance to Chord

37. Pastry II n Routing a message u If leaf set has the prefix --> send to local u else send to the identifier in the routing table with the longest common prefix (longer then the current node) u else query leaf set for a numerically closer node with the same prefix match as the current node u Used in event notification (Scribe)

38. Security Considerations n Malicious nodes u Attacker floods DHT with data u Attacker returns incorrect data F self-authenticating data u Attacker denies data exists or supplies incorrect routing info n Basic solution: using redundancy n What if attackers have quorum? u Need a way to control creation of node Ids u Solution: secure node identifiers F Use public keys

39. Applications for DHTs n DHTs are used as a basic building block for an application-level infrastructure u Internet Indirection Infrastructure (i3) F New forwarding infrastructure based on Chord u DOA (Delegation Oriented Architecture) F New naming and addressing infrastructure based on overlays

40. Internet Indirection Infrastructure (i3) n A DHT - based overlay network u Based on Chord n Aims to provide more flexible communication model than current IP addressing n Also a forwarding infrastructure u i3 packets are sent to identifiers u each identifier is routed to the i3 node responsible for that identifier u the node maintains triggers that are installed by receivers u when a matching trigger is found the packet is forwarded to the receiver

41. i3 II n An i3 identifier may be bound to a host, object, or a session n i3 has been extended with ROAM u Robut Overlay Architecture for Mobility u Allows end hosts to control the placement of rendezvous-points (indirection points) for efficient routing and handovers u Legacy application support F user level proxy for encapsulating IP packets to i3 packets

42. Source: http://i3.cs.berkeley.edu/ R inserts a trigger (id, R) and receives all packets with identifier id. the host changes its address from R1 to R2, it updates its trigger from (id, R1) to (id, R2). Mobility is transparent for the sender

43. Source: http://i3.cs.berkeley.edu/ A multicast tree using a hierarchy of triggers

44. Source: http://i3.cs.berkeley.edu/ Anycast using the longest matching prefix rule.

45. Source: http://i3.cs.berkeley.edu/ Sender-driven service composition using a stack of identifiers Receiver-driven service composition using a stack of identifiers

46. DOA (Delegation Oriented Architecture) n Proposed to circumvent the harmful side- effects of middleboxes n Middleboxes are used to improve scalability, but they also constrain the Internet architecture n Two changes u persistent host identifiers u a mechanism for resolving these n Each host has a unique 160 bit EID (End-point Identifier) n Works in a similar fashion to HIP u EID hash of a public key, identity/locator split, u Self-certification

47. DOA II n DOA header is located between IP and TCP headers and carries source and destination EIDs n The mapping service maps EIDs to IP addresses n This allows the introduction of various middleboxes to the routing of packets n Outsourcing of intermediaries u Ideally clients may select the most useful network elements to use n Differences to i3 u Not a forwarding infrastructure u i3 does not provide a primitive for specifying intermediaries

48. Layered-model Object API Firewall bypass End-to-end Routing Congestion control Presentation IP addresses Routing paths Host Identity, public key

49. Summary n Mobility and multi-homing require directories u Scalability, low-latency updates n Overlay networks have been proposed u Searching, storing, routing, notification,.. u Lookup (Chord, Tapestry, Pastry), coordination primitives (i3), middlebox support (DOA) u Logarithmic scalability, decentralised,… n Host/identity split and overlays for directories seem good candidates for solving many of the issues with mobility and multi-homing