Implementing Declarative Overlays From two talks by: Boon Thau Loo 1 Tyson Condie 1, Joseph M. Hellerstein 1,2, Petros Maniatis 2, Timothy Roscoe 2, Ion.

Slides:

Advertisements

Similar presentations

P2: Implementing Declarative Overlays

Advertisements

Declarative Networking: Language, Execution and Optimization Boon Thau Loo 1, Tyson Condie 1, Minos Garofalakis 2, David E. Gay 2, Joseph M. Hellerstein.

Implementing Declarative Overlays Timothy Roscoe Joint work with Boon Thau Loo, Tyson Condie, Joseph M. Hellerstein, Petros Maniatis, Ion Stoica Intel.

Declarative Networking: Extensible Networks with Declarative Queries Boon Thau Loo University of California, Berkeley.

Intel Research Timothy Roscoe P2: Implementing Declarative Overlays Timothy Roscoe Boon Thau Loo, Tyson Condie, Petros Maniatis, Ion Stoica, David Gay,

Implementing Declarative Overlays Boon Thau Loo 1 Tyson Condie 1, Joseph M. Hellerstein 1,2, Petros Maniatis 2, Timothy Roscoe 2, Ion Stoica 1 1 University.

Declarative Networking Mothy Joint work with Boon Thau Loo, Tyson Condie, Joseph M. Hellerstein, Petros Maniatis, Ion Stoica Intel Research and U.C. Berkeley.

Intel Research Petros Maniatis, IRB Declarative Overlays Petros Maniatis joint work with Tyson Condie, David Gay, Joseph M. Hellerstein, Boon Thau Loo,

The Design and Implementation of Declarative Networks Boon Thau Loo University of Pennsylvania, University of California-Berkeley * *This dissertation.

Implementing declarative overlays Boom Thau Loo Tyson Condie Joseph M. Hellerstein Petros Maniatis Timothy Roscoe Ion Stoica.

P2P data retrieval DHT (Distributed Hash Tables) Partially based on Hellerstein’s presentation at VLDB2004.

Ion Stoica, Robert Morris, David Karger, M. Frans Kaashoek, Hari Balakrishnan MIT and Berkeley presented by Daniel Figueiredo Chord: A Scalable Peer-to-peer.

Peer to Peer and Distributed Hash Tables

Pastry Peter Druschel, Rice University Antony Rowstron, Microsoft Research UK Some slides are borrowed from the original presentation by the authors.

Peter Druschel, Rice University Antony Rowstron, Microsoft Research UK

Towards a Declarative Language and System for Secure Networking Martín Abadi 1,2, Boon Thau Loo 3 1 Microsoft Research Silicon Valley, 2 UC Santa Cruz,

CHORD – peer to peer lookup protocol Shankar Karthik Vaithianathan & Aravind Sivaraman University of Central Florida.

Chord: A Scalable Peer-to-peer Lookup Protocol for Internet Applications Speaker: Cathrin Weiß 11/23/2004 Proseminar Peer-to-Peer Information Systems.

Lecture 5 - Routing On the Flat Labels M.Sc Ilya Nikolaevskiy Helsinki Institute for Information Technology (HIIT)

Chord: A scalable peer-to- peer lookup service for Internet applications Ion Stoica, Robert Morris, David Karger, M. Frans Kaashock, Hari Balakrishnan.

Presented by Pierre-Élie Fauché 1 XL: An Efficient Network Routing Algorithm Kirill Levchenko Geoffrey M. Voelker, Ramamohan Paturi, and Stefan Savage.

Robert Morris, M. Frans Kaashoek, David Karger, Hari Balakrishnan, Ion Stoica, David Liben-Nowell, Frank Dabek Chord: A scalable peer-to-peer look-up.

Robert Morris, M. Frans Kaashoek, David Karger, Hari Balakrishnan, Ion Stoica, David Liben-Nowell, Frank Dabek Chord: A scalable peer-to-peer look-up protocol.

Pastry Peter Druschel, Rice University Antony Rowstron, Microsoft Research UK Some slides are borrowed from the original presentation by the authors.

Presented by Elisavet Kozyri. A distributed application architecture that partitions tasks or work loads between peers Main actions: Find the owner of.

Peer to Peer File Sharing Huseyin Ozgur TAN. What is Peer-to-Peer?  Every node is designed to(but may not by user choice) provide some service that helps.

Building diagnosable distributed systems Petros Maniatis Intel Research Berkeley ICSI – Security Crystal Ball.

Chord: A Scalable Peer-to-Peer Lookup Protocol for Internet Applications Stoica et al. Presented by Tam Chantem March 30, 2007.

Overview Distributed vs. decentralized Why distributed databases

Secure Overlay Services Adam Hathcock Information Assurance Lab Auburn University.

1 CS 194: Distributed Systems Distributed Hash Tables Scott Shenker and Ion Stoica Computer Science Division Department of Electrical Engineering and Computer.

CECS 474 Computer Network Interoperability WAN Technologies & Routing

Effizientes Routing in P2P Netzwerken Chord: A Scalable Peer-to- peer Lookup Protocol for Internet Applications Dennis Schade.

Using Queries for Distributed Monitoring and Forensics Atul Singh Rice University Peter Druschel Max Planck Institute for Software Systems Timothy Roscoe.

Declarative Routing: Extensible Routing with Declarative Queries UC Berkeley: Boon Thau Loo, Joseph M. Hellerstein, Ion Stoica. Intel Research: Joseph.

Chord & CFS Presenter: Gang ZhouNov. 11th, University of Virginia.

PIER & PHI Overview of Challenges & Opportunities Ryan Huebsch † Joe Hellerstein † °, Boon Thau Loo †, Sam Mardanbeigi †, Scott Shenker †‡, Ion Stoica.

Technical Refresher Session 3. Overview Difference between communication between devices on a single logical network and communication between different.

1 Reading Report 5 Yin Chen 2 Mar 2004 Reference: Chord: A Scalable Peer-To-Peer Lookup Service for Internet Applications, Ion Stoica, Robert Morris, david.

Chord: A Scalable Peer-to-peer Lookup Protocol for Internet Applications Xiaozhou Li COS 461: Computer Networks (precept 04/06/12) Princeton University.

1 Distributed Hash Tables (DHTs) Lars Jørgen Lillehovde Jo Grimstad Bang Distributed Hash Tables (DHTs)

Ion Stoica, Robert Morris, David Karger, M. Frans Kaashoek, Hari Balakrishnan MIT and Berkeley presented by Daniel Figueiredo Chord: A Scalable Peer-to-peer.

Chord: A Scalable Peer-to-peer Lookup Service for Internet Applications.

Chord: A Scalable Peer-to-peer Lookup Service for Internet Applications Ion Stoica, Robert Morris, David Karger, M. Frans Kaashoek, Hari Balakrishnan Presented.

Scalable Content- Addressable Networks Prepared by Kuhan Paramsothy March 5, 2007.

Peer to Peer A Survey and comparison of peer-to-peer overlay network schemes And so on… Chulhyun Park

A Utility-based Approach to Scheduling Multimedia Streams in P2P Systems Fang Chen Computer Science Dept. University of California, Riverside

Customizable Routing with Declarative Queries Boon Thau Loo 1 Collaborators: Joseph M. Hellerstein 1,2, Karthik Lakshminarayanan 1, Raghu Ramakrishnan.

Chord Advanced issues. Analysis Theorem. Search takes O (log N) time (Note that in general, 2 m may be much larger than N) Proof. After log N forwarding.

Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring Principles of Reliable Distributed Systems Lecture 2: Distributed Hash.

Chord Advanced issues. Analysis Search takes O(log(N)) time –Proof 1 (intuition): At each step, distance between query and peer hosting the object reduces.

An Energy-Efficient Geographic Routing with Location Errors in Wireless Sensor Networks Julien Champ and Clement Saad I-SPAN 2008, Sydney (The international.

CS 6401 Overlay Networks Outline Overlay networks overview Routing overlays Resilient Overlay Networks Content Distribution Networks.

Algorithms and Techniques in Structured Scalable Peer-to-Peer Networks

Declarative Routing: Extensible Routing with Declarative Queries Boon Thau Loo 1 Joseph M. Hellerstein 1,2, Ion Stoica 1, Raghu Ramakrishnan 3, 1 University.

Two Peer-to-Peer Networking Approaches Ken Calvert Net Seminar, 23 October 2001 Note: Many slides “borrowed” from S. Ratnasamy’s Qualifying Exam talk.

CS694 - DHT1 Distributed Hash Table Systems Hui Zhang University of Southern California.

ECE 544 Protocol Design Project 2016 Chengyao Wen Hua Deng Xiaoyu Duan.

The Cost of Inconsistency in Chord Shelley Zhuang, Ion Stoica, Randy Katz OASIS/i3 Retreat, January 2005.

Chord: A Scalable Peer-to-Peer Lookup Service for Internet Applications * CS587x Lecture Department of Computer Science Iowa State University *I. Stoica,

1 Distributed Hash tables. 2 Overview r Objective  A distributed lookup service  Data items are distributed among n parties  Anyone in the network.

The Chord P2P Network Some slides taken from the original presentation by the authors.

Declarative Networking: Language, Execution and Optimization

Declarative Networking

A Scalable Peer-to-peer Lookup Service for Internet Applications

Declarative Routing: Extensible Routing with Declarative Queries

Chord Advanced issues.

Chord Advanced issues.

Declarative Networking

COS 461: Computer Networks

Presentation transcript:

Implementing Declarative Overlays From two talks by: Boon Thau Loo 1 Tyson Condie 1, Joseph M. Hellerstein 1,2, Petros Maniatis 2, Timothy Roscoe 2, Ion Stoica 1 1 University of California at Berkeley, 2 Intel Research Berkeley

Overlays Everywhere… Overlay networks are widely used today: Routing and forwarding component of large-scale distributed systems Provide new functionality over existing infrastructure Many examples, variety of requirements: Packet delivery: Multicast, RON Content delivery: CDNs, P2P file sharing, DHTs Enterprise systems: MS Exchange Overlay networks are an integral part of many large-scale distributed systems.

Problem Non-trivial to design, build and deploy an overlay correctly: Iterative design process:  Desired properties  Distributed algorithms and protocols  Simulation  Implementation  Deployment  Repeat… Each iteration takes significant time and utilizes a variety of expertise

The Goal of P2 Make overlay development more accessible: Focus on algorithms and protocol designs, not the implementation Tool for rapid prototyping of new overlays: Specify overlay network at a high level Automatically translate specification to protocol Provide execution engine for protocol Aim for “good enough” performance Focus on accelerating the iterative design process Can always hand-tune implementation later

Outline Overview of P2 Architecture By Example Data Model Dataflow framework Query Language Chord Additional Benefits Overlay Introspection Automatic Optimizations Conclusion

6 All-Pairs Reachability R2: reachable(S,D)  link(S,Z), reachable(Z,D) R1: reachable(S,D)  link(S,D) Input: link(source, destination) Output: reachable(source, destination) “For all nodes S,D, If there is a link from S to D, then S can reach D”. link(a,b) – “there is a link from node a to node b” reachable(a,b) – “node a can reach node b”

7 All-Pairs Reachability R2: reachable(S,D)  link(S,Z), reachable(Z,D) R1: reachable(S,D)  link(S,D) Input: link(source, destination) Output: reachable(source, destination) “For all nodes S,D and Z, If there is a link from S to Z, AND Z can reach D, then S can reach D”.

8 All-Pairs Reachability R2: reachable(S,D)  link(S,Z), reachable(Z,D) R1: reachable(S,D)  link(S,D) Input: link(source, destination) Output: reachable(source, destination) “For all nodes S,D, If there is a link from S to D, then S can reach D”. link(a,b) – “there is a link from node a to node b” reachable(a,b) – “node a can reach node b”

9 All-Pairs Reachability R2: reachable(S,D)  link(S,Z), reachable(Z,D) R1: reachable(S,D)  link(S,D) Input: link(source, destination) Output: reachable(source, destination) “For all nodes S,D and Z, If there is a link from S to Z, AND Z can reach D, then S can reach D”.

10 Towards Network Datalog Specify tuple placement Value-based partitioning of tables Tuples to be combined are co-located Rule rewrite ensures body is always single-site All communication is among neighbors No multihop routing during basic rule execution Link-restricted rules: Enforced via simple syntactic restrictions

11 All-Pairs Reachability R1:  R2:  Network c d reachable Output table: Input @dc link c b b c reachable Location Specifier Query:

12 All-Pairs Reachability R2: reachable(S,D)  link(S,Z), reachable(Z,D) R1: reachable(S,D)  link(S,D) Input: link(source, destination) Output: reachable(source, destination) “For all nodes S,D, If there is a link from S to D, then S can reach D”. link(a,b) – “there is a link from node a to node b” reachable(a,b) – “node a can reach node b”

13 All-Pairs Reachability R2: reachable(S,D)  link(S,Z), reachable(Z,D) R1: reachable(S,D)  link(S,D) Input: link(source, destination) Output: reachable(source, destination) “For all nodes S,D and Z, If there is a link from S to Z, AND Z can reach D, then S can reach D”.

14 R1:  P=(S,D). R2:  2 ), P=S  P Query DP Query: @dc link bdca path Forwarding table:

Chord Model Relational data: relational tables and tuples Two kinds of tables: Stored, soft state:  E.g. neighbor(Src,Dst), forward(Src,Dst,NxtHop) Transient streams:  Network messages: message (Rcvr, Dst)  Local timer-based events: periodic (NodeID,10)

Example: Ring Routing Each node has an address and an identifier Each object has an identifier. Every node knows its successor Objects “served” by successor

Ring State node(IP 40,40) succ(IP 40,58,IP 58 ) node(IP 58,58) succ(IP 58,60,IP 60 ) Stored tables: node(NAddr, N) succ(NAddr, Succ, SAddr)

Example: Ring lookup Find the responsible node for a given key k? n.lookup(k) if k in (n, n.successor) return n.successor.addr else return n.successor. lookup(k)

lookup(IP 40,IP 37,59) Ring Lookup Events node(IP 40,40) succ(IP 40,58,IP 58 ) Event streams: lookup(Addr, Req, K) response(Addr, K, Owner) lookup(IP 37,IP 37,59) response(IP 37,59,IP 60 ) lookup(IP 58,IP 37,59) node(IP 58,58) succ(IP 58,60,IP 60 ) n.lookup(k) if k in (n, n.successor) return n.successor.addr else return n.successor. lookup(k)

Query Language: Overlog Datalog rule syntax: ,, …,. Overlog rule syntax: ,, …,.

Query Language: Overlog Event: RECEIVE lookup(NAddr, Req, K) Condition: lookup(NAddr, Req, K) & node(NAddr, N) & succ(NAddr, Succ, SAddr) & K in (N, Succ] Action: SEND response(Req, K, SAddr) to Req K, SAddr)  Req, K), N), Succ, SAddr), K in (N,Succ]. Overlog rule syntax: ,, …,.

P2-Chord Chord Routing, including: Multiple successors Stabilization Optimized finger maintenance Failure recovery 47 OverLog rules 13 table definitions Other examples: Narada, flooding, routing protocols 10 pt font

Introspection with Queries Unifying framework for debugging and implementation Same query language, same platform Execution tracing/logging Rule and dataflow level Log entries stored as tuples and queried Correctness invariants, regression tests as queries: “Is the Chord ring well formed?” (3 rules) “What is the network diameter?” (5 rules) “Is Chord routing consistent?” (11 rules) With Atul Singh (Rice) and Peter Druschel (MPI)

Automatic Optimizations Application of traditional Datalog optimizations to network routing protocols (SIGCOMM 2005) Multi-query sharing: Common “subexpression” elimination Caching and reuse of previously computed results Opportunistically share message propagation across rules Join lookup.Addr = node.Addr Join lookup.Addr = succ.Addr lookup Select K not in (N,Succ) Project lookup(SAddr, Req,K) lookup Project Response(Req, K,SAddr) Select K in (N,Succ) response Join lookup.Addr = node.Addr lookup Join lookup.Addr = succ.Addr

Automatic Optimizations Cost-based optimizations Join ordering affects performance Join lookup.Addr = node.Addr Join lookup.Addr = succ.Addr Select K not in (N,Succ) lookup Project lookup(SAddr,R eq,K) lookup Project Response(Req, K,SAddr) Select K in (N,Succ) response Join lookup.Addr = node.Addr Join lookup.Addr = succ.Addr

Future Work “Right” language Formal data and query semantics Static analysis Optimizations Termination Correctness

Conclusion P2: Declarative Overlays Tool for rapid prototyping new overlay networks Declarative Networks Research agenda: Specify and construct networks declaratively Declarative Routing : Extensible Routing with Declarative Queries (SIGCOMM 2005)