The Design and Implementation of Declarative Networks Boon Thau Loo University of Pennsylvania, University of California-Berkeley * *This dissertation was completed as a graduate student at the University of California- Berkeley

Declarative Networking A declarative framework for networks: Network protocols are declaratively specified using a database query language Distributed query engine executes specifications to implement network protocols Success of database research: 70’s – today: Database research has revolutionized data management Today: Similar opportunity to revolutionize the Internet architecture

Motivation Internet faces many challenges today: Unwanted, harmful traffic Complexity/fragility in Internet routing Proliferation of new applications and services Efforts at improving the Internet: Evolutionary: App-level “Overlay” networks Revolutionary: Clean-slate designs  NSF GENI initiative, FIND program Opportunity: Software tools that can significantly accelerate network innovation

A Declarative Network Distributed recursive query Traditional Networks Declarative Networks Network State Distributed database Network protocol Recursive Query Execution Network messages Distributed Dataflow Dataflow messages Dataflow messages

The Case for Declarative Networking Ease of programming: Compact and high-level representation of protocols Orders of magnitude reduction in code size  Declarative Chord DHT is 48 lines instead of 10,000. Easy customization Safety: Queries are “sandboxed” within query processor Potential for static analysis of safety What about efficiency? No fundamental overhead when executing standard routing protocols Application of well-studied query optimizations Note: Same question was asked of relational databases in the 70’s.

Main Contributions of Dissertation Declarative Routing [SIGCOMM ’05]: Extensible Routers: balance of flexibility, efficiency and safety Declarative Overlays [SOSP ’05]: Rapid prototyping of new overlay networks Database Fundamentals [SIGMOD ‘06] : Network specific query language and semantics Distributed recursive query execution strategies Query optimizations, classical and new

A Breadth of Use Cases Example implementations to date: Textbook routing protocols Chord DHT Narada mesh for end-system multicast Distributed Gnutella/Web crawlers Pastry DHT Replication protocols Lamport/Chandy snapshots Paxos distributed consensus Overlays for host mobility Sensor network protocols P2 declarative networking system

Outline Background The Connection: Routing as a Query Execution Model Path-Vector Protocol Example  Query specification  protocol implementation Query Processing Beyond routing: Declarative Overlays Ongoing Penn

Traditional Router Packets Traditional Router Control Plane Forwarding Plane Routing Protocol Neighbor Table updates Forwarding Table updates

Declarative RouterTraditional Router Declarative Router Declarative Queries Control Plane Forwarding Plane Packets Query Engine Routing Protocol Neighbor Table updates Forwarding Table updates

All-Pairs Reachability R2: reachable(S,D)  link(S,Z), reachable(Z,D) R1: reachable(S,D)  link(S,D) Input: link(source, destination) Output: reachable(source, destination) “For all nodes S,D, If there is a link from S to D, then S can reach D”. link(a,b) – “there is a link from node a to node b” reachable(a,b) – “node a can reach node b”

All-Pairs Reachability R2: reachable(S,D)  link(S,Z), reachable(Z,D) R1: reachable(S,D)  link(S,D) Input: link(source, destination) Output: reachable(source, destination) “For all nodes S,D and Z, If there is a link from S to Z, AND Z can reach D, then S can reach D”.

Towards Network Datalog Specify tuple placement Value-based partitioning of tables Tuples to be combined are co-located Rule rewrite ensures body is always single-site All communication is among neighbors No multihop routing during basic rule execution Link-restricted rules: Enforced via simple syntactic restrictions

All-Pairs Reachability R1:  R2:  Network c d reachable Output table: Input @dc link c b b c reachable Location Specifier Query:

Path Vector in Network Datalog Input: destination) Query output: destination, pathVector) R1:  P=(S,D). R2: P=S  P 2. 2 ),  Query: Add S to front of P 2

R1:  P=(S,D). R2:  2 ), P=S  P Query DP Query: @dc link bdca path Forwarding table:

@SDP DP Query Execution bdca R1:  P=(S,D). R2:  2 ), P=S  P 2. Query: @dc Communication patterns are identical to those in the actual path vector protocol Matching variable Z = “Join”

Other Routing Examples Best-Path Routing Distance Vector Dynamic Source Routing (Wireless) Policy Decisions QoS-based Routing Link-state Multicast Overlays (Single-Source & CBT)

Outline Background The Connection: Routing as a Query Query Processing Beyond routing: Declarative Overlays Sampling of ongoing work

Recursive Query Evaluation Semi-naïve evaluation: Iterations (rounds) of synchronous computation Results from iteration i th used in (i+1) th Path Table hop hop hop 4 Link TableNetwork Problem: Unpredictable delays and failures 9

Pipelined Semi-naïve (PSN) Fully-asynchronous evaluation: Computed tuples in any iteration pipelined to next iteration Natural for network protocols Path Table Link TableNetwork Relaxation of semi-naïve

Pipelined Evaluation Challenges: Does PSN produce the correct answer? Is PSN bandwidth efficient?  I.e. does it make the minimum number of inferences? Proofs for Basic technique: local timestamps p(x,z) :- p 1 (x,y), p 2 (y,z), …, p n (y,z), q(z,w) recursive w.r.t. p

Execution Plan Strands Messages Network In Messages Network Out Single Node Nodes in execution plan (“operators”): Network operators (send/recv, cc, retry, rate limitation) Relational operators (selects, projects, joins, aggregates) Flow operators (mux, demux, queues)

Localization Rewrite Rules may have body predicates at different locations: R2:  2 ), P=S  P 2. R2b:  2 ), P=S  P 2. R2a:  Matching variable Z = “Join” Rewritten rules: Matching variable Z = “Join”

Localized Rule Compilation Execution Plan  path Join path.Z = linkD.Z linkD Project path(S,D,P) Send to path.S R2b:  2 ), P=S  P 2. Network In Network Out  linkD Join linkD.Z = path.Z path Project path(S,D,P) Send to path.S

Optimizations Traditional: evaluate in the NW context Aggregate Selections Magic Sets rewrite Predicate Reordering New: motivated by NW context Multi-query optimizations:  Query Results caching  Opportunistic message sharing Cost-based optimizations  Neighborhood density function  Hybrid rewrites PV/DV (Wired)  DSR (Wireless) Hybridized protocols: Zone Routing Protocol

Beyond Routing: Declarative Overlays Language extensions to support events and soft-state predicates Chord Routing, including: Multiple successors Stabilization Optimized finger maintenance Failure detection 48 rules 11 table definitions MIT-Chord: x100 more code Another example: Narada mesh in 22 rules 10 pt font

Outline Background The Connection: Routing as a Query Query Processing Beyond routing: Declarative Overlays Ongoing Penn

Ongoing Penn Declarative secure networking Difficult to design/implement/reason about secure networks Network Datalog + logic-based security languages [NetDB ’07] Authenticated path vector protocol, DNSSEC, secure DHTs,… Moving forward:  Exploit fine-grained control over networks and security policies Data-centric querying and routing in heterogeneous networks Internet: Wired infrastructure with wireless clouds at the edges Flexible network support for mobility [ACM MobiArch ’07]  Declarative queries for addressing and naming mobile hosts  Session-aware customizable QoS routing Moving forward:  Declarative wireless ad-hoc networks  Cost-based query optimizations to adapt protocols

Summary Declarative networking: Declarative Routing:  Extensible routing infrastructure Declarative Overlays  Rapid prototyping overlay networks Database fundamentals  Query language  New distributed query execution strategies and optimizations  Semantics in dynamic networks P2 declarative networking system (

Many Thanks… Advisors: Joseph M. Hellerstein, Ion Stoica Collaborators: UC Berkeley: Tyson Condie, Ryan Huebsch Intel Research: David Gay, Petros Maniatis, Timothy Roscoe Yahoo! Research: Minos Garofalakis, Raghu Ramakrishnan Rice University: Atul Singh Many others…