Republishers in a Publish/Subscribe Architecture for Data Streams Alasdair J G Gray and Werner Nutt School of Mathematical and Computer Sciences, Heriot-Watt University, Edinburgh 6 th July 2005

A.J.G. Gray and W. NuttBNCOD222 Overview  Motivation  Publish/Subscribe Architecture  Query planning

6 th July 2005A.J.G. Gray and W. NuttBNCOD223 Motivation Scenario:  Streams generated by distributed sensors  Users are also distributed  Use data integration to match users to streams For example,  Grid monitoring for logging and bookkeeping  Sensor networks Grid Job progress Bookkeeping Monitoring data

6 th July 2005A.J.G. Gray and W. NuttBNCOD224 Data Streams as Relations  Sensor readings can be viewed as tuples conforming to a relational schema  Example: Network ThroughPut NTP(from,to,tool,psize,latency,timestamp) ('hw','ral','ping',32,11.1, :05:34)

6 th July 2005A.J.G. Gray and W. NuttBNCOD225 Publish/Subscribe Architecture  Local as View Approach Consumers pose a query over the schema to request streams Producers describe their stream using a view on the schema  Queries and views are selections over a single relation Producers Registry Data Streams Consumers

6 th July 2005A.J.G. Gray and W. NuttBNCOD226 Query Planning: Consumer Query S 1 : from = 'hw' Λ tool = 'udp' S 2 : from = 'hw' Λ tool = 'ping' S 3 : from = 'ral' Λ tool = 'ping' S 4 : from = 'ral' Λ tool = 'udp' C: from = 'hw' Λ psize ≥ 1024 Problem: Approach does not scale to hundreds of producers and consumers.

6 th July 2005A.J.G. Gray and W. NuttBNCOD227 Republishers Provide Scalability S 1 : from = 'hw' Λ tool = 'udp' S 2 : from = 'hw' Λ tool = 'ping' S 3 : from = 'ral' Λ tool = 'ping' S 4 : from = 'ral' Λ tool = 'udp' R 3 : TRUE R 1 : from = 'hw'R 2 : from = 'ral' C: from = 'hw' Λ psize ≥ 1024

6 th July 2005A.J.G. Gray and W. NuttBNCOD228 Plans Need to be Maintained  Queries are long lived  Set of publishers can change  Query plans should reflect changes  What happens when we add a republisher? remove a republisher?

6 th July 2005A.J.G. Gray and W. NuttBNCOD229 Adding a Republisher: 1 st Attempt S 1 : from = 'hw' Λ tool = 'udp' S 2 : from = 'hw' Λ tool = 'ping' S 3 : from = 'ral' Λ tool = 'ping' S 4 : from = 'ral' Λ tool = 'udp' R 3 : TRUE R 1 : from = 'hw'R 2 : from = 'ral'R 4 : tool = 'ping' Problem: Republishers disconnected from Producers Cycle in data flow Relevant publishersMaximal relevant Replan R 3 Replan other queriesAdding a new publisher

6 th July 2005A.J.G. Gray and W. NuttBNCOD2210 Desirable Properties for a Hierarchy  Correctness: streams answer queries  Cycle freeness: loops can lead to duplicates  Uniqueness: hierarchy defined for a set of publishers  Local planning: Publishers and Consumers only need to communicate with the Registry

6 th July 2005A.J.G. Gray and W. NuttBNCOD2211 Adding a Republisher: 2 nd Attempt S 1 : from = 'hw' Λ tool = 'udp' S 2 : from = 'hw' Λ tool = 'ping' S 3 : from = 'ral' Λ tool = 'ping' S 4 : from = 'ral' Λ tool = 'udp' R 3 : TRUE R 1 : from = 'hw'R 2 : from = 'ral'R 4 : tool = 'ping' C: from = 'hw' Λ psize ≥ 1024 Relevant publishers

6 th July 2005A.J.G. Gray and W. NuttBNCOD2212 Removing a Republisher S 1 : from = 'hw' Λ tool = 'udp' S 2 : from = 'hw' Λ tool = 'ping' S 3 : from = 'ral' Λ tool = 'ping' S 4 : from = 'ral' Λ tool = 'udp' R 3 : TRUE R 1 : from = 'hw'R 2 : from = 'ral'R 4 : tool = 'ping' C: from = 'hw' Λ psize ≥ 1024

6 th July 2005A.J.G. Gray and W. NuttBNCOD2213 Conclusions  Republishers: Allow system to scale Complicate query answering problem  Republishers require special planning  We have developed algorithms that allows the system to adapt to changes in the set of publishers  Full details available in HW Technical Report sp?id=0031

6 th July 2005A.J.G. Gray and W. NuttBNCOD2214 Integrating Data Streams  Local as View Approach Consumers pose a query over the schema to request streams Producers describe their stream using a view on the schema  Queries and views are selections over a single relation

6 th July 2005A.J.G. Gray and W. NuttBNCOD2215 Example S 1 : from = 'hw' Λ tool = 'udp' S 2 : from = 'hw' Λ tool = 'ping' S 3 : from = 'ral' Λ tool = 'ping' S 4 : from = 'ral' Λ tool = 'udp' R 3 : TRUE R 1 : from = 'hw'R 2 : from = 'ral'R 4 : tool = 'ping' C: from = 'hw' Λ psize ≥ 1024