Sven Bittner, 12 April 2007 Talk at the 5th New Zealand Computer Science Research Student Conference NEWS ALERT: (Kiwi or Cow) and Chainsaw = (Kiwi and Chainsaw) or (Cow and Chainsaw)? YEAH RIGHT...

2/23 Structure of Talk Motivation & ProblemMotivation & Problem Undertaken ResearchUndertaken Research –Filtering –Event Routing –Advertisements Current Steps & SummaryCurrent Steps & Summary Sven Bittner – Expressive Subscriptions and Advertisements in Pub/Sub Systems

3/23 Structure of Talk Motivation & ProblemMotivation & Problem Undertaken ResearchUndertaken Research –Filtering –Event Routing –Advertisements Current Steps & SummaryCurrent Steps & Summary Sven Bittner – Expressive Subscriptions and Advertisements in Pub/Sub Systems

4/23 Problem: Information Overflow More and more electronically available informationMore and more electronically available information Users only want information they are interested inUsers only want information they are interested in Motivation Research Filtering Routing Advertisements Summary

5/23 Solution: Publish/Subscribe Incoming information Information of interest Filtering of information (event messages) according to user interests (subscriptions) Motivation Research Filtering Routing Advertisements Summary

6/23 Pub/Sub Systems: Details Publishers Subscribers Pub/sub system Publish event messages Registersubscriptions Sendsnotifications B7B7B7B7 B9B9B9B9 Filtering and routing B1B1B1B1 B3B3B3B3 B4B4B4B4 B5B5B5B5 B8B8B8B8 Advertise future eventmessages Event routing table … … Subs. routing table … … Event routing table … … Subs. routing table … … … … … … Advertisem. and subscript. index structures B6B6B6B6 B2B2B2B2 Motivation Research Filtering Routing Advertisements Summary

7/23 Problem/Hypothesis Focus on conjunctive subscr./advert.Focus on conjunctive subscr./advert. Argument: Boolean forms can be converted to DNFArgument: Boolean forms can be converted to DNF –DNF exponential in size –Already many subscr./advert. without conversion Hypothesis [B06]Hypothesis [B06] –Direct support of Boolean form decreases memory usage without degrading efficiency Motivation Research Filtering Routing Advertisements Summary Original: (Kiwi or Cow) and Chainsaw DNF: (Kiwi and Chainsaw) or (Cow and Chainsaw)

8/23 Structure of Talk Motivation & ProblemMotivation & Problem Undertaken ResearchUndertaken Research –Filtering –Event Routing –Advertisements Current Steps & SummaryCurrent Steps & Summary Sven Bittner – Expressive Subscriptions and Advertisements in Pub/Sub Systems

9/23 Three Research Areas 1. Central filtering of arbitrary Boolean subscriptions 2. Event routing optimizations for arbitrary Boolean subscriptions 3. Support of arbitrary Boolean advertisements a) Calculation of overlappings b) Subscription routing optimization Motivation Research Filtering Routing Advertisements Summary

10/23 Central Filtering Algorithm (1) Utilization of one-dimensional indexesUtilization of one-dimensional indexes Extension of conjunctive counting approachExtension of conjunctive counting approach Three-step filtering [BH05a]Three-step filtering [BH05a] Incomingevent MatchingpredicatesCandidatesubscriptions Matchingsubscriptions PredicatematchingCandidatesubscriptionmatchingRealsubscriptionmatchingSubscriptionindexesPredicateindexesSubscriptions Motivation Research Filtering Routing Advertisements Summary

11/23 Central Filtering Algorithm (2) Evaluation [BH05b]Evaluation [BH05b] –Memory requirements Development of characterization schemeDevelopment of characterization scheme Theoretical analysis based on schemeTheoretical analysis based on scheme Practical verificationPractical verification –Efficiency Empirical experiments, similar/better resultsEmpirical experiments, similar/better results  Proves hypothesis for central components Motivation Research Filtering Routing Advertisements Summary

12/23 Structure of Talk Motivation & ProblemMotivation & Problem Undertaken ResearchUndertaken Research –Filtering –Event Routing –Advertisements Current Steps & SummaryCurrent Steps & Summary Sven Bittner – Expressive Subscriptions and Advertisements in Pub/Sub Systems

13/23 Event Routing Optimization (1) Subscription pruning [BH06a]Subscription pruning [BH06a] –Applicable to all kinds of subscriptions –Tailored for various target parameters (me- mory usage, filter efficiency, network load) [BH06c] Optimization ideaOptimization idea –Broadening of subscriptions by pruning –No effect on filtering accuracy (only internal)  Reduction of complexity of routing table entries Motivation Research Filtering Routing Advertisements Summary

14/23 Event Routing Optimization (2) Subscription … Neighbor N 1 N 2 N 3 … Un-optimized routing table Subscription … Neighbor N 1 N 2 N 3 … Optimized routing table Selected pruning on selected entries Motivation Research Filtering Routing Advertisements Summary

15/23 Event Routing Optimization (3) Analysis (empirical experiments)Analysis (empirical experiments) –Strong reduction in table size (e.g., by 80%) –Strong increase in throughput (e.g., by 50%)  Promising optimization effect Comparison to covering optimizationComparison to covering optimization –Stable optimization behavior of pruning –Applicable if other optimizations fail (both subscription structure and relationships)  Proves hypothesis for distributed setting Motivation Research Filtering Routing Advertisements Summary

16/23 Structure of Talk Motivation & ProblemMotivation & Problem Undertaken ResearchUndertaken Research –Filtering –Event Routing –Advertisements Current Steps & SummaryCurrent Steps & Summary Sven Bittner – Expressive Subscriptions and Advertisements in Pub/Sub Systems

17/23 Support of Advertisements (1) 1.Calculation of overlappings (two directions) –Overlapping subscriptions: all subscriptions that potentially match messages described by advertisement –Similar to matching algorithm, three steps Disjoint predicate matchingDisjoint predicate matching Candidate overlapping subscription matchingCandidate overlapping subscription matching Real overlapping subscription matchingReal overlapping subscription matching Motivation Research Filtering Routing Advertisements Summary

18/23 Support of Advertisements (2) 1.Calculation of overlappings (two directions) –Evaluation and comparison [BH06b] Similar efficiency for function problem (all overlappings)Similar efficiency for function problem (all overlappings) Higher performance (e.g., 85% more efficient) for decision problem (at least one overlapping)Higher performance (e.g., 85% more efficient) for decision problem (at least one overlapping) Motivation Research Filtering Routing Advertisements Summary

19/23 Support of Advertisements (3) 2.Advertisement-based optimization –Advertisement pruning [BH06d] First designated subscription routing optimizationFirst designated subscription routing optimization Pruning of advertisementsPruning of advertisements Target parameter: minimal increase of overlappings when pruningTarget parameter: minimal increase of overlappings when pruning –Evaluation Proposed measure fulfils design goalProposed measure fulfils design goal  Proves hypothesis for advertisements Motivation Research Filtering Routing Advertisements Summary

20/23 Structure of Talk Motivation & ProblemMotivation & Problem Undertaken ResearchUndertaken Research –Filtering –Event Routing –Advertisements Current Steps & SummaryCurrent Steps & Summary Sven Bittner – Expressive Subscriptions and Advertisements in Pub/Sub Systems

21/23 Current Steps Detailed experimental studyDetailed experimental study Writing upWriting up RedraftingRedrafting Redrafting…Redrafting… Motivation Research Filtering Routing Advertisements Summary

22/23 Summary ClaimClaim Pub/sub systems should not convert subscriptions and advertisements to DNF for time and space efficiency reasons ProofProof –Filtering algorithm (central system) –Event routing optimization (distributed system) –Support for advertisements (if used) Motivation Research Filtering Routing Advertisements Summary

23/23 Conclusions Pub/sub systems should directly work on Boolean expressions (as opposed to DBMSs)Pub/sub systems should directly work on Boolean expressions (as opposed to DBMSs) (Kiwi or Cow) and Chainsaw = (Kiwi and Chainsaw) or (Cow and Chainsaw)? Motivation Research Filtering Routing Advertisements Summary  Not the same in pub/sub, because Boolean form More space efficientMore space efficient More time efficientMore time efficient

Sven Bittner, Talk: Expressive Subscriptions and Advertisements in Pub/Sub Systems Selected further reading: [BH05a] S. Bittner and A. Hinze. On the Benefits of Non-Canonical Filtering in Publish/Subscribe Systems. In Proceedings of the 25th IEEE International Conference on Distributed Computing Systems Workshops (ICDCSW '05), Columbus, USA, June [BH05b] S. Bittner and A. Hinze. A Detailed Investigation of Memory Requirements for Publish/Subscribe Filtering Algorithms. In Proceedings of the 13th International Conference on Cooperative Information Systems (CoopIS 2005), Agia Napa, Cyprus, 31 October-4 November, [BH06a] S. Bittner and A. Hinze. Pruning Subscriptions in Distributed Publish/Subscribe Systems. In Proc. of the 29th Australasian Computer Science Conference (ACSC 2006), Hobart, Australia, January, [BH06b] S. Bittner and A. Hinze. Arbitrary Boolean Advertisements: The Final Step in Supporting the Boolean Pub/Sub Model. Technical Report 06/2006. Computer Science Department, Waikato University, June [BH06c] S. Bittner and A. Hinze. Dimension-Based Subscription Pruning for Publish/Subscribe Systems. In Proceedings of the 26th IEEE International Conference on Distributed Computing Systems Workshops (ICDCSW '06), Lisbon, Portugal, July [BH06d] S. Bittner and A. Hinze. Optimizing Pub/Sub Systems by Advertisement Pruning. In Proceedings of the 8th International Symposium on Distributed Objects and Applications (DOA 2006), Montpellier, France, 30 October-1 November [B06] S. Bittner. Supporting Arbitrary Boolean Subscriptions in Distributed Pub/Sub Systems. In Proceedings of the 3rd Intern. Middleware Doctoral Symposium (MDS 2006), Australia, November Thank you for your attention!