1. Sven Bittner and Annika Hinze, 18 January 2006 Talk at the 29 th Australasian Computer Science Conference (ACSC2006) Pruning Subscriptions in Distributed Publish/Subscribe Systems

2. 2/29 Motivation: Publish/Subscribe Subscribers register subscriptionsSubscribers register subscriptions Publishers send event messagesPublishers send event messages System informs using notificationsSystem informs using notifications EBay TradeMeUserDistributedPub/SubSystem pub(item,price,timeLeft,…) pub(item,price,timeLeft,…) Notify about items of interest Subscription pub(item,...)Filtering Annika Hinze – Pruning Subscriptions in Distributed Publish/Subscribe Systems

3. 3/29 A subscriber is interested in books whose title contains the phrase “Harry Potter”.A subscriber is interested in books whose title contains the phrase “Harry Potter”. Motivation: Subscription Example title like “Harry Potter” AND condition = NEWcondition = USED price < 10.0 price < 15.0 AND ORendingWithin < 1 day According to the condition of the copy of the book (new, used), she wants to pay at most NZ$10.0 or NZ$15.0.According to the condition of the copy of the book (new, used), she wants to pay at most NZ$10.0 or NZ$15.0. To avoid unnecessary notifications, the subscriber will be notified not earlier than one day before the auction ends.To avoid unnecessary notifications, the subscriber will be notified not earlier than one day before the auction ends. Annika Hinze – Pruning Subscriptions in Distributed Publish/Subscribe Systems

4. 4/29 Motivation: Problem Sizes Online auctionsOnline auctions –Subscriptions: > 10 6 (no. of users) –Events: > 20 / sec (new items and bids) –Notifications: not time-critical, but events must be processed permanently Facility managementFacility management –Subscriptions: > 50,000 (today’s systems) –Events: > 1,000 / sec (from sensors, switches) –Notifications: delay < 0.1 sec  Time and space efficiency required Annika Hinze – Pruning Subscriptions in Distributed Publish/Subscribe Systems

5. 5/29 Structure MotivationMotivation Subscription PruningSubscription Pruning Selectivity EstimationSelectivity Estimation Evaluation of Subscription PruningEvaluation of Subscription Pruning Summary and OutlookSummary and Outlook Annika Hinze – Pruning Subscriptions in Distributed Publish/Subscribe Systems

6. 6/29 Structure MotivationMotivation Subscription PruningSubscription Pruning Selectivity EstimationSelectivity Estimation Evaluation of Subscription PruningEvaluation of Subscription Pruning Summary and OutlookSummary and Outlook Annika Hinze – Pruning Subscriptions in Distributed Publish/Subscribe Systems

7. 7/29 Current Optimizations Work on conjunctive subscriptions only (−)Work on conjunctive subscriptions only (−)  Restricted subscription language  Not applicable for general-purpose systems Strong assumptions (−)Strong assumptions (−) –Similarities/relationships among subscriptions –Evaluations too simplistic for high-level applications  We cannot generalize evaluation results Motivation Subscription Pruning Selectivity Estimation Evaluation Outlook

8. 8/29 Subscription Generalization Routing optimization for arbitrary Boolean subscriptions (+)Routing optimization for arbitrary Boolean subscriptions (+) Optimizes subscriptions independently of each other (+)Optimizes subscriptions independently of each other (+)  Optimization potential regardless of individual and collective subscription structure  Favourable routing optimization for general- purpose pub/sub systems Motivation Subscription Pruning Selectivity Estimation Evaluation Outlook

9. 9/29 Generalization by Pruning Goals of pruningGoals of pruning –Remove parts of subscription tree of non-local subscribers –Create more general subscription  Less predicates to filter on (+)  Less complex subscriptions (+)  More events to filter (−)  More time and space efficient filtering Motivation Subscription Pruning Selectivity Estimation Evaluation Outlook

10. 10/29 Application of Pruning (1) Forwarding of subscriptions for selective routingForwarding of subscriptions for selective routing Un-optimized routing: Routing table Subscriber Motivation Subscription Pruning Selectivity Estimation Evaluation Outlook

11. 11/29 Application of Pruning (2) Forwarding of subscriptions for selective routingForwarding of subscriptions for selective routing Routing table Less complex subscriptions  More time and space efficient filtering Subscriber Motivation Subscription Pruning Selectivity Estimation Evaluation Outlook

12. 12/29 Application of Pruning (3) Forwarding of subscriptions for selective routingForwarding of subscriptions for selective routing Routing table But more general subscriptions  More forwarded event messages  More event messages to filter Subscriber Motivation Subscription Pruning Selectivity Estimation Evaluation Outlook

13. 13/29 Example of Pruning (1) Valid pruning - Remove child of conjunctionValid pruning - Remove child of conjunction title like“Harry Potter”endingWithin< 1 day condition = NEWprice < 15.0 AND OR condition = USED AND OR title like“Harry Potter”endingWithin< 1 day condition = NEW price < 10.0 price < 15.0 AND OR condition = USED AND OR Remove unary operators Motivation Subscription Pruning Selectivity Estimation Evaluation Outlook

14. 14/29 Example of Pruning (2) Invalid pruning - Remove child of disjunctionInvalid pruning - Remove child of disjunction title like“Harry Potter”endingWithin< 1 day condition = NEWprice < 15.0 AND title like“Harry Potter”endingWithin< 1 day condition = NEW price < 10.0 price < 15.0 AND OR condition = USED AND OR Remove unary/ Summarize consecutive operators No filtering of used books anymore! Motivation Subscription Pruning Selectivity Estimation Evaluation Outlook

15. 15/29 Challenges of Pruning QuestionsQuestions 1.Which subscription should be pruned first? 2.Which part of a subscription should be pruned? AnswerAnswer The subscription (1.) supporting a pruning (2.) that minimally influences the network traffic  Utilize selectivities of subscriptions to determine effects of pruning on network load Motivation Subscription Pruning Selectivity Estimation Evaluation Outlook

16. 16/29 Structure MotivationMotivation Subscription PruningSubscription Pruning Selectivity EstimationSelectivity Estimation Evaluation of Subscription PruningEvaluation of Subscription Pruning Summary and OutlookSummary and Outlook Annika Hinze – Pruning Subscriptions in Distributed Publish/Subscribe Systems

17. 17/29 Selectivity of Subscriptions Calculation of selectivity forCalculation of selectivity for –Original subscriptions - Counting –Predicates - Counting/Approximation –Pruned subscriptions - No suitable method  Estimate selectivities for subscriptions Motivation Subscription Pruning Selectivity Estimation Evaluation Outlook

18. 18/29 Selectivity Estimation: Idea Using three easily computable estimatesUsing three easily computable estimates –Minimal sel min Worst case - smallest possible selectivity value for all distributions of events –Average sel avg Average case - assuming uniform distribution of all possible event messages and independent predicates in subscriptions –Maximal sel max Best case - largest possible selectivity value for all distributions of events Motivation Subscription Pruning Selectivity Estimation Evaluation Outlook

19. 19/29 Selectivity Estimation: Example Selectivity of predicates via countingSelectivity of predicates via counting Selectivity of subscriptions via estimationSelectivity of subscriptions via estimation title like “Harry Potter”endingWithin < 1 day condition = NEWcondition = USED price < 10.0 price < 15.0 AND OR (0.7, 0.72, 0.8)(0.13, 0.19, 0.2) (0.7, 0.77, 1.0) (0.0, 7.7e-4, 0.01) 0.20.930.90.8 0.010.1 Motivation Subscription Pruning Selectivity Estimation Evaluation Outlook

20. 20/29 Selectivity Degradation Absolute degradation when pruning s x to s yAbsolute degradation when pruning s x to s y –Describes expected influence on network load –Maximal difference between three components max(sel min (s y ) - sel min (s x ), sel avg (s y ) - sel avg (s x ), sel max (s y ) - sel max (s x )) Motivation Subscription Pruning Selectivity Estimation Evaluation Outlook

21. 21/29 Structure MotivationMotivation Subscription PruningSubscription Pruning Selectivity EstimationSelectivity Estimation Evaluation of Subscription PruningEvaluation of Subscription Pruning Summary and OutlookSummary and Outlook Annika Hinze – Pruning Subscriptions in Distributed Publish/Subscribe Systems

22. 22/29 Experiments: Goal Evaluate general settingEvaluate general setting –Real-world subscriptions –Real-world attribute domains Initial set of experimentsInitial set of experiments –Evaluation of memory usage and real selectivity –Real selectivity shows expected network load Motivation Subscription Pruning Selectivity Estimation Evaluation Outlook

23. 23/29 Experiments: Setup E-commerce setting (online book auctions)E-commerce setting (online book auctions) –Ten attributes, e.g., author, format and price EventsEvents –Analysis of real-world distributions –Average for 1,000,000 messages SubscriptionsSubscriptions –Three typical classes involving conjunctions and disjunctions –10,000 registered subscriptions Motivation Subscription Pruning Selectivity Estimation Evaluation Outlook

24. 24/29 Experiments: Results (1) Setting involving all three subscription classesSetting involving all three subscription classes Cut-off point  Memory usage Expected increase in network load Motivation Subscription Pruning Selectivity Estimation Evaluation Outlook

25. 25/29 At cut-off point (Column 4)At cut-off point (Column 4) –Slight increase in selectivity (Column 2) –Strong reduction in memory usage (Column 3) Experiments: Results (2) Subscriptionclass Class 1 Class 2 Class 3 Class 1–3 Increase in selectivity0.0090.0120.0160.026 Relief in memory 0.6670.8330.3680.663 Cut-off point at percent of prunings 0.7500.8750.5250.771 Motivation Subscription Pruning Selectivity Estimation Evaluation Outlook

26. 26/29 Structure MotivationMotivation Subscription PruningSubscription Pruning Selectivity EstimationSelectivity Estimation Evaluation of Subscription PruningEvaluation of Subscription Pruning Summary and OutlookSummary and Outlook Annika Hinze – Pruning Subscriptions in Distributed Publish/Subscribe Systems

27. 27/29 Summary (1) MotivationMotivation –Publish/subscribe (pub/sub) systems –Routing and optimizations in pub/sub Subscription pruningSubscription pruning –Drawbacks of current optimizations –Prune/remove parts of subscription trees –Pruning has to result in more general subscription Motivation Subscription Pruning Selectivity Estimation Evaluation Outlook

28. 28/29 Summary (2) Selectivity estimationSelectivity estimation –Three values easily computable values –Degradation measure  predicted influence of prunings Practical analysisPractical analysis –Evaluation of real-world scenario –Setting with all subscriptions Space usage decreased by 66% of maximal reductionSpace usage decreased by 66% of maximal reduction Only slight increase in network loadOnly slight increase in network load Motivation Subscription Pruning Selectivity Estimation Evaluation Outlook

29. 29/29 Future Work Integrate pruning in pub/sub prototypeIntegrate pruning in pub/sub prototype Extended experimentsExtended experiments –Measure network load, throughput and memory usage –Other real-world scenarios Motivation Subscription Pruning Selectivity Estimation Evaluation Outlook

30. Thank you for your attention! Contact: Annika Hinze a.hinze@cs.waikato.ac.nz