1. Sven Bittner and Annika Hinze, 31 October 2006 Talk at the 8th International Symposium on Distributed Objects and Applications (DOA 2006) Optimizing Publish/Subscribe Systems by Advertisement Pruning

2. 2/31 Context: Pub/Sub Systems Setting: Online book auctionsSetting: Online book auctions Sellers (publishers) Advertise future items Advertise future items Offer book items Offer book items Buyers (subscribers) Specify interests Specify interests Bid for items Bid for items Auctioning Site (pub/sub system) Filtering Filtering Routing Routing Notification Notification Motivation/Context Advert. Optimizations Advert. Pruning Experiments Summary

3. 3/31 Context: Advertisements Three types of specificationsThree types of specificationsAdvertisements –Describe future items (event messages) –Arbitrary Boolean combination of predicates (attribute-function-operand triples) –Sent by publishers title =“Harry Potter and the Half-Blood Prince” AND condition = NEWcondition = USED price > 10.0 price > 20.0 AND ORtitle =“Harry Potter and the Half-Blood Prince” AND condition = NEWcondition = USED price > 10.0 price > 20.0 AND OR Motivation/Context Advert. Optimizations Advert. Pruning Experiments Summary

4. 4/31 Context: Subscriptions Three types of specificationsThree types of specificationsSubscriptions –Describe interests –Arbitrary Boolean combination of predicates (attribute-function-operand triples) –Sent by subscribers title like“Harry Potter” AND condition = NEWcondition = USED price < 10.0 price < 15.0 AND OR AND condition = NEWcondition = USED price < 10.0 price < 15.0 AND OR condition = NEWcondition = USED price < 10.0 price < 15.0 AND ORendingWithin< 1 dayendingWithin< 1 day Motivation/Context Advert. Optimizations Advert. Pruning Experiments Summary

5. 5/31 Context: Event Messages Three types of specificationsThree types of specifications Event messages –Describe a state change/real-world event New and finished auctionsNew and finished auctions Biddings of subscribersBiddings of subscribers –Attribute-value pairs title: Harry Potter; endingWithin: 6 hours; condition: new; price: 15.00 Motivation/Context Advert. Optimizations Advert. Pruning Experiments Summary

6. 6/31 Context: Specification Overview Advertisements Event space Event messages Subscriptions Event space Motivation/Context Advert. Optimizations Advert. Pruning Experiments Summary

7. 7/31 Context: Routing Routing (event routing tables)Routing (event routing tables) –Distribution of subscriptions & advertisements –Subscriptions as routing entries (where to route event messages) –Advertisements state Where event messages will occur  Determine where to distribute subscriptions Motivation/Context Advert. Optimizations Advert. Pruning Experiments Summary

8. 8/31 Structure of Talk MotivationMotivation Advertisement-based OptimizationsAdvertisement-based Optimizations Advertisement PruningAdvertisement Pruning Evaluation of Advertisement PruningEvaluation of Advertisement Pruning Summary and OutlookSummary and Outlook Annika Hinze – Optimizing Publish/Subscribe Systems by Advertisement Pruning

9. 9/31 Concept: Overlapping (1) Subscriptions Event space Advertisements Subscription Subscription S Advertisement Advertisement A Overlapping Informal: At least one message described by A matches S Motivation/Context Advert. Optimizations Advert. Pruning Experiments Summary

10. 10/31 Concept: Overlapping (2) Function problemFunction problem –Determine all overlapping subscriptions for an advertisement –Or vice versa Decision problemDecision problem –Is there at least one overlapping? Overlappings determine subscription forwardingOverlappings determine subscription forwarding Motivation/Context Advert. Optimizations Advert. Pruning Experiments Summary

11. 11/31 Subscription S Overlapping: Application Motivation/Context Advert. Optimizations Advert. Pruning Experiments Summary Advertise-ments … … Subscription S Only forward to a neighbor if Only forward S to a neighbor if  overlapping from the respective neighbor  overlapping A from the respective neighborOverlappingadvertisement?Yes! Overlappingadvertisement?No!

12. 12/31 Optimization Goal Reduce size/amount of advertisementsReduce size/amount of advertisements Motivation/Context Advert. Optimizations Advert. Pruning Experiments Summary Advertise-ments Advertise-ments

13. 13/31 All advertisements Event space Subscription Optimization Requirements All advertisements Event space No/little effect on overlappingsNo/little effect on overlappings Little amount of additionally forwarded subscriptionsLittle amount of additionally forwarded subscriptions Motivation/Context Advert. Optimizations Advert. Pruning Experiments Summary Additionaloverlapping!  forwarding Subscription

14. 14/31 Current Optimization Approaches Merging and coveringMerging and covering –Merging: summarize advertisements –Covering: exploit subset relationships –Only for conjunctive advertisements (−) –NP-hard for Boolean expressions (−) –Analysis of all registered advertisements (−)  Not applicable for Boolean advertisements and subscriptions (−) Motivation/Context Advert. Optimizations Advert. Pruning Experiments Summary

15. 15/31 Structure of Talk MotivationMotivation Advertisement-based OptimizationsAdvertisement-based Optimizations Advertisement PruningAdvertisement Pruning Evaluation of Advertisement PruningEvaluation of Advertisement Pruning Summary and OutlookSummary and Outlook Annika Hinze – Optimizing Publish/Subscribe Systems by Advertisement Pruning

16. 16/31 Advertisement Pruning: Idea Generalize (broaden) advertisements individuallyGeneralize (broaden) advertisements individually  Efficiently to calculate for Boolean expressions (+)  Applicable regardless of registered advertisements (+) How to quantify the increase of overlappings? Motivation/Context Advert. Optimizations Advert. Pruning Experiments Summary Pruning?  Need characteristic measure!

17. 17/31 Overlapping Rank (1) Based on developed overlapping calculation algorithmBased on developed overlapping calculation algorithm –Algorithm uses non-overlapping predicates (=violating predicates) –Should incorporate number and effect of violating predicates Quantitative overlapping rankQuantitative overlapping rank –number of violating predicates Motivation/Context Advert. Optimizations Advert. Pruning Experiments Summary

18. 18/31 Overlapping Rank (2) Qualitative overlapping rankQualitative overlapping rank –Influence of violating predicates on overlappings –Assumption: The more selective a predicate in subscriptions, the higher its influence  Avoid removing highly selective violating predicates from advertisements Motivation/Context Advert. Optimizations Advert. Pruning Experiments Summary

19. 19/31 Overlapping Rank (3) Direct calculation for leaf nodes n lDirect calculation for leaf nodes n l Estimation based on Boolean operatorsEstimation based on Boolean operators –Three values –Three values rank min, rank avg, rank max Quantitative part (no. of subscriptions with p i ) Qualitative part (selectivity of p i ) Normalization (between 0 and 1) Motivation/Context Advert. Optimizations Advert. Pruning Experiments Summary

20. 20/31 Overlapping Rank Degradation Influence of a pruning operationInfluence of a pruning operation Relative change in overlapping rankRelative change in overlapping rank When pruning to :When pruning A i to A j :  The smaller, the better Motivation/Context Advert. Optimizations Advert. Pruning Experiments Summary

21. 21/31 Pruning: Application Registration: Determine “best” pruningRegistration: Determine “best” pruning Order advertisements by degradation (use priority queue)Order advertisements by degradation (use priority queue) Prune in this orderPrune in this order Motivation/Context Advert. Optimizations Advert. Pruning Experiments Summary Degradation Advertisement Insert O(1) * Delete O(log n) * * Assumes Fibonacci heap 0.001 A 76 0.005 A 43 0.01 A 123 0.02 A 98 0.6 A 5 ……………………

22. 22/31 Structure MotivationMotivation Advertisement-based OptimizationsAdvertisement-based Optimizations Advertisement PruningAdvertisement Pruning Evaluation of Advertisement PruningEvaluation of Advertisement Pruning Summary and OutlookSummary and Outlook Annika Hinze – Optimizing Publish/Subscribe Systems by Advertisement Pruning

23. 23/31 Experimental Setup Usage of semi-realistic data setUsage of semi-realistic data set –Identification of Eight advertisement classesEight advertisement classes Three subscription classesThree subscription classes –Classes define operator structure –Operands in predicates based on analysis of real-world book auctions (eBay) Motivation/Context Advert. Optimizations Advert. Pruning Experiments Summary

24. 24/31 Experimental Procedure Performing of prunings in order of rank degradationPerforming of prunings in order of rank degradation Analysis of influence onAnalysis of influence on –Space efficiency –Time efficiency Function problem (determining all overlapping subs)Function problem (determining all overlapping subs) Decision problem (determining existence of one sub)Decision problem (determining existence of one sub) –Amount of overlappings Motivation/Context Advert. Optimizations Advert. Pruning Experiments Summary

25. 25/31 Results: Most General Setting No prunings (un-optimized) All prunings Motivation/Context Advert. Optimizations Advert. Pruning Experiments Summary

26. 26/31 Results: Overview Cut-off Point Advertise- ment class Class 1 Class 2 Class 3 Class 4 Class 5 Class 6 Class 7 Class 8 Class 1 – 8 Time improve- ment (First, %) 1885561651382524 Decrease in memory (%) memory (%) 654880809135504449 Total increase in overlappings (%) 53111245215 Motivation/Context Advert. Optimizations Advert. Pruning Experiments Summary   

27. 27/31 Additional Experiments Simultaneous application of subscription and advertisement-based optimizationSimultaneous application of subscription and advertisement-based optimization ResultResult –Advertisement pruning better preserves overlappings –Detailed results in paper Motivation/Context Advert. Optimizations Advert. Pruning Experiments Summary

28. 28/31 Structure MotivationMotivation Advertisement-based OptimizationsAdvertisement-based Optimizations Advertisement PruningAdvertisement Pruning Evaluation of Advertisement PruningEvaluation of Advertisement Pruning Summary and OutlookSummary and Outlook Annika Hinze – Optimizing Publish/Subscribe Systems by Advertisement Pruning

29. 29/31 Summary (1) MotivationMotivation –Online auction scenario –Arbitrary Boolean advertisements/subscriptions Advertisement optimizationAdvertisement optimization –Only work for conjunctions –Strong assumptions and requirements –Idea: Independent handling of advertisements  Advertisement pruning Motivation/Context Advert. Optimizations Advert. Pruning Experiments Summary

30. 30/31 Summary (2) Advertisement PruningAdvertisement Pruning –Overlapping rank to characterize advertisements –Pruning based on overlapping rank degradation ExperimentsExperiments –Valuable optimization, general setting: 49% less memory49% less memory 24% more efficient24% more efficient Only 5% more overlappingsOnly 5% more overlappings Motivation/Context Advert. Optimizations Advert. Pruning Experiments Summary

31. 31/31 Future Work Experiments using other distributionsExperiments using other distributions Other heuristics for overlapping rankOther heuristics for overlapping rank Full support in prototypeFull support in prototype Motivation/Context Advert. Optimizations Advert. Pruning Experiments Summary

32. Thank you for your attention! Contact: Annika Hinze a.hinze@cs.waikato.ac.nz