Efficient Distribution-Based Event Filtering Annika Hinze, Sven Bittner Institute of Computer Science Freie Universität Berlin {hinze, Talk at the workshop on Distributed Event-based Systems At the Conference ICDCS

Efficient Distribution-Based Event Filtering 2/18 Event Notification Service Client Repository Profile Repository Event Message Repository Notification Provider Invoker Repository Information Object ObserverFilterNotifier Introduction: Components of an ENS

Efficient Distribution-Based Event Filtering 3/18 Introduction: Filtering Algorithms - Related work  Performance analysis of filter algorithms  Algorithms: - Tree-based algorithm by Gough et al [gough95] - Tree-based algorithm by Aguilera et al [aguilera99] - Counting algorithm [yan94] - Equality-preferred algorithm [fabret00] - Inequality-preferred algorithm [fabret00] … -Sequentially-test-all algorithm  Evaluations based on equally distributed data

Efficient Distribution-Based Event Filtering 4/18 Introduction: Filtering Algorithms – Problem statement  Applications: Environmental Monitoring: Events: Sensor readings produce wide range of data Profiles: Catastrophe Warning (not likely, but important) Logistics Support: Events: Car Locations, accident information Profiles: diverse, filtering+routing  Extreme distributions for events and profiles -High peaks for small range -Gauss distributed data, …

Efficient Distribution-Based Event Filtering 5/18 Introduction: Filtering Algorithms – Problem statement  Filter algorithms: performance differs with distributions!  Goal: Improving filter performance for certain applications based on the typical distributions

Efficient Distribution-Based Event Filtering 6/18 Roadmap  Motivation  Recap: Tree-based Algorithm  Performance Model  Measures for filter algorithms  Test results  Conclusions from testing  Outlook

Efficient Distribution-Based Event Filtering 7/ P4 P3 P1 P2 P Recap: Tree-based Algorithm  Fastest Filter Algorithm [gough95] : Profiles: P1[temp=30; hum=80] P2[temp=35; hum=85] P3[temp=20; hum=80] P4[temp=20; hum=90] P5[temp=35; hum=95] For each attribute sequentially test all branches Event: E[temp=20; hum=90] P4

Efficient Distribution-Based Event Filtering 8/18 Model  Performance measured in comparison steps  Events and profiles: certain probability P e, P p  Profile values naturally ordered in tree  For simplicity:  Profiles  possible attribute values (D 0 =  ) #steps = expectation for X = E(X) =  x i P e (x i ) xixi  Event distribution of one attribute a: -Event modeled as value of discrete random variable X -Values of X: x i :ordering number of attribute values

Efficient Distribution-Based Event Filtering 9/18 Model  Distributions of n attributes not independent  Conditional distributions: e.g., temperature and humidity related  # steps for each attribute a j : E (X j |X j-1,..,X 1 )  ( E (X j |X j-1,..,X 1 ) ) =  x i j P e (x i j | x j-1,..,x 1 ) jj  #steps for all attributes:  Strong dependence on order -Within each tree level -Of attributes (conditional distributions !)

Efficient Distribution-Based Event Filtering 10/18 Measures for filter algorithm Influence algorithm by V - Reorder edges according to value selectivity 1.Event distribution 2.Profile distribution 3.Profile distribution * event distribution A - Reorder nodes according to attribute selectivity 1.# values without profiles / # all values 2.Probability of values without profiles [simplified] 3.# values without profiles - conditional distributions

Efficient Distribution-Based Event Filtering 11/18 Roadmap  Motivation  Recap: Tree-based Algorithm  Performance Model  Measures for filter algorithm  Test results  Conclusions from testing  Outlook

Efficient Distribution-Based Event Filtering 12/18 Test setting  Algorithm implementation in Java  Simulated reordering  All permutations of 60 distributions  8 different orderings plus binary search  4 Test Groups: -Tree of 5,000 /10,000 profiles, tests until 95% precision -Full profile tree (n attributes), tests until 95% precision -Full profile tree, one attribute only, 4000 events -Full profile tree, one attribute only, all possible events

Efficient Distribution-Based Event Filtering 13/18 Test results: Value reordering

Efficient Distribution-Based Event Filtering 14/18 Conclusions from testing: Value reordering  V1 Event distribution -Faster than binary search if E (X ) < log 2 (#profiles) holds for event-distribution  V2 Profile distribution and  V3 Profile distribution * Event distribution -Faster notifications for profiles with high priority -Inferior average response time according to events -No queue if filter in optimal working point (f events  f filter )

Efficient Distribution-Based Event Filtering 15/18 Test results: Attribute reordering

Efficient Distribution-Based Event Filtering 16/18 Conclusions from testing: Attribute reordering  Idea: Reject unmatched events early  A1 # values without profiles / # all values -Faster for equally distributed data  A2 Probability of values without profiles -Faster for all event distributions  A3 conditional distributions -Faster for all event distributions -Costly to obtain, but best measure -Only sensible for applications with stable distributions

Efficient Distribution-Based Event Filtering 17/18 Conclusion  Fast distribution-dependent algorithm  Introduction of value-dependent and attribute- dependent selectivity measures  Tests show usefulness of measures  Introduction of criteria for reordering circumstances under which reordering improves performance

Efficient Distribution-Based Event Filtering 18/18 Outlook  Distribution-dependent Adaptive Service based on -Predefined distribution for events in application -History of events (to determine the event distribution)  Currently: implementation of GENAS  System adapts to -Distributions -Event semantics -Temporal correctnes

Thank you for your attention! Contact: Annika Hinze Sven Bittner

Efficient Distribution-Based Event Filtering 20/18 Bibliography: My Publications [hinze02] A.Hinze, A.Voisard: Composite Events in Notification Systems with Application to Logistics Support, submitted for review to Caise [faensen01] D. Faensen, L.Faulstich, H.Schweppe, A. Hinze, and A. Steidinger: Hermes -- A Notification Service for Digital Libraries, In Proceedings of the ACM/IEEE Joint Conference on Digital Libaries (JCDL), 2001, Roanoke, VA, USA, June [hinze01] A.Hinze: Does Alerting have special Requirements for Query Languages? in Martin Endig, Thomas Herstel (Hrsg.): Tagungsband zum13. GI-Workshop Grundlagen von Datenbanken.Uni. Magdeburg, Preprint Nr. 10, Juni 2001 [hinze01a] A. Hinze: How does the observation strategy influence the correctness of alerting services? Proceedings of the BTW 2001, Oldenburg, March [schweppe00] H.Schweppe, A. Hinze, D.Faensen: Event-based Notification on the Web, Tutorial at the 9th WWW9, May 15-19,2000, Amsterdam [schweppe00a] H.Schweppe, A. Hinze, D.Faensen: Database Systems as Middleware – Events, Notification, Messages. ADBIS-DASFAA 2000: [hinze99] A. Hinze, D.Faensen: A Unified Model of Internet Scale Alerting Services, Proceedings of the ICSC, December 13-15, 1999, Hong Kong, ©Springer-Verlag. [hinze99a] A. Hinze and H.Schweppe: Notification Services in Digital Libraries, Proc. of the Dagstuhl Seminar "Multimedia Database Support for Digital Libraries", August 1999, Dagstuhl [hinze99b] A. Hinze, D.Faensen: A Unified Model of Internet Scale Alerting Services, Technical Report, Number tr-b-99-15, Freie Universität Berlin, 1999 [hinze99c] A. Hinze: Alerting Services in a Digital Library Environment, Doctoral Consortium at CAiSE, 1999, Heidelberg, published as Technical Report, ETH Zürich [hinze99d] A. Hinze: A Model of Alerting Services in Wide Area Networks, in F.Hüsemann, K.Küspert, F.Mäurer (Eds.): 11. Workshop "Grundlagenvon Datenbanken", Luisenthal, Mai [faensen98] D.Faensen, A.Hinze, H.Schweppe: Alerting in a Digital Library environment - Do Channels meet the requirements? In Second ECDL, number 1513 inLNCS, ©Springer-Verlag. [faensen98a] D.Faensen, A.Hinze, H.Schweppe: Alerting in a Digital Library environment - Do Channels meet the requirements? TR, Number tr-b-98-08, Inst. of Computer Science, Freie Universität Berlin, 1998

Efficient Distribution-Based Event Filtering 21/18 Bibliography: Filtering Algorithms [hanson90] E.Hanson, M.Chaabouni, C.Kim, and Y.Wang: A predicate matching algorithm for database rule systems“, SIGMOD, 1990 [gough95] K.Gough and G.Smith: „Efficient Recognition of events in distriuted systems“, Proceedings of the ACSC-18, 1995 [aguilera99] M.K. Aguilera, R.Strom, D.Sturman, M.Astley, and T.Chandra: “Matching events in a content-based subscription system”, 18 th ACM PODC, 1999 [fabret00] F. Fabret, F. Llirbat, J. Pereira, and D. Shasha: “Efficient matching for content-based publish/subscribe systems", technical Report, INRIA, 2000