1. Adaptive Ordering of Pipelined Stream Filters S. Babu, R. Motwani, K. Munagala, I. Nishizawa, and J. Widom In Proc. of SIGMOD 2004, June 2004

2. Outline Introduction Ordering of filters Adaptive ordering algorithm Preliminaries Algorithms Experimental evaluation Conclusion

3. Introduction Why “Stream”?  Many modern applications deal with data that Many modern applications Is updated continuously Needs to be processed in real-time Why “Adaptive”?  Arrival characteristics of streams may vary significantly over time Characteristic of filters  Direct  Commutative

4. Ordering of filters Challenges  Selectivities across filters may be correlated.  An exhaustive algorithm becomes infeasible. Greedy approach  Arrival characteristics of streams may vary significantly over time Adaptive approach

5. Adaptive ordering algorithm Three-way tradeoff  Run-time overhead Monitor the changes of statistics Determine when to change the current order  Convergence properties If data and filter characteristics stabilize, adaptive algorithm should converge to a solution with desirable properties  Speed of adaptivity The time the convergence takes when data and filter characteristics change

6. Preliminaries symbolmeaning query input stream filters stream tuple ordering conditional drop probability Processing time per tuple total cost

7. Algorithms A-GREEDY algorithm The SWEEP algorithm The INDEPENDENT algorithm The LOCALSWAP algorithm

8. (Static) Greedy algorithm Greedy approach

9. A-GREEDY Profiler

10. A-GREEDY Reoptimizer violation

11. A-GREEDY Algorithm (1/2)

12. A-GREEDY Algorithm (2/2)

13. A-GREEDY properties Convergence  Good  Run-time overhead  Profile-tuple creation  Profile-window maintenance  Matrix-view update  Detection and correction of GI violations Adaptivity  rapidly

14. Algorithms A-GREEDY algorithm The SWEEP algorithm The INDEPENDENT algorithm The LOCALSWAP algorithm

15. The SWEEP Algorithm Rotating over

16. SWEEP properties Convergence  Good Run-time overhead  A-Greedy:  Sweep: Adaptivity  Violations may remain undetected for a relatively long time  Up to stages

17. Algorithms A-GREEDY algorithm The SWEEP algorithm The INDEPENDENT algorithm The LOCALSWAP algorithm

18. The INDEPENDENT Algorithm Assume the filters are independent  frequently in database literature  seldom true in practice 

19. INDEPENDENT properties Convergence  independent converge to the optimal ordering  dependent can be times worse than the GI ordering Run-time overhead  lower than A-Greedy Adaptivity  rapidly

20. Algorithms A-GREEDY algorithm The SWEEP algorithm The INDEPENDENT algorithm The LOCALSWAP algorithm

21. The LOCALSWAPS Algorithm Detect  a swap between adjacent filters in would improve performance 

22. LOCALSWAP properties Convergence  dependent on the way characteristics change  in some case, times higher cost than GI ordering Run-time overhead  lower than A-Greedy Adaptivity  may take more time to converge  may get stuck in a local optima

23. Experimental evaluation Three parts  Convergence experiments  Run-time overhead experiments  Adaptivity experiments

24. Convergence experiments Factors  Number of filters  Filter selectivities  Cost of filters  Correlation among filters Comparison  Optimal  A-Greedy  Independent

25. Convergence experiments (number) Factors  Number of filters

26. Convergence experiments (selectivity) Factors  Filter selectivities

27. Convergence experiments (correlation) Factors  Correlation among filters

28. Run-time overhead experiments Factors  Number of filters  Filter selectivities  Cost of filters Comparison  Optimal  A-Greedy  Sweep  LocalSwaps  Independent

29. Run-time overhead experiments (time) Factors  Spending time ≥ 98%

30. Run-time overhead experiments (number) Factors  Number of filters

31. Run-time overhead experiments (cost) Factors  Cost of filters

32. Adaptivity experiments Factors  Rate of change  Cost of filters Comparison  A-Greedy  Sweep  LocalSwaps  Independent

33. Adaptivity experiments (speed)

34. Adaptivity experiments (rate) Factors  Rate of change

35. Adaptivity experiments (cost) Tradeoff  Run-time overhead ↔ adaptivity

36. Conclusion Different points along the tradeoff spectrum  Fast Adaptivity A-Greedy  Low run-time overhead Slow adaptivity, good convergence  Sweep Independent filters  Independent Swap between adjacent filters, unpredicted convergence  LocalSwaps

37. Appendix

38. stable

39. correlation factor Γ filters are divided into groups  Each contains Γ filters Filters in  different groups independent  the same groups 80% the same result of input tuples  most correlated  completely independent

40. Example 1

41. Example 2 For, Total cost = 20 Input 12147254total Cost 4342131220

42. A-GREEDY one of + +permutation of others cost: Example 6 1 2 … 49 1 2 … 1, 2, 3, …, 100, 1, 2, …, 100, 1, 2, … 50 51 … 99 100 INDEPENDENT permutation of + cost:

43. A-GREEDY + permutation of others cost: Example 7 1, 2, 3, …, 100, 1, 2, …, 100, 1, 2, … LOCALSWAP cost: x x xx x x x Before, After, /100 ?