2015/7/21 Incremental Clustering for Mining in a Data Warehousing Environment Martin Ester Hans-Peter Kriegel J.Sander Michael Wimmer Xiaowei Xu Proceedings of the 24 th VLDB Conference New York,USA,1998 Modified Version

2015/7/22 Introduction Related Work The algorithm DBSCAN Incremental DBSCAN Performance Evaluation Conclusions Comment Outline

2015/7/23 Introduction Data Warehouse: collection of data from multiple sources two characteristic : (1) Derived information is present for the purpose of analysis (2) The environment is dynamic Data Mining: the application of data analysis and discovery algorithms that Under acceptable computational efficiency limitations Produce a particulate enumeration of patterns over the data

2015/7/24 Introduction(Cont.) The task in this paper is clustering - Grouping the objects of a database into a meaningful subclasses. Data warehouse updating - Periodically database update in a batch mode(ex:night). Due to the vary large size of the database,it is highly desirable to perform these updates incrementally Based on DBSCAN [EKSX 96] -The incremental algorithm has the same cluster result with DBSCAN and it speed up the daily updates in a data warehouse

2015/7/25 Related Work Partitioning algorithms: Construct various partitions and then evaluate then by some criterion Hierarchy algorithms: Create a hierarchy decomposition of the set of data(or objects) using some criterion Density-based: Based on connectivity and density functions

2015/7/26 Why choose DBSCAN The reason: -One of the most efficient algorithms on large databases -Be applied to any database containing data from a metric space (assuming a distance function)

2015/7/27 Introduction Related Work The algorithm DBSCAN Incremental DBSCAN Performance Evaluation Conclusions Comment Outline

2015/7/28 The algorithm DBSCAN Background Definition DBSCAN Algorithm DBSCAN Demo

2015/7/29 Background D: the dataset (i.e., a set of points) Eps: Maximum radius of the neighborhood MinPts: Minimum number of points in an Eps- neighborhood of that point N Eps (p) is the subset of D contained in the Eps- neighborhood of p N Eps (p)={q belong to D | dist(p,q) ≤ Eps} core object : | N Eps (p)| ≥ MinPts Eps p

2015/7/210 directly density-researchable ： p in N Eps (q) |N Eps (q)| ≥ MinPts p  q density-reachable ： p  D q because p  r r  q Definition q p Eps MinPts: 4 q p r

2015/7/211 Definition(Cont.) density-connected ： p and q are density-connected because p  D o q  D o cluster ： Maximality ：∀ p,q in D, if p ∈ C and q  D p, then q ∈ C Connectivity ：∀ p,q in C, p is density-connected to q noise ：｛ p in D ∣∀ i: p ∉ C i ｝ MinPts: 4 o p q core objects borderline objects noise

2015/7/212 DBSCAN Algorithm o p q

2015/7/213 DBSCAN Demo Stack Eps: MinPts: 3 noise

2015/7/214 References [EKSX 96] Ester M., Kriegel H.-P., Sander J., Xu X.: “A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise”, Proc. 2nd Int. Conf. on Knowledge Discovery and Data Mining, Portland, OR, 1996, pp [KR 90] Kaufman L., Rousseeuw P. J.: “Finding Groups in Data: An Introduction to Cluster Analysis”, John Wiley & Sons, [NH 94] Ng R. T., Han J.: “Efficient and Effective Clustering Methods for Spatial Data Mining”, Proc. 20th Int. Conf. on Very Large Data Bases, Santiago, Chile, 1994, pp [ZRL 96] Zhang T., Ramakrishnan R., Linvy M.: “BIRCH: An Efficient Data Clustering Method for Very Large Databases”, Proc. ACM SIGMOD Int. Conf. on Management of Data, 1996, pp

2015/7/215 References (con’t) [AF 96] Allard D. and Fraley C.:”Non Parametric Maximum Likelihood Estimation of Features in Saptial Point Process Using Voronoi Tessellation”, Journal of the American Statistical Association, December [alsohttp:// tr293R.ps]. [AS 94] Agrawal R., Srikant R.: “Fast Algorithms for Mining Association Rules”, Proc. 20th Int. Conf. on Very Large Data Bases, Santiago, Chile, 1994, pp [BKSS 90] Beckmann N., Kriegel H.-P., Schneider R., Seeger B.: “The R*-tree: An Efficient and Robust Access Method for Points and Rectangles”, Proc. ACM SIGMOD Int. Conf. on Management of Data, Atlantic City, NJ, 1990, pp [Bou 96] Bouguettaya A.: “On-Line Clustering”, IEEE Transactions on Knowledge and Data Engineering, Vol. 8, No. 2, 1996, pp [CHNW 96] Cheung D. W., Han J., Ng V. T., Wong Y.: “Maintenance of Discovered Association Rules in Large Databases: An Incremental Technique”, Proc. 12th Int. Conf. on Data Engineering, New Orleans, USA, 1996, pp [CPZ 97] Ciaccia P., Patella M., Zezula P.: “M-tree: An Efficient Access Method for Similarity Search in Metric Spaces”, Proc. 23rd Int. Conf. on Very Large Data Bases, Athens, Greece, 1997, pp

2015/7/216 References (con’t) [EKX 95] Ester M., Kriegel H.-P., Xu X.: “Knowledge Discovery in Large Spatial Databases: Focusing Techniques for Efficient Class Identification”, Proc. 4th Int. Symp. on Large Spatial Databases, Portland, ME, 1995, in: Lecture Notes in Computer Science, Vol. 951, Springer, 1995, pp [EW 98] Ester M., Wittmann R.: “Incremental Generalization for Mining in a Data Warehousing Environment”, Proc. 6th Int. Conf. on Extending Database Technology, Valencia, Spain, 1998, in: Lecture Notes in Computer Science, Vol. 1377, Springer, 1998, pp [FAAM 97] Feldman R., Aumann Y., Amir A., Mannila H.: “Efficient Algorithms for Discovering Frequent Sets in Incremental Databases”, Proc. ACM SIGMOD Workshop on Research Issues on Data Mining and Knowledge Discovery, Tucson, AZ, 1997, pp [FPS 96] Fayyad U., Piatetsky-Shapiro G., and Smyth P.: “Knowledge Discovery and Data Mining: Towards a Unifying Framework”, Proc. 2 nd Int. Conf. on Knowledge Discovery and Data Mining, Portland, OR, 1996, pp [Gue 94] Gueting R. H.: “An Introduction to Spatial Database Systems”, The VLDB Journal, Vol. 3, No. 4, October 1994, pp

2015/7/217 References (con’t) [HCC 93] Han J., Cai Y., Cercone N.: “Data-driven Discovery of Quantitative Rules in Relational Databases”, IEEE Transactions on Knowledge and Data Engineering, Vol.5, No. 1, 1993, pp [Huy 97] Huyn N.: “Multiple-View Self-Maintenance in Data Warehousing Environments”, Proc. 23 rd Int. Conf. on Very Large Data Bases, Athens, Greece, 1997, pp [Luo 95] Luotonen A.: “The common log file format”, [MJHS 96] Mombasher B., Jain N., Han E.-H., Srivastava J.: “Web Mining: Pattern Discovery from World Wide Web Transactions”, Technical Report , University of Minnesota, [MQM 97] Mumick I. S., Quass D., Mumick B. S.: “Maintenance of Data Cubes and Summary Tables in a Warehouse”, Proc. ACM SIGMOD Int. Conf. on Management of Data, 1997, pp [SEKX 98] Sander J., Ester M., Kriegel H.-P., Xu X.: “Density-Based Clustering in Spatial Databases: The Algorithm GDBSCAN and its Applications”, will appear in: Data Mining and Knowledge Discovery, Kluwer Acedemic Publishers, Vol. 2, [Sib 73] Sibson R.: “SLINK: an optimally efficient algorithm for the single-link cluster method”, The Computer Journal, Vol. 16, No. 1, 1973, pp