CMU SCS : Multimedia Databases and Data Mining Lecture #11: Fractals: M-trees and dim. curse (case studies – Part II) C. Faloutsos
CMU SCS Copyright: C. Faloutsos (2011)2 Must-read Material Alberto Belussi and Christos Faloutsos, Estimating the Selectivity of Spatial Queries Using the `Correlation' Fractal Dimension Proc. of VLDB, p , 1995 Estimating the Selectivity of Spatial Queries Using the `Correlation' Fractal Dimension
CMU SCS Copyright: C. Faloutsos (2011)3 Optional Material Optional, but very useful: Manfred Schroeder Fractals, Chaos, Power Laws: Minutes from an Infinite Paradise W.H. Freeman and Company, 1991
CMU SCS Copyright: C. Faloutsos (2011)4 Outline Goal: ‘Find similar / interesting things’ Intro to DB Indexing - similarity search Data Mining
CMU SCS Copyright: C. Faloutsos (2011)5 Indexing - Detailed outline primary key indexing secondary key / multi-key indexing spatial access methods –z-ordering –R-trees –misc fractals –intro –applications text...
CMU SCS Copyright: C. Faloutsos (2011)6 Indexing - Detailed outline fractals –intro –applications disk accesses for R-trees (range queries) dimensionality reduction selectivity in M-trees dim. curse revisited “fat fractals” quad-tree analysis [Gaede+]
CMU SCS Copyright: C. Faloutsos (2011)7 What else can they solve? separability [KDD’02] forecasting [CIKM’02] dimensionality reduction [SBBD’00] non-linear axis scaling [KDD’02] disk trace modeling [Wang+’02] selectivity of spatial/multimedia queries [PODS’94, VLDB’95, ICDE’00]...
CMU SCS Copyright: C. Faloutsos (2011)8 Metric trees - analysis Problem: How many disk accesses, for an M-tree? Given: – N (# of objects) – C (fanout of disk pages) – r (radius of range query - BIASED model) Optional
CMU SCS Copyright: C. Faloutsos (2011)9 Metric trees - analysis Problem: How many disk accesses, for an M-tree? Given: – N (# of objects) – C (fanout of disk pages) – r (radius of range query - BIASED model) NOT ENOUGH - what else do we need? Optional
CMU SCS Copyright: C. Faloutsos (2011)10 Metric trees - analysis A: something about the distribution Optional
CMU SCS Copyright: C. Faloutsos (2011)11 Metric trees - analysis A: something about the distribution [Ciaccia, Patella, Zezula, PODS98]: assumed that the distance distribution is the same, for every object: Paolo CiacciaMarco Patella Optional
CMU SCS Copyright: C. Faloutsos (2011)12 Metric trees - analysis A: something about the distribution [Ciaccia+, PODS98]: assumed that the distance distribution is the same, for every object: F1(d) = Prob(an object is within d from object #1) = F2(d) =... = F(d) Optional
CMU SCS Copyright: C. Faloutsos (2011)13 Metric trees - analysis A: something about the distribution Given our ‘fractal’ tools, we could try them - which one? Optional
CMU SCS Copyright: C. Faloutsos (2011)14 Metric trees - analysis A: something about the distribution Given our ‘fractal’ tools, we could try them - which one? A: Correlation integral [Traina+, ICDE2000] Optional
CMU SCS Copyright: C. Faloutsos (2011)15 Metric trees - analysis English dictionary Portuguese dictionary log(d) log(#pairs) log(d) log(#pairs) Optional
CMU SCS Copyright: C. Faloutsos (2011)16 Metric trees - analysis log(d) log(#pairs) Divina Comedia Eigenfaces Optional
CMU SCS Copyright: C. Faloutsos (2011)17
CMU SCS Copyright: C. Faloutsos (2011)18 Metric trees - analysis Optional
CMU SCS Copyright: C. Faloutsos (2011)19 Metric trees - analysis So, what is the # of disk accesses, for a node of radius r d, on a query of radius r q ? rdrd rqrq Optional
CMU SCS Copyright: C. Faloutsos (2011)20 Metric trees - analysis So, what is the # of disk accesses, for a node of radius r d, on a query of radius r q ? A: ~ (r d +r q ).... rdrd rqrq Optional
CMU SCS Copyright: C. Faloutsos (2011)21 Metric trees - analysis So, what is the # of disk accesses, for a node of radius r d, on a query of radius r q ? A: ~ (r d +r q )^D rdrd rqrq Optional
CMU SCS Copyright: C. Faloutsos (2011)22 Accuracy of selectivity formulas log(rq) log(#d.a.) Optional
CMU SCS Copyright: C. Faloutsos (2011)23 Fast estimation of D Normally, D takes O(N^2) time Anything faster? suppose we have already built an M-tree Optional
CMU SCS Copyright: C. Faloutsos (2011)24 Fast estimation of D Hint: Optional
CMU SCS Copyright: C. Faloutsos (2011)25 Fast estimation of D Hint: ratio of radii: r1^D * C = r2^D D ~ log(C) / log(r2/r1) r1 r2 Optional
CMU SCS Copyright: C. Faloutsos (2011)26 Indexing - Detailed outline fractals –intro –applications disk accesses for R-trees (range queries) dimensionality reduction selectivity in M-trees dim. curse revisited “fat fractals” quad-tree analysis [Gaede+]
CMU SCS Copyright: C. Faloutsos (2011)27 Dim. curse revisited (Q: how serious is the dim. curse, e.g.:) Q: what is the search effort for k-nn? –given N points, in E dimensions, in an R-tree, with k-nn queries (‘biased’ model) [Pagel, Korn + ICDE 2000]
CMU SCS Copyright: C. Faloutsos (2011)28 (Overview of proofs) assume that your points are uniformly distributed in a d-dimensional manifold (= hyper-plane) derive the formulas substitute d for the fractal dimension
CMU SCS Copyright: C. Faloutsos (2011)29 Reminder: Hausdorff Dimension (D 0 ) r = side length (each dimension) B(r) = # boxes containing points r D0 r = 1/2 B = 2r = 1/4 B = 4r = 1/8 B = 8 logr = -1 logB = 1 logr = -2 logB = 2 logr = -3 logB = 3 proof
CMU SCS Copyright: C. Faloutsos (2011)30 Reminder: Correlation Dimension (D 2 ) S(r) = p i 2 (squared % pts in box) r D2 #pairs( within <= r ) r = 1/2 S = 1/2r = 1/4 S = 1/4r = 1/8 S = 1/8 logr = -1 logS = -1 logr = -2 logS = -2 logr = -3 logS = -3 proof
CMU SCS Copyright: C. Faloutsos (2011)31 Observation #1 How to determine avg MBR side l? –N = #pts, C = MBR capacity Hausdorff dimension: B(r) r D0 B(l) = N/C = l -D0 l = (N/C) -1/D0 l proof
CMU SCS Copyright: C. Faloutsos (2011)32 Observation #2 k-NN query -range query –For k pts, what radius do we expect? Correlation dimension: S(r) r D2 22 proof
CMU SCS Copyright: C. Faloutsos (2011)33 Observation #3 Estimate avg # query-sensitive anchors: –How many expected q will touch avg page? –Page touch: q stabs -dilated MBR(p) p MBR(p) q l l q p proof
CMU SCS Copyright: C. Faloutsos (2011)34 Asymptotic Formula k-NN page accesses as N –C = page capacity –D = fractal dimension (=D0 ~ D2)
CMU SCS Copyright: C. Faloutsos (2011)35 Asymptotic Formula NO mention of the embedding dimensionality!! Still have dim. curse, but on f.d. D
CMU SCS Copyright: C. Faloutsos (2011)36 Synthetic Data plane –D 0 = D 2 = 2 –embedded in E-space –N = 100K manifold –E = 8 –D 0 =D 2 varies from 1-6 –line, plane, etc. (in 8-d)
CMU SCS Copyright: C. Faloutsos (2011)37 Accuracy of L Formula
CMU SCS Copyright: C. Faloutsos (2011)38 Embedding Dimension plane k = 50 L dist
CMU SCS Copyright: C. Faloutsos (2011)39 Intrinsic Dimensionality manifold k = 50 L dist
CMU SCS Copyright: C. Faloutsos (2011)40 Non-Euclidean Data Set sierpinski, k = 50, L dist
CMU SCS Copyright: C. Faloutsos (2011)41 Conclusions Worst-case theory is over-pessimistic High dimensional data can exhibit good performance if correlated, non-uniform Many real data sets are self-similar Determinant is intrinsic dimensionality –multiple fractal dimensions (D 0 and D 2 ) –indication of how far one can go
CMU SCS Copyright: C. Faloutsos (2011)42 References Ciaccia, P., M. Patella, et al. (1998). A Cost Model for Similarity Queries in Metric Spaces. PODS. Pagel, B.-U., F. Korn, et al. (2000). Deflating the Dimensionality Curse Using Multiple Fractal Dimensions. ICDE, San Diego, CA. Traina, C., A. J. M. Traina, et al. (2000). Distance Exponent: A New Concept for Selectivity Estimation in Metric Trees. ICDE, San Diego, CA.