Vertical Set Square Distance: A Fast and Scalable Technique to Compute Total Variation in Large Datasets Taufik Abidin, Amal Perera, Masum Serazi, William Perrizo 20 th International Conference on Computer and Their Applications March 16-18, 2005 Louisiana, New Orleans

20 th International Conference on CATA 2005 Outline:  Introduction  Motivation  Contribution  P-tree Vertical Data Structure  Vertical Set Square Distance (VSSD)  Experimental Results  Conclusion and Future Works

20 th International Conference on CATA 2005 Introduction  The determination of closeness (distance) of a point to another point or to a set of points (average distance) is required in data mining tasks.  For example:  Distance-based clustering. Distance is used to determine in which cluster a certain point should be placed, i.e. K-means.  Near nbr or case-based classification The assignment of class label to an unclassified point is based on the majority of class label of the nearest neighbor points. “Nearest” here implies the closest distance of points.

20 th International Conference on CATA 2005  One way to measure the distance from a certain point to a set of points is by examining the actual total separation of the set of points from the point in question  However, if the examination is done point by point when the set is very large, scanning the entire space causes the approach to be very costly, slow and non-scalable.  In this paper, we introduce a new technique called the Vertical Set Square Distance (VSSD) that can scalably, quickly and accurately compute the total length of separation (total variation) of a set of points about a point. Introduction (cont)

20 th International Conference on CATA 2005  In data mining, efforts have focused on finding techniques that can scalably deal with large cardinality of dataset due to the explosive growth of data stored in digital form.  Most existing approaches for measuring closeness of points are slow and computationally intensive (often relying upon sampling to become useable).  The availability of P-tree technology (vertical data structure) is an appropriate data structure to solve this curse of cardinality. Motivation

20 th International Conference on CATA 2005  We introduce a new vertical technique that scalably, quickly and accurately computes the total length of separation of a set of points about a fixed point.  We present empirical results based on real and synthetic datasets and show the one-to-one comparison in terms of scalability and speed (time) between our new method that uses the vertical approach (P-tree vertical data structure) and other method that uses the horizontal approach (record-based). Our Contributions

20 th International Conference on CATA 2005  P-tree vertical data representation consists of set structures essentially representing the data column-by-column rather than row-by-row (i.e. relational data).  The construction of P-trees from an existing relational table is typically by decomposing each attribute in the table into separate bit vectors (e.g., one for each bit position of a numeric attribute or one bitmap for each category in a categorical attribute). The construction for raw data coming from a sensor platform is done directly (without creating a horizontal relational table first. Note that many sensor platforms (e.g., RSI sensors), product raw vertical data. P-Tree Vertical Data Structure

20 th International Conference on CATA 2005 The Construction of P-Tree The construction of the P- Tree: 1. Vertically project each attribute. 2. Vertically project each bit position. 3. Compress each piece of bit slice into a P-tree. Logical operations are AND (  ), OR (  ) and complement ('). Root count operation is the count of 1- bits from the resulting P-trees logical operations R 11 R 12 R 13 R 21 R 22 R 23 R 31 R 32 R 33 R 41 R 42 R 43 P 11 P 12 P 13 P 21 P 22 P 23 P 31 P 32 P 33 P 41 P 42 P ^ ^^^ ^ ^ ^ ^^ = R[A 1 ] R[A 2 ] R[A 3 ] R[A 4 ] R (A 1 A 2 A 3 A 4 )

20 th International Conference on CATA 2005 Binary Representation Let x be a numerical domain of attribute A i of relation R(A 1, A 2, …, A d ), then x written in b bits binary number is shown in the left-hand side term of the equation below. The first subscript of x is the index of the attribute to which x belong. The second subscript of x indicates the bit position. The summation on the right-hand side of the equation is equal to the actual value of x as a decimal number. Vertical Set Square Distance

20 th International Conference on CATA 2005 Let X be a set of vectors in a relation R(A 1, A 2, …, A d ), let P X be the P-tree class mask of X, x is a vector  : and a is a target vector (also in d-dimensional space), then the vectors x and a in binary can be written as: The total variation of a set of vectors in X about a vector a can be computed vertically using Vertical Set Square Distance (VSSD) defined as follows: Vertical Set Square Dist. (Cont)

20 th International Conference on CATA 2005 Vertical Set Square Dist. (Cont) Term T 1, T 2 and T 3 can be computed separately and their summation is actually the total variation (the sum of the square length of separation of the vectors connecting X and a.

20 th International Conference on CATA 2005 Vertical Set Square Dist. (Cont) Or we can write T 1, expressing the diagonal terms (j=k) separately (noting also that x i,j 2 = x i,j as they are bits)

20 th International Conference on CATA 2005 Vertical Set Square Dist. (Cont) And T 3 is simply:

20 th International Conference on CATA 2005 Vertical Set Square Dist. (Cont) The root count operation is obviously independent of the vector a. These include the root count of single P-tree operand, P X, and the root count of two basic P-tree operands. and the root count of three P-tree operands which appear in T 1, T 2 and T 3.

20 th International Conference on CATA 2005 Vertical Set Square Dist. (Cont)  The independency allows us to pre-compute the root counts once in advance, during the construction of the P-tree, and can use them repeatedly as needed, regardless of the number of target vectors a.  This amortizes the cost of P-tree ANDing for high value datasets, e.g. cancer analysis, RSI, etc.

20 th International Conference on CATA 2005 Horizontal Set Square Distance (HSSD) as a comparison method Let X, a set of vectors in R(A 1,A 2 … A d ) and x = (x 1, x 2, …, x d ) is a vector belong to X and a = (a 1, a 2, …, a d ) is a target vector Then the horizontal set square distance (HSSD) is defined as follows:

20 th International Conference on CATA 2005 Experimental Results  The experiments were conducted using both real and synthetic datasets.  The goals are to compare the execution time (speed) and scalability of our algorithm employing a vertical approach (vertical data structure and horizontal bitwise AND operation) with a horizontal approach (horizontal data structure and vertical scan operation).  Performance of both algorithms was observed under different machine specifications, including an SGI Altix machine.

20 th International Conference on CATA 2005 Experimental Results (Cont.) The specification of machines used in the experiments. MachineSpecification AMD  1GB AMD Athlon K7 1.4GHz, 1GB RAM P4  2GB Intel P4 2.4GHz processor, 2GB RAM SGI Altix SGI Altix CC-NUMA 12 processor shared memory (12 x 4 GB RAM)

20 th International Conference on CATA 2005 Datasets  A set of aerial photographs from the Best Management Plot (BMP) of Oakes Irrigation Test Area (OITA) near Oakes, ND.  The original image size (dataset) is 1024x1024 pixels (cardinality of 1,048,576), contains 3 bands (RGB) plus synchronized data for soil moisture, soil nitrate and crop yield (total 6 attributes).

20 th International Conference on CATA 2005 Datasets (Cont.)  The datasets contain 4 different yield classes:  Low Yield ( 0 < intensity <= 63)  Medium Low Yield ( 63 < intensity <= 127)  Medium High Yield(127< intensity <= 191)  High Yield(191< intensity <= 255)  With super sampling five synthetic datasets are created:  2,097,152 rows  4,194,304 rows (2048x2048 pixels)  8,388,608 rows  16,777,216 rows (4096x4096 pixels)  25,160,256 rows (5016x5016 pixels)

20 th International Conference on CATA 2005 Timing and Scalability Results Observation 1.  The first performance evaluation was done on a P4 2 GB RAM machine using synthetic datasets having 4.1 and 8.3 million data tuples.  We use 100 arbitrary unclassified target vectors (resulting in 400 classification computations as to predicted yeild).  Datasets of sizes greater than 8.3 million rows cannot be executed in this machine due to out of memory problems when running Horizontal Set Square Distance (HSSD).

20 th International Conference on CATA 2005 Results Dataset size (# of Rows) Average Time to Compute Total Variation for Each Test Case in (Seconds) VSSDHSSD 4,193, ,388,

20 th International Conference on CATA 2005 Timing and Scalability Results Observation 1 (Continue) Dataset (Rows) Time (Seconds) VSSDHSSD Root Count Pre-Computation and P-trees Loading Horizontal Dataset Loading 1,048, ,097, ,194, ,388, This table shows the comparison of time required for loading the vertical data structure to memory and one time root count operations for VSSD, and loading horizontal records to memory needed for HSSD.

20 th International Conference on CATA 2005 Timing and Scalability Results Observation 2 This table shows the algorithm’s timing and scalability performances when they are executed on different machines. Cardinality of Dataset Average Running Time to Compute Total Variation for Each Test Case (Seconds) HSSDVSSD AMD 1GBP4 2GBSGI Altix 12x4GBAMD 1GB 1,048, ,097, ,194, ,388,608  ,777,216  ,160,256 

20 th International Conference on CATA 2005 Timing and Scalability Results Observation 2 (Continue)

20 th International Conference on CATA 2005 Conclusion  Vertical Set Square Distance (VSSD) is fast and accurate way to compute total variation for classification, clustering and rule mining, and scale well to very large datasets compare to the traditional horizontal approach (HSSD).  The complexity of the VSSD is O(d * b 2 ) where d is the number of dimensions and b is the maximum bit-width of the attributes (depends on the width of the data set only).  The VSSD is very fast because of the independency of root counts of P-tree operands to target vector a which allows the pre-computation of counts once in advance, during the construction of the P-tree.

20 th International Conference on CATA 2005 Future Work  Comprehensive study on the Vertical Set Square Distance (VSSD) in many data mining tasks, i.e. classification, clustering or outlier detection.  For classification using VSSD in the voting phase has already been shown to greatly accelerate the assignment of class since the calculation of class votes can be done entirely in one computation without having to visit each individual point, as is the case of the horizontal-based approach.

20 th International Conference on CATA 2005 Thank You…