Decomposition-by-Normalization (DBN): Leveraging Approximate Functional Dependencies for Efficient Tensor Decomposition Mijung Kim (Arizona State University) K. Selçuk Candan (Arizona State University) This work is supported by an NSF Grant # ‘MiNC: NSDL Middleware for Network- and Context-aware Recommendations’ and the NSF Grant # `RanKloud: Data Partitioning and Resource Allocation Strategies for Scalable Multimedia and Social Media Analysis 1

Tensor Decomposition 2 Tensor is a high-dimensional array Tensor decomposition is widely used for multi- aspect data analysis for multi-dimensional data

High cost of tensor decomposition Data is commonly high-dimensional and large- scale Dense tensor decomposition  The cost increases exponentially with the number of modes of the tensor. Sparse tensor decomposition  The cost increases more slowly (linearly with the number of nonzero entries in the tensor)  But still be very expensive for large data sets.  Parallelization for ALS method faces difficulties such as communication cost. How do we tackle this high computational cost of tensor decomposition? 3

Normalization Reduce the dimensionality and the size of the input tensor  based on functional dependencies (FD) of the relation (tensor) 4

Join-by-Decomposition [ Kim and Candan 2011] Step 1a: Decomposition of (user, movie, rating) relation Step 1b: Decomposition of (movie, genre) relation Step 2: Combination of the two decompositions into a final decomposition 5 M. Kim and K. S. Candan. Approximate tensor decomposition within a tensor- relational algebraic framework. In CIKM, Find all rank-R1 and rank-R2 decompositions of the two input tensors, where R1 × R2 = R and choose one pair where two decompositions are as independent from each other as possible.

Decomposition-by-Normalization (DBN) High-dimensional data set (5-mode tensor)

Decomposition-by-Normalization (DBN) High-dimensional data set (5-mode tensor) Normalization based on functional dependencies (vertical partitioning)

Decomposition-by-Normalization (DBN) 8 High-dimensional data set (5-mode tensor) Lower-dimensional data sets (two 3- mode tensors) Normalization based on functional dependencies (vertical partitioning)

Decomposition-by-Normalization (DBN) 9 High-dimensional data set (5-mode tensor) Lower-dimensional data sets (two 3- mode tensors) Tensor decomposition on each vertical partition (sub-tensor) Normalization based on functional dependencies (vertical partitioning)

Decomposition-by-Normalization (DBN) 10 High-dimensional data set (5-mode tensor) Lower-dimensional data sets (two 3- mode tensors) Tensor decomposition on each vertical partition (sub-tensor) Combined into the decomposition of the original data set (tensor) Normalization based on functional dependencies (vertical partitioning)

Task 1: Normalization Process 11 Y. Huhtala et al. TANE: An ecient algorithm for discovering functional and approximate dependencies. Comput. J., 42 (2): , 1999.

Task 2: Find Approximate FD 12 Many data sets may not have perfect FDs to leverage for normalization Thus we rely on approximate FDs in the data with support (the minimum fraction of tuples that must be removed for FDs to hold)

Task 3: Partitioning Partition the data into two partitions that will lead to least amount of errors. Find the partitions as independent from each other as possible.  minimize inter-partition (between partitions) pair- wise FDs  maximize intra-partition (within partitions) pair- wise FDs 13

Parallelized DBN We parallelize the entire DBN operation by associating each pair of rank decompositions to an individual processor core 14 Rank-1 × Rank-12Rank-2 × Rank-6Rank-3 × Rank-4 Rank-4 × Rank-3Rank-6 × Rank-2Rank-12 × Rank-1 Rank-12 Each pair can run in a separate core in a parallel manner.

Desiderata The vertical partitioning should be s.t.:  Approx. FDs need to have high support to prevent over- thinning of the relation R.  Case 1: join attribute X determines only a subset of the attributes of the relation R) (|R|=|R2|, |R1|<=|R2|) For dense tensors, the number of attributes in each partition should be balanced For sparse tensors, the total number of tuples of R1 and R2 are minimized  Case 2: join attribute X determines all attributes of the relation R (|R|=|R1|=|R2|) The support for the inter-partition FDs are minimized. (For dense tensors, the partitions should be balanced.) 15

Vertical Partitioning Strategies Partition with all the attributes determined with a support higher than the threshold (support) by the join attribute. 16

Desiderata The vertical partitioning should be s.t.:  Approx. FDs need to have high support to prevent over- thinning of the relation R.  Case 1: join attribute X determines only a subset of the attributes of the relation R) (|R|=|R2|, |R1|<=|R2|) For dense tensors, the number of attributes in each partition should be balanced For sparse tensors, the total number of tuples of R1 and R2 are minimized  Case 2: join attribute X determines all attributes of the relation R (|R|=|R1|=|R2|) The support for the inter-partition FDs are minimized. (For dense tensors, the partitions should be balanced.) 17

Vertical Partitioning Strategies (Case 1: join attribute X determines only a subset of the attributes of the relation R (|R|=|R2|, |R1|<=|R2|) ) Sparse tensors  The size of R1 (X and all determined attributes) can be minimized down to the number of unique values of X by eliminating all the duplicate tuples. Dense tensors  Promote balanced partitioning by relaxing or tightening the support threshold. If # attr. of R2 > # attr. of R1, move the attributes with the highest support of R2 to R1 (relaxing) or if # attr. of R1 > # attr. of R2, move the attributes with the lowest support of R1 to R2 (tightening). 18

Desiderata The vertical partitioning should be s.t.:  Approx. FDs need to have high support to prevent over- thinning of the relation R.  Case 1: join attribute X determines only a subset of the attributes of the relation R) (|R|=|R2|, |R1|<=|R2|) For dense tensors, the number of attributes in each partition should be balanced For sparse tensors, the total number of tuples of R1 and R2 are minimized  Case 2: join attribute X determines all attributes of the relation R (|R|=|R1|=|R2|) The support for the inter-partition FDs are minimized. (For dense tensors, the partitions should be balanced.) 19

Vertical Partitioning Strategies (Case 2: join attribute X determines all attributes of the relation R) We formulate the interFD-based partitioning as a graph partitioning problem.  pairwise FD graph, G pfd (V, E), where each vertex represents an attribute and the weight of the edge the average support of the approximate FDs between the attr.  The problem is then to locate a cut on G pfd with the minimum average weight. (For dense tensors, balance criterion is imposed)  We use a modified version of a minimum cut algorithm [Stoer and Wagner 1997] to seek a minimum average cut. 20 M. Stoer and F. Wagner. A simple min-cut algorithm. J. ACM, 44 (4):585-59, 1997

Rank Pruning based on Intra-Partition Dependencies The higher the overall dependency between the attributes in a partition, the smaller should be the decomposition rank of that partition. Thus, we only consider rank pairs (r1, r2) s.t. r1 < r2 if intra-partition FD support for R1 is larger than the support for R2, and vice versa. 21

Experimental Setup (Data Sets) UCI Machine Learning Repository [Frank and Asuncion 2010] 22 A. Frank and A. Asuncion. UCI Machine Learning Repository. Irvine, CA: U. of California, School of ICS, 2010.

Experimental Setup (Algorithms) NNCP (Non-Negative CP) vs. DBN Dense tensor [N-way Toolbox 2000]  NNCP-NWAY vs. DBN-NWAY Sparse tensor [MATLAB Tensor Toolbox 2007]  NNCP-CP vs. DBN-CP With parallelization  NNCP-NWAY/CP-GRID2,6 [Phan and Cichocki 2011] vs. pp-DBN-NWAY/CP DBN with intraFD-based rank pruning  DBN2,3 (2 pairs or 3 pairs selection) 23 C. A. Andersson and R. Bro. The N-way Toolbox for MATLAB. Chemometr. Intell. Lab., 52(1):1-4, B. W. Bader and T. G. Kolda. MATLAB Tensor Toolbox Ver. 2.2, A. H. Phan and A. Cichocki. PARAFAC algorithms for large-scale problems. Neurocomputing, 74(11): , 2011.

Experimental Setup (rank) rank-12 decomposition  DBN uses 6 combinations (1×12, 2×6, 3×4, 4×3, 6×2, and 12×1) 24

Experimental Setup (H/W and S/W) H/W  6 cores Intel(R) Xeon(R) CPU 2.66GHz with 24GB of RAM. S/W  MATLAB Version (R2010b) 64-bit (glnxa64) for the general implementation  MATLAB Parallel Computing Toolbox for the parallel implementation of DBN and NNCP 25

26 Key Results: Running Time (Dense Tensor) ( Case 1: join attribute X determines only a subset of the attributes of the relation R (|R|=|R2|, |R1|<=|R2|) ) NNCP vs. DBN with parallelization

27 NNCP vs. DBN and DBN2,3 (DBN2,3: DBN with intraFD- based rank pruning) Key Results: Running Time (Sparse Tensor) (Case 1: join attribute X determines only a subset of the attributes of the relation R (|R|=|R2|, |R1|<=|R2|) ) NNCP vs. DBN with parallelization

28 NOTE: In both cases, most of data points are located under the diagonal, which indicates that DBN outperforms NNCP. Key Results: Running Time (Case 2: join attribute X determines all attributes of the relation R (|R|=|R1|=|R2|)) Dense Tensor NNCP vs. DBN3 with parallelization Sparse Tensor NNCP vs. DBN3 with parallelization

Key Results: Accuracy 29 NOTE: THE HIGHER THE BETTER!!!!!!!

InterFD-based vertical partitioning 30 Note: The higher the closer to the optimal partitioning strategy!

intraFD-based rank pruning strategy 31 Note: The higher the better intraFD- based rank pruning works!

Lifecycle of data requires capture, integration, projection, decomposition, and data analysis. Tensor decomposition is a costly operation. We proposed:  highly efficient, effective, and easily parallelizable decomposition-by-normalization strategy for approximately evaluating decompositions  interFD-based partitioning  intraFD-based rank pruning strategies Conclusions 32