1. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL Bi-Clustering COMP 790-90 Seminar Spring 2011

2. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 2 Data Mining: Clustering Where K-means clustering minimizes

3. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 3 Clustering by Pattern Similarity (p-Clustering) The micro-array “raw” data shows 3 genes and their values in a multi-dimensional space  Parallel Coordinates Plots  Difficult to find their patterns “non-traditional” clustering

4. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 4 Clusters Are Clear After Projection

5. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 5 Motivation E-Commerce: collaborative filtering Movie 1 Movie 2 Movie 3 Movie 4 Movie 5 Movie 6 Movie 7 Viewer 112435 Viewer 24671 Viewer 323463 Viewer 43457 Viewer 55534

6. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 6 Motivation

7. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 7 Motivation Movie 1 Movie 2 Movie 3 Movie 4 Movie 5 Movie 6 Movie 7 Viewer 112435 Viewer 24671 Viewer 323463 Viewer 43457 Viewer 55534

8. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 8 Motivation

9. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 9 Motivation DNA microarray analysis CH1ICH1BCH1DCH2ICH2B CTFC343922844108280228 VPS8401281120275298 EFB131828037277215 SSA1401292109580238 FUN1428572852576271226 SP0722829048285224 MDM10538272266277236 CYS332228841278219 DEP131227240273232 NTG132929633274228

10. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 10 Motivation

11. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 11 Motivation Strong coherence exhibits by the selected objects on the selected attributes.  They are not necessarily close to each other but rather bear a constant shift.  Object/attribute bias bi-cluster

12. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 12 Challenges The set of objects and the set of attributes are usually unknown. Different objects/attributes may possess different biases and such biases  may be local to the set of selected objects/attributes  are usually unknown in advance May have many unspecified entries

13. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 13 Previous Work Subspace clustering  Identifying a set of objects and a set of attributes such that the set of objects are physically close to each other on the subspace formed by the set of attributes. Collaborative filtering: Pearson R  Only considers global offset of each object/attribute.

14. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 14 bi-cluster Consists of a (sub)set of objects and a (sub)set of attributes  Corresponds to a submatrix  Occupancy threshold   Each object/attribute has to be filled by a certain percentage.  Volume: number of specified entries in the submatrix  Base: average value of each object/attribute (in the bi-cluster)

15. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 15 bi-cluster CH1ICH1BCH1DCH2ICH2BObj base CTFC3 VPS8401120298273 EFB131837215190 SSA1 FUN14 SP07 MDM10 CYS332241219194 DEP1 NTG1 Attr base34766244219

16. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 16 bi-cluster Perfect  -cluster Imperfect  -cluster  Residue: d IJ d Ij d iJ d ij

17. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 17 bi-cluster The smaller the average residue, the stronger the coherence. Objective: identify  -clusters with residue smaller than a given threshold

18. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 18 Cheng-Church Algorithm Find one bi-cluster. Replace the data in the first bi-cluster with random data Find the second bi-cluster, and go on. The quality of the bi-cluster degrades (smaller volume, higher residue) due to the insertion of random data.

19. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 19 The FLOC algorithm Generating initial clusters Determine the best action for each row and each column Perform the best action of each row and column sequentially Improved? Y N

20. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 20 The FLOC algorithm Action: the change of membership of a row(or column) with respect to a cluster 34 4 13 22 3 2 2 04 column row 1 3 2 1 234 M+N actions are Performed at each iteration N=3 M=4

21. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 21 The FLOC algorithm Gain of an action: the residue reduction incurred by performing the action Order of action:  Fixed order  Random order  Weighted random order Complexity: O((M+N)MNkp) 

22. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 22 The FLOC algorithm Additional features  Maximum allowed overlap among clusters  Minimum coverage of clusters  Minimum volume of each cluster Can be enforced by “temporarily blocking” certain action during the mining process if such action would violate some constraint.

23. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 23 Performance Microarray data: 2884 genes, 17 conditions  100 bi-clusters with smallest residue were returned.  Average residue = 10.34  The average residue of clusters found via the state of the art method in computational biology field is 12.54  The average volume is 25% bigger  The response time is an order of magnitude faster

24. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 24 Conclusion Remark The model of bi-cluster is proposed to capture coherent objects with incomplete data set.  base  residue Many additional features can be accommodated (nearly for free).

25. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 25 Coherent Cluster Want to accommodate noise but not outliers

26. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 26 Coherent Cluster Coherent cluster  Subspace clustering pair-wise disparity  For a 2  2 (sub)matrix consisting of objects {x, y} and attributes {a, b} x y ab d xa d ya d xb d yb x y ab z attribute mutual bias of attribute a mutual bias of attribute b

27. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 27 Coherent Cluster  A 2  2 (sub)matrix is a  -coherent cluster if its D value is less than or equal to .  An m  n matrix X is a  -coherent cluster if every 2  2 submatrix of X is  -coherent cluster.  A  -coherent cluster is a maximum  -coherent cluster if it is not a submatrix of any other  -coherent cluster.  Objective: given a data matrix and a threshold , find all maximum  -coherent clusters.

28. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 28 Coherent Cluster Challenges:  Finding subspace clustering based on distance itself is already a difficult task due to the curse of dimensionality.  The (sub)set of objects and the (sub)set of attributes that form a cluster are unknown in advance and may not be adjacent to each other in the data matrix.  The actual values of the objects in a coherent cluster may be far apart from each other.  Each object or attribute in a coherent cluster may bear some relative bias (that are unknown in advance) and such bias may be local to the coherent cluster.

29. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 29 Coherent Cluster Compute the maximum coherent attribute sets for each pair of objects Construct the lexicographical tree Post-order traverse the tree to find maximum coherent clusters Two-way Pruning

30. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 30 Coherent Cluster Observation: Given a pair of objects {o 1, o 2 } and a (sub)set of attributes {a 1, a 2, …, a k }, the 2  k submatrix is a  -coherent cluster iff, for every attribute a i, the mutual bias (d o1ai – d o2ai ) does not differ from each other by more than . a1a1 a2a2 a3a3 a4a4 a5a5 1 3 5 7 323.522.5 o1o1 o2o2  [2, 3.5] If  = 1.5, then {a 1,a 2,a 3,a 4,a 5 } is a coherent attribute set (CAS) of (o 1,o 2 ).

31. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 31 Coherent Cluster Observation: given a subset of objects {o 1, o 2, …, o l } and a subset of attributes {a 1, a 2, …, a k }, the l  k submatrix is a  -coherent cluster iff {a 1, a 2, …, a k } is a coherent attribute set for every pair of objects (o i,o j ) where 1  i, j  l. a1a1 a5a5 a6a6 a7a7 a2a2 a3a3 a4a4 o1o1 o3o3 o4o4 o5o5 o6o6 o2o2

32. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 32 a1a1 a2a2 a3a3 a4a4 a5a5 1 3 5 7 323.522.5 r1r1 r2r2 Coherent Cluster Strategy: find the maximum coherent attribute sets for each pair of objects with respect to the given threshold .  = 1 3 5 7 r1r1 r2r2 a2a2 2 a3a3 3.5 a4a4 2 a5a5 2.5 a1a1 3 1 The maximum coherent attribute sets define the search space for maximum coherent clusters.

33. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 33 Two Way Pruning a0a1a2 o0142 o1255 o2365 o342007 o430076 (o0,o2) →(a0,a1,a2) (o1,o2) →(a0,a1,a2) (a0,a1) →(o0,o1,o2) (a0,a2) →(o1,o2,o3) (a1,a2) →(o1,o2,o4) (a1,a2) →(o0,o2,o4) (o0,o2) →(a0,a1,a2) (o1,o2) →(a0,a1,a2) (a0,a1) →(o0,o1,o2) (a0,a2) →(o1,o2,o3) (a1,a2) →(o1,o2,o4) (a1,a2) →(o0,o2,o4) delta=1 nc =3 nr = 3 MCAS MCOS

34. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 34 Coherent Cluster Strategy: grouping object pairs by their CAS and, for each group, find the maximum clique(s). Implementation: using a lexicographical tree to organize the object pairs and to generate all maximum coherent clusters with a single post-order traversal of the tree. objects attributes

35. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 35 (o 0,o 1 ) : {a 0,a 1 }, {a 2,a 3 } (o 0,o 2 ) : {a 0,a 1,a 2,a 3 } (o 0,o 4 ) : {a 1,a 2 } (o 1,o 2 ) : {a 0,a 1,a 2 }, {a 2,a 3 } (o 1,o 3 ) : {a 0,a 2 } (o 1,o 4 ) : {a 1,a 2 } (o 2,o 3 ) : {a 0,a 2 } (o 2,o 4 ) : {a 1,a 2 } a0a0 a1a1 a2a2 a3a3 o0o0 1425 o1o1 2558 o2o2 3657 o3o3 42072 o4o4 30766 a0a0 a1a1 a2a2 a2a2 a3a3 a1a1 a2a2 a2a2 a3a3 (o0,o1)(o0,o1) (o1,o2)(o1,o2) (o0,o2)(o0,o2) (o1,o3)(o1,o3) (o2,o3)(o2,o3) (o0,o4)(o0,o4) (o1,o4)(o1,o4) (o2,o4)(o2,o4) (o0,o1)(o0,o1) (o1,o2)(o1,o2) assume  = 1 {a 0,a 1 } : (o 0,o 1 ) {a 0,a 2 } : (o 1,o 3 ),(o 2,o 3 ) {a 1,a 2 } : (o 0,o 4 ),(o 1,o 4 ),(o 2,o 4 ) {a 2,a 3 } : (o 0,o 1 ),(o 1,o 2 ) {a 0,a 1,a 2 } : (o 1,o 2 ) {a 0,a 1,a 2,a 3 } : (o 0,o 2 ) (o1,o2)(o1,o2) (o1,o2)(o1,o2) (o1,o2)(o1,o2) (o0,o2)(o0,o2) (o0,o2)(o0,o2) (o0,o2)(o0,o2) (o0,o2)(o0,o2) (o0,o2)(o0,o2)

36. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 36 a0a0 a1a1 a2a2 a2a2 a3a3 a1a1 a2a2 a2a2 a3a3 (o0,o1)(o0,o1) (o1,o2)(o1,o2) (o0,o2)(o0,o2) (o1,o3)(o1,o3) (o2,o3)(o2,o3) (o0,o4)(o0,o4) (o1,o4)(o1,o4) (o2,o4)(o2,o4) (o0,o1)(o0,o1) (o1,o2)(o1,o2) (o0,o2)(o0,o2) (o0,o2)(o0,o2) a3a3 (o0,o2)(o0,o2) a3a3 (o0,o2)(o0,o2) a3a3 {o 0,o 2 }  {a 0,a 1,a 2,a 3 } (o1,o2)(o1,o2) (o0,o2)(o0,o2)(o1,o2)(o1,o2) (o0,o2)(o0,o2)(o1,o2)(o1,o2) (o0,o2)(o0,o2) {o 1,o 2 }  {a 0,a 1,a 2 } {o 0,o 1,o 2 }  {a 0,a 1 } a3a3 (o0,o2)(o0,o2) a3a3 (o0,o2)(o0,o2) (o0,o2)(o0,o2) {o 1,o 2,o 3 }  {a 0,a 2 } {o 0,o 2,o 4 }  {a 1,a 2 } {o 1,o 2,o 4 }  {a 1,a 2 } {o 0,o 1,o 2 }  {a 2,a 3 }

37. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 37 Coherent Cluster High expressive power  The coherent cluster can capture many interesting and meaningful patterns overlooked by previous clustering methods. Efficient and highly scalable Wide applications  Gene expression analysis  Collaborative filtering subspace cluster coherent cluster

38. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 38 Remark Comparing to Bicluster  Can well separate noises and outliers  No random data insertion and replacement  Produce optimal solution