1. Talk 2: Graph Mining Tools - SVD, ranking, proximity Christos Faloutsos CMU

2. Lipari 2010(C) 2010, C. Faloutsos 2 Outline Introduction – Motivation Task 1: Node importance Task 2: Recommendations Task 3: Connection sub-graphs Conclusions

3. Lipari 2010(C) 2010, C. Faloutsos 3 Node importance - Motivation: Given a graph (eg., web pages containing the desirable query word) Q: Which node is the most important?

4. Lipari 2010(C) 2010, C. Faloutsos 4 Node importance - Motivation: Given a graph (eg., web pages containing the desirable query word) Q: Which node is the most important? A1: HITS (SVD = Singular Value Decomposition) A2: eigenvector (PageRank)

5. Lipari 2010(C) 2010, C. Faloutsos 5 Node importance - motivation SVD and eigenvector analysis: very closely related

6. Lipari 2010(C) 2010, C. Faloutsos 6 SVD - Detailed outline Motivation Definition - properties Interpretation Complexity Case studies

7. Lipari 2010(C) 2010, C. Faloutsos 7 SVD - Motivation problem #1: text - LSI: find ‘concepts’ problem #2: compression / dim. reduction

8. Lipari 2010(C) 2010, C. Faloutsos 8 SVD - Motivation problem #1: text - LSI: find ‘concepts’

9. Lipari 2010(C) 2010, C. Faloutsos 9 SVD - Motivation Customer-product, for recommendation system: bread lettuce beef vegetarians meat eaters tomatos chicken

10. Lipari 2010(C) 2010, C. Faloutsos 10 SVD - Motivation problem #2: compress / reduce dimensionality

11. Lipari 2010(C) 2010, C. Faloutsos 11 Problem - specs ~10**6 rows; ~10**3 columns; no updates; random access to any cell(s) ; small error: OK

12. Lipari 2010(C) 2010, C. Faloutsos 12 SVD - Motivation

13. Lipari 2010(C) 2010, C. Faloutsos 13 SVD - Motivation

14. Lipari 2010(C) 2010, C. Faloutsos 14 SVD - Detailed outline Motivation Definition - properties Interpretation Complexity Case studies Additional properties

15. Lipari 2010(C) 2010, C. Faloutsos 15 SVD - Definition (reminder: matrix multiplication x 3 x 22 x 1 =

16. Lipari 2010(C) 2010, C. Faloutsos 16 SVD - Definition (reminder: matrix multiplication x 3 x 22 x 1 = 3 x 1

17. Lipari 2010(C) 2010, C. Faloutsos 17 SVD - Definition (reminder: matrix multiplication x 3 x 22 x 1 = 3 x 1

18. Lipari 2010(C) 2010, C. Faloutsos 18 SVD - Definition (reminder: matrix multiplication x 3 x 22 x 1 = 3 x 1

19. Lipari 2010(C) 2010, C. Faloutsos 19 SVD - Definition (reminder: matrix multiplication x=

20. Lipari 2010(C) 2010, C. Faloutsos 20 SVD - Definition A [n x m] = U [n x r]   r x r] (V [m x r] ) T A: n x m matrix (eg., n documents, m terms) U: n x r matrix (n documents, r concepts)  : r x r diagonal matrix (strength of each ‘concept’) (r : rank of the matrix) V: m x r matrix (m terms, r concepts)

21. Lipari 2010(C) 2010, C. Faloutsos 21 SVD - Definition A = U  V T - example:

22. Lipari 2010(C) 2010, C. Faloutsos 22 SVD - Properties THEOREM [Press+92]: always possible to decompose matrix A into A = U  V T, where U,  V: unique (*) U, V: column orthonormal (ie., columns are unit vectors, orthogonal to each other) –U T U = I; V T V = I (I: identity matrix)  : singular are positive, and sorted in decreasing order

23. Lipari 2010(C) 2010, C. Faloutsos 23 SVD - Example A = U  V T - example: data inf. retrieval brain lung = CS MD xx

24. Lipari 2010(C) 2010, C. Faloutsos 24 SVD - Example A = U  V T - example: data inf. retrieval brain lung = CS MD xx CS-concept MD-concept

25. Lipari 2010(C) 2010, C. Faloutsos 25 SVD - Example A = U  V T - example: data inf. retrieval brain lung = CS MD xx CS-concept MD-concept doc-to-concept similarity matrix

26. Lipari 2010(C) 2010, C. Faloutsos 26 SVD - Example A = U  V T - example: data inf. retrieval brain lung = CS MD xx ‘strength’ of CS-concept

27. Lipari 2010(C) 2010, C. Faloutsos 27 SVD - Example A = U  V T - example: data inf. retrieval brain lung = CS MD xx term-to-concept similarity matrix CS-concept

28. Lipari 2010(C) 2010, C. Faloutsos 28 SVD - Example A = U  V T - example: data inf. retrieval brain lung = CS MD xx term-to-concept similarity matrix CS-concept

29. Lipari 2010(C) 2010, C. Faloutsos 29 SVD - Detailed outline Motivation Definition - properties Interpretation Complexity Case studies Additional properties

30. Lipari 2010(C) 2010, C. Faloutsos 30 SVD - Interpretation #1 ‘documents’, ‘terms’ and ‘concepts’: U: document-to-concept similarity matrix V: term-to-concept sim. matrix  : its diagonal elements: ‘strength’ of each concept

31. Lipari 2010(C) 2010, C. Faloutsos 31 SVD – Interpretation #1 ‘documents’, ‘terms’ and ‘concepts’: Q: if A is the document-to-term matrix, what is A T A? A: Q: A A T ? A:

32. Lipari 2010(C) 2010, C. Faloutsos 32-32 SVD – Interpretation #1 ‘documents’, ‘terms’ and ‘concepts’: Q: if A is the document-to-term matrix, what is A T A? A: term-to-term ([m x m]) similarity matrix Q: A A T ? A: document-to-document ([n x n]) similarity matrix

33. Lipari 2010(C) 2010, C. Faloutsos 33 Copyright: Faloutsos, Tong (2009) 2-33 SVD properties V are the eigenvectors of the covariance matrix A T A U are the eigenvectors of the Gram (inner- product) matrix AA T Further reading: 1. Ian T. Jolliffe, Principal Component Analysis (2 nd ed), Springer, 2002. 2. Gilbert Strang, Linear Algebra and Its Applications (4 th ed), Brooks Cole, 2005.

34. Lipari 2010(C) 2010, C. Faloutsos 34 SVD - Interpretation #2 best axis to project on: (‘best’ = min sum of squares of projection errors)

35. Lipari 2010(C) 2010, C. Faloutsos 35 SVD - Motivation

36. Lipari 2010(C) 2010, C. Faloutsos 36 SVD - interpretation #2 minimum RMS error SVD: gives best axis to project v1 first singular vector

37. Lipari 2010(C) 2010, C. Faloutsos 37 SVD - Interpretation #2

38. Lipari 2010(C) 2010, C. Faloutsos 38 SVD - Interpretation #2 A = U  V T - example: = xx v1

39. Lipari 2010(C) 2010, C. Faloutsos 39 SVD - Interpretation #2 A = U  V T - example: = xx variance (‘spread’) on the v1 axis

40. Lipari 2010(C) 2010, C. Faloutsos 40 SVD - Interpretation #2 A = U  V T - example: –U  gives the coordinates of the points in the projection axis = xx

41. Lipari 2010(C) 2010, C. Faloutsos 41 SVD - Interpretation #2 More details Q: how exactly is dim. reduction done? = xx

42. Lipari 2010(C) 2010, C. Faloutsos 42 SVD - Interpretation #2 More details Q: how exactly is dim. reduction done? A: set the smallest singular values to zero: = xx

43. Lipari 2010(C) 2010, C. Faloutsos 43 SVD - Interpretation #2 ~ xx

44. Lipari 2010(C) 2010, C. Faloutsos 44 SVD - Interpretation #2 ~ xx

45. Lipari 2010(C) 2010, C. Faloutsos 45 SVD - Interpretation #2 ~ xx

46. Lipari 2010(C) 2010, C. Faloutsos 46 SVD - Interpretation #2 ~

47. Lipari 2010(C) 2010, C. Faloutsos 47 SVD - Interpretation #2 Exactly equivalent: ‘spectral decomposition’ of the matrix: = xx

48. Lipari 2010(C) 2010, C. Faloutsos 48 SVD - Interpretation #2 Exactly equivalent: ‘spectral decomposition’ of the matrix: = xx u1u1 u2u2 1 2 v1v1 v2v2

49. Lipari 2010(C) 2010, C. Faloutsos 49 SVD - Interpretation #2 Exactly equivalent: ‘spectral decomposition’ of the matrix: =u1u1 1 vT1vT1 u2u2 2 vT2vT2 + +... n m

50. Lipari 2010(C) 2010, C. Faloutsos 50 SVD - Interpretation #2 Exactly equivalent: ‘spectral decomposition’ of the matrix: =u1u1 1 vT1vT1 u2u2 2 vT2vT2 + +... n m n x 1 1 x m r terms

51. Lipari 2010(C) 2010, C. Faloutsos 51 SVD - Interpretation #2 approximation / dim. reduction: by keeping the first few terms (Q: how many?) =u1u1 1 vT1vT1 u2u2 2 vT2vT2 + +... n m assume: 1 >= 2 >=...

52. Lipari 2010(C) 2010, C. Faloutsos 52 SVD - Interpretation #2 A (heuristic - [Fukunaga]): keep 80-90% of ‘energy’ (= sum of squares of i ’s) =u1u1 1 vT1vT1 u2u2 2 vT2vT2 + +... n m assume: 1 >= 2 >=...

53. Lipari 2010(C) 2010, C. Faloutsos 53 SVD - Detailed outline Motivation Definition - properties Interpretation –#1: documents/terms/concepts –#2: dim. reduction –#3: picking non-zero, rectangular ‘blobs’ Complexity Case studies Additional properties

54. Lipari 2010(C) 2010, C. Faloutsos 54 SVD - Interpretation #3 finds non-zero ‘blobs’ in a data matrix = xx

55. Lipari 2010(C) 2010, C. Faloutsos 55 SVD - Interpretation #3 finds non-zero ‘blobs’ in a data matrix = xx

56. Lipari 2010(C) 2010, C. Faloutsos 56 SVD - Interpretation #3 finds non-zero ‘blobs’ in a data matrix = ‘communities’ (bi-partite cores, here) Row 1 Row 4 Col 1 Col 3 Col 4 Row 5 Row 7

57. Lipari 2010(C) 2010, C. Faloutsos 57 SVD - Detailed outline Motivation Definition - properties Interpretation Complexity Case studies Additional properties

58. Lipari 2010(C) 2010, C. Faloutsos 58 SVD - Complexity O( n * m * m) or O( n * n * m) (whichever is less) less work, if we just want singular values or if we want first k singular vectors or if the matrix is sparse [Berry] Implemented: in any linear algebra package (LINPACK, matlab, Splus, mathematica...)

59. Lipari 2010(C) 2010, C. Faloutsos 59 SVD - conclusions so far SVD: A= U  V T : unique (*) U: document-to-concept similarities V: term-to-concept similarities  : strength of each concept dim. reduction: keep the first few strongest singular values (80-90% of ‘energy’) –SVD: picks up linear correlations SVD: picks up non-zero ‘blobs’

60. Lipari 2010(C) 2010, C. Faloutsos 60 SVD - Detailed outline Motivation Definition - properties Interpretation Complexity SVD properties Case studies Conclusions

61. Lipari 2010(C) 2010, C. Faloutsos 61 SVD - Other properties - summary can produce orthogonal basis (obvious) (who cares?) can solve over- and under-determined linear problems (see C(1) property) can compute ‘fixed points’ (= ‘steady state prob. in Markov chains’) (see C(4) property)

62. Lipari 2010(C) 2010, C. Faloutsos 62 SVD -outline of properties (A): obvious (B): less obvious (C): least obvious (and most powerful!)

63. Lipari 2010(C) 2010, C. Faloutsos 63 Properties - by defn.: A(0): A [n x m] = U [ n x r ]  [ r x r ] V T [ r x m] A(1): U T [r x n] U [n x r ] = I [r x r ] (identity matrix) A(2): V T [r x n] V [n x r ] = I [r x r ] A(3):  k = diag( 1 k, 2 k,... r k ) (k: ANY real number) A(4): A T = V  U T

64. Lipari 2010(C) 2010, C. Faloutsos 64 Less obvious properties A(0): A [n x m] = U [ n x r ]  [ r x r ] V T [ r x m] B(1): A [n x m] (A T ) [m x n] = ??

65. Lipari 2010(C) 2010, C. Faloutsos 65 Less obvious properties A(0): A [n x m] = U [ n x r ]  [ r x r ] V T [ r x m] B(1): A [n x m] (A T ) [m x n] = U  2 U T symmetric; Intuition?

66. Lipari 2010(C) 2010, C. Faloutsos 66 Less obvious properties A(0): A [n x m] = U [ n x r ]  [ r x r ] V T [ r x m] B(1): A [n x m] (A T ) [m x n] = U  2 U T symmetric; Intuition? ‘document-to-document’ similarity matrix B(2): symmetrically, for ‘V’ (AT) [m x n] A [n x m] = V L2 VT Intuition?

67. Lipari 2010(C) 2010, C. Faloutsos 67 Less obvious properties A: term-to-term similarity matrix B(3): ( (A T ) [m x n] A [n x m] ) k = V  2k V T and B(4): (A T A ) k ~ v 1 1 2k v 1 T for k>>1 where v 1 : [m x 1] first column (singular-vector) of V 1 : strongest singular value

68. Lipari 2010(C) 2010, C. Faloutsos 68 Less obvious properties B(4): (A T A ) k ~ v 1 1 2k v 1 T for k>>1 B(5): (A T A ) k v’ ~ (constant) v 1 ie., for (almost) any v’, it converges to a vector parallel to v 1 Thus, useful to compute first singular vector/value (as well as the next ones, too...)

69. Lipari 2010(C) 2010, C. Faloutsos 69 Less obvious properties - repeated: A(0): A [n x m] = U [ n x r ]  [ r x r ] V T [ r x m] B(1): A [n x m] (A T ) [m x n] = U  2 U T B(2):(A T ) [m x n] A [n x m] = V  2 V T B(3): ( (A T ) [m x n] A [n x m] ) k = V  2k V T B(4): (A T A ) k ~ v 1 1 2k v 1 T B(5): (A T A ) k v’ ~ (constant) v 1

70. Lipari 2010(C) 2010, C. Faloutsos 70 Least obvious properties - cont’d A(0): A [n x m] = U [ n x r ]  [ r x r ] V T [ r x m] C(2): A [n x m] v 1 [m x 1] = 1 u 1 [n x 1] where v 1, u 1 the first (column) vectors of V, U. (v 1 == right-singular-vector) C(3): symmetrically: u 1 T A = 1 v 1 T u 1 == left-singular-vector Therefore:

71. Lipari 2010(C) 2010, C. Faloutsos 71 Least obvious properties - cont’d A(0): A [n x m] = U [ n x r ]  [ r x r ] V T [ r x m] C(4): A T A v 1 = 1 2 v 1 (fixed point - the dfn of eigenvector for a symmetric matrix)

72. Lipari 2010(C) 2010, C. Faloutsos 72 Least obvious properties - altogether A(0): A [n x m] = U [ n x r ]  [ r x r ] V T [ r x m] C(1): A [n x m] x [m x 1] = b [n x 1] then, x 0 = V  (-1) U T b: shortest, actual or least- squares solution C(2): A [n x m] v 1 [m x 1] = 1 u 1 [n x 1] C(3): u 1 T A = 1 v 1 T C(4): A T A v 1 = 1 2 v 1

73. Lipari 2010(C) 2010, C. Faloutsos 73 Properties - conclusions A(0): A [n x m] = U [ n x r ]  [ r x r ] V T [ r x m] B(5): (A T A ) k v’ ~ (constant) v 1 C(1): A [n x m] x [m x 1] = b [n x 1] then, x 0 = V  (-1) U T b: shortest, actual or least- squares solution C(4): A T A v 1 = 1 2 v 1

74. Lipari 2010(C) 2010, C. Faloutsos 74 SVD - detailed outline... SVD properties case studies –Kleinberg’s algorithm –Google’s algorithm Conclusions

75. Lipari 2010(C) 2010, C. Faloutsos 75 Kleinberg’s algo (HITS) Kleinberg, Jon (1998). Authoritative sources in a hyperlinked environment. Proc. 9th ACM-SIAM Symposium on Discrete Algorithms.

76. Lipari 2010(C) 2010, C. Faloutsos 76 Recall: problem dfn Given a graph (eg., web pages containing the desirable query word) Q: Which node is the most important?

77. Lipari 2010(C) 2010, C. Faloutsos 77 Kleinberg’s algorithm Problem dfn: given the web and a query find the most ‘authoritative’ web pages for this query Step 0: find all pages containing the query terms Step 1: expand by one move forward and backward

78. Lipari 2010(C) 2010, C. Faloutsos 78 Kleinberg’s algorithm Step 1: expand by one move forward and backward

79. Lipari 2010(C) 2010, C. Faloutsos 79 Kleinberg’s algorithm on the resulting graph, give high score (= ‘authorities’) to nodes that many important nodes point to give high importance score (‘hubs’) to nodes that point to good ‘authorities’) hubsauthorities

80. Lipari 2010(C) 2010, C. Faloutsos 80 Kleinberg’s algorithm observations recursive definition! each node (say, ‘i’-th node) has both an authoritativeness score a i and a hubness score h i

81. Lipari 2010(C) 2010, C. Faloutsos 81 Kleinberg’s algorithm Let E be the set of edges and A be the adjacency matrix: the (i,j) is 1 if the edge from i to j exists Let h and a be [n x 1] vectors with the ‘hubness’ and ‘authoritativiness’ scores. Then:

82. Lipari 2010(C) 2010, C. Faloutsos 82 Kleinberg’s algorithm Then: a i = h k + h l + h m that is a i = Sum (h j ) over all j that (j,i) edge exists or a = A T h k l m i

83. Lipari 2010(C) 2010, C. Faloutsos 83 Kleinberg’s algorithm symmetrically, for the ‘hubness’: h i = a n + a p + a q that is h i = Sum (q j ) over all j that (i,j) edge exists or h = A a p n q i

84. Lipari 2010(C) 2010, C. Faloutsos 84 Kleinberg’s algorithm In conclusion, we want vectors h and a such that: h = A a a = A T h Recall properties: C(2): A [n x m] v 1 [m x 1] = 1 u 1 [n x 1] C(3): u 1 T A = 1 v 1 T =

85. Lipari 2010(C) 2010, C. Faloutsos 85 Kleinberg’s algorithm In short, the solutions to h = A a a = A T h are the left- and right- singular-vectors of the adjacency matrix A. Starting from random a’ and iterating, we’ll eventually converge (Q: to which of all the singular-vectors? why?)

86. Lipari 2010(C) 2010, C. Faloutsos 86 Kleinberg’s algorithm (Q: to which of all the singular-vectors? why?) A: to the ones of the strongest singular-value, because of property B(5): B(5): (A T A ) k v’ ~ (constant) v 1

87. Lipari 2010(C) 2010, C. Faloutsos 87 Kleinberg’s algorithm - results Eg., for the query ‘java’: 0.328 www.gamelan.com 0.251 java.sun.com 0.190 www.digitalfocus.com (“the java developer”)

88. Lipari 2010(C) 2010, C. Faloutsos 88 Kleinberg’s algorithm - discussion ‘authority’ score can be used to find ‘similar pages’ (how?)

89. Lipari 2010(C) 2010, C. Faloutsos 89 SVD - detailed outline... Complexity SVD properties Case studies –Kleinberg’s algorithm (HITS) –Google’s algorithm Conclusions

90. Lipari 2010(C) 2010, C. Faloutsos 90 PageRank (google) Brin, Sergey and Lawrence Page (1998). Anatomy of a Large-Scale Hypertextual Web Search Engine. 7th Intl World Wide Web Conf. Larry Page Sergey Brin

91. Lipari 2010(C) 2010, C. Faloutsos 91 Problem: PageRank Given a directed graph, find its most interesting/central node A node is important, if it is connected with important nodes (recursive, but OK!)

92. Lipari 2010(C) 2010, C. Faloutsos 92 Problem: PageRank - solution Given a directed graph, find its most interesting/central node Proposed solution: Random walk; spot most ‘popular’ node (-> steady state prob. (ssp)) A node has high ssp, if it is connected with high ssp nodes (recursive, but OK!)

93. Lipari 2010(C) 2010, C. Faloutsos 93 (Simplified) PageRank algorithm Let A be the adjacency matrix; let B be the transition matrix: transpose, column-normalized - then 1 2 3 4 5 = To From B

94. Lipari 2010(C) 2010, C. Faloutsos 94 (Simplified) PageRank algorithm B p = p = 1 2 3 4 5

95. Lipari 2010(C) 2010, C. Faloutsos 95 Definitions AAdjacency matrix (from-to) DDegree matrix = (diag ( d1, d2, …, dn) ) BTransition matrix: to-from, column normalized B = A T D -1

96. Lipari 2010(C) 2010, C. Faloutsos 96 (Simplified) PageRank algorithm B p = 1 * p thus, p is the eigenvector that corresponds to the highest eigenvalue (=1, since the matrix is column-normalized ) Why does such a p exist? –p exists if B is nxn, nonnegative, irreducible [Perron–Frobenius theorem]

97. Lipari 2010(C) 2010, C. Faloutsos 97 (Simplified) PageRank algorithm In short: imagine a particle randomly moving along the edges compute its steady-state probabilities (ssp) Full version of algo: with occasional random jumps Why? To make the matrix irreducible

98. Lipari 2010(C) 2010, C. Faloutsos 98 Full Algorithm With probability 1-c, fly-out to a random node Then, we have p = c B p + (1-c)/n 1 => p = (1-c)/n [I - c B] -1 1

99. Lipari 2010(C) 2010, C. Faloutsos 99 Alternative notation MModified transition matrix M = c B + (1-c)/n 1 1 T Then p = M p That is: the steady state probabilities = PageRank scores form the first eigenvector of the ‘modified transition matrix’

100. Lipari 2010(C) 2010, C. Faloutsos 100 Parenthesis: intuition behind eigenvectors

101. Lipari 2010(C) 2010, C. Faloutsos 101 Formal definition If A is a (n x n) square matrix , x) is an eigenvalue/eigenvector pair of A if A x = x CLOSELY related to singular values:

102. Lipari 2010(C) 2010, C. Faloutsos 102 Property #1: Eigen- vs singular-values if B [n x m] = U [n x r]   r x r] (V [m x r] ) T then A = ( B T B ) is symmetric and C(4): B T B v i = i 2 v i ie, v 1, v 2,...: eigenvectors of A = (B T B)

103. Lipari 2010(C) 2010, C. Faloutsos 103 Property #2 If A [nxn] is a real, symmetric matrix Then it has n real eigenvalues (if A is not symmetric, some eigenvalues may be complex)

104. Lipari 2010(C) 2010, C. Faloutsos 104 Property #3 If A [nxn] is a real, symmetric matrix Then it has n real eigenvalues And they agree with its n singular values, except possibly for the sign

105. Lipari 2010(C) 2010, C. Faloutsos 105 Intuition A as vector transformation Axx’ = x 2 1 1 3

106. Lipari 2010(C) 2010, C. Faloutsos 106 Intuition By defn., eigenvectors remain parallel to themselves (‘fixed points’) Av1v1 v1v1 = 3.62 * 1

107. Lipari 2010(C) 2010, C. Faloutsos 107 Convergence Usually, fast:

108. Lipari 2010(C) 2010, C. Faloutsos 108 Convergence Usually, fast:

109. Lipari 2010(C) 2010, C. Faloutsos 109 Convergence Usually, fast: depends on ratio 1 : 2 1 2

110. Lipari 2010(C) 2010, C. Faloutsos 110 Kleinberg/google - conclusions SVD helps in graph analysis: hub/authority scores: strongest left- and right- singular-vectors of the adjacency matrix random walk on a graph: steady state probabilities are given by the strongest eigenvector of the (modified) transition matrix

111. Lipari 2010(C) 2010, C. Faloutsos 111 Conclusions SVD: a valuable tool given a document-term matrix, it finds ‘concepts’ (LSI)... and can find fixed-points or steady-state probabilities (google/ Kleinberg/ Markov Chains)

112. Lipari 2010(C) 2010, C. Faloutsos 112 Conclusions cont’d (We didn’t discuss/elaborate, but, SVD... can reduce dimensionality (KL)... and can find rules (PCA; RatioRules)... and can solve optimally over- and under- constraint linear systems (least squares / query feedbacks)

113. Lipari 2010(C) 2010, C. Faloutsos 113 References Berry, Michael: http://www.cs.utk.edu/~lsi/http://www.cs.utk.edu/~lsi/ Brin, S. and L. Page (1998). Anatomy of a Large-Scale Hypertextual Web Search Engine. 7th Intl World Wide Web Conf.

114. Lipari 2010(C) 2010, C. Faloutsos 114 References Christos Faloutsos, Searching Multimedia Databases by Content, Springer, 1996. (App. D)Searching Multimedia Databases by Content Fukunaga, K. (1990). Introduction to Statistical Pattern Recognition, Academic Press. I.T. Jolliffe Principal Component Analysis Springer, 2002 (2 nd ed.)

115. Lipari 2010(C) 2010, C. Faloutsos 115 References cont’d Kleinberg, J. (1998). Authoritative sources in a hyperlinked environment. Proc. 9th ACM-SIAM Symposium on Discrete Algorithms. Press, W. H., S. A. Teukolsky, et al. (1992). Numerical Recipes in C, Cambridge University Press. www.nr.com

116. Lipari 2010(C) 2010, C. Faloutsos 116 Outline Introduction – Motivation Task 1: Node importance Task 2: Recommendations & proximity Task 3: Connection sub-graphs Conclusions

117. Lipari 2010(C) 2010, C. Faloutsos 117 Acknowledgement : Most of the foils in ‘Task 2’ are by Hanghang TONG www.cs.cmu.edu/~htong

118. Lipari 2010(C) 2010, C. Faloutsos 118 Detailed outline Problem dfn and motivation Solution: Random walk with restarts Efficient computation Case study: image auto-captioning Extensions: bi-partite graphs; tracking Conclusions

119. Lipari 2010(C) 2010, C. Faloutsos 119 A B i i i i Motivation: Link Prediction Should we introduce Mr. A to Mr. B? ?

120. Lipari 2010(C) 2010, C. Faloutsos 120 Motivation - recommendations customers Products / movies ‘ smith ’ Terminator 2 ??

121. Lipari 2010(C) 2010, C. Faloutsos 121 Answer: proximity ‘yes’, if ‘A’ and ‘B’ are ‘close’ ‘yes’, if ‘smith’ and ‘terminator 2’ are ‘close’ QUESTIONS in this part: -How to measure ‘closeness’/proximity? -How to do it quickly? -What else can we do, given proximity scores?

122. Lipari 2010(C) 2010, C. Faloutsos 122 How close is ‘A’ to ‘B’? a.k.a Relevance, Closeness, ‘Similarity’…

123. Lipari 2010(C) 2010, C. Faloutsos 123 Why is it useful? Recommendation And many more Image captioning [Pan+] Conn. / CenterPiece subgraphs [Faloutsos+], [Tong+], [Koren+] and Link prediction [Liben-Nowell+], [Tong+] Ranking [Haveliwala], [Chakrabarti+] Email Management [Minkov+] Neighborhood Formulation [Sun+] Pattern matching [Tong+] Collaborative Filtering [Fouss+] …

124. Lipari 2010(C) 2010, C. Faloutsos 124 Test Image SeaSunSkyWaveCatForestTigerGrass Image Keyword Region Automatic Image Captioning Q: How to assign keywords to the test image? A: Proximity! [Pan+ 2004]

125. Lipari 2010(C) 2010, C. Faloutsos 125 Center-Piece Subgraph(CePS) Original Graph CePS Q: How to find hub for the black nodes? A: Proximity! [Tong+ KDD 2006] CePS guy Input Output

126. Detailed outline Problem dfn and motivation Solution: Random walk with restarts Efficient computation Case study: image auto-captioning Extensions: bi-partite graphs; tracking Conclusions Lipari 2010(C) 2010, C. Faloutsos 126

127. Lipari 2010(C) 2010, C. Faloutsos 127 How close is ‘A’ to ‘B’? Should be close, if they have - many, - short - ‘heavy’ paths

128. Lipari 2010(C) 2010, C. Faloutsos 128 Why not shortest path? A: ‘pizza delivery guy’ problem Some ``bad’’ proximities

129. Lipari 2010(C) 2010, C. Faloutsos 129 Why not max. netflow? A: No penalty for long paths Some ``bad’’ proximities

130. Lipari 2010(C) 2010, C. Faloutsos 130 What is a ``good’’ Proximity? Multiple Connections Quality of connection Direct & In-directed Conns Length, Degree, Weight… …

131. Lipari 2010(C) 2010, C. Faloutsos 131 1 4 3 2 5 6 7 9 10 8 1 1212 Random walk with restart [Haveliwala’02]

132. Lipari 2010(C) 2010, C. Faloutsos 132 Random walk with restart Node 4 Node 1 Node 2 Node 3 Node 4 Node 5 Node 6 Node 7 Node 8 Node 9 Node 10 Node 11 Node 12 0.13 0.10 0.13 0.22 0.13 0.05 0.08 0.04 0.03 0.04 0.02 1 4 3 2 5 6 7 9 10 8 1 1212 0.13 0.10 0.13 0.05 0.08 0.04 0.02 0.04 0.03 Ranking vector More red, more relevant Nearby nodes, higher scores

133. Lipari 2010(C) 2010, C. Faloutsos 133 Why RWR is a good score? all paths from i to j with length 1 all paths from i to j with length 2 all paths from i to j with length 3 : adjacency matrix. c: damping factor i j

134. Detailed outline Problem dfn and motivation Solution: Random walk with restarts –variants Efficient computation Case study: image auto-captioning Extensions: bi-partite graphs; tracking Conclusions Lipari 2010(C) 2010, C. Faloutsos 134

135. Lipari 2010(C) 2010, C. Faloutsos 135 Variant: escape probability Define Random Walk (RW) on the graph Esc_Prob(CMU  Paris) –Prob (starting at CMU, reaches Paris before returning to CMU) CMU Paris the remaining graph Esc_Prob = Pr (smile before cry)

136. Lipari 2010(C) 2010, C. Faloutsos 136 Other Variants Other measure by RWs –Community Time/Hitting Time [Fouss+] –SimRank [Jeh+] Equivalence of Random Walks –Electric Networks: EC [Doyle+]; SAEC[Faloutsos+]; CFEC[Koren+] –Spring Systems Katz [Katz], [Huang+], [Scholkopf+] Matrix-Forest-based Alg [Chobotarev+]

137. Lipari 2010(C) 2010, C. Faloutsos 137 Other Variants Other measure by RWs –Community Time/Hitting Time [Fouss+] –SimRank [Jeh+] Equivalence of Random Walks –Electric Networks: EC [Doyle+]; SAEC[Faloutsos+]; CFEC[Koren+] –Spring Systems Katz [Katz], [Huang+], [Scholkopf+] Matrix-Forest-based Alg [Chobotarev+] All are “related to” or “similar to” random walk with restart!

138. Lipari 2010(C) 2010, C. Faloutsos 138 Map of proximity measurements RWR Esc_Prob + Sink Hitting Time/ Commute Time Effective Conductance String System Regularized Un-constrained Quad Opt. Harmonic Func. Constrained Quad Opt. Mathematic Tools X out-degree “voltage = position” relax 4 ssp decides 1 esc_prob Katz Norma lize Physical Models

139. Lipari 2010(C) 2010, C. Faloutsos 139 Notice: Asymmetry (even in undirected graphs) A B C D E C-> A : high A-> C: low

140. Lipari 2010(C) 2010, C. Faloutsos 140 Summary of Proximity Definitions Goal: Summarize multiple relationships Solutions –Basic: Random Walk with Restarts [Haweliwala’02] [Pan+ 2004][Sun+ 2006][Tong+ 2006] –Properties: Asymmetry [Koren+ 2006][Tong+ 2007] [Tong+ 2008] –Variants: Esc_Prob and many others. [Faloutsos+ 2004] [Koren+ 2006][Tong+ 2007]

141. Detailed outline Problem dfn and motivation Solution: Random walk with restarts Efficient computation Case study: image auto-captioning Extensions: bi-partite graphs; tracking Conclusions Lipari 2010(C) 2010, C. Faloutsos 141

142. Lipari 2010(C) 2010, C. Faloutsos 142 Reminder: PageRank With probability 1-c, fly-out to a random node Then, we have p = c B p + (1-c)/n 1 => p = (1-c)/n [I - c B] -1 1

143. Lipari 2010(C) 2010, C. Faloutsos 143 Ranking vector Starting vector Adjacency matrix Restart p p = c B p + (1-c)/n 1 The only difference

144. Lipari 2010(C) 2010, C. Faloutsos 144 Computing RWR 1 4 3 2 5 6 7 9 10 8 1 1212 n x n n x 1 Ranking vector Starting vector Adjacency matrix 1 Restart p p = c B p + (1-c)/n 1

145. Lipari 2010(C) 2010, C. Faloutsos 145 Q: Given query i, how to solve it? ? ? Adjacency matrix Starting vector Ranking vector Query

146. Lipari 2010(C) 2010, C. Faloutsos 146 1 4 3 2 5 6 7 9 10 8 1 1212 0.13 0.10 0.13 0.05 0.08 0.04 0.02 0.04 0.03 OntheFly: 1 4 3 2 5 6 7 9 10 8 1 1212 No pre-computation/ light storage Slow on-line response O(mE)

147. Lipari 2010(C) 2010, C. Faloutsos 147 4 PreCompute 1 4 3 2 5 6 7 9 10 8 1 1212 0.13 0.10 0.13 0.05 0.08 0.04 0.02 0.04 0.03 1 3 2 5 6 7 9 10 8 1 1212 R:R: c x Q Q

148. Lipari 2010(C) 2010, C. Faloutsos 148 PreCompute: 1 4 3 2 5 6 7 9 10 8 1 1212 0.13 0.10 0.13 0.05 0.08 0.04 0.02 0.04 0.03 1 4 3 2 5 6 7 9 10 8 1 1212 Fast on-line response Heavy pre-computation/storage cost O(n ) 3 2

149. Lipari 2010(C) 2010, C. Faloutsos 149 Q: How to Balance? On-line Off-line

150. Lipari 2010(C) 2010, C. Faloutsos 150 How to balance? Idea (‘B-Lin’) Break into communities Pre-compute all, within a community Adjust (with S.M.) for ‘bridge edges’ H. Tong, C. Faloutsos, & J.Y. Pan. Fast Random Walk with Restart and Its Applications. ICDM, 613-622, 2006.

151. Lipari 2010(C) 2010, C. Faloutsos 151 Detailed outline Problem dfn and motivation Solution: Random walk with restarts Efficient computation Case study: image auto-captioning Extensions: bi-partite graphs; tracking Conclusions

152. Lipari 2010(C) 2010, C. Faloutsos 152 gCaP: Automatic Image Caption Q … SeaSunSkyWave {} {} CatForestGrassTiger {?, ?, ?,} A: Proximity! [Pan+ KDD2004]

153. Lipari 2010(C) 2010, C. Faloutsos 153 Test Image SeaSunSkyWaveCatForestTigerGrass Image Keyword Region

154. Lipari 2010(C) 2010, C. Faloutsos 154 Test Image SeaSunSkyWaveCatForestTigerGrass Image Keyword Region {Grass, Forest, Cat, Tiger}

155. Lipari 2010(C) 2010, C. Faloutsos 155 C-DEM (Screen-shot)

156. Lipari 2010(C) 2010, C. Faloutsos 156 C-DEM: Multi-Modal Query System for Drosophila Embryo Databases [Fan+ VLDB 2008]

157. Detailed outline Problem dfn and motivation Solution: Random walk with restarts Efficient computation Case study: image auto-captioning Extensions: bi-partite graphs; tracking Conclusions Lipari 2010(C) 2010, C. Faloutsos 157

158. Lipari 2010(C) 2010, C. Faloutsos 158 Problem: update E’ edges changed Involves n’ authors, m’ confs. n authors m Conferences

159. Lipari 2010(C) 2010, C. Faloutsos 159 Solution: Use Sherman-Morrison Lemma to quickly update the inverse matrix

160. Lipari 2010(C) 2010, C. Faloutsos 160 Fast-Single-Update 176x speedup 40x speedup log(Time) (Seconds) Datasets Our method

161. Lipari 2010(C) 2010, C. Faloutsos 161 pTrack: Philip S. Yu’s Top-5 conferences up to each year ICDE ICDCS SIGMETRICS PDIS VLDB CIKM ICDCS ICDE SIGMETRICS ICMCS KDD SIGMOD ICDM CIKM ICDCS ICDM KDD ICDE SDM VLDB 1992199720022007 Databases Performance Distributed Sys. Databases Data Mining DBLP: (Au. x Conf.) - 400k aus, - 3.5k confs - 20 yrs

162. Lipari 2010(C) 2010, C. Faloutsos 162 pTrack: Philip S. Yu’s Top-5 conferences up to each year ICDE ICDCS SIGMETRICS PDIS VLDB CIKM ICDCS ICDE SIGMETRICS ICMCS KDD SIGMOD ICDM CIKM ICDCS ICDM KDD ICDE SDM VLDB 1992199720022007 Databases Performance Distributed Sys. Databases Data Mining DBLP: (Au. x Conf.) - 400k aus, - 3.5k confs - 20 yrs

163. Lipari 2010(C) 2010, C. Faloutsos 163 KDD’s Rank wrt. VLDB over years Prox. Rank Year Data Mining and Databases are getting closer & closer

164. Lipari 2010(C) 2010, C. Faloutsos 164 cTrack:10 most influential authors in NIPS community up to each year Author-paper bipartite graph from NIPS 1987-1999. 3k. 1740 papers, 2037 authors, spreading over 13 years T. Sejnowski M. Jordan

165. Lipari 2010(C) 2010, C. Faloutsos 165 Conclusions - Take-home messages Proximity Definitions –RWR –and a lot of variants Computation –Sherman–Morrison Lemma –Fast Incremental Computation Applications –Recommendations; auto-captioning; tracking –Center-piece Subgraphs (next) –E-mail management; anomaly detection, …

166. Lipari 2010(C) 2010, C. Faloutsos 166 References L. Page, S. Brin, R. Motwani, & T. Winograd. (1998), The PageRank Citation Ranking: Bringing Order to the Web, Technical report, Stanford Library. T.H. Haveliwala. (2002) Topic-Sensitive PageRank. In WWW, 517-526, 2002 J.Y. Pan, H.J. Yang, C. Faloutsos & P. Duygulu. (2004) Automatic multimedia cross-modal correlation discovery. In KDD, 653-658, 2004.

167. Lipari 2010(C) 2010, C. Faloutsos 167 References C. Faloutsos, K. S. McCurley & A. Tomkins. (2002) Fast discovery of connection subgraphs. In KDD, 118-127, 2004. J. Sun, H. Qu, D. Chakrabarti & C. Faloutsos. (2005) Neighborhood Formation and Anomaly Detection in Bipartite Graphs. In ICDM, 418-425, 2005. W. Cohen. (2007) Graph Walks and Graphical Models. Draft.

168. Lipari 2010(C) 2010, C. Faloutsos 168 References P. Doyle & J. Snell. (1984) Random walks and electric networks, volume 22. Mathematical Association America, New York. Y. Koren, S. C. North, and C. Volinsky. (2006) Measuring and extracting proximity in networks. In KDD, 245–255, 2006. A. Agarwal, S. Chakrabarti & S. Aggarwal. (2006) Learning to rank networked entities. In KDD, 14-23, 2006.

169. Lipari 2010(C) 2010, C. Faloutsos 169 References S. Chakrabarti. (2007) Dynamic personalized pagerank in entity-relation graphs. In WWW, 571-580, 2007. F. Fouss, A. Pirotte, J.-M. Renders, & M. Saerens. (2007) Random-Walk Computation of Similarities between Nodes of a Graph with Application to Collaborative Recommendation. IEEE Trans. Knowl. Data Eng. 19(3), 355-369 2007.

170. Lipari 2010(C) 2010, C. Faloutsos 170 References H. Tong & C. Faloutsos. (2006) Center-piece subgraphs: problem definition and fast solutions. In KDD, 404-413, 2006. H. Tong, C. Faloutsos, & J.Y. Pan. (2006) Fast Random Walk with Restart and Its Applications. In ICDM, 613- 622, 2006. H. Tong, Y. Koren, & C. Faloutsos. (2007) Fast direction-aware proximity for graph mining. In KDD, 747-756, 2007.

171. Lipari 2010(C) 2010, C. Faloutsos 171 References H. Tong, B. Gallagher, C. Faloutsos, & T. Eliassi- Rad. (2007) Fast best-effort pattern matching in large attributed graphs. In KDD, 737-746, 2007. H. Tong, S. Papadimitriou, P.S. Yu & C. Faloutsos. (2008) Proximity Tracking on Time-Evolving Bipartite Graphs. SDM 2008.

172. Lipari 2010(C) 2010, C. Faloutsos 172 References B. Gallagher, H. Tong, T. Eliassi-Rad, C. Faloutsos. Using Ghost Edges for Classification in Sparsely Labeled Networks. KDD 2008 H. Tong, Y. Sakurai, T. Eliassi-Rad, and C. Faloutsos. Fast Mining of Complex Time-Stamped Events CIKM 08 H. Tong, H. Qu, and H. Jamjoom. Measuring Proximity on Graphs with Side Information. ICDM 2008

173. Lipari 2010(C) 2010, C. Faloutsos 173 Resources www.cs.cmu.edu/~htong/soft.htm For software, papers, and ppt of presentations www.cs.cmu.edu/~htong/tut/cikm2008/cikm_tutor ial.htmlwww.cs.cmu.edu/~htong/tut/cikm2008/cikm_tutor ial.html For the CIKM’08 tutorial on graphs and proximity Again, thanks to Hanghang TONG for permission to use his foils in this part

174. Lipari 2010(C) 2010, C. Faloutsos 174 Outline Introduction – Motivation Task 1: Node importance Task 2: Recommendations & proximity Task 3: Connection sub-graphs Conclusions

175. Lipari 2010(C) 2010, C. Faloutsos 175 Detailed outline Problem definition Solution Results H. Tong & C. Faloutsos Center-piece subgraphs: problem definition and fast solutions. In KDD, 404-413, 2006.

176. Lipari 2010(C) 2010, C. Faloutsos 176 Center-Piece Subgraph(Ceps) Given Q query nodes Find Center-piece ( ) Input of Ceps –Q Query nodes –Budget b –k softAnd number App. –Social Network –Law Inforcement –Gene Network –… B A CB A C

177. Lipari 2010(C) 2010, C. Faloutsos 177 Challenges in Ceps Q1: How to measure importance? (Q2: How to extract connection subgraph? Q3: How to do it efficiently?)

178. Lipari 2010(C) 2010, C. Faloutsos 178 Challenges in Ceps Q1: How to measure importance? A: “proximity” – but how to combine scores? (Q2: How to extract connection subgraph? Q3: How to do it efficiently?)

179. Lipari 2010(C) 2010, C. Faloutsos 179 AND: Combine Scores Q: How to combine scores?

180. Lipari 2010(C) 2010, C. Faloutsos 180 AND: Combine Scores Q: How to combine scores? A: Multiply …= prob. 3 random particles coincide on node j

181. Lipari 2010(C) 2010, C. Faloutsos 181 Detailed outline Problem definition Solution Results

182. Lipari 2010(C) 2010, C. Faloutsos 182 Case Study: AND query

183. Lipari 2010(C) 2010, C. Faloutsos 183 Case Study: AND query

184. Lipari 2010(C) 2010, C. Faloutsos 184 Conclusions Proximity (e.g., w/ RWR) helps answer ‘AND’ and ‘k_softAnd’ queries

185. Overall conclusions SVD: a powerful tool –HITS/ pageRank –(dimensionality reduction) Proximity: Random Walk with Restarts –Recommendation systems –Auto-captioning –Center-Piece Subgraphs Lipari 2010(C) 2010, C. Faloutsos 185