1. DATA MINING LECTURE 7 Dimensionality Reduction PCA – SVD
(Thanks to Jure Leskovec, Evimaria Terzi)

2. The curse of dimensionality
Real data usually have thousands, or millions of dimensions
E.g., web documents, where the dimensionality is the vocabulary of words
Facebook graph, where the dimensionality is the number of users
Huge number of dimensions causes problems
Data becomes very sparse, some algorithms become meaningless (e.g. density based clustering)
The complexity of several algorithms depends on the dimensionality and they become infeasible (e.g. nearest neighbor search).

3. Dimensionality Reduction
Usually the data can be described with fewer dimensions, without losing much of the meaning of the data.
The data reside in a space of lower dimensionality

4. Example In this data matrix the dimension is essentially 3
There are three types of products and three types of users

5. Example Cloud of points 3D space:
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets,
Example
Cloud of points 3D space:
Think of point positions as a matrix:
We can rewrite coordinates more efficiently!
Old basis vectors: [1 0 0] [0 1 0] [0 0 1]
New basis vectors: [1 2 1] [ ]
Then A has new coordinates: [1 0]. B: [0 1], C: [1 1]
Notice: We reduced the number of coordinates! A B C A
1 row per point:

6. Dimensionality Reduction
Find the “true dimension” of the data
In reality things are never as clear and simple as in this example, but we can still reduce the dimension.
Essentially, we assume that some of the data is noise, and we can approximate the useful part with a lower dimensionality space.
Dimensionality reduction does not just reduce the amount of data, it often brings out the useful part of the data

7. Dimensionality Reduction
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets,
Dimensionality Reduction
Goal of dimensionality reduction is to discover the axis of data!
Rather than representing every point with 2 coordinates we represent each point with
1 coordinate (corresponding to the position of the point on the red line).
By doing this we incur a bit of error as the points do not exactly lie on the line

8. Why Reduce Dimensions? Discover hidden correlations/topics
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets,
Why Reduce Dimensions?
Discover hidden correlations/topics
Words that occur commonly together
Remove redundant and noisy features
Not all words are useful
Interpretation and visualization
Easier storage and processing of the data

9. Dimensionality Reduction
We have already seen a form of dimensionality reduction
LSH, and random projections reduce the dimension while preserving the distances

10. Data in the form of a matrix
We are given 𝑛 objects and 𝑑 attributes describing the objects. Each object has 𝑑 numeric values describing it.
We will represent the data as a 𝑛×𝑑 real matrix A.
We can now use tools from linear algebra to process the data matrix
Our goal is to produce a new 𝑛×𝑘 matrix B such that
It preserves as much of the information in the original matrix A as possible
It reveals something about the structure of the data in A

11. Example: Document matrices
d terms
(e.g., theorem, proof, etc.)
Example: Document matrices
n documents
Aij = frequency of the j-th term in the i-th document
Find subsets of terms that bring documents together

12. Example: Recommendation systems
d movies
Example: Recommendation systems
n customers
Aij = rating of j-th product by the i-th customer
Find subsets of movies that capture the behavior or the customers

13. Linear algebra We assume that vectors are column vectors.
We use 𝑣 𝑇 for the transpose of vector 𝑣 (row vector)
Dot product: 𝑢 𝑇 𝑣 (1𝑛, 𝑛1 → 11)
The dot product is the projection of vector 𝑣 on 𝑢 (and vice versa)
1, 2, =12
𝑢 𝑇 𝑣= 𝑣 𝑢 cos (𝑢,𝑣)
If ||𝑢|| = 1 (unit vector) then 𝑢 𝑇 𝑣 is the projection length of 𝑣 on 𝑢
−1, 2, −1 2 =0 : orthogonal vectors
Orthonormal vectors: two unit vectors that are orthogonal

14. Matrices
An nm matrix A is a collection of n row vectors and m column vectors
𝐴= | | | 𝑎 1 𝑎 2 𝑎 3 | | | 𝐴= − 𝛼 1 𝑇 − − 𝛼 2 𝑇 − − 𝛼 3 𝑇 −
Matrix-vector multiplication
Right multiplication 𝐴𝑢: projection of u onto the row vectors of 𝐴, or projection of row vectors of 𝐴 onto 𝑢.
Left-multiplication 𝑢 𝑇 𝐴: projection of 𝑢 onto the column vectors of 𝐴, or projection of column vectors of 𝐴 onto 𝑢
Example:
1,2, =[1,2]

15. Rank
Row space of A: The set of vectors that can be written as a linear combination of the rows of A
All vectors of the form 𝑣= 𝑢 𝑇 𝐴
Column space of A: The set of vectors that can be written as a linear combination of the columns of A
All vectors of the form 𝑣=𝐴𝑢.
Rank of A: the number of linearly independent row (or column) vectors
These vectors define a basis for the row (or column) space of A
Example
Matrix A = has rank r=2
Why? The first two rows are linearly independent, so the rank is at least 2, but all three rows are linearly dependent (the first is equal to the sum of the second and third) so the rank must be less than 3.

16. Rank-1 matrices
In a rank-1 matrix, all columns (or rows) are multiples of the same column (or row) vector
𝐴= −1 2 4 −2 3 6 −3
All rows are multiples of 𝑟=[1,2,−1]
All columns are multiples of 𝑐= 1,2,3 𝑇
External product: 𝑢 𝑣 𝑇 (𝑛1 ,1𝑚 → 𝑛𝑚)
The resulting 𝑛𝑚 has rank 1: all rows (or columns) are linearly dependent
𝐴=𝑟 𝑐 𝑇

17. Eigenvectors
(Right) Eigenvector of matrix A: a vector v such that 𝐴𝑣=𝜆𝑣
𝜆: eigenvalue of eigenvector 𝑣
A square symmetric matrix A of rank r, has r orthonormal eigenvectors 𝑢 1 , 𝑢 2 ,…, 𝑢 𝑟 with eigenvalues 𝜆 1 , 𝜆 2 ,…, 𝜆 𝑟 .
Eigenvectors define an orthonormal basis for the column space of A
We can write:
𝐴=𝑈Λ 𝑈 𝑇

18. Singular Value Decomposition
𝐴=𝑈 Σ 𝑉 𝑇 = 𝑢 1 , 𝑢 2 , ⋯, 𝑢 𝑟 𝜎 𝜎 ⋱ 𝜎 𝑟 𝑣 1 𝑇 𝑣 2 𝑇 ⋮ 𝑣 𝑟 𝑇
𝜎 1 ,≥ 𝜎 2 ≥…≥ 𝜎 𝑟 : singular values of matrix 𝐴
𝑢 1 , 𝑢 2 ,…, 𝑢 𝑟 : left singular vectors of 𝐴
𝑣 1 , 𝑣 2 ,…, 𝑣 𝑟 : right singular vectors of 𝐴
𝐴= 𝜎 1 𝑢 1 𝑣 1 𝑇 + 𝜎 2 𝑢 2 𝑣 2 𝑇 +⋯+ 𝜎 𝑟 𝑢 𝑟 𝑣 𝑟 𝑇
[n×m] =
[n×r]
[r×r]
[r×m]
r: rank of matrix A

19. Singular Value Decomposition
The left singular vectors are an orthonormal basis for the row space of A.
The right singular vectors are an orthonormal basis for the column space of A.
If A has rank r, then A can be written as the sum of r rank-1 matrices
There are r “linear components” (trends) in A.
Linear trend: the tendency of the row vectors of A to align with vector v
Strength of the i-th linear trend: ||𝐴 𝒗 𝒊 || = 𝝈 𝒊

20. Symmetric matrices
Special case: A is symmetric positive definite matrix
𝐴= 𝜆 1 𝑢 1 𝑢 1 𝑇 + 𝜆 2 𝑢 2 𝑢 2 𝑇 +⋯+ 𝜆 𝑟 𝑢 𝑟 𝑢 𝑟 𝑇
𝜆 1 ≥ 𝜆 2 ≥⋯≥ 𝜆 𝑟 ≥0: Eigenvalues of A
𝑢 1 , 𝑢 2 ,…, 𝑢 𝑟 : Eigenvectors of A

21. An (extreme) example A = User-Movie matrix
Blue and Red rows (colums) are linearly dependent
There are two prototype users (vectors of movies): blue and red
To describe the data is enough to describe the two prototypes, and the projection weights for each row
A is a rank-2 matrix
𝐴= 𝑤 1 , 𝑤 𝑑 1 𝑇 𝑑 2 𝑇
A =

22. SVD – Example: Users-to-Movies
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets,
SVD – Example: Users-to-Movies
A = U  VT - example: Users to Movies
Casablanca
Serenity
Matrix
Amelie
Alien n m
SciFi  VT =
Romance U
“Concepts” AKA Latent dimensions AKA Latent factors

23. SVD – Example: Users-to-Movies
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets,
SVD – Example: Users-to-Movies
A = U  VT - example: Users to Movies
Casablanca
Serenity
Matrix
Amelie
Alien
SciFi-concept
Romance-concept
SciFi = x x
Romance

24. SVD – Example: Users-to-Movies
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets,
SVD – Example: Users-to-Movies
A = U  VT - example: Users to Movies
Casablanca
Serenity
Matrix
Amelie
Alien
SciFi-concept
Romance-concept
U is “user-to-concept”
similarity matrix
SciFi = x x
Romance

25. SVD – Example: Users-to-Movies
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets,
SVD – Example: Users-to-Movies
A = U  VT - example: Users to Movies
Casablanca
Serenity
Matrix
Amelie
Alien
SciFi-concept
Romance-concept
V is “movie to concept” similarity matrix
SciFi = x x
Romance

26. SVD – Example: Users-to-Movies
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets,
SVD – Example: Users-to-Movies
A = U  VT - example: Users to Movies
Casablanca
Serenity
Matrix
Amelie
Alien
SciFi-concept
Romance-concept
Σ is the “concept strength” matrix
SciFi = x x
Romance

27. SVD – Example: Users-to-Movies
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets,
SVD – Example: Users-to-Movies
A = U  VT - example: Users to Movies
Casablanca
Serenity
Matrix
Amelie
Alien
Movie 2
1st singular vector
Movie 1
SciFi = x x
Σ is the “spread (variance)” matrix
Romance

28. An (more realistic) example
User-Movie matrix
There are two prototype users and movies but they are noisy
A =

29. SVD – Example: Users-to-Movies
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets,
SVD – Example: Users-to-Movies
A = U  VT - example: Users to Movies
Casablanca
Serenity
Matrix
Amelie
Alien n m
SciFi  VT =
Romnce U
“Concepts” AKA Latent dimensions AKA Latent factors

30. SVD – Example: Users-to-Movies
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets,
SVD – Example: Users-to-Movies
A = U  VT - example: Users to Movies =
SciFi
Romance x
Casablanca
Serenity
Matrix
Amelie
Alien

31. SVD – Example: Users-to-Movies
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets,
SVD – Example: Users-to-Movies
A = U  VT - example: Users to Movies =
SciFi
Romance x
Casablanca
Serenity
Matrix
Amelie
Alien
SciFi-concept
Romance-concept
The first two vectors are more or less unchanged

32. SVD – Example: Users-to-Movies
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets,
SVD – Example: Users-to-Movies
A = U  VT - example: Users to Movies =
SciFi
Romance x
Casablanca
Serenity
Matrix
Amelie
Alien
The third vector has a very low singular value

33. SVD - Interpretation #1 ‘movies’, ‘users’ and ‘concepts’:
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets,
SVD - Interpretation #1
‘movies’, ‘users’ and ‘concepts’:
𝑼: user-to-concept similarity matrix
𝑽: movie-to-concept similarity matrix
: its diagonal elements: ‘strength’ of each concept

34. Rank-k approximation A = In this User-Movie matrix
We have more than two singular vectors, but the strongest ones are still about the two types.
The third models the noise in the data
By keeping the two strongest singular vectors we obtain most of the information in the data.
This is a rank-2 approximation of the matrix A
A =

35. Rank-k approximations (Ak)
n x d
n x k
k x k
k x d
𝑼𝒌 (𝑽𝒌): orthogonal matrix containing the top k left (right) singular vectors of A.
𝜮 𝒌 : diagonal matrix containing the top k singular values of A
Ak is an approximation of A
Ak is the best approximation of A

36. SVD as an optimization
The rank-k approximation matrix 𝐴 𝑘 produced by the top-k singular vectors of A minimizes the Frobenious norm of the difference with the matrix A
𝐴 𝑘 = arg max 𝐵:𝑟𝑎𝑛𝑘 𝐵 =𝑘 𝐴 −𝐵 𝐹 2
𝐴−𝐵 𝐹 2 = 𝑖,𝑗 𝐴 𝑖𝑗 − 𝐵 𝑖𝑗 2

37. What does this mean?
We can project the row (and column) vectors of the matrix A into a k-dimensional space and preserve most of the information
(Ideally) The k dimensions reveal latent features/aspects/topics of the term (document) space.
(Ideally) The 𝐴 𝑘 approximation of matrix A, contains all the useful information, and what is discarded is noise

38. Latent factor model
Rows (columns) are linear combinations of k latent factors
E.g., in our extreme document example there are two factors
Some noise is added to this rank-k matrix resulting in higher rank
SVD retrieves the latent factors (hopefully).

39. SVD and Rank-k approximations= U  VT
features
sig.
significant
noise
noise =
significant
noise
objects

40. Example = x More details Q: How exactly is dim. reduction done?
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets,
Example
More details
Q: How exactly is dim. reduction done?
A: Compute SVD = x

41. Example = x More details Q: How exactly is dim. reduction done?
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets,
Example
More details
Q: How exactly is dim. reduction done?
A: Set smallest singular values to zero = x

42. Example  x More details Q: How exactly is dim. reduction done?
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets,
Example
More details
Q: How exactly is dim. reduction done?
A: Set smallest singular values to zero x 

43. Example  x More details Q: How exactly is dim. reduction done?
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets,
Example
More details
Q: How exactly is dim. reduction done?
A: Set smallest singular values to zero x 

44. Example  x More details Q: How exactly is dim. reduction done?
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets,
Example
More details
Q: How exactly is dim. reduction done?
A: Set smallest singular values to zero  x

45. Example  More details Q: How exactly is dim. reduction done?
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets,
Example
More details
Q: How exactly is dim. reduction done?
A: Set smallest singular values to zero 
Frobenius norm:
ǁMǁF = Σij Mij2
ǁA-BǁF =  Σij (Aij-Bij)2
is “small”

46. Application: Recommender systems
Data: Users rating movies
Sparse and often noisy
Assumption: There are k basic user profiles, and each user is a linear combination of these profiles
E.g., action, comedy, drama, romance
Each user is a weighted cobination of these profiles
The “true” matrix has rank k
If we had the matrix A with all ratings of all users for all movies, the matrix 𝐴 𝑘 would tell us the true preferences of the users for the movies
What we observe is a noisy, and incomplete version of this matrix 𝐴

47. Model-based Recommendation Systems
We are given the incomplete matrix 𝐴 and we would like to get the missing ratings that 𝐴 𝑘 would produce
Algorithm: compute the rank-k approximation 𝐴 𝑘 of and matrix 𝐴 predict for user 𝑢 and movie 𝑚, the value 𝐴 𝑘 [𝑚,𝑢].
The rank-k approximation 𝐴 𝑘 is provably close to 𝐴 𝑘
Model-based collaborative filtering

48. Example = x Missing ratings 1 1 1 0 0 0 3 3 0 0 4 4 0 0 0 5 5 5 0 0
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets,
Example
Missing ratings = x

49. Example = x Missing ratings 1 1 1 0 0 0 3 3 0 0 4 4 0 0 0 5 5 5 0 0
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets,
Example
Missing ratings = x

50. Example Reconstruction of missing ratings 0.96 1.14 0.82 -0.01 -0.01

51. SVD and PCA
PCA is a special case of SVD on the centered covariance matrix.

52. Covariance matrix
Goal: reduce the dimensionality while preserving the “information in the data”
Information in the data: variability in the data
We measure variability using the covariance matrix.
Sample covariance of variables X and Y
𝑖 𝑥 𝑖 − 𝜇 𝑋 𝑇 ( 𝑦 𝑖 − 𝜇 𝑌 )
Given matrix A, remove the mean of each column from the column vectors to get the centered matrix C
The matrix 𝑉 = 𝐶 𝑇 𝐶 is the covariance matrix of the row vectors of A.

53. PCA: Principal Component Analysis
We will project the rows of matrix C into a new set of attributes (dimensions) such that:
The attributes have zero covariance to each other (they are orthogonal)
Each attribute captures the most remaining variance in the data, while orthogonal to the existing attributes
The first attribute should capture the most variance in the data
For matrix C, the variance of the rows of C when projected to vector x is given by 𝜎 2 = 𝐶𝑥 2
The right singular vector of C maximizes 𝜎 2 !

54. PCA Input: 2-d dimensional points Output: 1st (right) singular vector:
2nd (right) singular vector
2nd (right) singular vector:
direction of maximal variance, after removing the projection of the data along the first singular vector.
1st (right) singular vector
1st (right) singular vector:
direction of maximal variance,

55. Singular values
2nd (right) singular vector
1: measures how much of the data variance is explained by the first singular vector.
2: measures how much of the data variance is explained by the second singular vector.
1st (right) singular vector 1

56. Singular values tell us something about the variance
The variance in the direction of the k-th principal component is given by the corresponding singular value σk2
Singular values can be used to estimate how many components to keep
Rule of thumb: keep enough to explain 85% of the variation:

57. Example 𝐴= 𝑎 11 ⋯ 𝑎 1𝑛 ⋮ ⋱ ⋮ 𝑎 𝑛1 ⋯ 𝑎 𝑛𝑛 𝐴=𝑈Σ 𝑉 𝑇
drugs
𝐴= 𝑎 11 ⋯ 𝑎 1𝑛 ⋮ ⋱ ⋮ 𝑎 𝑛1 ⋯ 𝑎 𝑛𝑛
𝐴=𝑈Σ 𝑉 𝑇
First right singular vector 𝑣 1
More or less same weight to all drugs
Discriminates heavy from light users
Second right singular vector
Positive values for legal drugs, negative for illegal
students
legal
illegal
𝑎 𝑖𝑗 : usage of student i of drug j
Drug 1
Drug 2

58. Another property of PCA/SVD
The chosen vectors are such that minimize the sum of square differences between the data vectors and the low-dimensional projections
1st (right) singular vector

59. Application Latent Semantic Indexing (LSI):
Apply PCA on the document-term matrix, and index the k-dimensional vectors
When a query comes, project it onto the k-dimensional space and compute cosine similarity in this space
Principal components capture main topics, and enrich the document representation

60. SVD is “the Rolls-Royce and the Swiss Army Knife of Numerical Linear Algebra.”*
*Dianne O’Leary, MMDS ’06

61. Computation of eigenvectors
Consider a symmetric square matrix 𝑀
Power-method:
Start with the vector 𝑣 of all 1’s
Compute 𝑣 = 𝑀𝑣
Normalize by the length of v
Repeat until the vector does not change
This will give us the first eigenvector.
The first eigenvalue is 𝜆= 𝑣 𝑇 𝑀𝑣
For the second one, compute the first eigenvector of the matrix 𝑀 ∗ =𝑀 −𝜆𝑣 𝑣 𝑇

62. Singular Values and Eigenvalues
The left singular vectors of 𝐴 are also the eigenvectors of 𝐴 𝐴 𝑇
The right singular vectors of 𝐴 are also the eigenvectors of 𝐴 𝑇 𝐴
The singular values of matrix 𝐴 are also the square roots of eigenvalues of 𝐴 𝐴 𝑇 and 𝐴 𝑇 𝐴

63. Computing singular vectors
Compute the eigenvectors and eigenvalues of the matrices 𝑀 𝑀 𝑇 and 𝑀 𝑇 𝑀