1. Dimensionality Reduction Part 2: Nonlinear Methods
Comp
Spring 2007

2. Why Dimensionality Reduction
Two approaches to reduce number of features
Feature selection: select the salient features by some criteria
Feature extraction: obtain a reduced set of features by a transformation of all features
Data visualization and exploratory data analysis also need to reduce dimension
Usually reduce to 2D or 3D

3. Deficiencies of Linear Methods
Data may not be best summarized by linear combination of features
Example: PCA cannot discover 1D structure of a helix

4. Intuition: how does your brain store these pictures?

5. Brain Representation

6. Brain Representation Every pixel?
Or perceptually meaningful structure?
Up-down pose
Left-right pose
Lighting direction
So, your brain successfully reduced the high-dimensional inputs to an intrinsically 3-dimensional manifold!

7. Manifold Learning Y X
latent
observed
Discover low dimensional representations (smooth manifold) for data in high dimension.
Linear approaches(PCA, MDS)
Non-linear approaches (ISOMAP, LLE, others)

8. Linear Approach- PCA
PCA Finds subspace linear projections of input data.

9. Linear Method Linear Methods for Dimensionality Reduction
PCA: rotate data so that principal axes lie in direction of maximum variance
MDS: find coordinates that best preserve pairwise distances
PCA

10. Motivation Linear Dimensionality Reduction doesn’t always work
Data violates underlying “linear” assumptions
Data is not accurately modeled by “affine” combinations of measurements
Structure of data, while apparent, is not simple
In the end, linear methods do nothing more than “globally transform” (rate, translate, and scale) all of the data, sometime what’s needed is to “unwrap” the data first

11. Stopgap Remedies Local PCA Neural Networks
Compute PCA models for small overlapping item neighborhoods
Requires a clustering preprocess
Fast and simple, but results in no global parameterization
Neural Networks
Assumes a solution of a given dimension
Uses relaxation methods to deform given solution to find a better fit
Relaxation step is modeled as “layers” in a network where properties of future iterations are computed based on information from the current structure
Many successes, but a bit of an art

12. Why Linear Modeling Fails
Suppose that your sample data lies on some low-dimensional surface embedded within the high-dimensional measurement space.
Linear models allow ALL affine combinations
Often, certain combinations are atypical of the actual data
Recognizing this is harder as dimensionality increases

13. What does PCA Really Model?
Principle Component Analysis assumptions
Mean-centered distribution
What if the mean, itself is atypical?
Eigenvectors of Covariance
Basis vectors aligned with successive directions of greatest variance
Classic 1st Order statistical model
Distribution is characterized by its mean and variance (Gaussian Hyperspheres)

14. Non-Linear Dimensionality Reduction
Non-linear Manifold Learning
Instead of preserving global pairwise distances, non-linear dimensionality reduction tries to preserve only the geometric properties of local neighborhoods
Discover a lower-dimensional “embedding” manifold
Find a parameterization over that manifold
Linear parameter space
Projection mapping from original M-D space to d-D embedding space
“reprojection, elevating, or lifting”
“projection”
Linear Embedding Space

15. Nonlinear DimRedux Steps
Discover a low-dimensional embedding manifold
Find a parameterization over the manifold
Project data into parameter space
Analyze, interpolate, and compress in embedding space
Orient (by linear transformation) the parameter space to align axes with salient features
Linear (affine) combinations are valid here
In the case of interpolation and compression use “lifting” to estimate M-D original data

16. Nonlinear Methods Local Linear Embeddings [Roweis 2000]
Isomaps [Tenenbaum 2000]
These two papers ignited the field
Principled approach (Asymptotically, as the amount of data goes to infinity they have been proven to find the “real” manifold)
Widely applied
Hotly contested

17. Nonlinear Approaches- Isomap
Josh. Tenenbaum, Vin de Silva, John langford 2000
Constructing neighbourhood graph G
For each pair of points in G, Computing shortest path distances ---- geodesic distances.
Use Classical MDS with geodesic distances.
Euclidean distance Geodesic distance

18. Sample points with Swiss Roll
Altogether there are 20,000 points in the “Swiss roll” data set. We sample 1000 out of 20,000.

19. Construct neighborhood graph G
K- nearest neighborhood (K=7)
DG is 1000 by 1000 (Euclidean) distance matrix of two neighbors (figure A)

20. Compute all-points shortest path in G
Now DG is 1000 by 1000 geodesic distance matrix of two arbitrary points along the manifold (figure B)

21. Use MDS to embed graph in Rd
Find a d-dimensional Euclidean space Y (Figure c) to preserve the pariwise diatances.

22. Isomap
Small Euclidean distance
Key Observation: On a manifold distances are measured using geodesic distances rather than Euclidean distances
Large geodesic distance

23. Problem: How to Get Geodesics
Without knowledge of the manifold it is difficult to compute the geodesic distance between points
It is even difficult if you know the manifold
Solution
Use a discrete geodesic approximation
Apply a graph algorithm to approximate the geodesic distances

24. Dijkstra’s Algorithm
Efficient Solution to all-points-shortest path problem
Greedy breath-first algorithm

25. Dijkstra’s Algorithm
Efficient Solution to all-points-shortest path problem
Greedy breath-first algorithm

26. Dijkstra’s Algorithm
Efficient Solution to all-points-shortest path problem
Greedy breath-first algorithm

27. Dijkstra’s Algorithm
Efficient Solution to all-points-shortest path problem
Greedy breath-first algorithm

28. Isomap algorithm
Compute fully-connected neighborhood of points for each item
Can be k nearest neighbors or ε-ball
Neighborhoods must be symmetric
Test that resulting graph is fully-connected, if not increase either K or 
Calculate pairwise Euclidean distances within each neighborhood
Use Dijkstra’s Algorithm to compute shortest path from each point to non-neighboring points
Run MDS on resulting distance matrix

29. Isomap Results
Find a 2D embedding of the 3D S-curve (also shown for LLE)
Isomap does a good job of preserving metric structure (not surprising)
The affine structure is also well preserved

30. Residual Fitting Error

31. Neighborhood Graph

32. More Isomap Results

33. More Isomap Results

34. Isomap Failures
Isomap also has problems on closed manifolds of arbitrary topology

35. Non-Linear Example
A Data-Driven Reflectance Model (Matusik et al, Siggraph2003)
Bidirectional Reflectance Distribution Functions(BRDF)
Define ratio of the reflected radiance in a particular direction to the incident irradiance from direction.
Isotropic BRDF

36. Measurement
Modeling Bidirectional Reflectance Distribution Functions (BRDFs)

37. Measurement
A “fast” BRDF measurement device inspired by Marshner[1998]

38. Measurement 20-80 million reflectance measurements per material
Each tabulated BRDF entails 90x90x180x3=4,374,000 measurement bins

39. Measurement 20-80 million reflectance measurements per material
Each tabulated BRDF entails 90x90x180x3=4,374,000 measurement bins

40. Rendering from Tabulated BRDFs
Even without further analysis, our BRDFs are immediately useful
Renderings made with Henrik Wann Jensen’s Dali renderer
Nickel
Hematite
Gold Paint
Pink Felt

41. BRDFs as Vectors in High-Dimensional Space
Each tabulated BRDF is a vector in 90x90x180x3 =4,374,000 dimensional space
180
Unroll 90 90
4,374,000

42. Linear Analysis (PCA) Find optimal “linear basis” for our data set
Eigenvalue
magnitude
Find optimal “linear basis” for our data set
45 components needed to reduce
residue to under
measurement error 20 40 60 80
100
120
Dimension
mean 5 10 20 30 45 60
all

43. Problems with Linear Subspace Modeling
Large number of basis vectors (45)
Some linear combinations yield invalid or unlikely BRDFs (outside convex hull)

44. Problems with Linear Subspace Modeling
Large number of basis vectors (45)
Some linear combinations yield invalid or unlikely BRDFs (inside convex hull)

45. Results of Non-Linear Manifold Learning
At 15 dimensions
reconstruction error
is less than 1%
Parameter count
similar to
analytical models
Error 5 10 15
Dimensionality

46. Non-Linear Advantages
15-dimensional parameter space
More robust than linear model
More extrapolations are plausible
Linear Model Extrapolation
Non-linear Model
Extrapolation

47. Non-Linear Model Results

48. Non-Linear Model Results

49. Non-Linear Model Results

50. Representing Physical Processes
Steel Oxidation

51. Local Linear Embeddings
First Insight
Locally, at a fine enough scale, everything looks linear

52. Local Linear Embeddings
First Insight
Find an affine combination the “neighborhood” about a point that best approximates it

53. Finding a Good Neighborhood
This is the remaining “Art” aspect of nonlinear methods
Common choices
-ball: find all items that lie within an epsilon ball of the target item as measured under some metric
Best if density of items is high and every point has a sufficient number of neighbors
K-nearest neighbors: find the k-closest neighbors to a point under some metric
Guarantees all items are similarly represented, limits dimension to K-1

54. Characterictics of a Manifoldx1 x2 R2 Rn M x1 x2 R2 Rn z x
x: coordinate for z
Locally it is a linear patch
Key: how to combine all local
patches together?

55. LLE: Intuition
Assumption: manifold is approximately “linear” when viewed locally, that is, in a small neighborhood
Approximation error, e(W), can be made small
Local neighborhood is effected by the constraint Wij=0 if zi is not a neighbor of zj
A good projection should preserve this local geometric property as much as possible

56. LLE: Intuition We expect each data point and its
neighbors to lie on or close
to a locally linear patch of the
manifold.
Each point can be written as a linear combination of its neighbors.
The weights chosen to
minimize the reconstruction
Error.

57. LLE: Intuition
The weights that minimize the reconstruction errors are invariant to rotation, rescaling and translation of the data points.
Invariance to translation is enforced by adding the constraint that the weights sum to one.
The weights characterize the intrinsic geometric properties of each neighborhood.
The same weights that reconstruct the data points in D dimensions should reconstruct it in the manifold in d dimensions.
Local geometry is preserved

58. LLE: Intuition Use the same weights from the original space
Low-dimensional embedding
the i-th row of W
Use the same weights
from the original space

59. Local Linear Embedding (LLE)
Assumption: manifold is approximately “linear” when viewed locally, that is, in a small neighborhood
Approximation error, e(W), can be made small
Meaning of W: a linear representation of every data point by its neighbors
This is an intrinsic geometrical property of the manifold
A good projection should preserve this geometric property as much as possible

60. Constrained Least Square problem
Compute the optimal weight for each point individually:
Neightbors of x
Zero for all non-neighbors of x

61. Finding a Map to a Lower Dimensional Space
Yi in Rk: projected vector for Xi
The geometrical property is best preserved if the error below is small
Y is given by the eigenvectors of the lowest d non-zero eigenvalues of the matrix
Use the same weights
computed above

62. Find Weights Rewriting as a matrix for all x Reorganizing
Want to find W that minimizes , and satisfies “sum-to-one” constraint
Ends up as constrained “least-squares” problem
“Unknown W matrix” N N N M M N

63. Find Linear Embedding Space
Now that we have the weight matrix W, find the linear vector that satisfies the following where W is N x N and X is M x N
This can be found by finding the null space of
Classic problem: run SVD on and find the orthogonal vector associated with the smallest d singular values (the smallest singular value will be zero and represent the system’s invariance to translation)

64. Numerical Issues Numerical problems can arise in computing LLEs
The least-squared covariance matrix that arises in the computation of the weighting matrix, W, solution can be ill-conditioned
Regularization (rescale the measurements by adding a small multiple of the Identity to covariance matrix)
Finding small singular (eigen) values is not as well conditioned as finding large ones. The small ones are subject to numerical precision errors, and to get mixed
Good (but slow) solvers exist, you have to use them

65. Results
The resulting parameter vector, yi, gives the coordinates associated with the item xi
The dth embedding coordinate is formed from the orthogonal vector associated with the dst singular value of A.

66. Reprojection Often, for data analysis, a parameterization is enough
For interpolation and compression we might want to map points from the parameter space back to the “original” space
No perfect solution, but a few approximations
Delauney triangulate the points in the embedding space, find the triangle that the desired parameter setting falls into, and compute the baricenric coordinates of it, and use them as weights
Interpolate by using a radially symmetric kernel centered about the desired parameter setting
Works, but mappings might not be one-to-one

67. LLE Example 3-D S-Curve manifold with points color-coded
Compute a 2-D embedding
The local affine structure is well maintained
The metric structure is okay locally, but can drift slowly over the domain (this causes the manifold to taper)

68. More LLE Examples

69. More LLE Examples

70. LLE Failures Does not work on to closed manifolds
Cannot recognize Topology

71. Summary Non-Linear Dimensionality Reduction Methods Comparisons Issues
These methods are considerably more powerful and temperamental than linear method
Applications of these methods are a hot area of research
Comparisons
LLE is generally faster, but more brittle than Isomaps
Isomaps tends to work better on smaller data sets (i.e. less dense sampling)
Isomaps tends to be less sensitive to noise (perturbation of the input vectors)
Issues
Neither method handles closed manifolds and topological variations well