1. Nonlinear Dimension Reduction: Semi-Definite Embedding vs. Local Linear Embedding Li Zhang and Lin Liao

2. Outline Nonlinear Dimension Reduction Semi-Definite Embedding Local Linear Embedding Experiments

3. Dimension Reduction To understand images in terms of their basic modes of variability. Unsupervised learning problem: Given N high dimensional input X i R D, find a faithful one-to-one mapping to N low dimensional output Y i R d and d<D. Methods: Linear methods (PCA, MDS): subspace Nonlinear methods (SDE, LLE): manifold

4. Semi-Definite Embedding Given input X=(X 1,...,X N ) and k Find the k nearest neighbors for each input X i Formulate and solve a corresponding semi-definite programming problem; find optimal Gram matrix of output K=Y T Y Extract approximately a low dimensional embedding Y from the eigenvectors and eigenvalues of Gram matrix K

5. Semi-Definite Programming Maximize C·X Subject to AX=b matrix(X) is positive semi-definite where X is a vector with size n 2, and matrix(X) is a n by n matrix reshaped from X

6. Semi-Definite Programming Constraints: Maintain the distance between neighbors |Y i -Y j | 2= |X i -X j | 2 for each pair of neighbor (i,j)  K ii +K jj -K ij -K ji = G ii +G jj -G ij -G ji where K=Y T Y,G=X T X Constrain the output centered on the origin ΣY i =0  ΣK ij =0 K is positive semidefinite

7. Semi-Definite Programming Objective function Maximize the sum of pairwise squared distance between outputs Σ ij |Y i -Y j | 2  Tr(K)

8. Semi-Definite Programming Solve the best K using any SDP solver CSDP (fast, stable) SeDuMi (stable, slow) SDPT3 (new, fastest, not well tested)

9. Locally Linear Embedding

10. Swiss Roll N=800 SDE, k=4 LLE, k=18

11. LLE on Swiss Roll, varying K K=5K=6 K=8K=10

12. LLE on Swiss Roll, varying K K=12K=14 K=16K=18

13. LLE on Swiss Roll, varying K K=20K=30 K=40K=60

14. Twos N=638 SDE, k=4 LLE, k=18

15. Teapots N=400 SDE, k=4 LLE, k=12

16. LLE on Teapot, varying N N=400N=200 N=100N=50

17. Faces N=1900 SDE, failed LLE, k=12

18. SDE versus LLE Similar idea First, compute neighborhoods in the input space Second, construct a square matrix to characterize local relationship between input data. Finally, compute low-dimension embedding using the eigenvectors of the matrix

19. SDE versus LLE Different performance SDE: good quality, more robust to sparse samples, but optimization is slow and hard to scale to large data set LLE: fast, scalable to large data set, but low quality when samples are sparse, due to locally linear assumption