1. Graph Based Semi- Supervised Learning Fei Wang Department of Statistical Science Cornell University

2. Research Overview Machine Learning AlgorithmsApplications OptimizationProbabilistic Information- Theoretic Computer Vision Multimedia Analysis Information Retrieval … … Bioinformatics

3. Research Overview 2003.9-2008.7 Department of Automation Tsinghua University ICML06 CVPR06 Graph Based Semi-supervised Learning ICDM06

4. Research Overview 2003.9-2008.7 Department of Automation Tsinghua University SDM08 ICML08 Maximum Margin Clustering KDD08: Semi-supervised Support Vector Machine ICDM08: Maximum Margin Feature Extraction CVPR09: Maximum Margin Feature Selection with Manifold Regularization

5. Research Overview IJCAI09, ICDM09 DMKD, submitted 2003.9-2008.7 Department of Automation Tsinghua University 2008.9-2009.8 School of CIS FIU

6. Research Overview ISMIR2009

7. Research Overview 2008.9-2009.8 School of CIS FIU2009.8-present Department of Statistical Science Cornell University Large Scale Statistical Machine Learning Random Projection for NMF

8. Linear Neighborhood Propagation (LNP) Graph Construction

9. Machine Learning Supervised Learning Unsupervised Learning Semi-supervised Learning

10. Data Relationships Traditional machine learning algorithms usually make the i.i.d. assumption There are relationships among data points Relationship is everywhere

11. Graph Is Everywhere Internet Graph Internet Graph http://www.religioustolerance.org/pagegraph.htm Friendship Graph http://aicozyzawa.files.wordpress.com/2009/05/march08- friend-graph-facebo1.jpg Protein Interaction Graph http://www.systemsbiology.org.au/images/geomi_network.jpg

12. Graph Based SSL The graph nodes are the data points The graph edges correspond to the data relationships

13. Label Propagation Initial label vector if is labeled, otherwise Zhu & Gaharamani, 2002

14. Label Propagation If is labeled, ; otherwise Matrix form W y f

15. Label Propagation If is labeled, ; otherwise Matrix form

16. Label Propagation The process will finally converge.

17. The Construction of W (Zhu el al, ICML 2003) (Zhou et al, NIPS 2004) Similarity Matrix Degree Matrix

18. Linear Neighborhood Each data point can be linearly reconstructed from its neighborhood

19. A Toy Example

20. Application on Image Segmentation

21. Comparisons LNP Partially Labeled Original Graph Cut Random Walk For more examples see “Linear Neighborhood Propagation and Its Applications”. PAMI 2009

22. Application on Text Classification

23. Poisson Propagation Problem Solution

24. Optimization Framework Local Predicted Label Variation Predicted Label Smoothness

25. Data Manifold The high-dimensional data points are not everywhere in the data space They usually (nearly) reside on a low dimensional manifold embedded in the high dimensional space A manifold is a mathematical space that on a small enough scale resembles the Euclidean space of a certain dimension, called the dimension of the manifold From Wiki

26. Laplace-Beltrami Operator Laplace Operator : a second order differential operator defined in an Euclidean space Hessian Laplace-Beltrami Operator is a second order differential operator in a Riemannian manifold, it is an analog of the Laplace operator in the Euclidean space

27. Graph Laplacian An operator in the continuous space will be degenerated to a matrix in the discrete space Graph Laplacian is the discrete analog of the Laplace Beltrami operator on continuous manifold Similarity Matrix Degree Matrix Theorem: Assume the data set is sampled from a continuous manifold, and the neighborhood points are uniformly distributed on the sphere around the center point. If W is constructed by LNP, then statistically L = I - W provides a discrete approximation of the L-B Operator

28. Laplace’s Equation Dirichlet Boundary Condition LNP Traditional GBSSL

29. Label Field vs. Electric Field r The data graph can be viewed as the discretized form of the data manifold There is a label field on the data manifold. The predicted data labels are just the label potentials at their corresponding places Vacuum Permittivity Q Q

30. Poisson’s Equation Assume that the charges are continuously distributed in the Euclidean space with charge density, then the electric potential satisfies Laplace Operator Poisson’s Equation Consider a Riemannian space, where the Laplace operator becomes the Laplace-Beltrami operator

31. Laplace’s Equation vs. Poisson’s Equation Poisson’s equation The value of the electric potential V in an electric field on a Riemannian manifold with charge density satisfies the following Poisson’s equation Laplace’s equation Generally SSL on the data manifold solves the following Laplace’s equation There is no label sources on the data manifold Where does the label come from?

32. GBSSL by Solving Poisson’s Equation Assume the label sources are placed at the positions of the labeled points, then the label source distribution becomes Green’s Function Green’s function

33. Discrete Green’s Function The discrete Green’s function is defined as the inverse of the graph Laplacian by discarding its zero eigen-mode, i.e. Chung & Yau. Discrete Green’s Functions. J. Combinatorial Theory. 2000

34. Poisson Propagation Predicted Label Vector Discrete Green’s Function Initial Label Vector

35. Experiments SVMTSVMGRFPPcLNPPPl g214c47.3224.7239.9632.1334.5632.05 g241d46.7250.0846.5544.3345.8544.72 Digit130.6017.779.804.287.324.11 COIL68.3667.5059.6352.3057.1650.19 USPS23.2124.9814.3713.1213.9612.89 BCI49.8549.1550.3647.7750.3247.68 Text63.2344.5252.7738.2649.7136.54 http://www.kyb.tuebingen.mpg.de/ssl-book/benchmarks.html

36. Conclusions Linear Neighborhood Propagation: Construct the graphs through linear reconstruction of the neighborhoods Poisson Propagation: Get the data label predictions through solving a Poisson’s Equation, rather than Laplace’s Equation Efficient Implementation: 1.Approximating the eigen-system of the graph Laplacian 2.Algebraic Multigrid

37. Thank You Q&A