Graph Based Semi- Supervised Learning Fei Wang Department of Statistical Science Cornell University

Research Overview Machine Learning AlgorithmsApplications OptimizationProbabilistic Information- Theoretic Computer Vision Multimedia Analysis Information Retrieval … … Bioinformatics

Research Overview Department of Automation Tsinghua University ICML06 CVPR06 Graph Based Semi-supervised Learning ICDM06

Research Overview Department of Automation Tsinghua University SDM08 ICML08 Maximum Margin Clustering KDD08: Semi-supervised Support Vector Machine ICDM08: Maximum Margin Feature Extraction CVPR09: Maximum Margin Feature Selection with Manifold Regularization

Research Overview IJCAI09, ICDM09 DMKD, submitted Department of Automation Tsinghua University School of CIS FIU

Research Overview ISMIR2009

Research Overview School of CIS FIU present Department of Statistical Science Cornell University Large Scale Statistical Machine Learning Random Projection for NMF

Linear Neighborhood Propagation (LNP) Graph Construction

Machine Learning Supervised Learning Unsupervised Learning Semi-supervised Learning

Data Relationships Traditional machine learning algorithms usually make the i.i.d. assumption There are relationships among data points Relationship is everywhere

Graph Is Everywhere Internet Graph Internet Graph Friendship Graph friend-graph-facebo1.jpg Protein Interaction Graph

Graph Based SSL The graph nodes are the data points The graph edges correspond to the data relationships

Label Propagation Initial label vector if is labeled, otherwise Zhu & Gaharamani, 2002

Label Propagation If is labeled, ; otherwise Matrix form W y f

Label Propagation If is labeled, ; otherwise Matrix form

Label Propagation The process will finally converge.

The Construction of W (Zhu el al, ICML 2003) (Zhou et al, NIPS 2004) Similarity Matrix Degree Matrix

Linear Neighborhood Each data point can be linearly reconstructed from its neighborhood

A Toy Example

Application on Image Segmentation

Comparisons LNP Partially Labeled Original Graph Cut Random Walk For more examples see “Linear Neighborhood Propagation and Its Applications”. PAMI 2009

Application on Text Classification

Poisson Propagation Problem Solution

Optimization Framework Local Predicted Label Variation Predicted Label Smoothness

Data Manifold The high-dimensional data points are not everywhere in the data space They usually (nearly) reside on a low dimensional manifold embedded in the high dimensional space A manifold is a mathematical space that on a small enough scale resembles the Euclidean space of a certain dimension, called the dimension of the manifold From Wiki

Laplace-Beltrami Operator Laplace Operator : a second order differential operator defined in an Euclidean space Hessian Laplace-Beltrami Operator is a second order differential operator in a Riemannian manifold, it is an analog of the Laplace operator in the Euclidean space

Graph Laplacian An operator in the continuous space will be degenerated to a matrix in the discrete space Graph Laplacian is the discrete analog of the Laplace Beltrami operator on continuous manifold Similarity Matrix Degree Matrix Theorem: Assume the data set is sampled from a continuous manifold, and the neighborhood points are uniformly distributed on the sphere around the center point. If W is constructed by LNP, then statistically L = I - W provides a discrete approximation of the L-B Operator

Laplace’s Equation Dirichlet Boundary Condition LNP Traditional GBSSL

Label Field vs. Electric Field r The data graph can be viewed as the discretized form of the data manifold There is a label field on the data manifold. The predicted data labels are just the label potentials at their corresponding places Vacuum Permittivity Q Q

Poisson’s Equation Assume that the charges are continuously distributed in the Euclidean space with charge density, then the electric potential satisfies Laplace Operator Poisson’s Equation Consider a Riemannian space, where the Laplace operator becomes the Laplace-Beltrami operator

Laplace’s Equation vs. Poisson’s Equation Poisson’s equation The value of the electric potential V in an electric field on a Riemannian manifold with charge density satisfies the following Poisson’s equation Laplace’s equation Generally SSL on the data manifold solves the following Laplace’s equation There is no label sources on the data manifold Where does the label come from?

GBSSL by Solving Poisson’s Equation Assume the label sources are placed at the positions of the labeled points, then the label source distribution becomes Green’s Function Green’s function

Discrete Green’s Function The discrete Green’s function is defined as the inverse of the graph Laplacian by discarding its zero eigen-mode, i.e. Chung & Yau. Discrete Green’s Functions. J. Combinatorial Theory. 2000

Poisson Propagation Predicted Label Vector Discrete Green’s Function Initial Label Vector

Experiments SVMTSVMGRFPPcLNPPPl g214c g241d Digit COIL USPS BCI Text

Conclusions Linear Neighborhood Propagation: Construct the graphs through linear reconstruction of the neighborhoods Poisson Propagation: Get the data label predictions through solving a Poisson’s Equation, rather than Laplace’s Equation Efficient Implementation: 1.Approximating the eigen-system of the graph Laplacian 2.Algebraic Multigrid

Thank You Q&A