Variational Graph Embedding for Globally and Locally Consistent Feature Extraction Shuang Hong Yang College of Computing, Georgia Tech, USA Hongyuan Zha S. Kevin Zhou Siemens Corporate Research Inc., USA Bao-Gang Hu Chinese Academy of Sciences, China Presented by Yang (shy@gatech.edu) on 09/08/2009 at ECML PKDD 2009

Motivation No.1 Graph-Based Learning Learning by exploiting relationships in data: Kernel / (dis-)similarity learning Metric Learning Dimensionality reduction, Manifold Learning Spectral Clustering Semi-Supervised Classification / Clustering Relational Learning Collaborative filtering / co-clustering … … Data graph: G= (V, E, W) V: node set a node v  a data x E: edge set W: edge weights wij=w(xi,xj)

Motivation No.1 Graph-Based Learning Learning by exploiting relationships in data: Kernel / (dis-)similarity learning Metric Learning Dimensionality reduction, Manifold Learning Spectral Clustering Semi-Supervised Classification / Clustering Relational Learning Collaborative filtering / co-clustering … … Where comes a graph? Existing graph construction: Prior: prior knowledge, side information, etc Naive: gaussian similarity, ε-ball, kNN, b-matching, etc. Data graph: G= (V, E, W) V: node set a node v  a data x E: edge set W: edge weights wij=w(xi,xj)

Motivation No.1 Graph-Based Learning Learning by exploiting relationships in data: Kernel / (dis-)similarity learning Metric Learning Dimensionality reduction, Manifold Learning Spectral Clustering Semi-Supervised Classification / Clustering Relational Learning Collaborative filtering / co-clustering … … Where comes a graph? Existing graph construction: Prior: prior knowledge, side information, etc Naive: gaussian similarity, ε-ball, kNN, b-matching, etc. GBL is very sensitive to graph structure and edge weight. Q1: How to construct a reliable & effective graph? Data graph: G= (V, E, W) V: node set a node v  a data x E: edge set W: edge weights wij=w(xi,xj)

Motivation No.2 Feature Extraction find a mapping such that the high-dimensional data x is compressed to a low-dimensional one z Existing FE methods: either of the two classes Global/Statistical: optimizing globally defined statistical measures (variance, entropy, correlation, Fisher information, etc.) E.g.: PCA, FDA, other classical methods Good to preserv global properties but suboptimal when the underlying assuptions are violated (e.g., multi-modality) Local/Geometric Preserving the local geometric (submanifold) structure from which the data are sampled E.g.: ISOMAP, LLE, Laplacian Eigenmap, LPP Good to preserv local/geometric structure, but neglecting the overall properties of the data, e.g., relavance to label, variance, redundancy

Motivation No.2 Feature Extraction find a mapping such that the high-dimensional data x is compressed to a low-dimensional one z Existing FE methods: either of the two classes Global/Statistical: optimizing globally defined statistical measures (variance, entropy, correlation, Fisher information, etc.) E.g.: PCA, FDA, other classical methods Good to preserv global properties but suboptimal when the underlying assuptions are violated (e.g., multi-modality) Local/Geometric Preserving the local geometric (submanifold) structure from which the data are sampled E.g.: ISOMAP, LLE, Laplacian Eigenmap, LPP Good to preserv local/geometric structure, but neglecting the overall properties of the data, e.g., relavance to label, variance, redundancy

Motivation No.2 Feature Extraction find a mapping such that the high-dimensional data x is compressed to a low-dimensional one z Existing FE methods: either of the two classes Global/Statistical: optimizing globally defined statistical measures (variance, entropy, correlation, Fisher information, etc.) E.g.: PCA, FDA, other classical methods Good to preserv global properties but suboptimal when the underlying assuptions are violated (e.g., multi-modality) Local/Geometric Preserving the local geometric (submanifold) structure from which the data are sampled E.g.: ISOMAP, LLE, Laplacian Eigenmap, LPP Good to preserv local/geometric structure, but neglecting the overall properties of the data, e.g., relavance to label, variance, redundancy

Motivation No.2 Feature Extraction find a mapping such that the high-dimensional data x is compressed to a low-dimensional one z Existing FE methods: either of the two classes Global/Statistical: optimizing globally defined statistical measures (variance, entropy, correlation, Fisher information, etc.) E.g.: PCA, FDA, other classical methods Good to preserv global properties but suboptimal when the underlying assuptions are violated (e.g., multi-modality) Local/Geometric Preserving the local geometric (submanifold) structure from which the data are sampled E.g.: ISOMAP, LLE, Laplacian Eigenmap, LPP Good to preserv local/geometric structure, but neglecting the overall properties of the data, e.g., relavance to label, variance, redundancy Q2: Is it possible to combine the two pros into one framework?

This Paper Well-defined graph can be established based on theoretically justified learning measures. Take feature extraction as an example Graph-based Learning task, Mutual Information (MI) and Bayes Error Rate (BER) as example measures, we show that: Both global/statistical and local/geometric structures of the data can be captured using a single objective, leading to high-quality feature learner with various appealing properties Algorithm: Variational Graph-Embedding: iterative between graph learning and spectral graph embedding

Variational Graph Embedding for FE Outline Motivation Variational Graph Embedding for FE Theoretically Optimal Measures: MI and BER Variational Graph Embedding (MI) Variational Graph Embedding (BER) Experiment

Optimal Feature Extraction Theoretical Optimal Measures for feature learning Mutual Information (MI) accounts for high-order statistics, i.e., complete dependency y--z Bayes Error Rate (BER) directly maximizing the discrimination/generalization ability Practical prohibition Estimation: Both involve unkown distributions, and numerical integration of high-dimensional data Optimization: coupled variables, complicated objective

Two steps making things easy Our Solution Two steps making things easy Nonparametric Estimation Eliminates the numerical integration of unknown high-dimensional distributions, reduces the task into kernel-based optimization Variational Kernel Approximation Turns complex optimization into variational graph embedding The resultant algorithm is a EM-style iteration between learning a graph (optimizing variational parameters) and embedding a learned graph (optimizing learner parameters)

Variational Graph Embedding: MI 1. Max Nonparametric Quadratic MI 1.1 Shannon entropy  Renyi’s quadratic entropy 1.2 Kernel density estimation: isotropic Gaussian kernel Optimizing a big sum of nonconvex functions!

Variational Graph Embedding: MI 1. Max Nonparametric Quadratic MI 2. Max Variational Nonparametric QMI 1.1 Shannon entropy  Renyi’s quadratic entropy 1.2 Kernel density estimation: isotropic Gaussian kernel 2. kernel term  variational kernel term (Jaakkola-Jordan Variational Lower Bound for exp function) Variational Graph Embedding (MI) E-Step: optimizing variational parameters (equivalent to learning a weighted graph) M-Step: linearly embedding graph (can be solved by spectral analysis in linear or kernel space)

Variational Graph Embedding: MI Initial Graph Approximate each kernel term by its first-order Taylor expansion

Variational Graph Embedding: MI Justification Max-Relevance-Min-Redundancy with nature tradeoff

Variational Graph Embedding: MI Justification Max-Relevance-Min-Redundancy with nature tradeoff Max Discrimination

Variational Graph Embedding: MI Justification Max-Relevance-Min-Redundancy with nature tradeoff Max Discrimination Locality Preserving

Variational Graph Embedding: MI Justification Max-Relevance-Min-Redundancy with nature tradeoff Max Discrimination Locality Preserving

Variational Graph Embedding: MI Justification Max-Relevance-Min-Redundancy with nature tradeoff Max Discrimination Locality Preserving Connection with LPP and FDA

Variational Graph Embedding: BER Bayes Error Rate Nonparametric Estimation Variational kernel approximation Variational Graph Embedding =

Variational Graph Embedding: BER Variational Graph Embedding (MI) E-Step: optimizing variational parameters (equivalent to learning a weighted graph) M-Step: linearly embedding graph (can be solved by spectral analysis in linear or kernel space)

Experiments Face recognition Compared with both global/statistical methods (PCA/KPCA, LDA/KDA) and local/geometric methods (LPP/KLPP, MFA/KMFA) Compared with both supervised (LDA, MFA) and unsupervised (PCA, LPP) Three benchmark facial image sets (Yale, ORL and CMU Pie) are used. All the images were taken in different environments, at different times, with different poses, facial expressions and details. All the raw images are normalized to 32x32. For each data set, we randomly select v images of each person as training data, and leave others for testing. Only the training data are used to learn features. To evaluate the effectiveness of different methods, the classification accuracy of a k-NN classifier on testing data is used as the evaluation metric.

Experiments Face recognition

Experiments Face recognition On average, MIE is 36% over PCA, 8% over FDA, and 4% over MFA; BERE is 39% over PCA, 11% over FDA and 6% over MFA. The improvements are even more significant in the kernel case: kMIE (31%,9%,7%) and kBERE(33%,10%,8%) over (KPCA, KDA, KMFA)

Discussion 1. It is possible to capture both global/statistical and local/geometric structure into one well-defined objective. 2. Graph construction: What makes a good graph? Predictive: relevant to the target concept we are inferring Locally and globally consistent: account for both local & global information revealed by the data, and strive for a nature tradeoff Computationally convenient: easy and inexpensive for learning and testing A feasible graph construction approach: nonparametric measure + variational kernel approximation Kernel / (dis-)similarity learning Metric Learning, Dimensionality reduction Spectral Clustering Semi-Supervised Classification / Clustering Relational Learning Collaborative filtering / co-clustering, … …

Thanks!