High Dimensional Probabilistic Modelling through Manifolds Neil Lawrence Machine Learning Group Department of Computer Science University of Sheffield, U.K.

Overview Motivation Mathematical Foundations Some Results Extensions Modelling high dimensional data. Smooth low dimensional embedded spaces. Mathematical Foundations Probabilistic PCA Gaussian Processes Some Results Extensions

Motivation

High Dimensional Data Handwritten digit: 3648 dimensions. Space contains more than just this digit.

Handwritten Digit A simple model of the digit – rotate the ‘prototype’. 10 20 30 40 50 60 10 20 30 40 50 60 10 20 30 40 50 60

Projection onto Principal Components

Discontinuities

Low Dimensional Manifolds Pure rotation of a prototype is too simple. In practice the data may go through several distortions, e.g. digits undergo thinning, translation and rotation. For data with ‘structure’: we expect fewer distortions than dimensions; we therefore expect the data to live in a lower dimension manifold. Deal with high dimensional data by looking for lower dimensional non-linear embedding.

Our Options Spectral Approaches Non-spectral approaches Classical Multidimensional Scaling (MDS) Uses eigenvectors of similarity matrix. LLE and Isomap are MDS with particular proximity measures. Kernel PCA Provides an embedding and a mapping from the high dimensional space to the embedding. The mapping is implied through the use of a kernel function as the similarity matrix. Non-spectral approaches Non-metric MDS and Sammon Mappings Iterative optimisation of a stress function. A mapping can be forced (e.g. Neuroscale).

Our Options Probabilistic Approaches Probabilistic PCA A linear method. Density Networks Use importance sampling and a multi-layer perceptron. GTM Uses a grid based sample and an RBF network. The difficulty for probabilistic approaches: propagate a distribution through a non-linear mapping.

The New Model PCA has a probabilistic interpretation. It is difficult to ‘non-linearise’. We present a new probabilistic interpretation of PCA. This can be made non-linear. The result is non-linear probabilistic PCA.

Mathematical Foundations

Notation q – dimension of latent/embedded space. d – dimension of data space. N – number of data points. centred data, . latent variables, . mapping matrix, W 2 <d£q. a(i) is vector from i th row of A ai is a vector from i th column of A

Reading Notation X and Y are design matrices. Covariance given by N-1YTY. Inner product matrix given by YYT.

Linear Embeddings Represent data, Y, with a lower dimensional embedding X. Assume a linear relationship of the form where

Probabilistic PCA X W Y W Y

Maximum Likelihood Solution If Uq are first q eigenvectors of N-1YTY and the corresponding eigenvalues are q. where V is an arbitrary rotation matrix.

PCA – Probabilistic Interpretation X W Y W Y

Dual Probabilistic PCA X W Y X Y

Maximum Likelihood Solution If Uq are first q eigenvectors of d-1YYT and the corresponding eigenvalues are q. where V is an arbitrary rotation matrix.

Maximum Likelihood Solution If Uq are first q eigenvectors of N-1YTY and the corresponding eigenvalues are q. where V is an arbitrary rotation matrix.

Equivalence of PPCA Formulations Solution for PPCA: Solution for Dual PPCA: Equivalence is from

Gaussian Processes

Gaussian Process (GP) Prior over functions. Functions are infinite dimensional. Distribution over instantiations: finite dimensional objects. Can prove by induction that GP is `consistent’. GP is defined by mean function and covariance function. Mean function often taken to be zero. Covariance function must be positive definite. Class of valid covariances is the same as Mercer Kernels.

Gaussian Processes A (zero mean) Gaussian Process likelihood is of the form where K is the covariance function or kernel. The linear kernel has the form

Dual Probabilistic PCA (revisited) X W Gaussian Process Y X Y

Dual Probabilistic PCA is a GPLVM Log-likelihood:

Non-linear Kernel Instead of linear kernel function. Use, for example, RBF kernel function. Leads to non-linear embeddings.

Pros & Cons of GPLVM Pros Cons Probabilistic Missing data straightforward. Can sample from model given X. Different noise models can be handled. Kernel parameters can be optimised. Cons Speed of optimisation. Optimisation is non-convex cf Classical MDS, kernel PCA.

Benchmark Examples

GP-LVM Optimisation Gradient based optimisation wrt X, , ,  (SCG). Example data-set Oil flow data Three phases of flow (stratified annular homogenous) Twelve measurement probes 1000 data-points We sub-sampled to 100 data-points.

PCA non-Metric MDS metric MDS GTM KPCA GP-LVM

Nearest Neighbour in X Number of errors for each method. PCA GP-LVM Non-metric MDS 20 4 13 Metric MDS GTM* Kernel PCA* 6 7 * These models required parameter selection.

Full Oil Data

Nearest Neighbour in X Number of errors for each method. PCA GTM GP-LVM 162 11 1

Applications

Applications Grochow et al Urtasun et al We’ve been looking at faces … Style Based Inverse Kinematics Urtasun et al A prior for tracking We’ve been looking at faces …

Face Animation Data from Electronic Arts OpenGL Code by Manuel Sanchez (now at Electronic Arts).

Extensions

Back Constraints GP-LVM Gives a smooth mapping from X to Y. Points close together in X will be close in Y. It does not imply points close in Y will be close in X. Kernel PCA gives a smooth mapping from Y to X. Points close together in Y will be close in X. It does not imply points close in X will be close in Y. (joint work with Joaquin Quiňonero Candela)

Back Constraints Maximise likelihood with constraint. Each latent point is given by a mapping from data space. For example the mapping could be a kernel:

Back Constrained GP-LVM X Gives a mapping in both directions, a GP mapping from X to Y and a reverse constraining mapping from Y to X. X constrained to be function of Y Y

Motion Capture with Back Constraints MATLAB demo Example in motion capture with RBF back constraints

Linear Back Constraints X X =YB Learn the projection matrix: B2<d£q Y As motivation consider PCA on a digit data set.

Reconstruction with GP

Linear Projection with GP-LVM PCA Constrained GP-LVM

Linear constrained GP-LVM Nearest Neighbour in X latent dim 2 3 4 PCA 131 115 47 Linear constrained GP-LVM 79 60 39 (c.f 24 errors for nearest neighbour in Y)

Ongoing Work Improving quality of learning in large data sets.

Conclusions

Conclusion Probabilistic non-linear interpretation of PCA. A probabilistic model for high dimensions. Back constraints can be introduced to improve visualisation. seek better linear projections