1. High Dimensional Probabilistic Modelling through Manifolds
Neil Lawrence
Machine Learning Group
Department of Computer Science
University of Sheffield, U.K.

2. Overview Motivation Mathematical Foundations Some Results Extensions
Modelling high dimensional data.
Smooth low dimensional embedded spaces.
Mathematical Foundations
Probabilistic PCA
Gaussian Processes
Some Results
Extensions

3. Motivation

4. High Dimensional Data Handwritten digit:
3648 dimensions.
Space contains more than just this digit.

5. Handwritten Digit
A simple model of the digit – rotate the ‘prototype’. 10 20 30 40 50 60 10 20 30 40 50 60 10 20 30 40 50 60

6. Projection onto Principal Components

7. Discontinuities

8. Low Dimensional Manifolds
Pure rotation of a prototype is too simple.
In practice the data may go through several distortions,
e.g. digits undergo thinning, translation and rotation.
For data with ‘structure’:
we expect fewer distortions than dimensions;
we therefore expect the data to live in a lower dimension manifold.
Deal with high dimensional data by looking for lower dimensional non-linear embedding.

9. Our Options Spectral Approaches Non-spectral approaches
Classical Multidimensional Scaling (MDS)
Uses eigenvectors of similarity matrix.
LLE and Isomap are MDS with particular proximity measures.
Kernel PCA
Provides an embedding and a mapping from the high dimensional space to the embedding.
The mapping is implied through the use of a kernel function as the similarity matrix.
Non-spectral approaches
Non-metric MDS and Sammon Mappings
Iterative optimisation of a stress function.
A mapping can be forced (e.g. Neuroscale).

10. Our Options Probabilistic Approaches
Probabilistic PCA
A linear method.
Density Networks
Use importance sampling and a multi-layer perceptron.
GTM
Uses a grid based sample and an RBF network.
The difficulty for probabilistic approaches:
propagate a distribution through a non-linear mapping.

11. The New Model PCA has a probabilistic interpretation.
It is difficult to ‘non-linearise’.
We present a new probabilistic interpretation of PCA.
This can be made non-linear.
The result is non-linear probabilistic PCA.

12. Mathematical Foundations

13. Notation q – dimension of latent/embedded space.
d – dimension of data space.
N – number of data points.
centred data,
latent variables,
mapping matrix, W 2 <d£q.
a(i) is vector from i th row of A
ai is a vector from i th column of A

14. Reading Notation X and Y are design matrices.
Covariance given by N-1YTY.
Inner product matrix given by YYT.

15. Linear Embeddings
Represent data, Y, with a lower dimensional embedding X.
Assume a linear relationship of the form
where

16. Probabilistic PCA X W Y W Y

17. Maximum Likelihood Solution
If Uq are first q eigenvectors of N-1YTY and the corresponding eigenvalues are q.
where V is an arbitrary rotation matrix.

18. PCA – Probabilistic InterpretationX W Y W Y

19. Dual Probabilistic PCAX W Y X Y

20. Maximum Likelihood Solution
If Uq are first q eigenvectors of d-1YYT and the corresponding eigenvalues are q.
where V is an arbitrary rotation matrix.

21. Maximum Likelihood Solution
If Uq are first q eigenvectors of N-1YTY and the corresponding eigenvalues are q.
where V is an arbitrary rotation matrix.

22. Equivalence of PPCA Formulations
Solution for PPCA:
Solution for Dual PPCA:
Equivalence is from

23. Gaussian Processes

24. Gaussian Process (GP) Prior over functions.
Functions are infinite dimensional.
Distribution over instantiations: finite dimensional objects.
Can prove by induction that GP is `consistent’.
GP is defined by mean function and covariance function.
Mean function often taken to be zero.
Covariance function must be positive definite.
Class of valid covariances is the same as Mercer Kernels.

25. Gaussian Processes
A (zero mean) Gaussian Process likelihood is of the form
where K is the covariance function or kernel.
The linear kernel has the form

26. Dual Probabilistic PCA (revisited)X W
Gaussian Process Y X Y

27. Dual Probabilistic PCA is a GPLVM
Log-likelihood:

28. Non-linear Kernel Instead of linear kernel function.
Use, for example, RBF kernel function.
Leads to non-linear embeddings.

29. Pros & Cons of GPLVM Pros Cons Probabilistic
Missing data straightforward.
Can sample from model given X.
Different noise models can be handled.
Kernel parameters can be optimised.
Cons
Speed of optimisation.
Optimisation is non-convex cf Classical MDS, kernel PCA.

30. Benchmark Examples

31. GP-LVM Optimisation Gradient based optimisation wrt X, , ,  (SCG).
Example data-set
Oil flow data
Three phases of flow (stratified annular homogenous)
Twelve measurement probes
1000 data-points
We sub-sampled to 100 data-points.

32. PCA
non-Metric MDS
metric MDS
GTM
KPCA
GP-LVM

33. Nearest Neighbour in X Number of errors for each method.
PCA
GP-LVM
Non-metric MDS 20 4 13
Metric MDS
GTM*
Kernel PCA* 6 7
* These models required parameter selection.

34. Full Oil Data

35. Nearest Neighbour in X Number of errors for each method. PCA GTM
GP-LVM
162 11 1

36. Applications

37. Applications Grochow et al Urtasun et al We’ve been looking at faces …
Style Based Inverse Kinematics
Urtasun et al
A prior for tracking
We’ve been looking at faces …

38. Face Animation Data from Electronic Arts
OpenGL Code by Manuel Sanchez (now at Electronic Arts).

39. Extensions

40. Back Constraints GP-LVM Gives a smooth mapping from X to Y.
Points close together in X will be close in Y.
It does not imply points close in Y will be close in X.
Kernel PCA gives a smooth mapping from Y to X.
Points close together in Y will be close in X.
It does not imply points close in X will be close in Y.
(joint work with Joaquin Quiňonero Candela)

41. Back Constraints Maximise likelihood with constraint.
Each latent point is given by a mapping from data space.
For example the mapping could be a kernel:

42. Back Constrained GP-LVMX
Gives a mapping in both
directions, a GP mapping
from X to Y and a reverse
constraining mapping
from Y to X.
X constrained to be function of Y Y

43. Motion Capture with Back Constraints
MATLAB demo
Example in motion capture with RBF back constraints

44. Linear Back ConstraintsX
X =YB
Learn the projection matrix: B2<d£q Y
As motivation consider PCA on a digit data set.

45. Reconstruction with GP

46. Linear Projection with GP-LVM
PCA
Constrained GP-LVM

47. Linear constrained GP-LVM
Nearest Neighbour in X
latent dim 2 3 4
PCA
131
115 47
Linear constrained GP-LVM 79 60 39
(c.f 24 errors for nearest neighbour in Y)

48. Ongoing Work
Improving quality of learning in large data sets.

49. Conclusions

50. Conclusion Probabilistic non-linear interpretation of PCA.
A probabilistic model for high dimensions.
Back constraints can be introduced to
improve visualisation.
seek better linear projections