1. Probabilistic Dimensional Reduction with the Gaussian Process Latent Variable Model
Neil Lawrence
Machine Learning Group
Department of Computer Science
University of Sheffield, U.K.

2. Overview Motivation Mathematical Foundations Results and Extensions
Modelling high dimensional data.
Smooth low dimensional embedded spaces.
Mathematical Foundations
Probabilistic PCA
Gaussian Processes
Results and Extensions

3. Motivation

4. High Dimensional Data Handwritten digit:
3648 dimensions.
Space contains more than just this digit.

5. Handwritten Digit
A simple model of the digit – rotate the ‘prototype’. 10 20 30 40 50 60 10 20 30 40 50 60 10 20 30 40 50 60

6. MATLAB Demo demDigitsManifold([2 3], ‘all’)
demDigitsManifold([2 3], ‘sixnine’)

7. Projection onto Principal Components

8. Discontinuities

9. Low Dimensional Manifolds
Pure rotation of a prototype is too simple.
In practice the data may go through several distortions,
e.g. digits undergo thinning, translation and rotation.
For data with ‘structure’:
we expect fewer distortions than dimensions;
we therefore expect the data to live in a lower dimension manifold.
Deal with high dimensional data by looking for lower dimensional non-linear embedding.

10. Our Options Spectral Approaches Non-spectral approaches
Classical Multidimensional Scaling (MDS)
Uses eigenvectors of similarity matrix.
LLE and Isomap are MDS with particular proximity measures.
Kernel PCA
Provides an embedding and a mapping from the high dimensional space to the embedding.
The mapping is implied through the use of a kernel function as the similarity matrix.
Non-spectral approaches
Non-metric MDS and Sammon Mappings
Iterative optimisation of a stress function.
A mapping can be forced (e.g. Neuroscale).

11. Our Options Probabilistic Approaches
Probabilistic PCA
A linear method.
Density Networks
Use importance sampling and a multi-layer perceptron.
GTM
Uses a grid based sample and an RBF network.
The difficulty for probabilistic approaches:
propagate a distribution through a non-linear mapping.

12. The New Model PCA has a probabilistic interpretation.
It is difficult to ‘non-linearise’.
We present a new probabilistic interpretation of PCA.
This can be made non-linear.
The result is non-linear probabilistic PCA.

13. Mathematical Foundations

14. Notation q – dimension of latent/embedded space.
d – dimension of data space.
N – number of data points.
centred data,
latent variables,
mapping matrix, W 2 <d£q.
ai,: is vector from i th row of A
a:,i is a vector from i th column of A

15. Reading Notation X and Y are design matrices.
Covariance given by N-1YTY.
Inner product matrix given by YYT.

16. Linear Embeddings
Represent data, Y, with a lower dimensional embedding X.
Assume a linear relationship of the form
where

17. Probabilistic PCA X W Y W Y

18. Maximum Likelihood Solution
If Uq are first q eigenvectors of N-1YTY and the corresponding eigenvalues are q.
where V is an arbitrary rotation matrix.

19. PCA – Probabilistic InterpretationX W Y W Y

20. Dual Probabilistic PCAX W Y X Y

21. Maximum Likelihood Solution
If Uq are first q eigenvectors of d-1YYT and the corresponding eigenvalues are q.
where V is an arbitrary rotation matrix.

22. Maximum Likelihood Solution
If Uq are first q eigenvectors of N-1YTY and the corresponding eigenvalues are q.
where V is an arbitrary rotation matrix.

23. Equivalence of PPCA Formulations
Solution for PPCA:
Solution for Dual PPCA:
Equivalence is from

24. Gaussian Processes

25. Gaussian Process (GP) Prior over functions.
Functions are infinite dimensional.
Distribution over instantiations: finite dimensional objects.
Can prove by induction that GP is `consistent’.
GP is defined by mean function and covariance function.
Mean function often taken to be zero.
Covariance function must be positive definite.
Class of valid covariances is the same as Mercer Kernels.

26. Gaussian Processes
A (zero mean) Gaussian Process likelihood is of the form
where K is the covariance function or kernel.
The linear kernel has the form

27. GP Covariance Functions
RBF
RBF
RBF
linear
MLP
MLP
Bias
Combo

28. Dual Probabilistic PCA (revisited)X W
Gaussian Process Y X Y

29. Dual Probabilistic PCA is a GPLVM
Log-likelihood:

30. Non-linear Kernel Instead of linear kernel function.
Use, for example, RBF kernel function.
Leads to non-linear embeddings.

31. Pros & Cons of GPLVM Pros Cons Probabilistic
Missing data straightforward.
Can sample from model given X.
Different noise models can be handled.
Kernel parameters can be optimised.
Cons
Speed of optimisation.
Optimisation is non-convex cf Classical MDS, kernel PCA.

32. Benchmark Examples

33. GP-LVM Optimisation Gradient based optimisation wrt X, , ,  (SCG).
Example data-set
Oil flow data
Three phases of flow (stratified annular homogenous)
Twelve measurement probes
1000 data-points
We sub-sampled to 100 data-points.

34. PCA
non-Metric MDS
metric MDS
GTM
KPCA
GP-LVM

35. Nearest Neighbour in X Number of errors for each method.
PCA
GP-LVM
Non-metric MDS 20 4 13
Metric MDS
GTM*
Kernel PCA* 6 7
* These models required parameter selection.

36. Full Oil Data

37. Nearest Neighbour in X Number of errors for each method.
PCA
GTM
GP-LVM
162 11 1
cf. 2 errors for nearest neighbour in Y

38. Applications

39. Applications Grochow et al Urtasun et al We’ve been looking at faces.
Style Based Inverse Kinematics
Urtasun et al
A prior for tracking
We’ve been looking at faces.

40. Face Animation Data from Electronic Arts
OpenGL Code by Manuel Sanchez (now at Electronic Arts).

41. Extensions

42. Back Constraints GP-LVM Gives a smooth mapping from X to Y.
Points close together in X will be close in Y.
It does not imply points close in Y will be close in X.
Kernel PCA gives a smooth mapping from Y to X.
Points close together in Y will be close in X.
It does not imply points close in X will be close in Y.
This characteristic is more common in visualisation methods.
(joint work with Joaquin Quiňonero Candela)

43. Back Constraints Maximise likelihood with constraint.
Each latent point is given by a mapping from data space.
For example the mapping could be a kernel:

44. Back Constrained GP-LVMX
Gives a mapping in both
directions, a GP mapping
from X to Y and a reverse
constraining mapping
from Y to X.
X constrained to be function of Y Y

45. Motion Capture Data Data from Ohio State University
Motion capture data of a man running.
217 data points in 102 dimensions.
Down sample to 55 data points in 102 dimensions.
Model with Pure GP-LVM and Back constrained GP-LVM (BC-GP-LVM).

46. MATLAB Demo
demStickResults.m

47. Pure GP-LVM

48. BC-GP-LVM

49. Running Angle
(a)
(b)
(c)
(d)

50. Vocal Joystick Data Vowel data from Jon Malkin & Jeff Bilmes.
Aim is to use vowel data in a vocal joystick.
Data consists of MFCCs and deltas.
Each data point is one frame (d=24).
2,700 data points (300 from each vowel).

51. PCA - Vocal Joystick
/a/ red cross, /ae/ green circle /ao/ blue plus /e/ cyan asterix /i/ magenta square /ibar/ yellow diamond /o/ red down triangle /schwa/ green up triangle and /u/ blue left triangle.

52. Pure GP-LVM
/a/ red cross, /ae/ green circle /ao/ blue plus /e/ cyan asterix /i/ magenta square /ibar/ yellow diamond /o/ red down triangle /schwa/ green up triangle and /u/ blue left triangle.

53. Back Constrained GP-LVM
/a/ red cross, /ae/ green circle /ao/ blue plus /e/ cyan asterix /i/ magenta square /ibar/ yellow diamond /o/ red down triangle /schwa/ green up triangle and /u/ blue left triangle.

54. Isomap (7 neighbours)
/a/ red cross, /ae/ green circle /ao/ blue plus /e/ cyan asterix /i/ magenta square /ibar/ yellow diamond /o/ red down triangle /schwa/ green up triangle and /u/ blue left triangle.

55. Nearest Neighbour in X Number of errors for each method.
PCA
Pure
GP-LVM
BC-GP-LVM
Isomap
1613
226
155
458
cf. 24 errors in data space

56. Motion Capture with Back Constraints
MATLAB demo
Example in motion capture with RBF back constraints
In OXFORD toolbox: demStickResults

57. Linear Back ConstraintsX
X =YB
Learn the projection matrix: B2<d£q Y
As motivation consider PCA on a digit data set.

58. Reconstruction with GP

59. Linear Projection with GP-LVM
PCA
Constrained GP-LVM

60. Linear constrained GP-LVM
Nearest Neighbour in X
latent dim 2 3 4
PCA
131
115 47
Linear constrained GP-LVM 79 60 39
cf. 24 errors for nearest neighbour in Y

61. Dynamics

62. Adding Dynamics Data often has a temporal ordering.
Markov-based dynamics are often used.
For the GP-LVM
Marginalising such dynamics is intractable.
But: MAP solutions are trivial to implement.
Many choices: Kalman filter, Markov chains etc.
Recent work by Wang, Fleet and Hertzmann uses GP.

63. GP-LVM with Dynamics
Gaussian process mapping in latent space between time points
Time point 1
Time point 2

64. Dynamics with Stick Man
demStickResults.m

65. Stick Man with Dynamics

66. Robot SLAM Joint work with Brian Ferris and Dieter Fox
Data consists of 215 signal strength readings from 30 wireless access points.
Expect the data to be inherently 2-D.
Ideally Loop closure should occur.

67. Robot Localisation Pure GP-LVM BC-GP-LVM GP-LVM + dynamics BC-GP-LVM

68. Ongoing Work Improving quality of learning in large data sets.
Based on work by Snelson & Ghahramani.
Already used in presented results and available on-line.

69. Conclusions

70. Conclusion Probabilistic non-linear interpretation of PCA.
A probabilistic model for high dimensions.
Back constraints can be introduced to
improve visualisation.
seek better linear projections