1. Cognitive Computer Vision Kingsley Sage khs20@sussex.ac.uk and Hilary Buxton hilaryb@sussex.ac.uk Prepared under ECVision Specific Action 8-3 http://www.ecvision.org

2. Lecture 8 Gaussian Mixture Models Using Mixture Models in continuous valued Hidden Markov Models Coursework

3. So why are GMMs relevant to Cognitive CV? Provides a well-founded methodology for representing e.g.: – Arbitrarily complex functions – Images Combined with HMMs (previous lecture) provided one basis for a model of expectation capable of dealing with continuous valued data

4. The Gaussian function In 1-dimension, the Gaussian Function is the probability function of the normal distribution  p(x)  : mean  : std deviation  2 : variance

5. The Gaussian function Can extend this idea to k-dimensions (the multivariate normal distribution) k: # of dimensions x,  : vectors of k data values  : covariance matrix T: matrix transpose

6. Variance and co-variance reminder For single dimension (k=1) X with N observations = {x 1,x 2,…x N } For small samples (say N < 20), a better unbiased estimate is given by replacing 1/N terms with 1/(N-1)

7. Variance and co-variance reminder For k-dimensional X with N observations …

8. Variance and co-variance reminder Covariance matrix is symmetric I.e. cov(x k,1 )=cov(x 1,k ) Diagonal matrix is variance of 1  k parameters Matrix encodes variability between variables Uncorrelated dimensions will have low co-variance

9. Effect of co-variance

10. Mixtures of Gaussians We might represent set of points (like a trajectory) as a sum of a number of Gaussian components. We decide how many components and fit to data …                      m=1 m=2 m=3 Here, each mixture m has its own  (m) vector (length k=2) and  matrix (2  2 matrix)

11. Mixtures of Gaussians Or, a common trick with HMMs, we can use a single co-variance matrix …                     

12. Mixtures of Gaussians Or we can treat a set of variables (like pixels in an image) as a set of Gaussian function (one or more per pixel) to describe its variability over a set (e.g. time) In this example, each pixel position has one Gaussian function, so we can view the mean and variance maps as images X={,,,,, …}

13. The Hidden Markov Model (HMM) reminder of model structure from previous lecture 3 1 2 x1x1 x2x2 x3x3 Observable variable Hidden states

14. Hidden states are indexes to Gaussian mixtures Adapted from an example by Sam Roweis Data maps time between eruptions of a geyser vs. duration of eruption This is modelled as a HMM with 3 hidden states

15. What is a Hidden Markov Model? Formally a Hidden Markov Model = ( , A, B)  vector and A matrix as before 11 22 1 4.438355.4517 2 4.082478.8017 3 1.993083.2375 N hidden states The B (confusion) parameters Each state has a  (mean) vector with length k ( 2 in this case) There is one mixture per hidden state There is either a covariance matrix per state, or (more usually) one overall co-variance matrix  NB: We now use N for the # of hidden nodes. The # of observations is now shown as T

16. Forward evaluation (1) 123 1 00.30470.6953 2 00.61670.3833 3 0.98350.01650 O = {o 1,o 2, …,o T } 11 22 1 4.438355.4517 2 4.082478.8017 3 1.993083.2375 N hidden states State at t me t-1 State at time t k=1k=2 k=10.1052-0.0829 k=2-0.082938.9357  =

17. Forward evaluation (2)  t=1Norm 11/3 1/3 * N(o 1 ;  (1),  ) = 0.0084 0.9698 21/3 1/3 * N(o 1 ;  (2),  ) = 2.6084e-4 0.0302 31/3 1/3 * N(o 1 ;  (3),  ) = 3.0514e-13 0 Here O = {o 1 = [4.0,60.0]} Normalisation solves the problem with the Gaussian functions constants 11 22 1 4.438355.4517 2 4.082478.8017 3 1.993083.2375 N hidden states k=1k=2 k=10.1052-0.0829 k=2-0.082938.9357  =

18. Forward evaluation (3)  t=1t=2 1 1/3 0.9698 {(0.9698*0) + (0.0302*0) + (0.00*0.9835)} * N(o 2 ;  (1),  ) 2 1/3 0.0302 {(0.9698*0.3047) + (0.0302*0.6167) + (0.00*0.0165)} * N(o 2 ;  (2),  ) 3 1/3 0.00 {(0.9698*0.6953) + (0.0302*0.3833) + (0.00*0)} * N(o 2 ;  (3),  ) Here O = {o 1 = [4.0,60.0], o 2 = [2.0,80.0]} 123 1 00.30470.6953 2 00.61670.3833 3 0.98350.01650 State at t me t-1 State at time t

19. Coursework The first coursework deals with using HMMs The lab seminar dealt with practical issues around the discrete HMM case These ideas all apply equally to the continuous valued HMMs Coursework requires you to write forwards and backwards evaluation routines and use them to measure statistics for a simple visual control task

20. Further reading Variable length Markov Models: “The Power of Amnesia”, D. Ron, Y. Singer and N. Tishby, In Advances in Neural Information Processing Systems (NIPS), vol 6, pp 176- 183, 1994 Extending VLMMs to work with continuous valued data: “Learning temporal structure for task-based control”, check for latest version at www.informatics.sussex.ac.uk/users/khs20 www.informatics.sussex.ac.uk/users/khs20

21. Summary We can combine multi-dimensional Gaussian mixture models with simple HMMs to interpret continuous data The hidden states then index the Gaussian mixtures Forward and backwards evaluation perform the same functions as in the discrete case

22. Next time … Dynamic Bayes nets for recognising visual behaviour …