Real-time on-line learning of transformed hidden Markov models Nemanja Petrovic, Nebojsa Jojic, Brendan Frey and Thomas Huang Microsoft, University of Toronto, University of Illinois

2 Six break points vs. six things in video Traditional video segmentation: Find breakpoints Example: MovieMaker (cut and paste) Our goal: Find possibly recurring scenes or objects timeline REPRESENTATIVE FRAMES

3 Transformed hidden Markov model z T x c Class with prior P(c=k) = π k P(z|c) = N(z;μ c,Φ c ) x = Tz Ex = Tμ Var x = TΦT T p(x|c,T) = N(x; Tμ c, TΦ c T T ) Generation is repeated for each frame of the sequence, with the pair (T,c) being the state of a Markov chain. Translation T with uniform prior Latent image z Observed frame x

4 Goal: maximize total likelihood of a dataset log p(X) = log Σ {T,c} Σ z p(X,{T,c},Z) = log Σ {T,c} Σ z q({T,c},Z)p(X,{T,c},Z)/q({T,c},Z) Σ {T,c} Σ z q({T,c},Z)log p(X,{T,c},Z) - Σ {T,c} Σ z q({T,c},Z)log q({T,c},Z) = B We express q(T,c,z) = q({T,c}) * q(Z|{T,c}) {T,c} represents values of transformation and class for all frames, i.e., the path that the video sequence takes through the state space of the model. Instead of the likelihood, we optimize the bound B, which is tight for q=p({T,c},Z|X)

5 Posterior approximation We allow q({T,c}) to have a non-zero probability only on M most probable paths: q({T,c}) = Σ m=1:M r m δ({T,c} - {T,c} * m ) (Viterbi 1982) This reduces a number of problems with adaptive scaling in the exact forward-backward inference.

6 Expensive part of the E step Find quick way to calculate log p(x|c,T) = -N/2 log(2π) – ½ log|TΦ c T T | - ½ (x-Tμ c ) T (TΦ c T T ) -1 (x-Tμ c ) for all possible shifts T in the E step of EM algorithm Shifted cluster mean Tμ Frame x T log p

7 Computing Mahalanobis distance using FFTs = Σ.* T = sum x.*T μ Φ = IFFT FFT(x).* conj ( FFT ) x T (TΦT T ) -1 Tμ = x T T(Φ -1 μ) = x T T(diag Φ -1.* μ) All terms that have to be evaluated for all T can be expressed as correlations, e.g. : μ Φ (where summation is over pixels) N log N versus N 2 !

8 Parameter optimization F = Σ T Σ c Σ z q(T,c,z)log p(X,T,c,z) = Σ T Σ c Σ z q({T,c}) * q(z|{T,c}) x ( logπ {Tc} + Σ time log p(x t,z t |T t,c t ) + Σ time log p(c t+1 |c t ) log p(T t+1 |T t,c t ) ) Solve F/( )=0 for an estimated q.

9 On-line vs. batch EM Example: Update equation for the class mean Σ t Σ T q(T t,c t )E[z|x t,c t,T t ] = Σ t q(c t ) μ c t Batch EM: –solve for μ using all frames. –Inference and parameter optimization iterated. On-line EM: –rewrite the equation for one extra frame –establish the relationsip between μ (t+1) and μ (t). –Parameters updated after each frame. No need for iteration.

10 Reducing the complexity of the M step Σ T q(T t,c t )E[z|x t,c t,T t ] can be expressed as a sum of convolutions. For example, when there is no observation noise, E[z|x t,c t,T t ]= T t T x t, and Σ T q(T t,c t )E[z|x t,c t,T t ] = IFFT (FFT(q).* FFT(x)) (similar trick applies to variance estiamates)

11 Represent pixels on a polar grid! Shifts in the log-polar coordinates correspond to scale and rotation changes in the Cartesian coordiante system How to deal with scale and rotation? rotation scale

12 Estimating the number of classes The algorithm is initialized with a single class A new class is introduced whenever the frame likelihood drops bellow a threshold The classes can be merged in the end to achieve a more compact representation

13 Clustering a 20-minute whale watching video

Clustering a 20-minute beach video

15 0 min9 min Shots from the first class

16 Discovering objects using motion priors Different motion prior predefined for each of the classes Three characteristic frames from 240x320 input sequence Learned means and variances

17 Tracking results

18 Summary Before - CVPR 99/00 28x44 images Grayscale images 1 day of computation for 15 sec video Batch EM Exact inference Fixed number of clusters Limited number of translations Memory inefficient Now 120x160 images Full color images 5-10 frames/sec On-line EM Approximate inference Variable number of clusters All possible translations Memory efficient

19 Sneak preview: Panoramic THMMs z T x c P(c=k) = π k P(z|c) = N(z;μ c,Φ c ) x = WTz Ex = WTμ Var x = WTΦT T W T p(x|c,T) = N(x; WTμ c, WTΦ c T T W T ) WT

20 Video clustering - model Appearancemean variance Camera/object motion Temporal constraints Unsupervised learning – the only input is the video

21 Current implementation DirectShow filter for frame clustering (5-15 frames/sec!) Translation invariance On-line learning algorithm Classes repeating across video Potential applications: –Video segmentation –Content based search/retrieval –Short video summary creation –DVD chapter creation

22 Comparing with layered sprites Perfect segmentation Layered sprites. Jojic, CVPR 01 But, THMM is hundreds/thousands of time faster!

Example with more content