Bilinear models for action and identity recognition Oxford Brookes Vision Group 26/01/2009 Fabio Cuzzolin

Bilinear models for invariant gaitID The identity recognition problem View-invariance in gaitID Bilinear models HMMs and a three-layer model Four experiments on the Mobo database

Identity recognition from gait biometrics increasingly popular cooperative methods: face recognition, retinal analysis surveillance context: non-cooperative users the problem: recognizing the identity of humans from their gait methods: dimensionality reduction, silhouette analysis issues: nuisance factors, viewpoint dependence

A brief review gait signatures: silhouettes [Collins 02, Wang 03], optical flow, velocity moments, shape symmetry, static body parameters baseline algorithm [Sarkar 05] computes similarity scores between a probe sequence and each gallery (training) sequence by pairwise frame correlation methodologies: mostly pattern recognition after dimensionality reduction eigenspaces [Abdelkader 01], PCA/MDA [Tolliver 03, Han 04] stochastic models (HMMs): [Kale 02, Debrunner 00] KL-divergence between Markov models

Bilinear models for invariant gaitID The identity recognition problem View-invariance in gaitID Bilinear models HMMs and a three-layer model Four experiments on the Mobo database

The view-invariance issue nuisance factors many different nuisance factors are involved viewpoint illumination clothes, shoes, carried objects trajectory view-invariance big issue: view-invariance possible approaches: 3D tracking virtual view reconstruction static body parameters

Approches to view-invariant gait ID [Cunado 99]: Evidence gathering technique coupled oscillators, Fourier description, inclination of thigh and leg [Urtasun,Fua 04]: fitting 3D temporal motion models to synchronized video sequences Motion parameters: coefficients of the singular value decomposition of the estimated model angles [Bhanu,Han 02] matching a 3D kinematic model to 2D silhouettes extracting a number of feature angles from the fitted model [Kale 03]: synthetic side-view of the moving person using a single camera [Shakhnarovich 01]: view-normalization from volumetric intersection of the visual hulls [Johnson, Bobick 01]: static body parameters recovered across multiple views

Bilinear models for invariant gaitID The identity recognition problem View-invariance in gaitID Bilinear models HMMs and a three-layer model Four experiments on the Mobo database

Bilinear models style invariance From view-invariance to style invariance motions usually possess several labels: action, identity, viewpoint, emotional state, etc. Bilinear models Bilinear models (Tenenbaum) can be used to separate the influence of two of those factors, called style and content (the label to classify) y SC is a training set of k-dimensional observations with labels S and C b C is a parameter vector representing content, while A S is a style-specific linear map mapping the content space onto the observation space

Bilinear models the content (identity, action) of an observation can be thought of as a vector in an abstract content space of some dimension J bCbC ASAS y SC observations are then derived from content vector linearly, through a map which depends on the style parameter S

Learning an asymmetric bilinear model given an observation sequence y SC …... an asymmetric bilinear model can fitted to the data through the SVD Y=SUV of a stacked observation matrix the symmetric model can be written as Y=AB where least-squares optimal style and content parameters are

Content classification of unknown style consider a training set in which persons (content=ID) are seen walking from different viewpoints (style=viewpoint) when new motions are acquired in which a known person is walking from a different viewpoint (unknown style)… … an iterative EM procedure can be set up to classify the content (identity) E step -> estimation of p(c|s), the prob. of the content given the current estimate s of the style M step -> estimation of the linear map for the unknown style s

Bilinear models for invariant gaitID The identity recognition problem View-invariance in gaitID Bilinear models HMMs and a three-layer model Four experiments on the Mobo database

Hidden Markov models finite-state representation of an observation process state process {X k } is a Markov chain given a sequence os observations (feature matrix) EM algorithm for parameter learning (Moore) A->transition probabilities (motion dynamics) C-> means of state-output distributions (poses)

Motions as stacked HMMs interpretation of the C matrix: columns of C are means of the output distributions associated with the states of the model in gaitID (cyclic motions) the dynamics is the same for all sequences (A neglected) stacked columns of the C matrix a sequence can then be represented as a collection of poses: stacked columns of the C matrix

Three-layer model First layer (feature representation): projection of the contour of the silhouette on a sheaf of lines passing through the center 1 Third layer: bilinear model of HMMs 3 2 In the second layer each sequence is encoded as a Markov model, its C matrix is stacked in an observation vector, and a bilinear model is trained over those vectors

Bilinear models for invariant gaitID The identity recognition problem View-invariance in gaitID Bilinear models HMMs and a three-layer model Four experiments on the Mobo database

Mobo database: 25 people performing 4 different walking actions, from 6 cameras action, id, view each sequence has three labels: action, id, view MOBO database

Four experiments one label is chosen as contentanother one as style we can then set up four experiments in which one label is chosen as content, another one as style, and the remaining is considered as a nuisance factor contentstylenuisance action view-invariant action recognition viewIDaction ID-invariant action recognition IDviewID action-invariant gaitID actionviewID view-invariant gaitID viewaction

Results – ID versus VIEW baseline algorithm Compared performances with baseline algorithm and straight k-NN on sequence HMMs

Results – ID versus action ID vs action experiment performance of the bilinear classifier in the ID vs action experiment as a function of the nuisance (view=1:5), averaged over all the possible choices of the test action the average best-match performance of the bilinear classifier is shown in solid red, (minimum and maximum in magenta). Best-3 matches ratio in dotted red

Feature extraction projection of the contour Type 1: projection of the contour of the silhouette on a sheaf of lines passing through the center Type 2: size functions [Frosini 90] Lees moments Type 3: Lees moments

Results - influence of features ID-invariant action recognition Left: ID-invariant action recognition using the bilinear classifier. The entire dataset is considered, regardless the viewpoint. The correct classification percentage is shown as a function of the test identity in black (for models using Lee's features) and red (contour projections). Related mean levels are drawn as dotted lines. View-invariant action recognition Right: View-invariant action recognition.

Conclusions covariance factors covariance factors of paramount importance in gaitID bilinear-multilinear models bilinear-multilinear models provide a way to separate different factors three-layer model we proposed a three-layer model in which sequence are represented through HMMs expensive and sensitive some approaches to view-invariance are expensive and sensitive experiments on the Mobo database show how much separating factor is effective for motion classification future: multilinear models, testing on more realistic setups (many factors, USF database)