University of South Florida, Tampa1 Gait Recognition and Inverse Biometrics Sudeep Sarkar (Zongyi Liu, Pranab Mohanty) Computer Science and Engineering University of South Florida, Tampa

2 Biometrics from far

University of South Florida, Tampa 3 Paul Taylor on Walking “And walking is the most revealing. A walk is like a fingerprint. No two people walk the same.” – Paul Taylor

University of South Florida, Tampa 4 Gait Research Gait Challenge Problem Human Perception  Point light displays  Johansson (1973), Cutting & Kozlowski, Mather & Mudock., Neri, Pavlova  Gender discrimination  Stevenage and Nixon (1999) Gait Recognition publications/year (Google)

University of South Florida, Tampa 5 The HumanID Gait Challenge Problem Data set of gait video  Number of subjects (122)  Exercise 5 covariates view, shoe, surface, carrying condition, time  1.2 TB of data Challenge experiments of increasing difficulty Baseline recognition algorithm to measure progress Joint effort of USF, NIST, and ND.

University of South Florida, Tampa 6 Samples within bounding boxes Grass View Shoe Surface Briefcase Time

University of South Florida, Tampa 7 Gallery and Probes C,A,L, BF G,A,L, BF C,B,L, BF G,B,L, BF C,A,R, BF G,A,R, BF C,B,R, BF G,B,R, BF Briefcase ConcreteGrass Concrete Shoe A C,A,L, NB G,A,L, NB C,B,L, NB G,B,L NB C,A,R, NB G,A,R, NB C,B,R, NB G,B,R, NB No Briefcase Grass B A B Left Camera Right Camera M 1 + N 1 C,A,L, NB G,A,L, NB C,A,L, BF G,A,L, BF C,B,L, NB G,B,L NB C,B,L, BF G,B,L, BF C,A,R, NB G,A,R, NB C,A,R, BF G,A,R, BF C,B,R, NB G,B,R, NB C,B,R, BF G,B,R, BF No Briefcase Briefcase Concrete Grass Left Camera Right Camera N2N2

University of South Florida, Tampa 8 Challenge Experiments Gallery Size: 122 May Subject + Nov New Subjects Repeat Subjects In Nov

University of South Florida, Tampa 9 Baseline Algorithm: Silhouette Detection Estimate background  RGB mean, covariance Compute Mahanalobis distance Smooth the distances Expectation maximization  Foreground vs. background  Feature – Mahanalobis distance  Assume Gaussian distribution for each class  Initialize labels by random threshold  Iterate to maximize likelihood

University of South Florida, Tampa 10 Post Processing of Silhouettes Select the largest connected component Scale the silhouettes  Final size 128 by 88.  The height scaled to 128 pixels  Horizontally centered so that the column with most number of foreground pixel is at column 44 Some amount of scale invariance

University of South Florida, Tampa 11 Baseline Algorithm: Similarity Measure N probe Gallery (N frames)Probe (M frames) Correlate Median (or Mean, Maximum) SIMILARITY MEASURE

University of South Florida, Tampa 12 Impact of the Challenge Problem

University of South Florida, Tampa 13 Impact on full dataset

University of South Florida, Tampa 14 Summary of Identification Rates

University of South Florida, Tampa 15 Significant findings based on this dataset Walking surface and Time are the most significant covariates Shape over dynamics (Maryland, CMU, USF) Quality of silhouettes do not seem to be the bottleneck (USF) 3D approaches are starting to emerge (Maryland)

University of South Florida, Tampa 16 Dynamics Normalized Gait Recognition Emphasize gait shape over gait dynamics  Shape => silhouette shape for each stance  Dynamics => transition between stances Normalize the dynamics of any given gait sequence into a generic one Emphasize differences in gait shape between subjects

University of South Florida, Tampa 17 Generic Gait Model: Population HMM Learnt from the manually specified silhouettes of 71 subjects. (Baum-Welch) Exponential Observation Model

University of South Florida, Tampa 18 Dynamics Normalized Gait Map given frames to generic stances  Viterbi (dynamic programming) For each stance, average all frames mapped to it Final: Dynamics normalized gait cycle over 20 stances

University of South Florida, Tampa 19 Similarity Computation Gallery Probe LDA Subspaces S1S1 S2S2 S 20 Project

University of South Florida, Tampa 20 Performance Gait Challenge (122 subjects)

University of South Florida, Tampa 21 Performance on UMD Data (55 subjects) Without retraining

University of South Florida, Tampa 22 Recognition Performance: Speed CMU Mobo Database (25 subjects) Without retraining

University of South Florida, Tampa 23 Face, Gait Publications/Year FERETGAIT CHALLENGE

University of South Florida, Tampa 24 Inverse Biometrics: From Scores to Templates Gallery Break in set Recognition Algorithm Score Can you recreate this template?

University of South Florida, Tampa 25 Two step process Face Recognition System Affine transformation that approximates the recognition algorithm Break-in Set Face Recognition System Unknown Target Match Scores Embedding Reconstruction Reconstructed Target Break-in Set Embedding & Reconstruction Modeling Distance Matrix

University of South Florida, Tampa 26 Modeling Recognition Algorithm Computed distances between faces in the break-in set using the face recognition method to be modeled Define an Transformation (A) that operates on a face (x i ) to embed it into a lower dimensional space not necessarily orthogonal Distance Matrix Break-in Set

University of South Florida, Tampa 27 Two part transformation a rigid transformation independent of the recognition algorithm derived from the orthonormal subspace analysis e.g. PCA of images in break-in set non-rigid transformation depends on the specific recognition algorithm approximate the recognition algorithm through sheer and stretching of the image space derived using classical MDS A = A nr A r

University of South Florida, Tampa 28 Embedding & Reconstruction ? ? Original image space Modeled Space Inverse Transformation Unknown target Observe the distances of selected templates from break-in set to unknown target Calculate the co-ordinates of the unknown target in the transformed space Use inverse transformation to reconstruct the unknown target template in the original affine space

University of South Florida, Tampa 29 Dataset & Recognition Algorithms Databases: FERET and FRGC Modeling results Break-In Results Algorithms  FRGC baseline algorithm (PCA)  Independent Component Analysis (ICA)  Bayesian intrapersonal/extrapersonal classifier (template based)  Elastic Bunch Graph Matching (EBGM)  Commercial Face Recognition System (feature based)

University of South Florida, Tampa 30 Modeling results Quality of modeling is evaluated using ROC curves Training set: 600 images from150 subjects from FERET  Subjects different from the test set FERET data set with fa-fb and dup I probe set  Bayesian, Commercial, PCA+Mahacosine (ICA, EBGM) On FRGC dataset  Only for the commercial and baseline

University of South Florida, Tampa 31 Affine modeling quality (Bayesian) On FERET with 1196 gallery subjects Different expressions Different Days

University of South Florida, Tampa 32 Affine modeling quality (Com.) On FERET with 1196 gallery subjects Different expressions Different Days

University of South Florida, Tampa 33 Affine modeling quality (Com.) On FRGC data Exp II (indoor, time) Exp III (indoor vs. outdoor)

University of South Florida, Tampa 34 Face Template Reconstruction On FERET Gallery of 1196 subjects using a FRGC break-in set On FRGC Gallery of 466 subjects using a FERET break-in set Break-in in presence of score quantization Break-in performance: Probability of breaking a randomly chosen face template Algorithms are set to operate at 1% False Acceptance Rate  Commercial and Baseline (PCA)

University of South Florida, Tampa 35 Template reconstruction (FERET) Original Baseline Commercial Bayesian Reconstructed

University of South Florida, Tampa 36 Template reconstruction (FRGC) Original Reconstruction Baseline Commercial

University of South Florida, Tampa 37 Template reconstruction (contd.) Baseline Bayesian Commercial Original No. of Images in Break-in Set

University of South Florida, Tampa 38 Break-in probability (FERET) On FERET Gallery (1196 subjects) Probability of breaking a randomly chosen face template

University of South Florida, Tampa 39 Break-in probability (FRGC) On FRGC Gallery (466 subjects) Probability of breaking a randomly chosen face template

University of South Florida, Tampa 40 Break-in probability (Quantization) FERET data Score quantization is a countermeasure against the hill climbing attack Probability of breaking a randomly chosen face template (10 levels)(100 levels)(1000 levels)(10000 levels) Number of quantized levels

University of South Florida, Tampa 41 Questions Thank You