An Illumination Invariant Face Recognition System for Access Control using Video Ognjen Arandjelović Roberto Cipolla Funded by Toshiba Corp. and Trinity College, Cambridge

Face Recognition Single-shot recognition – a popular area of research since 1970s Many methods have been developed Bad performance in presence of: –Illumination variation –Pose variation –Facial expression –Occlusions (glasses, hair etc.) Eigenfaces Wavelet methods 3D Morphable Models

Face Recognition from Video Face motion helps resolve ambiguities of single shot recognition – implicit 3D Video information often available (surveillance, authentication etc.) Recognition setupTraining streamNovel stream

Face Manifolds Face patterns describe manifolds which are: –Highly nonlinear, and –Noisy, but –Smooth Facial featuresFace pattern manifoldFace region

Limitations of Previous Work In this work we address 3 fundamental questions: –How to model nonlinear manifolds of face motion –How to achieve illumination and pose robustness –How to choose the distance measure ?

Face Motion Manifolds: Revisited Unchanging identity, changing illumination Changing identity, unchanging illumination Motivation: How can we use the prior knowledge on the shape of the manifolds?

Pose Clusters Face motion manifolds are nonlinear, but: –Low-dimensional (c.f. registration for the reduction of the dimensionality), and –Key observation: can be described well using only 3 linear pose clusters Colour-coded pose clusters for 3 manifolds

Determining Pose Clusters Pose clusters are semantic clusters: –K-means and similar algorithms are unsuitable –We are using a simple method based on the motion parallax –Membership decided based on Maximum Likelihood Pupils Discrepancy η Image plane Yaw measure Distribution for 3 clusters

Pose Clusters: Example Input manifold and colour-coded pose clusters Sample frames from the 3 pose clusters

Illumination compensation Performed in two stages: –Coarse illumination compensation (exploiting face smoothness) –Fine illumination compensation (exploiting low dimensionality of the face illumination subspace) InputOutput

Region-based GIC Gamma Intensity Correction (GIC) Canonical image Region-based GIC (RGIC): faces are (roughly) divided into regions with smoothly varying surface normal Solved by 1D non-linear optimization Face regions Varying Gamma

Region-based GIC: Artefacts Region-based GIC suffers from artefacts at region boundaries Mean face γ value mapSmoothed γ map Input faceRGIC faceOur method Boundary artefacts Artefacts removed

Illumination Subspace Each input frame corrected for a linear Pose Illumination Subspace component to match the reference distribution of the same pose –Illumination subspace is high-dimensional –Constrained to expected variations by Mahalanobis distance Input manifold Reference manifold Illumination Subspace

Illumination Compensation Results Original/input frames Illumination-corrected frames Reference frames Strong side lighting And in face pattern space…

Comparing Pose Clusters “Distribution-based” distances (Kullback-Leibler divergence, Resistor Average Distance etc.) unsuitable We use the simple Euclidean distance between cluster centres Reference cluster Novel cluster Reduced spread Cluster centres

Unified Manifold Similarity Recognition based based on the likelihood ratio: Manifolds belong to the same person Distances between pose clusters Learn likelihoods from ground truth training data Likelihood histogram Undefined value regions RBF-interpolated likelihood Two-pose interpolated likelihood Likelihood now monotonically decreasing

Face Video Database Revisited Testing performed under extreme, varying illuminations 10 illumination conditions used (random 5 for training, others for testing)

Registration Linear operations on images are highly nonlinear in the pattern space Translation/rotation and weak perspective can be easily corrected for directly from point correspondences –We use the locations of pupils and nostrils to robustly estimate the optimal affine registration parameters Translation manifold Skew manifold Rotation manifold

Registration Method Used Feature localization based on the combination of shape and pattern matching (Fukui et al. 1998) Detect features Crop & affine register faces

Results Very high recognition rates attainted (96% average) under extreme variations in illumination Other methods showed little to no illumination invariance

Results, continued The method was shown to give promising results for authentication uses: –Good separability of inter- and intra- class manifold distances was found –It can provide a secure system with only 0.1% false positive rate and 8% false negative rate Cumulative distributions of inter- and intra- class manifold distances The ROC curve for the proposed method

Future Research Non-constant illumination within a single sequence causes problems Illumination compensation is still not perfect – pose illumination subspaces have unnecessarily high dimensions Pose estimation is too primitive – outliers cause problems in estimation of linear subspaces Complete pose invariance is still not achieved (what if there are no corresponding pose clusters?) For suggestions, questions etc. please contact me at: