Person-Specific Domain Adaptation with Applications to Heterogeneous Face Recognition (HFR) Presenter: Yao-Hung Tsai Dept. of Electrical Engineering, NTU Oral Presentation:

Dept. of Electrical Engineering 2 Yao-Hung Tsai ( 蔡曜宏 ) Outline Face Recognition Conventional Approach Heterogeneous Face Recognition Domain Adaptation Approach Proposed Approach –Domain-independent Component Analysis –Person-specific Classifier –Combinational Framework Experiments

Dept. of Electrical Engineering 3 Yao-Hung Tsai ( 蔡曜宏 ) Outline Face Recognition Conventional Approach Heterogeneous Face Recognition Domain Adaptation Approach Proposed Approach –Domain-independent Component Analysis –Person-specific Classifier –Combinational Framework Experiments

Dept. of Electrical Engineering 4 Yao-Hung Tsai ( 蔡曜宏 ) Heterogeneous Face Recognition Face Recognition – Face Identification –Identify the subject from the captured images

Dept. of Electrical Engineering 5 Yao-Hung Tsai ( 蔡曜宏 ) Heterogeneous Face Recognition Face Recognition – Face Verification –Verify a specific subject with respect to the captured image

Dept. of Electrical Engineering 6 Yao-Hung Tsai ( 蔡曜宏 ) Heterogeneous Face Recognition Face Recognition Application Access Control System Photo auto-tagging Crime investigation ……

Dept. of Electrical Engineering 7 Yao-Hung Tsai ( 蔡曜宏 ) Outline Face Recognition Conventional Approach Heterogeneous Face Recognition Domain Adaptation Approach Proposed Approach –Domain-independent Component Analysis –Person-specific Classifier –Combinational Framework Experiments

Dept. of Electrical Engineering 8 Yao-Hung Tsai ( 蔡曜宏 ) Conventional Approach Direct method –Direct compare two images based on their pixel values v.s. –Advantages : Naïve, simple to implement –Disadvantages Require lots of computation effort

Dept. of Electrical Engineering 9 Yao-Hung Tsai ( 蔡曜宏 ) Conventional Approach A common method : Eigenface method –Representation: pixel intensity –Collecting several images as the training set: –Then we apply PCA to this set. = … n d

Dept. of Electrical Engineering 10 Yao-Hung Tsai ( 蔡曜宏 ) Conventional Approach PCA – PCA projects columns of X from high-dimension ( ) to low dimension ( ). – PCA make projection variance maximized by optimize: – After solving the optimization we will get a set of basis vectors (faces): – We can reconstruct the images by: Ex: 2 dim to 1 dim Note: v 1 will capture most data variance

Dept. of Electrical Engineering 11 Yao-Hung Tsai ( 蔡曜宏 ) Conventional Approach –The combinational coefficients will be the new feature of face: –For recognition, we simply project all images into this k- dimensional space and apply classifiers. Note: Same class cluster together.

Dept. of Electrical Engineering 12 Yao-Hung Tsai ( 蔡曜宏 ) Conventional Approach However, there exist several problems –Traditional pattern recognition problems typically deal with Training and test data collected from the same feature space –In real word applications, these data are Collected from different feature domains Exhibiting distinct feature distributions We call this cross-domain recognition problems –Also called Heterogeneous Face Recognition (HFR)

Dept. of Electrical Engineering 13 Yao-Hung Tsai ( 蔡曜宏 ) Outline Face Recognition Conventional Approach Heterogeneous Face Recognition Domain Adaptation Approach Proposed Approach –Domain-independent Component Analysis –Person-specific Classifier –Combinational Framework Experiments

Dept. of Electrical Engineering 14 Yao-Hung Tsai ( 蔡曜宏 ) Heterogeneous Face Recognition HFR is an emerging task in biometrics Sketches in Criminal CasesNight Vision Camera

Dept. of Electrical Engineering 15 Yao-Hung Tsai ( 蔡曜宏 ) Heterogeneous Face Recognition Face recognition conduct on different domains

Dept. of Electrical Engineering 16 Yao-Hung Tsai ( 蔡曜宏 ) Heterogeneous Face Recognition When conventional FR meets HFR … –If directly apply PCA on images cross domains (e.x. infra-red v.s. visible spectrum) –We visualize the data distribution of first 3 dimensions : VIS NIR VIS Domain NIR Domain

Dept. of Electrical Engineering 17 Yao-Hung Tsai ( 蔡曜宏 ) Heterogeneous Face Recognition –Observing the difference between domains : Instances with same class are far from each others. Data from same domain close to each others. That is, Domain difference dominates the data variance. –So, we need to conduct domain adaptation approach For comparing images from source and target domain and

Dept. of Electrical Engineering 18 Yao-Hung Tsai ( 蔡曜宏 ) Outline Face Recognition Conventional Approach Heterogeneous Face Recognition Domain Adaptation Approach Proposed Approach –Domain-independent Component Analysis –Person-specific Classifier –Combinational Framework Experiments

Dept. of Electrical Engineering 19 Yao-Hung Tsai ( 蔡曜宏 ) Domain Adaptation Approach There are numerous approaches of domain adaptation –Observing domain invariant features Local Binary Patterns (LBP) – PAMI 2006 –Projecting images on common feature space Canonical Correlation Analysis (CCA) Partial Least Squares (PLS) – CVPR 2011 Semi-coupled Dictionary Learning (SCDL) – CVPR 2012 Coupled Dictionary Learning (CDL) – ICCV 2013 –Match distributions between cross domain images Match marginal distributions (TCA) – TNN 2011 Match also joint distributions (JDA) – ICCV 2014

Dept. of Electrical Engineering 20 Yao-Hung Tsai ( 蔡曜宏 ) Domain Adaptation Approach Illustrate the notation of external data –Take access control system (ACS) as an example –At first, we usually cannot get the user’s images in advance –Thus, we need to use images from other subjects collected in advance to model the system –Let us call it external data Note: Images from both domains need to be collected. External Data … …

Dept. of Electrical Engineering 21 Yao-Hung Tsai ( 蔡曜宏 ) Domain Adaptation Approach Most of the approaches require a large number of paired external data –However, it is very difficult to collect them ! –Collecting external data with no labeled information is much easier Moreover, direct use of external data might be non- preferable –There’s no guarantee of the same feature distribution among external data and test data –For example, the common feature space observed from the face images of females will not generalize well to those of males.

Dept. of Electrical Engineering 22 Yao-Hung Tsai ( 蔡曜宏 ) Domain Adaptation Approach So, I proposed an approach with the following properties –Require no labeled information in external data –Advocate the learning of person-specific domain adaptation model for HFR –DiCA ( Domain-independent Components Analysis) is proposed to build a common feature space

Dept. of Electrical Engineering 23 Yao-Hung Tsai ( 蔡曜宏 ) Outline Face Recognition Conventional Approach Heterogeneous Face Recognition Domain Adaptation Approach Proposed Approach –Domain-independent Component Analysis –Person-specific Classifier –Combinational Framework Experiments

Dept. of Electrical Engineering 24 Yao-Hung Tsai ( 蔡曜宏 ) Domain-independent Component Analysis Review the observations in HFR problems – –Domain difference dominates the data variance. We check differences of projected means (MMD) for every dimensions of PCA space: and … : mean of NIR : mean of VIS MMD: -

Dept. of Electrical Engineering 25 Yao-Hung Tsai ( 蔡曜宏 ) Domain-independent Component Analysis Then we can discard the components with high MMD value. We get the final domain-independent projection matrix: External Data …… Domain-independent Components Analysis: DiCA

Dept. of Electrical Engineering 26 Yao-Hung Tsai ( 蔡曜宏 ) Domain-independent Component Analysis So far, we can directly project user’s images to DiCA space and match test images. However, to address the issue that subjects from external data are different from users and to improve the performance. I further proposed –Person-specific Classifier (PC)

Dept. of Electrical Engineering 27 Yao-Hung Tsai ( 蔡曜宏 ) Outline Face Recognition Conventional Approach Heterogeneous Face Recognition Domain Adaptation Approach Proposed Approach –Domain-independent Component Analysis –Person-specific Classifier –Combinational Framework Experiments

Dept. of Electrical Engineering 28 Yao-Hung Tsai ( 蔡曜宏 ) Person-specific Classifier Forming a specific classifier for the input test image, for this specific classifier outperforms than the general one SVM (support vector machine) classifier is chose to be this person-specific classifier –Choose test data as positive instance. –User defined negative instances could be chosen for different usage. Positive Negative Person-specific classifier

Dept. of Electrical Engineering 29 Yao-Hung Tsai ( 蔡曜宏 ) Person-specific Classifier Support Vector Machines (SVM) –Classifier to discriminate two categories data –Training dataset x i ∈ A + ⇔ y i = 1 & x i ∈ A - ⇔ y i = -1

Dept. of Electrical Engineering 30 Yao-Hung Tsai ( 蔡曜宏 ) Person-specific Classifier Goal : Predict the unseen class label for new data –Find a function f : R n → R by learning from data f(x) ≥ 0 ⇒ x ∈ A + and f(x) < 0 ⇒ x ∈ A - –Simplest function is linear : f (x) = w ⊤ x + b

Dept. of Electrical Engineering 31 Yao-Hung Tsai ( 蔡曜宏 ) Outline Face Recognition Conventional Approach Heterogeneous Face Recognition Domain Adaptation Approach Proposed Approach –Domain-independent Component Analysis –Person-specific Classifier –Combinational Framework Experiments

Dept. of Electrical Engineering 32 Yao-Hung Tsai ( 蔡曜宏 ) Combinational Framework External Data … … NIR Data VIS Data User’s images … Test positivenegative Similarity Score User’s images Form DiCA Subspace NIR VIS

Dept. of Electrical Engineering 33 Yao-Hung Tsai ( 蔡曜宏 ) Outline Face Recognition Conventional Approach Heterogeneous Face Recognition Domain Adaptation Approach Proposed Approach –Domain-independent Component Analysis –Person-specific Classifier –Combinational Framework Experiments

Dept. of Electrical Engineering 34 Yao-Hung Tsai ( 蔡曜宏 ) Experiments Two HFR scenario: –Photo – sketch (CUHK database) –VIS – NIR (CASIA 2.0 database) Identification Task –For photo-sketch, there are 100 gallery images and 100 test images. –For VIS-NIR, there are 359 gallery images and 6200 test images (with different occlusions)

Dept. of Electrical Engineering 35 Yao-Hung Tsai ( 蔡曜宏 ) Experiments Two HFR scenario: –Photo – sketch (CUHK database) –VIS – NIR (CASIA 2.0 database) Identification Task –For photo-sketch, there are 100 gallery images and 100 test images. –For VIS-NIR, there are 359 gallery images and 6200 test images (with different occlusions)

Dept. of Electrical Engineering 36 Yao-Hung Tsai ( 蔡曜宏 ) Experiments Sketch-to-photo Dataset NIR-to-VIS Dataset

Dept. of Electrical Engineering 37 Yao-Hung Tsai ( 蔡曜宏 ) The End Thank You!