Inter-modality Face Sketch Recognition Hamed Kiani

Outline Overview Previous Works Proposed Approach Results Summary Inter-modality Face Sketch Recognition ICME'12 2

Overview Face Recognition Inter-modality Face Sketch Recognition ICME'12 3 Known Face Images Face Recognition System Identity Input Face

Overview Face sketch recognition Inter-modality Face Sketch Recognition ICME'12 4 Known Face Photos (Mug shot) Photo-Sketch Matching Suspect’s identity Viewing Verbal description Drawing Eyewitness Police artistSketch

Overview – Modality Gap: the difference of visual cues between face sketch and photo. Inter-modality Face Sketch Recognition ICME'12 5 Intra-modality approachesInter-modality approaches Matching face photos and sketches in a same modality by (photo or sketch) Synthesis Matching photo and sketch of different modalities (direct matching). Tang and Wang [ECCV’03] Liu et al. [CVPR’5] Wang and Tang [PAMI’09] Klare and Jain [SPIE ‘10] Klare et al. [PAMI’11] Zhang et al. [CVPR’11] Image synthesis No modality gap Modality gap No image synthesis

Overview Visual cues of face come from: – Fine texture (appearance): low contrast details, flaws, moles, wrinkles, etc. – Coarse texture (shape): high contrast boundaries of facial components eyes, mouth, etc Inter-modality Face Sketch Recognition ICME'12 6

Overview Face textures and modality gap:  Fine textures of a face photo captured by camera (true pixels)  Fine texture of a sketch is rendered by artist, depending on drawing style and tools  Fine textures of face photo and sketch are not equivalent: high amount of modality gap  Coarse texture (facial component and boundaries) exists in both sketch and photo  modality gap is not affected significantly by coarse texture Inter-modality Face Sketch Recognition ICME'12 7

Proposed Approach Histogram of Averaged Oriented Gradients (HAOG): a modified version of Histogram of Oriented Gradients (HOG) HOG for sketch recognition: Modeling local appearance and shape Based on fine and coarse textures. “Fine texture leads to a high amount of modality gap” Inter-modality Face Sketch Recognition ICME'12 8

Proposed Approach Idea of HAOG: E mphasizing coarse texture much more than fine texture in feature extraction. How? By averaged gradient vector (dominant gradient) instead of pixel’s gradient vector (orientation and magnitude). Inter-modality Face Sketch Recognition ICME'12 9

Proposed Approach But: Local gradients cannot directly be averaged, opposite gradient vectors cancel each other Solution: Doubling the angles of the gradient vectors before averaging: equal to squaring the length of gradient vectors [Bazen and Grez, 2002]. Inter-modality Face Sketch Recognition ICME'12 10

Proposed Approach Thus, we define squared gradient vectors Inter-modality Face Sketch Recognition ICME'12 11

Proposed Approach HAOG Inter-modality Face Sketch Recognition ICME'12 12 x-gradient y-gradient

Proposed Approach HAOG Inter-modality Face Sketch Recognition ICME'12 13 HAOG

Proposed Approach Given a query sketch and a gallery of face photos, face sketch recognition is done by: Inter-modality Face Sketch Recognition ICME'12 14 : HAOG descriptor, :chi-square

Proposed Approach Inter-modality Face Sketch Recognition ICME'12 15 Figure 1. (a1) Face photo, (a2) Face sketch, (b1,b2) Gradient magnitudes of (a1,a2), Squared gradient magnitudes of (a1,a2).

Proposed Approach Inter-modality Face Sketch Recognition ICME'12 16 Figure 2. Face sketch (top), photo (bottom), (b,c,d) local patches (first row), HAOG descriptors (second row) and HOG descriptors (third row).

Results Results on CUHK dataset with 606 pairs of face photo/sketch Inter-modality Face Sketch Recognition ICME'12 17

Summary Face sketch recognition vs. face recognition Modality gap HOG vs. HAOG Future work Inter-modality Face Sketch Recognition ICME'12 18

Inter-modality Face Sketch Recognition ICME'12 19