Computer Vision Spring ,-685 Instructor: S. Narasimhan WH 5409 T-R 10:30am – 11:50am Lecture #22

Face Detection Thanks to Guru Krishnan and Shree Nayar

I.1 Locate human faces in images. What is Face Detection?

Face Detection Topics: (1) Features for Face Detection (2) Integral Images (3) Nearest Neighbor Classifier (4) Support Vector Machine Locate human faces in images.

Where is Face Detection Used? Cameras (for Autofocus) I.16

Where is Face Detection Used? Search Engine Face Detection Function I.2

Where is Face Detection Used? Only Faces of Gates now! Search Engine I.3

Where is Face Detection Used? Photo Tagging in Social Networks I.4

Where is Face Detection Used? Security Surveillance in Public I.6

How Humans Detect Faces? We do not know yet! Memory-prediction model Match faces with the face model in memory. Parallel computing Detect faces at multiple location/scale combination. Some Conjectures:

Face Detection in Computers Basic Idea: Slide windows of different sizes across image. At each location match the window to a face model. I.1

Face Detection in Computers Basic Idea: Slide windows of different sizes across image. At each location match the window to a face model. I.1

Basic Framework For each window Extract Features Match Face Model Yes / No Features: Which features represent faces well? Classifier: How to construct/match the face model? F I.12

What are Good Features? Facial Components ? SIFT?

Characteristics of Good Features Discriminate Face/Non-Face ≠ Extremely Fast to Compute Need to evaluate tens of thousands windows in an image. I.7 I.8 I.9 I.10 I.11 I.12

Haar Features A Set of Responses to Rectangular Haar Filters *

Response to Haar Filter White = 1, Black = -1 *

Haar Feature: Discriminative Ability V A = 64V A ≈ 0 Haar features capture the Directional Pattern V A = 16V A = -127 I.13 I.14 I.12

Haar Features * Feature Vector

Face Detection at Different Scales Use filters of different sizes to find faces at corresponding scale

Haar Feature: Computation Cost Value = ∑ (pixels in white) – ∑ (pixels in black) = Pixel sum in White Rect – Pixel sum in Black Rect If filter size is N×M pixels: Computation cost = (N×M-1) additions Still too expensive! Integral Image to the Rescue!

Integral Image For example: ImageIntegral Image A table that holds the sum of all pixel values to the left and top of a given pixel, inclusive.

Integral Image For example: ImageIntegral Image A table that holds the sum of all pixel values to the left and top of a given pixel, inclusive.

Integral Image For example: ImageIntegral Image A table that holds the sum of all pixel values to the left and top of a given pixel, inclusive.

Integral Image A table that holds the sum of all pixel values to the left and top of a given pixel, inclusive. For example: ImageIntegral Image

Summation Within a Rectangle ImageIntegral Image (II) Fast summations of arbitrary rectangles using integral images.

Summation Within a Rectangle ImageIntegral Image (II) Fast summations of arbitrary rectangles using integral images. Sum = II P +… = … P

Summation Within a Rectangle ImageIntegral Image (II) Fast summations of arbitrary rectangles using integral images. Sum = II P – II Q + … = 3490 – … Q P

Summation Within a Rectangle ImageIntegral Image (II) Fast summations of arbitrary rectangles using integral images. Sum = II P – II Q – II S + … = 3490 – 1137 – … Q PS

Summation Within a Rectangle ImageIntegral Image (II) Sum = II P – II Q – II S + II R = 3490 – 1137 – = 1521 QR PS Can be computed in constant time with only 4 references

Haar Response using Integral Image V A = ∑ (pixels in white area) – ∑ (pixels in black area) = (II O – II T + II R – II S ) – (II P – II Q + II T – II O ) = (2061–329+98–584) – (3490– –2061) = ImageIntegral Image (II) QR PS T O

Computing Integral Image Raster Scanning Let I A and II A be the values of the Image and Integral Image, respectively, at pixel A. A DB C II A = II B + II C – II D + I A

Haar Features for Face Detection * Feature Vector

Classifier for Face Detection Given the features for a window, how to decide whether it contains a face or not? ? ? ? ?

Feature Space f2f2 fNfN f1f1 ° ° ° ° ° Training Data of Face Training Data of Non-Face °

f2f2 fNfN f1f1 Nearest Neighbor Classifier ° ° ° ° ° Training Data of Face Training Data of Non-Face ° Nearest samples decide the result of the classifier. Test Image

f2f2 fNfN f1f1 Nearest Neighbor Classifier ° ° ° ° ° Training Data of Face Training Data of Non-Face ° Nearest samples decide the result of the classifier. Face

f2f2 fNfN f1f1 Nearest Neighbor Classifier ° ° ° ° ° Training Data of Face Training Data of Non-Face ° Nearest samples decide the result of the classifier. Test Image

f2f2 fNfN f1f1 Nearest Neighbor Classifier ° ° ° ° ° Training Data of Face Training Data of Non-Face ° Nearest samples decide the result of the classifier. Not Face

f2f2 fNfN f1f1 Nearest Neighbor Classifier ° ° ° ° ° Training Data of Face Training Data of Non-Face ° Test Image Nearest samples decide the result of the classifier.

f2f2 fNfN f1f1 Nearest Neighbor Classifier ° ° ° ° ° Training Data of Face Training Data of Non-Face ° False Positive Larger the training set, more robust the NN classifier

f2f2 fNfN f1f1 Nearest Neighbor Classifier ° ° ° ° ° Training Data of Face Training Data of Non-Face ° Larger the training set, slower the NN classifier

f2f2 fNfN f1f1 Decision Boundary ° Training Data of Face Training Data of Non-Face A simple decision boundary separating the face and non-face classes will suffice. ° ° ° ° °

Decision Boundary ° ° ° ° ° ° ° ° ° ° ° Faces Non-Faces

Decision Boundary How to find the optimal decision boundary? ° ° ° ° ° ° ° ° ° ° ° Face Class Non-Face Class

Evaluating a Decision Boundary Margin or Safe Zone: The width that the boundary could be increased by before hitting a data point. Margin ° ° ° ° ° ° ° ° ° ° °

Evaluating a Decision Boundary Choose Decision Boundary with the Maximum Margin! Margin I ° ° ° ° ° ° ° ° Margin II ° ° ° ° ° ° ° ° + + Decision I: FaceDecision II: Non-Face

Support Vector Machine (SVM) Margin ° ° ° ° ° ° ° ° ° ° ° Classifier optimized to maximize Margin Support Vectors: Closest data samples to the boundary. Decision Boundary and the Margin depend only on the Support Vectors.

Support Vector Machine (SVM) Equation of Boundary: ° ° ° ° ° ° Find:

° ° ° ° ° °

iPhone App using Face Detection Application ： Game I.15

Comments  Current face detection system: mature but not perfect  Critical to face-related vision/graphics tasks  Increasing requirement on accuracy and speed

References [Burges 1998] Burges, C. J. C. “A Tutorial on Support Vector Machines for Pattern Recognition” [Freund 1996] Freund, Y. and Schapire, R.E. “Experiments with a new boosting algorithm” [Viola 2004] Viola, P. and Jones, M.J. “Robust real-time face detection”

Image Credits I.1http://images2.fanpop.com/images/photos/ /TBBT-Cast-the- big-bang-theory jpg I.2http:// I.3http:// I.4http://pictures.labho.com/facebook-launches-facial-recognition- worldwide/ I.5http:// I.6http:// I.7http:// I.8http:// with-nadine/2110-what-your-face-says-about-you.html I.9http:// I.10http:// I.11http://

Image Credits I.12http://stereogum.com/48421scarlett_johansson_covers_jeff_buckley/ mp3s/ I.13http:// solution-for-multitasking / I.14http://animal-world.com/dogs/Non-Sporting-Dog-Breeds/ NonSportingDogBreeds.php I.15http:// I.16http:// 2008/12/23/nikons60_2.jpg