Object Detection Using Semi- Naïve Bayes to Model Sparse Structure Henry Schneiderman Robotics Institute Carnegie Mellon University

Object Detection Find all instances of object X (e.g. X = human faces)

Examples of Detected Objects

Sparse Structure of Statistical Dependency Chosen variable

Sparse Structure of Statistical Dependency Chosen coefficient

Sparse Structure of Statistical Dependency Chosen coefficient

Detection using a Classifier “Object is present” (at fixed size and alignment) “Object is NOT present” (at fixed size and alignment) Classifier

Proposed Model: Semi-Naïve Bayes Kononenko (1991), Pazzini (1996), Domingos and Pazzini (1997), Rokach and Maimon (2001) e.g. S 1 = (x 21, x 34, x 65, x 73, x 123 ) S 2 = (x 3, x 8, x 17, x 65, x 73, x 111 ) subsets input variables

Goal: Automatic subset grouping S 1 = (x 21, x 34, x 65, x 73, x 123 ) S 2 = (x 3, x 8, x 17, x 65, x 73, x 111 )... S n = (x 14, x 16, x 17, x 23, x 85, x 101, x 103, x 107 )

Approach: Selection by Competition x 1 x 2 x 3... x m S 1 S 2... S q Generate q candidate subsets log [p 1 (S 1 |  1 ) / p 1 (S 1 |  2 )] log [p 2 (S 2 |  1 ) / p 2 (S 2 |  2 )]... log [p q (S q |  1 ) / p q (S q |  2 )] Train q log likelihood function Select combination of n candidates log [p j1 (S j1 |  1 ) / p j1 (S j1 |  2 )] log [p j2 (S j2 |  1 ) / p j2 (S j2 |  2 )]... log [p jn (S jn |  1 ) / p jn (S jn |  2 )] H(x 1,…,x r ) = log [p j1 (S j1 |  1 ) / p j1 (S j1 |  2 )]+log [p j2 (S j2 |  1 ) / p j2 (S j2 |  2 )] log [p jn (S jn |  1 ) / p jn (S jn |  2 )] q functions n functions, n << q

Approach: Selection by Competition x 1 x 2 x 3... x m S 1 S 2... S q Generate q candidate subsets log [p 1 (S 1 |  1 ) / p 1 (S 1 |  2 )] log [p 2 (S 2 |  1 ) / p 2 (S 2 |  2 )]... log [p q (S q |  1 ) / p q (S q |  2 )] Train q log likelihood function Select combination of n candidates log [p j1 (S j1 |  1 ) / p j1 (S j1 |  2 )] log [p j2 (S j2 |  1 ) / p j2 (S j2 |  2 )]... log [p jn (S jn |  1 ) / p jn (S jn |  2 )] H(x 1,…,x r ) = log [p j1 (S j1 |  1 ) / p j1 (S j1 |  2 )]+log [p j2 (S j2 |  1 ) / p j2 (S j2 |  2 )] log [p jn (S jn |  1 ) / p jn (S jn |  2 )]

Generation of Subsets “modeling error for assuming independence” q is size of the subset

Generation of Subsets Selection of variables - “discrimination power” q is size of the subset

Pair-Wise Measurement pair-wise measurements Pair-affinity

Visualization of C(x,*) (frontal faces) x x x

Measure over a Subset Subset-affinity

Generation of Candidate Subsets x 1 x 2 x x m S 1 S S p Heuristic search and selective evaluation of D(S i ) C(x 1, x 2 ) C(x 1, x 3 ) C(x m-1, x m )

subset size vs. modeling power Model complexity limited by number of training examples, etc. Examples of limited modeling power –5 modes in a mixture model –7 projection onto principal components

Approach: Selection by Competition x 1 x 2 x 3... x m S 1 S 2... S q Generate q candidate subsets log [p 1 (S 1 |  1 ) / p 1 (S 1 |  2 )] log [p 2 (S 2 |  1 ) / p 2 (S 2 |  2 )]... log [p q (S q |  1 ) / p q (S q |  2 )] Train q log likelihood function Select combination of n candidates log [p j1 (S j1 |  1 ) / p j1 (S j1 |  2 )] log [p j2 (S j2 |  1 ) / p j2 (S j2 |  2 )]... log [p jn (S jn |  1 ) / p jn (S jn |  2 )] H(x 1,…,x r ) = log [p j1 (S j1 |  1 ) / p j1 (S j1 |  2 )]+log [p j2 (S j2 |  1 ) / p j2 (S j2 |  2 )] log [p jn (S jn |  1 ) / p jn (S jn |  2 )]

Log-likelihood function = Table S i = (x i1, x i2,..., x iq ) vector quantization table look-up

Sub-Classifier Training by Counting P i (f i |  1 ) P i (f i |  2 ) fifi fifi

Example of VQ x i1 x i2 x i3... x iq c 1 c 2 c 3 z 1 z 2 z 3 f = z 1 m 0 + z 2 m 1 + z 3 m 2 quantization to m levels projection on to 3 principal components

Approach: Selection by Competition x 1 x 2 x 3... x m S 1 S 2... S q Generate q candidate subsets log [p 1 (S 1 |  1 ) / p 1 (S 1 |  2 )] log [p 2 (S 2 |  1 ) / p 2 (S 2 |  2 )]... log [p q (S q |  1 ) / p q (S q |  2 )] Train q log likelihood function Select combination of n candidates log [p j1 (S j1 |  1 ) / p j1 (S j1 |  2 )] log [p j2 (S j2 |  1 ) / p j2 (S j2 |  2 )]... log [p jn (S jn |  1 ) / p jn (S jn |  2 )] H(x 1,…,x r ) = log [p j1 (S j1 |  1 ) / p j1 (S j1 |  2 )]+log [p j2 (S j2 |  1 ) / p j2 (S j2 |  2 )] log [p jn (S jn |  1 ) / p jn (S jn |  2 )]

h 1 (S 1 ) h 2 (S 2 )... h P (S P ) E 1,  1 E 1,  2 E 2,  1 E 2,  2... E p,  1 E p,  2 Evaluate on training data ROC 1 ROC 2... ROC P Evaluate ROCs Order top Q log-likelihood functions h j1 (S j1 ) h j2 (S j2 )... h jQ (S jQ ) Candidate log-likelihood functions

h j1 (S j1 ) + h 1 (S 1 )... h jQ (S jQ ) + h p (S p ) Form pQ pairs of log-likelihood functions E j1,  1 + E 1,  1 E j1,  2 + E 1,  2... E jQ,  1 + E p,  1 E jQ,  2 + E p,  2 Sum Evaluations ROC 1... ROC QP Evaluate ROCs Order top Q pairs of log-likelihood functions h k1,1 (S k1,1 ) + h k1,2 (S k1,2 )... h kQ,1 (S kQ,1 ) + h kQ,2 (S kQ,2 )... Repeat for n iterations

H 1 (x 1, x 2,..., x r )... H Q (x 1, x 2,..., x r ) Cross-Validation Selects Classifier Q Candidates: H 1 (x 1, x 2,..., x r ) = h k1,1 (S k1,1 ) + h k1,2 (S k1,2 ) h k1,n (S k1,n )... H Q (x 1, x 2,..., x r ) = h kQ,1 (S kQ,1 ) + h kQ,2 (S kQ,2 ) h Q,n (S kQ,n ) H * (x 1, x 2,..., x r ) Cross-validation

Example subsets learned for telephones

Evaluation of Classifier “Object is present” (at fixed size and alignment) “Object is NOT present” (at fixed size and alignment) Classifier

1) Compute feature values f 2 = #3214 f 1 = #5710 f n = #723

2) Look-Up Log-Likelihoods P 1 ( #5710 |  1 ) P 1 ( #5710 |  2 ) f 2 = #3214 f 1 = #5710 f n = #723 = 0.53 P 2 ( #3214 |  1 ) P 2 ( #3214 |  2 ) = 0.03 log P n ( #723 |  1 ) P n ( #723 |  2 ) = 0.23 log

3) Make Decision P 1 ( #5710 |  1 ) P 1 ( #5710 |  2 ) = 0.53 P 2 ( #3214 |  1 ) P 2 ( #3214 |  2 ) = 0.03 log P n ( #723 |  1 ) P n ( #723 |  2 ) = 0.23 log  > <

Detection using a Classifier “Object is present” (at fixed size and alignment) “Object is NOT present” (at fixed size and alignment) Classifier

View-based Classifiers Face Classifier #1 Face Classifier #2 Face Classifier #3

Detection: Apply Classifier Exhaustively Search in position Search in scale

Decision can be made by partial evaluation P 1 ( #5710 |  1 ) P 1 ( #5710 |  2 ) = 0.53 P 2 ( #3214 |  1 ) P 2 ( #3214 |  2 ) = 0.03 log P n ( #723 |  1 ) P n ( #723 |  2 ) = 0.23 log  > <

Detection Computational Strategy Computational strategy changes with size of search space Apply log [p 1 (S 1 |  1 ) / p 1 (S 1 |  2 )] exhaustively to scaled input image Apply log [p 2 (S 2 |  1 ) / p 2 (S 2 |  2 )] reduced search space Apply log [p 3 (S 3 |  1 ) / p 3 (S 3 |  2 )] further reduced search space

Repeat for N 2 Candidates Compute M 2 feature valuesLook-up M 2 log-likelihood values Candidate-Based Evaluation

Repeat for N 2 Candidates Compute N 2 + M 2 +2MN feature valuesLook-up M 2 log-likelihood values Feature-Based Evaluation

Create candidate subsets Train candidate log-likelihood functions Select log-likelihood functions Retrain selected log-likelihood functions using Adaboost Training images of object Cross-validation images Training images of non- object Determine detection threshold Automatically select non-object examples for next stage Images that do not contain object Increment stage Cascade Implementation Adaboost using confidence-rated predictions [Shapire and Singer, 1999] Bootstrapping [Sung and Poggio, 1995]

Boosting... H(x 1, x 2,..., x r ) = h j1 (S j1 ) + h j2 (S j2 ) h jn (S jn ) x 1 x 2 x 3... x m S 1 S 2... S p Generate p candidate subsets; p >> n h 1 (S 1 ) h 2 (S 2 )... h p (S p ) Train p candidate sub-classifiers h j1 (S j1 ) h j2 (S j2 )... h jr (S jn ) Select combination of n candidates Retrain with Adaboost using confidence-rated predictions [Shapire and Singer, 1999]

Face, eye, ear detection

Frontal Face Detection Recognition rate85.2%89.7%92.1%93.7%94.2% False detections (this method) False Detections [Viola and Jones, CVPR, 2001] MIT-CMU Frontal Face Test Set [Sung and Poggio, 1995; Rowley, Baluja and Kanade, 1997] –180 ms 300x200 image –400 ms 300x500 image Top Rank Video TREC 2002 Face Detection Top Rank 2002 ARDA VACE Face Detection algorithm evaluation AMD Athalon 1.2GHz

Face & Eye Detection for Red-Eye Removal from Consumer Photos CMU Face Detector

Eye Detection Experiments performed independently at NIST Sequested data set: 29,627 mugshots Eyes correctly located (radius of 15 pixels) 98.2% (assumed one face per image) Thanks to Jonathon Phillips, Patrick Grother, and Sam Trahan for their assistance in running these experiments

Realistic Facial Manipulation: Earring Example With Jason Pinto

Telephone Detection

Cart, pose 1

Cart, pose 2

Cart, pose 3

Door Handle Detection

Summary of Classifier Design Sparse structure of statistical dependency in many image classification problem Semi-naïve Bayes Model Automatic learning structure of semi-naïve Bayes classifier: –Generation of many candidate subsets –Competition among many log-likelihood functions to find best combination CMU on-line face detector: