“Secret” of Object Detection Zheng Wu (Summer intern in MSRNE) Sep. 3, 2010 Joint work with Ce Liu (MSRNE) William T. Freeman (MIT) Adam Kalai (MSRNE) Jianxiong Xiao (MIT)

Intro Sliding Windows Features Cascade Classifier PASCAL Con 2 motorbike person

Intro Sliding Windows Features Cascade Classifier PASCAL Con 3 Outline Introduction Sliding-Window-based Object Detection Window Generation Feature Extraction Cascade Classifier PASCAL Challenge Conclusion

Intro Sliding Windows Features Cascade Classifier PASCAL Con 4 Related Topic Image Matching Image Classification Object Detection

Intro Sliding Windows Features Cascade Classifier PASCAL Con Object Detection Single PatternMultiple Patterns 5 Viola & Jones Face Detector Dalal & Triggs Pedestrian Detector Felzenszwalb’s Part-based Detector

Intro Sliding Windows Features Cascade Classifier PASCAL Con 6 Outline Introduction Sliding-Window-based Object Detection Window Generation Feature Extraction Cascade Classifier PASCAL Challenge Conclusion

Intro Sliding Windows Features Cascade Classifier PASCAL Con 7 Sliding Windows Search over: Location Scale Aspect Ratio Millions of windows!

Intro Sliding Windows Features Cascade Classifier PASCAL Con 8 Sliding Windows Subsample on grid - have to set “optimal” step size manually Fix Aspect Ratio - assume single pattern detection Fix Scale - assume object’s resolution does not change much between training and test sets. Search with branch-and-bound method - have to use special scoring function

Intro Sliding Windows Features Cascade Classifier PASCAL Con 9 Sliding Windows We propose sliding windows from segmentation Superpixel Segmentation [Levinshtein et al, PAMI09] Region Segmentation [Felzenszwalb & Huttenlocher, IJCV03] < 100,000 sliding windows / image on PASCAL Dataset

Intro Sliding Windows Features Cascade Classifier PASCAL Con 10 Outline Introduction Sliding-Window-based Object Detection Window Generation Feature Extraction Cascade Classifier PASCAL Challenge Conclusion

Intro Sliding Windows Features Cascade Classifier PASCAL Con 11 Generic Feature “Objectness” features [Alexe et al, CVPR10]

Intro Sliding Windows Features Cascade Classifier PASCAL Con 12 Generic Feature Each type of generic feature is weak, but combination is stronger Low dimensional feature (=8) Not suitable for objects with “concave” shape, i.e. table, chair

Intro Sliding Windows Features Cascade Classifier PASCAL Con 13 Generic Feature

Intro Sliding Windows Features Cascade Classifier PASCAL Con 14 Class-specific Feature Histogram of Orientated Gradients

Intro Sliding Windows Features Cascade Classifier PASCAL Con 15 Class-specific Feature Dense grid (>10*10) (secret 1) Overlapping cells Histogram bin size High dimensional feature (>1000) - redundant or overfitting? Normalization No spatial relationship maintained

Intro Sliding Windows Features Cascade Classifier PASCAL Con 16 Outline Introduction Sliding-Window-based Object Detection Window Generation Feature Extraction Cascade Classifier PASCAL Challenge Conclusion

Intro Sliding Windows Features Cascade Classifier PASCAL Con 17 Cascade Classifier Same type of classifier with different features Viola & Jones Face Detector, IJCV01 Different types of classifier with same features Harzallah et al, ICCV09 (INRIA) Vedaldi et al, ICCV09 (Oxford)

Intro Sliding Windows Features Cascade Classifier PASCAL Con 18 Cascade Classifier Training SVM is slow… to train 20,000 examples with 4000 dimensions: >15min for Linear SVM >3 hours for Nonlinear SVM Training SVM requires a lot of memory… design matrix: 20,000*20,000 matrix Training with Imbalance data a few hundreds of positive examples billions of negative examples

Intro Sliding Windows Features Cascade Classifier PASCAL Con 19 Boosted SVM

Intro Sliding Windows Features Cascade Classifier PASCAL Con examples, half for training, half for testing Training error is 0.05 for all boosted classifiers Boosted SVM

Intro Sliding Windows Features Cascade Classifier PASCAL Con 21 Positive Training Set

Intro Sliding Windows Features Cascade Classifier PASCAL Con 22 Negative Training Set Random Samples SVM ver. 1 Training Sample Pool False Positives SVM ver. 2 … Secret 2

Intro Sliding Windows Features Cascade Classifier PASCAL Con 23 Outline Introduction Sliding-Window-based Object Detection Window Generation Feature Extraction Cascade Classifier PASCAL Challenge Conclusion

Intro Sliding Windows Features Cascade Classifier PASCAL Con 24 PASCAL Dataset 2009

Intro Sliding Windows Features Cascade Classifier PASCAL Con 25 PASCAL Dataset 2009

Intro Sliding Windows Features Cascade Classifier PASCAL Con 26 PASCAL Dataset 2009

Intro Sliding Windows Features Cascade Classifier PASCAL Con 27 PASCAL Dataset 2009

Intro Sliding Windows Features Cascade Classifier PASCAL Con 28 PASCAL Dataset 2009

Intro Sliding Windows Features Cascade Classifier PASCAL Con 29 PASCAL Dataset 2009

Intro Sliding Windows Features Cascade Classifier PASCAL Con 30 PASCAL Dataset 2009

Intro Sliding Windows Features Cascade Classifier PASCAL Con 31 PASCAL Dataset 2009

Intro Sliding Windows Features Cascade Classifier PASCAL Con 32 PASCAL 2009 (trainval + test)

Intro Sliding Windows Features Cascade Classifier PASCAL Con 33 PASCAL 2009 (train+val)

Intro Sliding Windows Features Cascade Classifier PASCAL Con 34 PASCAL 2009 (train+val,1/image)

Intro Sliding Windows Features Cascade Classifier PASCAL Con 35 PASCAL 2009 (train+val,5/image)

Intro Sliding Windows Features Cascade Classifier PASCAL Con 36 PASCAL 2009 (train+val,10/image)

Intro Sliding Windows Features Cascade Classifier PASCAL Con 37 True Positives - aeroplane

Intro Sliding Windows Features Cascade Classifier PASCAL Con 38 False Positives - aeroplane

Intro Sliding Windows Features Cascade Classifier PASCAL Con 39 True Positives - bicycle

Intro Sliding Windows Features Cascade Classifier PASCAL Con 40 False Positives - bicycle

Intro Sliding Windows Features Cascade Classifier PASCAL Con 41 True Positives - horse

Intro Sliding Windows Features Cascade Classifier PASCAL Con 42 False Positives - horse

Intro Sliding Windows Features Cascade Classifier PASCAL Con 43 Conclusion Proposing sliding windows without fixing scale or aspect ratio is possible. Simple feature (saliency, contrast, etc) is only useful for certain object classes. Histogram-based feature is not sufficient to represent real world object, no matter what learning algorithm is used. Boosting is helpful to speed up SVM-training and reduce the memory usage. Digging out “hard” negative examples. Throwing away “hard” positive examples.

Intro Sliding Windows Features Cascade Classifier PASCAL Con 44 Future Work It is time to go beyond the histogram-of-X feature - not every pixel within bounding box is informative - the appearance of object’s part is more robust Evolve the classifier - even PASCAL dataset is too small - the right decision boundary is still far away… - Active learning? Online learning?

Thank You ! 45