1. Video Surveillance E6998 -007 Senior/Feris/Tian 1 Emerging Topics in Video Surveillance Rogerio Feris IBM TJ Watson Research Center rsferis@us.ibm.com http://rogerioferis.com

2. Video Surveillance E6998 -007 Senior/Feris/Tian 2 Outline Video Surveillance in Crowded Scenarios Online Learning – Self-adaptation in Surveillance Other Recent Topics

3. Video Surveillance E6998 -007 Senior/Feris/Tian 3 Simple Scenarios Few Objects – Background Subtraction + Tracking + High-level Event/Alert Detection Current systems work well

4. Video Surveillance E6998 -007 Senior/Feris/Tian 4 Crowded Scenarios Many objects, occlusions, shadows, etc. Object Segmentation, Tracking and Event Analysis in crowded scenarios: Open Problem!

5. Video Surveillance E6998 -007 Senior/Feris/Tian 5 Parts-based Detectors [Pedro et al, A discriminatively trained, multiscale, deformable part model, CVPR08] Root filter (low-res) + Parts filters (high-res) Occlusion Handling

6. Video Surveillance E6998 -007 Senior/Feris/Tian 6 Parts-based Detectors Score of a window: score of root + score of parts Score of Parts: Appearance + Geometry Efficient localization of parts through Dynamic Programming SVM Classification (Structured prediction)

7. Video Surveillance E6998 -007 Senior/Feris/Tian 7 Detecting Pedestrians in Crowds [Leibe et al, Pedestrian Detection in Crowded Scenes, CVPR05] Combination of different models: bag of features, segmentation, and chamfer matching

8. Video Surveillance E6998 -007 Senior/Feris/Tian 8 Tracking in Crowds [Andriluka et al, People-tracking-by-detection and people-detection-by- tracking, CVPR08] Extends [Leibe et al, CVPR05] to temporal-domain and person articulation (parts) estimation Click for Video Demo

9. Video Surveillance E6998 -007 Senior/Feris/Tian 9 Crowd Segmentation [Dong et al, Fast Crowd Segmentation Using Shape Indexing, ICCV07]

10. Video Surveillance E6998 -007 Senior/Feris/Tian 10 Crowd Analysis [Ali & Shah, A Lagrangian Particle Dynamics Approach for Crowd Flow Segmentation and Stability Analysis, CVPR07]

11. Video Surveillance E6998 -007 Senior/Feris/Tian 11 Online Learning

12. Video Surveillance E6998 -007 Senior/Feris/Tian 12 Initialize sample weights For each cycle: Find a classifier/rectangle feature that performs well on the weighted samples Increase weights of misclassified examples Return a weighted combination of classifiers Adaboost ensembles many weak classifiers into one single strong classifier Offline Adaboost Learning

13. Video Surveillance E6998 -007 Senior/Feris/Tian 13 Offline Adaboost Learning Major Problems: Large number of examples required to train a robust classifier time consuming to label data slow training (may take several days) No Adaptation to particular surveillance scenarios

14. Video Surveillance E6998 -007 Senior/Feris/Tian 14 Learning from Small Sets Choice of Features (Levi & Weiss, CVPR04) Co-Training (Levin & Viola, ICCV2003) Online Adaptation: Online Boosting (Oza01, Javed05, Bischof06, Pham07)

15. Video Surveillance E6998 -007 Senior/Feris/Tian 15 Online Boosting [Oza,2001] Train a generic strong classifier (set of weak classifiers, # of weak classifiers fixed) on a small training set. Online Process: Given one single example with known label: Slide the example over each weak classifier When the weak classifier receives the example update the weak classifier online update the weight of the example and pass to the next weak classifier

16. Video Surveillance E6998 -007 Senior/Feris/Tian 16 Online Boosting [Oza,2001]

17. Video Surveillance E6998 -007 Senior/Feris/Tian 17 Online Boosting [Oza,2001]

18. Video Surveillance E6998 -007 Senior/Feris/Tian 18 Online Boosting Car and People Detection [Omar Javed, CVPR05] Train a generic strong classifier (set of weak classifiers, # of weak classifiers fixed) on a small training set. While running the classifier on unlabeled data, if an example is confidently predicted by a subset of weak classifiers use it for online learning Co-training framework BGS used for efficiency, for using more expensive features, and for balancing the number of positive and negative examples

19. Video Surveillance E6998 -007 Senior/Feris/Tian 19 Online Boosting Car and People Detection [Omar Javed, CVPR05]

20. Video Surveillance E6998 -007 Senior/Feris/Tian 20 More Recent Work [Bo Wu & Nevatia, Improving Part-based Object Detection by Unsupervised, Online Boosting, CVPR07] [Helmut & Hurst, Online Boosting and Vision, CVPR06] [Pham & Cham, Online Learning Asymmetric Boosted Classifiers for Object Detection, CVPR07] [Huang et al.,Incremental Learning of Boosted Face Detector, ICCV07] – Boosting Adaptation IEEE Online Learning for Classification Workshop (CVPR08)

21. Video Surveillance E6998 -007 Senior/Feris/Tian 21 Other Recent Topics

22. Video Surveillance E6998 -007 Senior/Feris/Tian 22 High-Resolution Imagery [Kopf et al, Capturing and Viewing Gigapixel Images, SIGGRAPH07] How can we make use of high-resolution in video analytics? Much more info – e.g., in face reco: skin texture, iris, etc.

23. Video Surveillance E6998 -007 Senior/Feris/Tian 23 Next Generation Neural Networks [Hinton, Reducing the dimensionality of data with neural networks, Science 2006] New algorithm for learning deep belief nets State-of-the art results in MNIST digit dataset (better than SVMs) Youtube talk at Google: http://www.youtube.com/watch?v=AyzOUbkUf3M http://www.youtube.com/watch?v=AyzOUbkUf3M Matlab Code: http://www.cs.toronto.edu/~hinton/

24. Video Surveillance E6998 -007 Senior/Feris/Tian 24 Learning with lots of data How can we recognize thousands of products in a retail store for loss prevention? 80 Million Tiny Images (http://www.cs.nyu.edu/~fergus/) Surveillance with Moving Cameras Cameras in vehicles, or even wearable cameras. New challenges: object detection, etc. [Leibe et al, Dynamic 3D Scene Analysis from a Moving Vehicle, CVPR 2007]

25. Video Surveillance E6998 -007 Senior/Feris/Tian 25 Many more recent topics: Check for papers in recent computer vision conferences (like CVPR, ICCV, and ECCV) and also specialized workshops/conferences such as AVSS and PETS