1. Context-based vision system for place and object recognition Antonio Torralba Kevin Murphy Bill Freeman Mark Rubin Presented by David Lee Some slides borrowed from Kevin Murphy

2. Object out of context

3. Object in context

4. Wearable test-bed

5. System diagram

6. Computing the features

7. 24 filtered Images Downsample to 4x4 4x4x24 =384 dim 80 dim

8. Visualizing the filter bank output Images 80-dimensional representation

9. Place recognition system

10. Hidden Markov Model Hidden states = location (63 values) Observations = v G t ∈ R 80 Transition model encodes topology of environment Observation model is a mixture of Gaussians (100 views per place)

11. Hidden Markov Model Observation Likelihood Prediction Prior Transition Matrix Mixture of Gaussians MLE (counting)

12. Scene Categorization 17 Categories (Office, Corridor, Street, etc) Train a separate HMM on category labels

13. Place recognition demo

14. Specific location Location category Indoor/outdoor Ground truth System estimate Performance on known env.

15. Performance on new env.

16. Comparison of features Recognition Categorization

17. Effect of HMM on recognition With Without (But with temporal smoothing)

18. From place to object recognition

19. Object priming Predict object properties based on context (top-down signals):  Visual gist, v t G  Specific Location, Q t  Kind of location, C t

20. Object Priming Again… MLE Probability of object i Probability of object i in image v i given entire video sequence Probability of object i Given current observation & place Estimate of current place (Output of HMM) Mixture of Gaussians Observation Likelihood Prior probability of object i being in place q

21. Predicting object presence

22. ROC curves for object detection

23. Predicting object position and scale

24. Estimate of mask Probability of an object i being present and location being q (Output of previous system) Estimate of mask given current gist, place, and object delta Gaussian

25. Predicted segmentation

26. Conclusion Real world problem (and it works!) Uses only global feature (context) How much did {HMM / place prior} affect {place recognition / object detection}? Can we really say “context” did the job?