1. Context Neelima Chavali ECE6504 02/21/2013

2. Roadmap Introduction Paper1 – Motivation – Problem statement – Approach – Experiments & Results Paper 2 Experiments Comments Questions?

3. What is context? Any data or meta-data not directly produced by the presence of an object – Nearby image data Context Derek Hoeim

4. What is context? Any data or meta-data not directly produced by the presence of an object – Nearby image data – Scene information Context Derek Hoeim

5. What is context? Any data or meta-data not directly produced by the presence of an object – Nearby image data – Scene information – Presence, locations of other objects Tree Derek Hoeim

6. How do we use context? What is this? Now can you tell?

7. How is context used in Compuer Vision

8. AN EMPIRICAL STUDY OF CONTEXT IN OBJECT DETECTION: SANTOSH DUVVALA, DEREK HOIEM, JAMES HAYS, ALEXEI EFROS, MARTIAL HEBERT Paper 1

9. Motivation Lack of standardization Little agreement about what constitutes “context” Isolate contribution of contextual information

10. Problem statement Evaluate context on the challenging PASCAL VOC 2008 dataset using state-of-the-art object detectors Analyze –different sources of context –different uses of context Novel algorithms using geographic context, and local pixel context

11. Sources and Uses of Context Sources: local pixel context, 2D scene gist, 3D geometric, semantic, geographic, photogrammetric, cultural Uses: Object presence, location, size, spatial support Santosh Duvvala

12. Approach Local Appearance detector: Deformable parts model

13. Object presence gist: Torralba Oliva 2003 geom context: Hoiem et al. 2005 im2gps: Hays and Efros 2008 Santosh Duvvala

14. Semantic and Geographic context Santosh Duvvala

15. Object location Santosh Duvvala

16. Object size Santosh Duvvala

17. Combining Contexts Santosh Duvvala

18. Object Spatial Support

19. Experimental Results and Analysis Detection results on PASCAL VOC 2008 testset

20. Results Change in accuracy with respect to Size and occlusion

21. Confusion matrices

22. OBJECT GRAPHS FOR CONTEXT-AWARE CATEGORY DISCOVERY: YONG JAE LEE AND KRISTEN GRAUMAN Paper 2

23. Motivation Unlabeled Image DataDiscovered categories 1) reveal structure in very large image collections 2) greatly reduce annotation time and effort 3) Mitigate the biases. 23Yong Jae Lee and Kristen Grauman

24. Can seeing previously learned objects in novel images help to discover new categories? 1 34 2 Main idea 24 Can you identify the recurring pattern? Yong Jae Lee and Kristen Grauman

25. Problem Statement Discover novel categories that occur amidst known objects within un-annotated images 25Yong Jae Lee and Kristen Grauman

26. drive- way sky house ? grass Context-aware visual discovery grass sky truck house ? drive- way grass sky house drive- way fence ? ? ?? 26 Context in supervised recognition: [Torralba 2003], [Hoiem et al. 2006], [He et al. 2004], [Shotton et al. 2006], [Heitz & Koller 2008], [Rabinovich et al. 2007], [Galleguillos et al. 2008], [Tu 2008], [Parikh et al. 2008], [Gould et al. 2009], [Malisiewicz & Efros 2009], [Lazebnik 2009] Yong Jae Lee and Kristen Grauman

27. Approach Overview 27 Learn category models for some classes Detect unknowns in unlabeled images Describe object-level context via Object-Graph Group regions to discover new categories Yong Jae Lee and Kristen Grauman

28. Learn “Known” Categories Offline: Train region-based classifiers for N “known” categories using labeled training data. sky road building tree 28 Detect Unknowns Object-level Context Discovery Learn Models Yong Jae Lee and Kristen Grauman

29. Identifying Unknown Objects Input: unlabeled pool of novel images Compute multiple-segmentations for each unlabeled image 29 Detect Unknowns Object-level Context Discovery Learn Models e.g., [Hoiem et al. 2006], [Russell et al. 2006], [Rabinovich et al. 2007] Yong Jae Lee and Kristen Grauman

30. P(class | region) bldg tree sky road P(class | region) bldg tree sky road P(class | region) bldg tree sky road P(class | region) bldg tree sky road Prediction: known High entropy → Prediction: unknown For all segments, use classifiers to compute posteriors for the N “known” categories. Deem each segment as “known” or “unknown” based on resulting entropy. 30 Identifying Unknown Objects Detect Unknowns Object-level Context Discovery Learn Models Yong Jae Lee and Kristen Grauman

31. 31 Previous Work: [Scholkopf 2000], [Markou & Singh 2003], [Weinshall et al. 2008] ImageGT known/unknown Multiple-Segmentation Entropy Maps unknowns building tree knowns sky road Identifying Unknown Objects Detect Unknowns Object-level Context Discovery Learn Models Yong Jae Lee and Kristen Grauman

32. An unknown region within an image 0 Closest nodes in its object-graph 2a 2b 1b 1a 3a 3b Consider spatially near regions above and below, record distributions for each known class. S b t s r 1a above 1b below H 1 (s) b t s r H 0 (s) 0 self g(s) = [,,, ] H R (s) b t s r Ra above Rb below 1 st nearest regionout to R th nearest b t s r 0 self Object-Graphs Detect Unknowns Object-level Context Discovery Learn Models 32Yong Jae Lee and Kristen Grauman

33. Object-Graphs Average across segmentations N posterior prob.’s per pixel b t s r N posterior prob.’s per superpixel b t s r Obtain per-pixel measures of class posteriors on larger spatial extents. 33 Detect Unknowns Object-level Context Discovery Learn Models Yong Jae Lee and Kristen Grauman

34. g(s 1 ) = [ :,, : ] b t g r abovebelow H R (s)H 1 (s) abovebelow b t g r g(s 2 ) = [ :,, : ] b t g r abovebelow H R (s)H 1 (s) abovebelow b t g r Object-graphs are very similar  produces a strong match Known classes b: building t: tree g: grass r: road 34 Object-Graph matching Detect Unknowns Object-level Context Discovery Learn Models building ? road building / road building / road tree / road building ? road building / road Yong Jae Lee and Kristen Grauman

35. Unknown Regions Clusters from region-region affinities 35 Detect Unknowns Object-level Context Discovery Learn Models Yong Jae Lee and Kristen Grauman

36. Object Discovery Accuracy Four datasets Multiple splits for each dataset; varying categories and number of knowns/unknowns Train 40% (for known categories), Test 60% of data Textons, Color histograms, and pHOG Features MSRC-v2 PASCAL 2008 Corel MSRC-v0 36Yong Jae Lee and Kristen Grauman

37. 37 MSRC-v2 PASCAL 2008 Corel MSRC-v0 Object Discovery Accuracy Yong Jae Lee and Kristen Grauman

38. Comparison with State-of-the-art Russell et al., 2006: Topic model (LDA) to discover categories among multiple segmentations using appearance only. Significant improvement over existing state-of-the-art. 38 MSRC-v2 Yong Jae Lee and Kristen Grauman

39. MAP comparision Yong Jae Lee and Kristen Grauman39

40. Example Object-Graphs buildingsky roadunknown 40 Color in superpixel nodes indicate the predicted known category. Yong Jae Lee and Kristen Grauman

41. Conclusions Discover new categories in the context of those that have already been directly taught. Substantial improvement over traditional unsupervised appearance-based methods. Future work: Continuously expand the object- level context for future discoveries. 41Yong Jae Lee and Kristen Grauman

42. EXPERIMENTS 42

43. 10 Unknowns: Randomly selected groups 43

44. 10 Unknowns: Manmade objects 44

45. 15 Unknowns: Known Animals 45

46. 5 Unknowns : animals 46

47. Acknowledgements Dr. Yong Lee Dr. Devi Parikh 47