Presentation is loading. Please wait.

Presentation is loading. Please wait.

Object-Graphs for Context-Aware Category Discovery

Published byHerman Kusuma Modified over 6 years ago

Similar presentations

Presentation on theme: "Object-Graphs for Context-Aware Category Discovery"— Presentation transcript:

1 Object-Graphs for Context-Aware Category Discovery
Yong Jae Lee and Kristen Grauman University of Texas at Austin CVPR 2010

2 Discovered categories
Motivation Unlabeled Image Data Discovered categories 1) reveal structure in very large image collections 2) greatly reduce annotation time and effort 3) training data is not always available

3 Existing approaches - Topic models e.g., pLSA, LDA.
Previous work treats unsupervised visual discovery as an appearance-grouping problem. - Topic models e.g., pLSA, LDA. [Fergus et al. 2005], [Sivic et al. 2005], [Quelhas et al. 2005], [Fei-Fei & Perona 2005], [Liu & Chen 2007], [Russell et al. 2006] - Partitioning of the image data. [Grauman & Darrell 2006], [Dueck & Frey 2007], [Kim et al. 2008], [Lee & Grauman 2008], [Lee & Grauman 2009]

4 Can you identify the recurring pattern?
Existing approaches Previous work treats unsupervised visual discovery as an appearance-grouping problem. 1 3 4 2 Can you identify the recurring pattern?

5 Can you identify the recurring pattern?
Our idea How can seeing previously learned objects in novel images help to discover new categories? 1 2 3 4 Can you identify the recurring pattern?

6 Can you identify the recurring pattern?
Our idea Discover visual categories within unlabeled images by modeling interactions between the unfamiliar regions and familiar objects. 1 3 4 2 Can you identify the recurring pattern?

7 Context-aware visual discovery
? drive-way sky house ? grass grass sky truck house ? drive-way grass sky house drive-way fence ? Context in supervised recognition: [Torralba 2003], [Hoiem et al. 2006], [He et al. 2004], [Shotton et al. 2006], [Heitz & Koller 2008], [Rabinovich et al. 2007], [Galleguillos et al. 2008], [Tu 2008], [Parikh et al. 2008], [Gould et al. 2009], [Malisiewicz & Efros 2009], [Lazebnik 2009]

8 Key Ideas Context-aware category discovery treating previously learned categories as object-level context. Object-Graph descriptor to encode surrounding object-level context. Note: Different from semi-supervised learning – unlabeled data do not necessarily belong to categories of the labeled data.

9 Learn category models for some classes
Approach Overview Learn category models for some classes Detect unknowns in unlabeled images Describe object-level context via Object-Graph Group regions to discover new categories

10 Learn “Known” Categories
Detect Unknowns Object-level Context Discovery Learn Models Learn “Known” Categories sky road building tree Offline: Train region-based classifiers for N “known” categories using labeled training data.

11 Identifying Unknown Objects
Detect Unknowns Object-level Context Discovery Learn Models Identifying Unknown Objects Compute multiple-segmentations for each unlabeled image Input: unlabeled pool of novel images e.g., [Hoiem et al. 2006], [Russell et al. 2006], [Rabinovich et al. 2007]

12 Identifying Unknown Objects
Detect Unknowns Object-level Context Discovery Learn Models Identifying Unknown Objects P(class | region) bldg tree sky road Prediction: known High entropy → Prediction:unknown For all segments, use classifiers to compute posteriors for the N “known” categories. Deem each segment as “known” or “unknown” based on resulting entropy.

13 An unknown region within an image
Detect Unknowns Object-level Context Discovery Learn Models Object-Graphs An unknown region within an image Model the topology of category predictions relative to the unknown (unfamiliar) region. Incorporate uncertainty from classifiers.

14 Detect Unknowns Object-level Context Discovery Learn Models Object-Graphs Closest nodes in its object-graph An unknown region within an image 1b 1a 3a 3b 2a 2b S Consider spatially near regions above and below, record distributions for each known class. b t s r 1a above 1b below H1(s) H0(s) self g(s) = [ , , , ] HR(s) Ra Rb 1st nearest region out to Rth nearest

15 Detect Unknowns Object-level Context Discovery Learn Models Object-Graphs Average across segmentations N posterior prob.’s per pixel b t s r N posterior prob.’s per superpixel b t s r Obtain per-pixel measures of class posteriors on larger spatial extents.

16 Object-Graph matching
Detect Unknowns Object-level Context Discovery Learn Models Object-Graph matching Known classes b: building t: tree g: grass r: road building ? road building / road / road tree / road building ? road g(s1) = [ : , , : ] b t g r above below HR(s) H1(s) g(s2) = [ : , , : ] b t g r above below HR(s) H1(s) Object-graphs are very similar  produces a strong match

17 Object-Graph matching
Detect Unknowns Object-level Context Discovery Learn Models Object-Graph matching Known classes b: building t: tree g: grass r: road building / road building building grass ? ? building / road road road road g(s1) = [ : , , : ] b t g r above below HR(s) H1(s) g(s2) = [ : , , : ] b t g r above below HR(s) H1(s) Object-graphs are partially similar  produces a fair match

18 Clusters from region-region affinities
Detect Unknowns Object-level Context Discovery Learn Models Clusters from region-region affinities Unknown Regions

19 Object Discovery Accuracy
MSRC-v2 PASCAL 2008 Corel MSRC-v0 Four datasets Multiple splits for each dataset; varying categories and number of knowns/unknowns Train 40% (for known categories), Test 60% of data Textons, Color histograms, and pHOG Features

20 Object Discovery Accuracy
MSRC-v2 PASCAL 2008 Corel MSRC-v0

21 Comparison with State-of-the-art
MSRC-v2 Russell et al., 2006: Topic model (LDA) to discover categories among multiple segmentations using appearance only. Significant improvement over existing state-of-the-art.

22 Example Object-Graphs
unknown building sky road Color in superpixel nodes indicate the predicted known category.

23 Examples of Discovered Categories

24 Collect-Cut (poster Thursday)
Unsupervised Segmentation Examples Best Bottom-up (with multi-segs) Collect-Cut (ours) Discovered Ensemble from Unlabeled Multi-Object Images Unlabeled Images Use discovered shared top-down cues to refine both the segments and discovered categories with an energy function that can be minimized with graph cuts.

25 Conclusions Discover new categories in the context of those that have already been directly taught. Substantial improvement over traditional unsupervised appearance-based methods. Future work: Continuously expand the object-level context for future discoveries.

26

27 Category Retrieval Results

28 Impact of Known/Unknown Decisions
Red star denotes the cutoff (half of max possible entropy value). Regions considered for discovery are almost all true unknowns (and vice versa), at some expense of misclassification.

29 Impact of Object-Graph Descriptor
Appearance-level context Object-level context How does the object-graph descriptor compare to a simpler alternative that directly encodes the surrounding appearance features?

30 Perfect Known/Unknown Separation
Performance attainable were we able to perfectly separate segments according to whether they are known or unknown.

31 Random Splits of Known/Unknown

32 Identifying Unknown Objects
Detect Unknowns Object-level Context Discovery Learn Models Identifying Unknown Objects Image GT known/unknown unknowns building tree knowns sky road Multiple-Segmentation Entropy Maps Previous Work: [Scholkopf 2000], [Markou & Singh 2003], [Weinshall et al. 2008]

Download ppt "Object-Graphs for Context-Aware Category Discovery"

Similar presentations

Automatic Photo Pop-up Derek Hoiem Alexei A.Efros Martial Hebert Carnegie Mellon University.

Automatic Photo Pop-up Derek Hoiem Alexei A.Efros Martial Hebert Carnegie Mellon University.
Weakly supervised learning of MRF models for image region labeling Jakob Verbeek LEAR team, INRIA Rhône-Alpes.

Weakly supervised learning of MRF models for image region labeling Jakob Verbeek LEAR team, INRIA Rhône-Alpes.
Context-based object-class recognition and retrieval by generalized correlograms by J. Amores, N. Sebe and P. Radeva Discussion led by Qi An Duke University.

Context-based object-class recognition and retrieval by generalized correlograms by J. Amores, N. Sebe and P. Radeva Discussion led by Qi An Duke University.
Adding Unlabeled Samples to Categories by Learned Attributes Jonghyun Choi Mohammad Rastegari Ali Farhadi Larry S. Davis PPT Modified By Elliot Crowley.

Adding Unlabeled Samples to Categories by Learned Attributes Jonghyun Choi Mohammad Rastegari Ali Farhadi Larry S. Davis PPT Modified By Elliot Crowley.
Foreground Focus: Finding Meaningful Features in Unlabeled Images Yong Jae Lee and Kristen Grauman University of Texas at Austin.

Foreground Focus: Finding Meaningful Features in Unlabeled Images Yong Jae Lee and Kristen Grauman University of Texas at Austin.
Interactively Co-segmentating Topically Related Images with Intelligent Scribble Guidance Dhruv Batra, Carnegie Mellon University Adarsh Kowdle, Cornell.

Interactively Co-segmentating Topically Related Images with Intelligent Scribble Guidance Dhruv Batra, Carnegie Mellon University Adarsh Kowdle, Cornell.
Shape Sharing for Object Segmentation

Shape Sharing for Object Segmentation
Carolina Galleguillos, Brian McFee, Serge Belongie, Gert Lanckriet Computer Science and Engineering Department Electrical and Computer Engineering Department.

Carolina Galleguillos, Brian McFee, Serge Belongie, Gert Lanckriet Computer Science and Engineering Department Electrical and Computer Engineering Department.
Wrap Up. We talked about Filters Edges Corners Interest Points Descriptors Image Stitching Stereo SFM.

Wrap Up. We talked about Filters Edges Corners Interest Points Descriptors Image Stitching Stereo SFM.
Capturing Human Insight for Visual Learning Kristen Grauman Department of Computer Science University of Texas at Austin Work with Sudheendra Vijayanarasimhan,

Capturing Human Insight for Visual Learning Kristen Grauman Department of Computer Science University of Texas at Austin Work with Sudheendra Vijayanarasimhan,
LARGE-SCALE IMAGE PARSING Joseph Tighe and Svetlana Lazebnik University of North Carolina at Chapel Hill road building car sky.

LARGE-SCALE IMAGE PARSING Joseph Tighe and Svetlana Lazebnik University of North Carolina at Chapel Hill road building car sky.
Patch to the Future: Unsupervised Visual Prediction

Patch to the Future: Unsupervised Visual Prediction
INTRODUCTION Heesoo Myeong, Ju Yong Chang, and Kyoung Mu Lee Department of EECS, ASRI, Seoul National University, Seoul, Korea Learning.

INTRODUCTION Heesoo Myeong, Ju Yong Chang, and Kyoung Mu Lee Department of EECS, ASRI, Seoul National University, Seoul, Korea Learning.
Boundary Preserving Dense Local Regions

Boundary Preserving Dense Local Regions
Watching Unlabeled Video Helps Learn New Human Actions from Very Few Labeled Snapshots Chao-Yeh Chen and Kristen Grauman University of Texas at Austin.

Watching Unlabeled Video Helps Learn New Human Actions from Very Few Labeled Snapshots Chao-Yeh Chen and Kristen Grauman University of Texas at Austin.
Unsupervised Learning of Visual Taxonomies IEEE conference on CVPR 2008 Evgeniy Bart – Caltech Ian Porteous – UC Irvine Pietro Perona – Caltech Max Welling.

Unsupervised Learning of Visual Taxonomies IEEE conference on CVPR 2008 Evgeniy Bart – Caltech Ian Porteous – UC Irvine Pietro Perona – Caltech Max Welling.
Recognition: A machine learning approach

Recognition: A machine learning approach
Robust Higher Order Potentials For Enforcing Label Consistency

Robust Higher Order Potentials For Enforcing Label Consistency
LARGE-SCALE NONPARAMETRIC IMAGE PARSING Joseph Tighe and Svetlana Lazebnik University of North Carolina at Chapel Hill CVPR 2011Workshop on Large-Scale.

LARGE-SCALE NONPARAMETRIC IMAGE PARSING Joseph Tighe and Svetlana Lazebnik University of North Carolina at Chapel Hill CVPR 2011Workshop on Large-Scale.
Learning Spatial Context: Using stuff to find things Geremy Heitz Daphne Koller Stanford University October 13, 2008 ECCV 2008.

Learning Spatial Context: Using stuff to find things Geremy Heitz Daphne Koller Stanford University October 13, 2008 ECCV 2008.

Similar presentations

Ads by Google