Presentation is loading. Please wait.

Presentation is loading. Please wait.

Robust Higher Order Potentials For Enforcing Label Consistency

Published by Modified over 9 years ago

Similar presentations

Presentation on theme: "Robust Higher Order Potentials For Enforcing Label Consistency"— Presentation transcript:

1 Robust Higher Order Potentials For Enforcing Label Consistency
Pushmeet Kohli Microsoft Research Cambridge Lubor Ladicky Philip Torr Oxford Brookes University, Oxford CVPR 2008

2 Image labelling Problems
Assign a label to each image pixel Geometry Estimation Image Denoising Object Segmentation Sky Building Tree Grass

3 Object Segmentation using CRFs
(Shotton et al. ECCV 2006) CRF Energy Unary potentials based on Colour, Location and Texture features Encourages label consistency in adjacent pixels

4 Limitations of Pairwise CRFs
Encourages short boundaries (Shrinkage bias) Can only enforce label consistency in adjacent pixels Inability to incorporate region based features Image Unary Potential MAP-CRF Solution

5 Label Consistency in Image Regions
Pixels constituting some regions belong to Same plane (Orientation) (Hoiem, Efros, & Herbert, ICCV’05) Same object (Russel, Efros, Sivic, Freeman, & Zisserman, CVPR06) Image (MSRC) Segmentation (Mean shift)

6 Image labelling using segments
Unsupervised Segmentation Object Labelling Geometric Context [Hoiem et al, ICCV05] Object Segmentation [He et al. ECCV06, Yang et al. CVPR07, Rabinovich et al. ICCV07, Batra et al. CVPR08] Interactive Video Segmentation [Wang, SIGGRAPH 2005 ] Not robust to Inconsistent Segments!

7 Our Higher Order CRF Model
Encourages label consistency in regions Multiple Segmentations c

8 Label Consistency in Segments
Encourages consistency within super-pixels Takes the form of a PN Potts model [Kohli et al. CVPR 2007] c

9 Label Consistency in Segments
Encourages consistency within super-pixels Takes the form of a PN Potts model [Kohli et al. CVPR 2007] c Cost: 0

10 Label Consistency in Segments
Encourages consistency within super-pixels Takes the form of a PN Potts model [Kohli et al. CVPR 2007] c Cost: f (|c|)

11 Label Consistency in Segments
Encourages consistency within super-pixels Takes the form of a PN Potts model [Kohli et al. CVPR 2007] Does not distinguish between Good/Bad Segments ! c Cost: f (|c|)

12 Quality based Label Consistency
Label inconsistency cost depends on segment quality How to measure quality G(c)? [Ren and Malik ICCV03, Rabinovich et al. ICCV07, many others] Colour and Texture Similarity Contour Energy Measure quality from variance in feature responses Higher order generalization of contrast-sensitive pairwise potential

13 Quality based Label Consistency
Mean shift segmentation Segment Quality (darker is better) MSRC image

14 Robust Consistency Potentials
gmax PN Potts 1 Too Rigid! Inconsistent Pixels gmax 1 T Robust Inconsistent Pixels

15 Robust Consistency Potentials
Maximum Inconsistency Cost Number of Inconsistent Pixels Slope gmax 1 T Robust Inconsistent Pixels

16 Minimizing Higher order Energy Functions
Message passing is computationally expensive High runtime and space complexity - O(LN) L = Number of Labels, N = Size of Clique Efficient BP for Higher Order MRFs [Lan et al. ECCV 06, Potetz CVPR 2007] 2x2 clique potentials for Image Denoising Take minutes per iteration (Hours to converge)

17 Minimizing Higher order Energy Functions
Graph Cut based move making algorithm [Kohli et al. CVPR 2007] Can handle very high order energy functions Extremely efficient: computation time in the order of seconds Only applicable to some classes of functions (PN Potts) Cannot handle robust consistency potential This paper Can minimize a much larger class of higher order energy functions Same time complexity as [Kohli et al. CVPR 2007]

18 Move making algorithms
Expansion and Swap move algorithms [Boykov Veksler and Zabih, PAMI 2001] Makes a series of changes to the solution (moves) Each move results in a solution with smaller energy Current Solution How to minimize move functions? Move to new solution Generate pseudo-boolean move function Minimize move function to get optimal move

19 Minimizing Move Functions using Graph Cuts
Most pairwise CRF models used in Computer Vision lead to submodular move functions Second order Pseudo-boolean Function Minimization (submodular) st-mincut (Positive weights) Optimal moves can be found extremely efficiently using graphs cuts

20 Minimizing Higher Order Energy Functions: Our results
We show that a large class of higher order potentials lead to higher order submodular move functions Can be minimized in polynomial time Submodular Function Minimization Minimizing general submodular functions is computationally expensive Complexity O(n6) Cannot handle large problems! Details in Technical Report

21 Minimizing Higher Order Energy Functions: Our results
Minimizing Higher order functions using Graph cuts Higher order functions can be transformed to second order functions by adding auxillary variables Exponential number of auxillary variables needed in general Result 2 Our higher order functions can be transformed to second-order functions using ≤2 auxillary variables per potential. Can be minimized extremely efficiently Complexity << O(n6) Details in Technical Report

22 Overview of our Method + + Higher Order Energy Segmentation Solution
Unary Potentials [Shotton et al. ECCV 2006] + Energy Minimization Contrast Sensitive Pairwise Potentials + Segmentation Solution Higher Order Potentials (Multiple Segmentations)

23 Experimental results Datasets: MSRC (21), Sowerby (7)
[Shotton et al. ECCV 2006] [He et al. CVPR 04]

24 Qualitative Results Image (MSRC-21) Pairwise CRF Higher order CRF
Ground Truth Grass Sheep

25 Qualitative Results (Contd..)
Image (MSRC-21) Pairwise CRF Higher order CRF Ground Truth Results can be improved using image specific colour models Rother et al. SIGGRAPH 2004 Shotton et al. ECCV 2006

26 Quantitative Results: Problems
Rough ground truth segmentations Fine structures have small influence on overall pixel accuracy

27 Generating Accurate Segmentations
Generated accurate segmentation of 27 images 30 minutes per image Image (MSRC-21) Original Segmentation New Segmentation

28 Relationship between Qualitative and Quantitative Results
Pairwise CRF Higher order CRF Ground Truth Image (MSRC-21) Overall Pixel Accuracy 95.8% 98.7% Small changes in pixel accuracy can lead to large improvements in segmentation results.

29 Quantitative Accuracy
Measure accuracy in labelling boundary pixels. Accuracy evaluated in boundary bands of variable width Hand-labelled Segmentation Trimap (8-pixels) Trimap (16-pixels) Image (MSRC-21)

30 Quantitative Accuracy
Measure accuracy in labelling boundary pixels. Accuracy evaluated in boundary bands of variable width

31 Conclusions Method to enforce label consistency in image regions
Generalization of the commonly used Pairwise CRF model Allows integration of pixel and region level features for image labelling problems

32 Thanks

33 Number of Segmentations
Running Time Results Time (sec) Number of Segmentations Inconsistency Cost

34 Qualitative Results (Contd..)
Image (MSRC-21) Pairwise CRF Higher order CRF Ground Truth

35 Transformation to second order functions
Auxiliary variables

Download ppt "Robust Higher Order Potentials For Enforcing Label Consistency"

Similar presentations

Using Strong Shape Priors for Multiview Reconstruction Yunda SunPushmeet Kohli Mathieu BrayPhilip HS Torr Department of Computing Oxford Brookes University.

Using Strong Shape Priors for Multiview Reconstruction Yunda SunPushmeet Kohli Mathieu BrayPhilip HS Torr Department of Computing Oxford Brookes University.
POSE–CUT Simultaneous Segmentation and 3D Pose Estimation of Humans using Dynamic Graph Cuts Mathieu Bray Pushmeet Kohli Philip H.S. Torr Department of.

POSE–CUT Simultaneous Segmentation and 3D Pose Estimation of Humans using Dynamic Graph Cuts Mathieu Bray Pushmeet Kohli Philip H.S. Torr Department of.
Mean-Field Theory and Its Applications In Computer Vision1 1.

Mean-Field Theory and Its Applications In Computer Vision1 1.
O BJ C UT M. Pawan Kumar Philip Torr Andrew Zisserman UNIVERSITY OF OXFORD.

O BJ C UT M. Pawan Kumar Philip Torr Andrew Zisserman UNIVERSITY OF OXFORD.
Automatic Photo Pop-up Derek Hoiem Alexei A.Efros Martial Hebert Carnegie Mellon University.

Automatic Photo Pop-up Derek Hoiem Alexei A.Efros Martial Hebert Carnegie Mellon University.
HOPS: Efficient Region Labeling using Higher Order Proxy Neighborhoods Albert Y. C. Chen 1, Jason J. Corso 1, and Le Wang 2 1 Dept. of Computer Science.

HOPS: Efficient Region Labeling using Higher Order Proxy Neighborhoods Albert Y. C. Chen 1, Jason J. Corso 1, and Le Wang 2 1 Dept. of Computer Science.
Pose Estimation and Segmentation of People in 3D Movies Karteek Alahari, Guillaume Seguin, Josef Sivic, Ivan Laptev Inria, Ecole Normale Superieure ICCV.

Pose Estimation and Segmentation of People in 3D Movies Karteek Alahari, Guillaume Seguin, Josef Sivic, Ivan Laptev Inria, Ecole Normale Superieure ICCV.
Foreground Focus: Finding Meaningful Features in Unlabeled Images Yong Jae Lee and Kristen Grauman University of Texas at Austin.

Foreground Focus: Finding Meaningful Features in Unlabeled Images Yong Jae Lee and Kristen Grauman University of Texas at Austin.
Spectral graph reduction for image and streaming video segmentation Fabio Galasso 1 Margret Keuper 2 Thomas Brox 2 Bernt Schiele 1 1 Max Planck Institute.

Spectral graph reduction for image and streaming video segmentation Fabio Galasso 1 Margret Keuper 2 Thomas Brox 2 Bernt Schiele 1 1 Max Planck Institute.
Joint Optimisation for Object Class Segmentation and Dense Stereo Reconstruction Ľubor Ladický, Paul Sturgess, Christopher Russell, Sunando Sengupta, Yalin.

Joint Optimisation for Object Class Segmentation and Dense Stereo Reconstruction Ľubor Ladický, Paul Sturgess, Christopher Russell, Sunando Sengupta, Yalin.
Agenda Introduction Bag-of-words models Visual words with spatial location Part-based models Discriminative methods Segmentation and recognition Recognition-based.

Agenda Introduction Bag-of-words models Visual words with spatial location Part-based models Discriminative methods Segmentation and recognition Recognition-based.
Scene Labeling Using Beam Search Under Mutex Constraints ID: O-2B-6 Anirban Roy and Sinisa Todorovic Oregon State University 1.

Scene Labeling Using Beam Search Under Mutex Constraints ID: O-2B-6 Anirban Roy and Sinisa Todorovic Oregon State University 1.
Recovering Human Body Configurations: Combining Segmentation and Recognition Greg Mori, Xiaofeng Ren, and Jitentendra Malik (UC Berkeley) Alexei A. Efros.

Recovering Human Body Configurations: Combining Segmentation and Recognition Greg Mori, Xiaofeng Ren, and Jitentendra Malik (UC Berkeley) Alexei A. Efros.
I Images as graphs Fully-connected graph – node for every pixel – link between every pair of pixels, p,q – similarity w ij for each link j w ij c Source:

I Images as graphs Fully-connected graph – node for every pixel – link between every pair of pixels, p,q – similarity w ij for each link j w ij c Source:
The University of Ontario CS 4487/9687 Algorithms for Image Analysis Multi-Label Image Analysis Problems.

The University of Ontario CS 4487/9687 Algorithms for Image Analysis Multi-Label Image Analysis Problems.
ICCV 2007 tutorial on Discrete Optimization Methods in Computer Vision part I Basic overview of graph cuts.

ICCV 2007 tutorial on Discrete Optimization Methods in Computer Vision part I Basic overview of graph cuts.
Efficient Inference for Fully-Connected CRFs with Stationarity

Efficient Inference for Fully-Connected CRFs with Stationarity
GrabCut Interactive Image (and Stereo) Segmentation Joon Jae Lee Keimyung University Welcome. I will present Grabcut – an Interactive tool for foreground.

GrabCut Interactive Image (and Stereo) Segmentation Joon Jae Lee Keimyung University Welcome. I will present Grabcut – an Interactive tool for foreground.
GraphCut-based Optimisation for Computer Vision Ľubor Ladický.

GraphCut-based Optimisation for Computer Vision Ľubor Ladický.
Optimization & Learning for Registration of Moving Dynamic Textures Junzhou Huang 1, Xiaolei Huang 2, Dimitris Metaxas 1 Rutgers University 1, Lehigh University.

Optimization & Learning for Registration of Moving Dynamic Textures Junzhou Huang 1, Xiaolei Huang 2, Dimitris Metaxas 1 Rutgers University 1, Lehigh University.

Similar presentations

Ads by Google