Download presentation
Presentation is loading. Please wait.
1
Outline Image Segmentation by Data-Driven Markov Chain Monte Carlo
Z. Tu and S. C. Zhu, “Image segmentation by data-driven Markov chain Monte Carlo,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 24, no. 5, , 2003
2
Image Segmentation The process to decompose an image into its constituent regions November 12, 2018 Computer Vision
3
Image Segmentation Motivation – cont.
November 12, 2018 Computer Vision
4
Image Segmentation Image Lattice: Image: For any point either or
11/12/2018 Image Segmentation Image Lattice: Image: For any point either or Lattice partition into K disjoint regions: Region is discrete label map: Region Boundary is Continuous: Regions are treated as discrete label maps (easier for maintaining topology) Boundaries are treated as continuous (easier for diffusion) November 12, 2018 Computer Vision
5
11/12/2018 Bayesian Formulation Each Image Region is a realization from a probabilistic model are parameters of model indexed by A segmentation is denoted by a vector of hidden variables W; K is number of regions Bayesian Framework: We will characterize the space of all segmentations in a future slide Posterior Likelihood Prior November 12, 2018 Computer Vision
6
Prior Over Segmentations
11/12/2018 Prior Over Segmentations Want fewer regions Want regions with smooth boundaries ~ uniform Want less complex models The product in the prior corresponds to the assumption that the prior for each region are independent. |\Theta_i| is the number of parameters of theta C is 0.9 (hard-coded) Gamma is the only free parameter (the only free parameter that they vary) What is \lambda_0 and \mu and \nu ? (not defined in the paper) Want large regions November 12, 2018 Computer Vision
7
Likelihood for Images Visual Patterns are independent stochastic processes is model-type index is model parameter vector is image appearance in of the ith region Grayscale Color November 12, 2018 Computer Vision
8
Four Gray-level Models
11/12/2018 Four Gray-level Models Uniform Clutter Texture Shading Uniform: Gaussian Intensity Histogram FB Response Histogram B-Spline November 12, 2018 Computer Vision
9
Four Gray-level Models
11/12/2018 Four Gray-level Models Uniform Clutter Texture Shading Clutter: Gaussian Intensity Histogram FB Response Histogram B-Spline November 12, 2018 Computer Vision
10
Four Gray-level Models
11/12/2018 Four Gray-level Models Uniform Clutter Texture Shading Texture: Gaussian Intensity Histogram FB Response Histogram B-Spline November 12, 2018 Computer Vision
11
Four Gray-level Models
11/12/2018 Four Gray-level Models Uniform Clutter Texture Shading Shading: Gaussian Intensity Histogram FB Response Histogram B-Spline November 12, 2018 Computer Vision
12
Four Gray-level Models
11/12/2018 Four Gray-level Models Uniform Clutter Texture Shading Gray-level model space: Gaussian Intensity Histogram FB Response Histogram B-Spline November 12, 2018 Computer Vision
13
Calibration Likelihoods are calibrated using empirical study
11/12/2018 Calibration Likelihoods are calibrated using empirical study Calibration required to make likelihoods for different models comparable (necessary for model competition) This is a hack. November 12, 2018 Computer Vision
14
Calibration – cont. November 12, 2018 Computer Vision
15
Three Color Models (L*,u*,v*)
11/12/2018 Three Color Models (L*,u*,v*) Gaussian Mixture of 2 Gaussians Bezier Spline Color model space: November 12, 2018 Computer Vision
16
What do we do with scores?
Search November 12, 2018 Computer Vision
17
Anatomy of Solution Space
11/12/2018 Anatomy of Solution Space Space of all k-partitions General partition space Space of all segmentations or Scene Space Partition space K Model spaces Space of all segmentation is the union of all k-segmentations A k-segmentation is the product of one k-partition space and k spaces for the image models A model space is made up of cue-spaces November 12, 2018 Computer Vision
18
Anatomy of the Solution Space
November 12, 2018 Computer Vision
19
Searching Through Segmentations
Exhaustive Enumeration of all segmentations Takes too long! Greedy Search (Gradient Ascent) Local minima! Stochastic Search Takes too long MCMC based exploration November 12, 2018 Computer Vision
20
Why MCMC What is it? What does it do?
- A clever way of searching through a high-dimensional space - A general purpose technique of generating samples from a probability - Iteratively searches through space of all segmentations by constructing a Markov Chain which converges to stationary distribution November 12, 2018 Computer Vision
23
Designing Markov Chains
Three Markov Chain requirements Ergodic: from an initial segmentation W0, any other state W can be visited in finite time (no greedy algorithms); ensured by jump-diffusion dynamics Aperiodic: ensured by random dynamics Detailed Balance: every move is reversible November 12, 2018 Computer Vision
24
5 Dynamics 1.) Boundary Diffusion 2.) Model Adaptation
3.) Split Region 4.) Merge Region 5.) Switch Region Model At each iteration, we choose a dynamic with probability q(1),q(2),q(3),q(4),q(5) November 12, 2018 Computer Vision
25
Dynamics 1: Boundary Diffusion
11/12/2018 Dynamics 1: Boundary Diffusion Diffusion* within Temperature Decreases over Time Brownian Motion Along Curve Normal Boundary Between Regions i and j The motion of {x(s),y(s)} follows steepest ascent equation of log(p(W | I)) plus brownian motion Brownian Motion is a Normal distribution *Movement within partition space November 12, 2018 Computer Vision
26
Dynamics 2: Model Adaptation
11/12/2018 Dynamics 2: Model Adaptation Fit the parameters* of a region by steepest ascent How is this not greedy? It appears to be greedy. *Movement within cue space November 12, 2018 Computer Vision
27
Dynamics 3-4: Split and Merge
11/12/2018 Dynamics 3-4: Split and Merge Remaining Variables Are unchanged Split one region into two Probability of Proposed Split q(3) is just the probability of choosing dynamics 3 Conditional Probability of how likely chain proposes to move to W’ from W Data-Driven Speedup November 12, 2018 Computer Vision
28
Dynamics 3-4: Split and Merge
11/12/2018 Dynamics 3-4: Split and Merge Remaining Variables Are unchanged Merge two Regions Data-Driven Speedup Probability of Proposed Merge q(3) is just the probability of choosing dynamics 3 November 12, 2018 Computer Vision
29
Dynamics 5: Model Switching
11/12/2018 Dynamics 5: Model Switching Change models Proposal Probabilities Data-Driven Speedup q(5) is just the probability of choosing dynamics 5 November 12, 2018 Computer Vision
30
11/12/2018 Motivation of DD Region Splitting: How to decide where to split a region? Model Switching: Once we switch to a new model, what parameters do we jump to? Vs. Page 18 stuff Model Adaptation Required some initial parameter vectors November 12, 2018 Computer Vision
31
Data Driven Methods Focus on boundaries and model parameters derived from data: compute these before MCMC starts Cue Particles: Clustering in Model Space K-partition Particles: Edge Detection Particles Encode Probabilities Parzen Window Style November 12, 2018 Computer Vision
32
Cue Particles In Action
11/12/2018 Cue Particles In Action Clustering in Color Space This figure shows a few color clusters in (L,u,v) color space. The size of the ball represents the weights of each particle. Each cluster has an associated saliency map November 12, 2018 Computer Vision
33
Cue Particles In Action
November 12, 2018 Computer Vision
34
Cue Particles In Action
November 12, 2018 Computer Vision
35
Cue Particles In Action
November 12, 2018 Computer Vision
36
Cue Particles In Action
November 12, 2018 Computer Vision
37
Cue Particles In Action
November 12, 2018 Computer Vision
38
Cue Particles In Action
November 12, 2018 Computer Vision
39
Cue Particles In Action
November 12, 2018 Computer Vision
40
Edge Detection November 12, 2018 Computer Vision
41
Edge Detection November 12, 2018 Computer Vision
42
K-partition Particles in Action
Edge detection gives us a good idea of where we expect a boundary to be located November 12, 2018 Computer Vision
43
Particles or Parzen Window* Locations?
What is this particle about? A particle is just the position of a Parzen-window which is used for density estimation 1D particles November 12, 2018 Computer Vision
44
Multiple Solutions MAP gives us one solution Output of MCMC sampling
11/12/2018 Multiple Solutions MAP gives us one solution Output of MCMC sampling How do we get multiple solutions? Distance between two segmentations W1 and W2 is last paragraph of paper: it is a non-principled way of doing this They simply measure the distance between W1 and W2 by accumulating the difference in the number of regions in W1 and W2 and the types of Image models used at each pixel by W1 and W2 Parzen Windows: Again Scene Particles November 12, 2018 Computer Vision
45
Why multiple solutions?
Segmentation is often not the final stage of computation A higher level task such as recognition can utilize a segmentation We don’t want to make any hard decision before recognition multiple segmentations = good idea November 12, 2018 Computer Vision
46
11/12/2018 K-adventurers We want to keep a fixed number K of segmentations but we don’t want to keep trivially different segmentations Goal: Keep the K segmentations that best preserve the posterior probability in KL-sense Greedy Algorithm: - Add new particle, remove worst particle The papers shows no results on K-adventurers. Maybe it doesn’t work too well. November 12, 2018 Computer Vision
47
Results (Multiple Solutions)
11/12/2018 Results (Multiple Solutions) This is the result of using different scales not K-adventurers result. November 12, 2018 Computer Vision
48
Results November 12, 2018 Computer Vision
49
Results (Color Images)
November 12, 2018 Computer Vision
50
Conclusions DDMCMC: Combines generative (top-down) and discriminative (bottom-up) approaches Traverse the space of segmentations via Markov Chains November 12, 2018 Computer Vision
51
11/12/2018 References DDMCMC Paper: DDMCMC Website: MCMC Tutorial by Authors: Some nice segmentation results on the DDMCMC website. Funny Anecdote: Reading this paper was similar to performing some type MCMC. After first iteration I only understood 15% of the paper, and after a few iterations I got to 40%, but then another careful reading and I was back down to 30%. Finally, my understanding of this paper converged to a stable high-80% (I believe). It took many hours to understand this work and present it to a broad audience of 1st and 2nd year Robotics PhD/Masters Students at CMU. November 12, 2018 Computer Vision
Similar presentations
