1. STEREO MATCHING USING POPULATION-BASED MCMC
Joonyoung Park1
Wonsik Kim2
Kyoung Mu Lee2
1LG Electronics Inc.
2Computer Vision Lab.
School of EECS
Seoul National University, Korea
Good afternoon. My name is Joonyoung Park.
This work was co-worked by Wonsik Kim and my professor Kyoung Mu Lee.
I will talk about stereo matching using population-based MCMC,
which is one of the sampling-based methods.

2. Low-Level Vision as an Optimization Problem
Stereo
Photomontage
Segmentation
Denoising /Inpainting .
In various computer vision problems like stereo, Photomontage, Segmentation, Denoising, and Inpainting,
In order to solve them,
we design some energy models like this form,
Then we can change the problems into finding a solution that minimizes Energy.
That is an optimization problem.
So optimization is one of the most important part in the computer vision area.
General pairwise MRF model:

3. Existing Methods for Optimization Problem
fast
easy to get stuck at local minimum
Graph cuts
α–Expansion / Swap
Message passing
Belief Propagation (BP) / Tree-ReWeighted Message Passing (TRW)
Sampling based method
Gibbs sampler
Swendsen-Wang Cuts
There have been many tries to solve the optimization problem,
And recently, there are two kinds of methods which are widely used.
They are Graph cuts based method and message passing based method.
But they have a risk to get stuck in local minima because they are deterministic methods,
On the other hand, We have an alternative, which is sampling based method.
Because it is statistical method, it is able to find global minimum.
But there is a big problem in sampling based approach. It’s too slow.
So we use some acceleration techniques, giving up the guarantee of global minimum.
slow
global minimum

4. Our Study Population based MCMC We develop a new efficient
sampling based optimization method.
Population based MCMC
In our work, we propose to use a new kind of sampling-based method, Population-based MCMC.
From now, I will call it Pop-MCMC
We adopt this method from statistics,
and develop it for vision problems in a proper way.

5. What is sampling ?
I’ll start with the concept of sampling.

6. Sampling To generate samples from a target distribution
The original goal of sampling is to generate samples from a given distribution.
For example, the distribution is given like this,
and we extract samples like this figure by sampling.
Given
probability distribution
Obtained samples

7. Optimization Two important concepts Annealing + Sample move
previous sampling based methods
Two important concepts
Annealing + Sample move
But here, we are trying to use sampling in the optimization problem.
For optimization, there are two important concepts.
One is annealing and the other is sample move.
I will give you more explanation about these concepts in detail.

8. ... ... Optimization target distribution: temperature:
previous sampling based methods
Annealing
target distribution:
temperature:
...
The First part is annealing.
The target distribution is given by pi, and we introduce a new variable for temperature T.
At first we start from high temperature, and as time goes by, the temperature decreases.
At high temperature, the distribution is almost flat,
and at low temperature, the distribution becomes concentrated on global optimum.
...

9. ... ... Optimization target distribution: temperature:
previous sampling based methods
Annealing
target distribution:
temperature:
...
We perform the sampling from high to low temperature,
At high temperature, the samples are wandering freely,
But as the temperature becomes low,
the samples become concentrated on the global optimum.
...

10. ... ... Optimization target distribution: temperature: Most Probable
previous sampling based methods
Annealing
target distribution:
temperature:
...
Most
Probable
State!
So finally at the lowest temperature, we obtain the global optimum.
...

11. Optimization Previous methods Gibbs sampler Swendsen–Wang Cuts
previous sampling based methods
Sample move
Previous methods
Gibbs sampler
Swendsen–Wang Cuts
Then, how can we obtain the samples?
We usually get the next sample from the current sample,
and we call it sample move.
I’ll talk about previous methods of sample move.

12. Optimization Flip the label of single node Too slow (only local move)
previous sampling based methods
Sample move
Prev. Method①: Gibbs Sampler
The left one represents the current sample,
And the right one represents the next sample.
Gibbs sampler is the most primitive method.
In Gibbs sampler, we can flip the label of only one node,
So it takes very long time to converge to global optimum.
Flip the label of single node
Too slow (only local move)

13. Optimization Flip the label of a cluster
previous sampling based methods
Sample move
Prev. Method②: Swendsen-Wang Cuts
And in Swendsen-Wang cuts, we can flip the labels of multiple nodes at the same time.
At first, we make a cluster with some probabilistic rule.
And then we flip the label of the cluster.
Here, All the nodes of one cluster should have same label,
it is one of the limitation in SWC.
Anyway The move becomes rather bigger than Gibbs sampler,
but it is only local move, so it is still slow
Flip the label of a cluster
Still too slow (only local move)

14. Limitation of previous methods
Optimization
previous sampling based methods
Limitation of previous methods
slow convergence rate
Fast cooling:
easy to get stuck at local minima
As I said, the limitation of these methods is slow convergence rate.
So in practical case, we use fast cooling schedule in the process of annealing.
And then, it is easy to get stuck in local optima, not in global optimum.

15. ... ... Optimization Parallel tempering Conventional annealing:
time
Parallel tempering:
time
time
To overcome this problem, parallel tempering was proposed.
Unlike conventional annealing, it uses multiple chains simultaneously.
time
...
time

16. ... ... Optimization Parallel tempering Conventional annealing:
time
Parallel tempering:
time
interaction!
time
And there occur some interactions between chains.
It makes global move,
Therefore we can make convergence rate faster.
But it is not enough, so Pop-MCMC is proposed.
interaction!
time
...
interaction!
time

17. Population based MCMC It was originated from parallel tempering
It uses multiple chains
It allows more active interactions between chains.
We designed 3 sample moves
Mutation : The move of single chain
Exchange :
Crossover : Ways of interaction between two chains
Now, we are ready to talk about Pop-MCMC.
It was originated from parallel tempering, so it also uses multiple chains.
But it allows more active interations between chains.
And it uses three kinds of move.
mutation, exchange, crossover are those.
And the new design of mutation and crossover moves are our contribution.
I will talk about these moves in more detail.
Our contribution!!

18. PopMCMC: Move① – Mutation
Optimization
Population based MCMC
PopMCMC: Move① – Mutation
The first move is mutation.

19. ... Optimization Population based MCMC PopMCMC: Move① – Mutation time
In mutation move,
At first, one chain is chosen randomly among the multiple chains.
...
time

20. ... Optimization Population based MCMC PopMCMC: Move① – Mutation time
And for the sample of chosen chain,
we perform a sample move like conventional sampling methods.
Here, SWC which I said before is applied.
...
time

21. PopMCMC: Move② – Exchange
Optimization
Population based MCMC
PopMCMC: Move② – Exchange
The second move is exchange.

22. ... Optimization Population based MCMC PopMCMC: Move② – Exchange time
In exchange move,
Two chains are randomly chosen.
...
time

23. ... Optimization Population based MCMC PopMCMC: Move② – Exchange
time
time
time
And for the samples of two chosen chains, we exchange two samples.
...
time

24. PopMCMC: Move③ – Crossover
Optimization
Population based MCMC
PopMCMC: Move③ – Crossover
The third move is crossover.

25. ... Optimization Population based MCMC PopMCMC: Move③ – Crossover time
In crossover move, two chains are randomly chosen like the case of exchange move.
...
time

26. ... Optimization Population based MCMC PopMCMC: Move③ – Crossover time
And for the samples of the chosen chains, we decide a cluster to exchange.
In SWC, The cluster must be composed of same labels.
But in this crossover move, there is no limitations in making cluster.
I’ll skip the mathematical part which makes this possible.
If you want it, you can refer the paper.
We can make a cluster just randomly, and it enables global move.
After making cluster, the crossover moves are performed by exchanging two parts like this.
...
time

27. ... PopMCMC: Overall mutation mutation mutation mutation mutation
crossover /exchange
crossover /exchange
mutation
mutation
mutation
mutation
mutation
crossover /exchange
crossover /exchange
mutation
mutation
mutation
mutation
mutation
This is the overall mechanism of popmcmc.
For each chain, the mutation moves occur,
and between randomly chosen two chains,
the crossover or exchange moves occur.
...
crossover /exchange
crossover /exchange
mutation
mutation
mutation
mutation
mutation

28. Proposed: Population-based MCMC
Advantages
Global move
Unlike Swendsen-Wang Cuts that performs only local move, our exchange and crossover moves allow global move.
Fast Mixing
Global move and parallel tempering enable fast mixing.
The advantage of Pop-MCMC are like this.
Unlike previous methods, we can perform global moves by exchange and crossover moves,
And the global move makes mixing rates faster,
it means that we can approach global optimum faster.

29. using Segment-based Energy Model
Application to Stereo
Stereo Problem
using Segment-based Energy Model
(A. Klaus et. al., ICPR 2006)
We apply Pop-MCMC to stereo problem.
The used Energy model is adopted from Klaus’ paper in ICPR 2006

30. Segment-based Energy Model
Application to Stereo
Segment-based Energy Model
At first we over-segmented the reference image using mean-shift algorithm,
And then we consider each segment as a node, and adjacent segments as neighbors.
Each node will have a label, which is a plane of disparities.
So the energy function is given like this.
= MRF DESIGN = NODES: segments NEIGHBORS: adjacent segments
LABELS: planes of disparities

31. From Middlebury web-site
Application to Stereo
Test Images
The test images were taken from middlebury website,
Tsukuba
Venus
Teddy
Cones
From Middlebury web-site

32. Experimental Results Disparity maps Tsukuba Venus Teddy Cones 1.38
Bad pixels(%)
1.38
1.21
14.7
13.1
And these are the results of disparity maps.
In fact, the rate of bad pixels are not so low comparing with the state of the art.
It is because we just use the simple energy model.

33. Experimental Results Convergence Tsukuba Cones
However what is important is these graphs.
In the case of using the same energy model,
Comparing with the other methods like Belief Propagation or SW-Cut,
We can find lower energy state by using Pop-MCMC.

34. Conclusion Future works
We develop a new efficient Population-based MCMC for optimization and applied it to stereo problem.
It finds lower energy state compared with other methods.
Other applications / More complex energy models
Future works
I’ll summarize my talk.
In our work, we develop a new efficient Pop-MCMC and applied it to the stereo problem.
And it finds lower energy sate compared with other methods.
So it shows a possibility to be used as an optimization tool.
For future work, we can apply it to other application,
and we can use the more complex energy models for improving the quality of disparity maps

35. Seoul National University
Thank You
Computer Vision Lab.
Seoul National University
Thank you for your attention.