1. #MOTION ESTIMATION AND OCCLUSION DETECTION #BLURRED VIDEO WITH LAYERS
OPTICAL FLOW – PART 2
#MOTION ESTIMATION AND OCCLUSION DETECTION
#BLURRED VIDEO WITH LAYERS
Liron Gruber

2. OUTLINE
Local Layering for Joint Motion Estimation and Occlusion Detection (Sun1 Liu2 Pfister1):
The problem
Some layers models history
Local layers model
Probabilistic model
Results
Modeling Blurred Video with Layers (Jonas Wulff and Michael J. Black):
The two layered model
Initialization and optimization

3. Joint Motion Estimation and Occlusion Detection
The problem:
Most motion estimation algorithms (optical flow, layered models) cannot handle large amount of occlusion
Their solution:
Local layering model where motion and occlusion relationships are inferred jointly
Problem:
motion is often initialized with no occlusion assumption despite that occlusion may be included in the final objective
Solution:
the uncertainties of occlusion relationships are retained so that motion is inferred by considering all the possibilities of local occlusion relationships.

4. What we have today Optical flow methods:
-> X and T junctions problem
Layered models :
1. Number of layers
2. The layer ownership
3. Depth ordering
4. The motion for each layer
Inferring the number of layers and the relative depth ordering among all the layers is very challenging as more layers are expected.

5. Layers ‘HISTORY’
Previous work on layers relies on either motion or color cues to initialize layer segmentation
For example:
1. optical flow algorithm -> motion vectors
clustering -> layer primitive (Limor’s Sun)
2. optical flow algorithms -> different features
‘random forest’ classifiers (Humayun)
optical flow algorithm is applied first to extract motion vectors and then clustering is used to get layer primitive
2. optical flow estimated by a number of different algorithms to generate features for the random forest classifiers (Humayun)
3. ( estimate the motion of superpixels and the occlusion boundaries between superpixels for epipolar constrained optical flow estimation )

6. The Problem:
[1]
[2]
Their method

7. Mid - summary So – Local Layers… Global layers :
all layers methods separate motion estimation
(Do not use the detected occlusion to improve optical flow)
(exp 3 – does that , but not vise versa...)
Global layers :
limited in capturing mutual or self occlusions
often only contain a few number of layers (complexity explodes)
So – Local Layers…

8. local layering model Jointly infer motion and occlusion:
superpixel representations (from over segmentation)
Each superpixel’s occlusion-relationships with its neighbors
In the inference - keeping the uncertainties of
both motion and occlusion relationships
motion is inferred by considering all the possibilities of local occlusion relationships and vice versa

9. (after segmentation of I1 to local layers)
In practice – given I1, I2
Unknowns:
(after segmentation of I1 to local layers)
m - movement of each layer
o - occlusion map for I1
R=1,0,-1 - occlusion relationship between
spatially close local layers
To simplify the problem, they pre-segment the first frame using the SLIC superpixel algorithm

10. R
The omegas depend on the unknown motion ofcourse..

11. From R to Pseudo Depth (d)A:
A(i ,i) =| Ni |
A(i, j) = -1 if j in Ni
0 otherwise b:
b(i) = sum of Rij (over all j in Ni )
The set Ni contains all the spatially close superpixels
that may bump into i in the next frame

12. Probabilistic Model Data term Motion and occlusion Motion prior
EM (expactation maximization ) algorithm to maximize the posterior probability density function of the motion and the occlusion relationship.
while marginalizing over the per-pixel occlusion map

13. Probabilistic Model – Data term
(intuition only…)
sum squared difference
Occlusion modified

14. Probabilistic Model - Motion prior
(intuition only…)
Similar to optical flow:
Motion is smooth and slow
Occasionally abrupt near object boundaries
EM (expactation maximization ) algorithm to maximize the posterior probability density
function (p.d.f.) of the motion and the occlusion relationship,
while marginalizing over the per-pixel occlusion map

15. Probabilistic Model - Motion and occlusion
There are pixels in the set
overlap
No pixels in the set
No overlap
O seems to be determined from R,m - High order interactions
Only Make sure o is consistent with R,m :

16. Inference process
I have an extra slide…

17. Results Classical /baseline optical flow methods (motion)
State-of-the-art learning based approach (occlusion)
(Average EPE)
By analyzing the occlusions locally on a per-occurrence basis, the proposed method detects most of the occlusions. The detected occlusions help recover the motion in the large occlusion regions, such as the background occluded by the hand, the leg, and the small dragon.

18. Summary
Local layering model can handle motion and occlusion well for both challenging synthetic and real sequences
(“two bars” sequence and the MPI Sintel dataset)
This method improves the baseline algorithms that provide the motion estimations and performs comparably with one learning-based occlusion detection algorithm

19. OUTLINE
Local Layering for Joint Motion Estimation and Occlusion Detection (Sun1 Liu2 Pfister1):
The problem
Some layers models history
Local layers model
Probabilistic model
Results
Modeling Blurred Video with Layers (Jonas Wulff and Michael J. Black):
The two layered model
Initialization and optimization

20. Modeling Blurred Video with Layers
The problem: Videos contain complex spatially-varying motion blur due to finite shutter speeds. Existing methods (to estimate optical flow, deblur the images, and segment the scene) fail in such cases and fail specifically at object boundaries. Their solution: A novel 2 layered model of scenes in motion. Jointly estimate the layer segmentation and each layer's appearance and motion.
the motion blur of a surface is completely determined by the motion of that
surface (estimating the optical flow gives us the blur kernel).

21. Example

22. Notation a blur matrix (s is the shutter speed) observed color image
unblurred color “appearance" of layer l
segmentation mask for l
* assumed to be constant across the sequence
* only consider opaque layers (is binary)
the transformation (motion) parameters for layer l at frame t
*Note that while Gl does have three color channels, they enforce all three to be the same for a given pixel.
*focus on affine motion, a linear Transformation, perspective effects from frame to frame in a video are often negligible
a blur matrix (s is the shutter speed)

23. A walk through A Single Layer with Motion Blur
Two Layers without Motion Blur
Two Layers with Foreground Motion and Blur
……. ->

24. The Two-Layer Model They minimized: + Regularization term
observed image
segmentation mask
blur matrix
unblurred “appearance"
transformation matrices
(according to )
“rho” – charbonneir penalty..( Limor) sqrt(x^2 +eps^2)
Not getting into the regularization…..
They minimized:
+ Regularization term

25. (Regularization term)
Spatial smoothness:
Spatial Smoothness - spatial derivatives of natural images exhibit a heavy-tail distribution
Background Preference – more pixels in the background (penalty function)
Background preference:

26. The generative model - example

27. Initialization Good initialization is important -
(the choice of initial dense optical flow algorithm is not
critical, they use MDP-Flow) [b]
2 dominant motions are robustly estimated [c]
MDP – motion detail preserving
After these 2 steps: they combine this with a spatial consistency term, and optimize via graph cuts and use a simple non-blind deblurring method (last step not necessary..)
Pyramids levels
(large motions)

28. Optimization Iterative, alternating optimization method:
1. optimize one variable at a time (using gradient
descent)
2. Terminate the optimization after 3 iterations (to
avoid reaching local optima) and switch to the
next variable
Relax the binary-valued
To deal with large motions - a Gaussian pyramid
The initialization is done at the highest resolution, and the estimated starting values are rescaled to the highest pyramid level.
The complete optimization schedule is then performed at each pyramid level, and the results form the input of the next level.

29. “…the shape of the person and rims of the bicycle being evident”

30. Results accurate optical flow method (no layers, no blur)
layered optical flow model (no blur)
They also compared in a table (extra slides) – themselves without the blur model…
motion blur in an algorithm for optical flow (no layers)

31. Summary
They developed a principled formulation of motion blur in layers.
They jointly estimated parametric motion, deblurred appearance, and scene segmentation.
The layered model captures the blur at boundaries and by modeling the blur process one achieves better motion estimation, layer segmentation, and layer deblurring.

32. The end
Food for thought : can you combine the two methods? Can you use o,R,m in the equation of this paper?
Thank you

34. EXTRA SLIDES (1)

35. Extra equations +explanations 1

36. Extra equations +explanations 2

37. Extra equations +explanations 3

38. Inference

39. EXTRA SLIDES (2)

40. Evaluation 2