#MOTION ESTIMATION AND OCCLUSION DETECTION #BLURRED VIDEO WITH LAYERS OPTICAL FLOW – PART 2 #MOTION ESTIMATION AND OCCLUSION DETECTION #BLURRED VIDEO WITH LAYERS Liron Gruber

OUTLINE Local Layering for Joint Motion Estimation and Occlusion Detection (Sun1 Liu2 Pfister1): The problem Some layers models history Local layers model Probabilistic model Results Modeling Blurred Video with Layers (Jonas Wulff and Michael J. Black): The two layered model Initialization and optimization

Joint Motion Estimation and Occlusion Detection The problem: Most motion estimation algorithms (optical flow, layered models) cannot handle large amount of occlusion Their solution: Local layering model where motion and occlusion relationships are inferred jointly Problem: motion is often initialized with no occlusion assumption despite that occlusion may be included in the final objective Solution: the uncertainties of occlusion relationships are retained so that motion is inferred by considering all the possibilities of local occlusion relationships.

What we have today Optical flow methods: -> X and T junctions problem Layered models : 1. Number of layers 2. The layer ownership 3. Depth ordering 4. The motion for each layer Inferring the number of layers and the relative depth ordering among all the layers is very challenging as more layers are expected.

Layers ‘HISTORY’ Previous work on layers relies on either motion or color cues to initialize layer segmentation For example: 1. optical flow algorithm -> motion vectors clustering -> layer primitive (Limor’s Sun) 2. optical flow algorithms -> different features ‘random forest’ classifiers (Humayun) optical flow algorithm is applied first to extract motion vectors and then clustering is used to get layer primitive 2. optical flow estimated by a number of different algorithms to generate features for the random forest classifiers (Humayun) 3. ( estimate the motion of superpixels and the occlusion boundaries between superpixels for epipolar constrained optical flow estimation )

The Problem: [1] [2] Their method

Mid - summary So – Local Layers… Global layers : all layers methods separate motion estimation (Do not use the detected occlusion to improve optical flow) (exp 3 – does that , but not vise versa...) Global layers : limited in capturing mutual or self occlusions often only contain a few number of layers (complexity explodes) So – Local Layers…

local layering model Jointly infer motion and occlusion: superpixel representations (from over segmentation) Each superpixel’s occlusion-relationships with its neighbors In the inference - keeping the uncertainties of both motion and occlusion relationships motion is inferred by considering all the possibilities of local occlusion relationships and vice versa

(after segmentation of I1 to local layers) In practice – given I1, I2 Unknowns: (after segmentation of I1 to local layers) m - movement of each layer o - occlusion map for I1 R=1,0,-1 - occlusion relationship between spatially close local layers To simplify the problem, they pre-segment the first frame using the SLIC superpixel algorithm

R The omegas depend on the unknown motion ofcourse..

From R to Pseudo Depth (d) A: A(i ,i) =| Ni | A(i, j) = -1 if j in Ni 0 otherwise b: b(i) = sum of Rij (over all j in Ni ) The set Ni contains all the spatially close superpixels that may bump into i in the next frame

Probabilistic Model Data term Motion and occlusion Motion prior EM (expactation maximization ) algorithm to maximize the posterior probability density function of the motion and the occlusion relationship. while marginalizing over the per-pixel occlusion map

Probabilistic Model – Data term (intuition only…) sum squared difference Occlusion modified

Probabilistic Model - Motion prior (intuition only…) Similar to optical flow: Motion is smooth and slow Occasionally abrupt near object boundaries EM (expactation maximization ) algorithm to maximize the posterior probability density function (p.d.f.) of the motion and the occlusion relationship, while marginalizing over the per-pixel occlusion map

Probabilistic Model - Motion and occlusion There are pixels in the set overlap No pixels in the set No overlap O seems to be determined from R,m - High order interactions Only Make sure o is consistent with R,m :

Inference process I have an extra slide…

Results Classical /baseline optical flow methods (motion) State-of-the-art learning based approach (occlusion) (Average EPE) By analyzing the occlusions locally on a per-occurrence basis, the proposed method detects most of the occlusions. The detected occlusions help recover the motion in the large occlusion regions, such as the background occluded by the hand, the leg, and the small dragon.

Summary Local layering model can handle motion and occlusion well for both challenging synthetic and real sequences (“two bars” sequence and the MPI Sintel dataset) This method improves the baseline algorithms that provide the motion estimations and performs comparably with one learning-based occlusion detection algorithm

OUTLINE Local Layering for Joint Motion Estimation and Occlusion Detection (Sun1 Liu2 Pfister1): The problem Some layers models history Local layers model Probabilistic model Results Modeling Blurred Video with Layers (Jonas Wulff and Michael J. Black): The two layered model Initialization and optimization

Modeling Blurred Video with Layers The problem: Videos contain complex spatially-varying motion blur due to finite shutter speeds. Existing methods (to estimate optical flow, deblur the images, and segment the scene) fail in such cases and fail specifically at object boundaries. Their solution: A novel 2 layered model of scenes in motion. Jointly estimate the layer segmentation and each layer's appearance and motion. the motion blur of a surface is completely determined by the motion of that surface (estimating the optical flow gives us the blur kernel).

Example

Notation a blur matrix (s is the shutter speed) observed color image unblurred color “appearance" of layer l segmentation mask for l * assumed to be constant across the sequence * only consider opaque layers (is binary) the transformation (motion) parameters for layer l at frame t *Note that while Gl does have three color channels, they enforce all three to be the same for a given pixel. *focus on affine motion, a linear Transformation, perspective effects from frame to frame in a video are often negligible a blur matrix (s is the shutter speed)

A walk through A Single Layer with Motion Blur Two Layers without Motion Blur Two Layers with Foreground Motion and Blur ……. ->

The Two-Layer Model They minimized: + Regularization term observed image segmentation mask blur matrix unblurred “appearance" transformation matrices (according to ) “rho” – charbonneir penalty..( Limor) sqrt(x^2 +eps^2) Not getting into the regularization….. They minimized: + Regularization term

(Regularization term) Spatial smoothness: Spatial Smoothness - spatial derivatives of natural images exhibit a heavy-tail distribution Background Preference – more pixels in the background (penalty function) Background preference:

The generative model - example

Initialization Good initialization is important - (the choice of initial dense optical flow algorithm is not critical, they use MDP-Flow) [b] 2 dominant motions are robustly estimated [c] MDP – motion detail preserving After these 2 steps: they combine this with a spatial consistency term, and optimize via graph cuts and use a simple non-blind deblurring method (last step not necessary..) Pyramids levels (large motions)

Optimization Iterative, alternating optimization method: 1. optimize one variable at a time (using gradient descent) 2. Terminate the optimization after 3 iterations (to avoid reaching local optima) and switch to the next variable Relax the binary-valued To deal with large motions - a Gaussian pyramid The initialization is done at the highest resolution, and the estimated starting values are rescaled to the highest pyramid level. The complete optimization schedule is then performed at each pyramid level, and the results form the input of the next level.

“…the shape of the person and rims of the bicycle being evident”

Results accurate optical flow method (no layers, no blur) layered optical flow model (no blur) They also compared in a table (extra slides) – themselves without the blur model… motion blur in an algorithm for optical flow (no layers)

Summary They developed a principled formulation of motion blur in layers. They jointly estimated parametric motion, deblurred appearance, and scene segmentation. The layered model captures the blur at boundaries and by modeling the blur process one achieves better motion estimation, layer segmentation, and layer deblurring.

The end Food for thought : can you combine the two methods? Can you use o,R,m in the equation of this paper? Thank you

EXTRA SLIDES (1)

Extra equations +explanations 1

Extra equations +explanations 2

Extra equations +explanations 3

Inference

EXTRA SLIDES (2)

Evaluation 2