Background vs. foreground segmentation of video sequences = +

The Problem Separate video into two layers: –stationary background –moving foreground Sequence is very noisy; reference image (background) is not given

Simple approach (1) temporal mean background temporal median

Simple approach (2) threshold

Simple approach: noise can spoil everything

Variational approach Find the background and foreground simultaneously by minimizing energy functional Bonus: remove noise

Notations  [0,t max ] N(x,t) original noisy sequence B(x) background image C(x,t) background mask (1 on background, 0 on foreground) given need to find

Energy functional: data term BN B - NC

Energy functional: data term Degeneracy: can be trivially minimized by C  0 (everything is foreground) B  N (take original image as background)

Energy functional: data term C1

there should be enough of background original images should be close to the restored background image in the background areas

Energy functional: smoothness For background image B For background mask C

Energy functional

Edge-preserving smoothness Regularization term Quadratic regularization [Tikhonov, Arsenin 1977] ELE: Known to produce very strong isotropic smoothing

Edge-preserving smoothness Regularization term Change regularization ELE:

Edge-preserving smoothness Regularization term ELE:

Edge-preserving smoothness Regularization term ELE: n  Change the coordinate system: across the edge along the edge Compare:

Edge-preserving smoothness Regularization term Weak edge (s  0) Conditions on  Isotropic smoothing  s) is quadratic at zero  (s) s

Edge-preserving smoothness Regularization term Strong edge (s   ) Conditions on  no smoothing across the edge: more smoothing along the edge: Anisotropic smoothing  s) does not grow too fast at infinity  (s) s

Edge-preserving smoothness Regularization term Conclusion Using regularization term of the form: we can achieve both isotropic smoothness in uniform regions and anisotropic smoothness on edges with one function 

Edge-preserving smoothness Regularization term Example of an edge-preserving function:

Edge-preserving smoothness Space of Bounded Variations Even if we have an edge-preserving functional: if the space of solutions {u} contains only smooth functions, we may not achieve the desired minimum:

Edge-preserving smoothness Space of Bounded Variations which one is “better”?

Bounded Variation – ND case bounded open subset, function Variation of over where φ

Edge-preserving smoothness Space of Bounded Variations integrable (absolute value) and with bounded variation Functions are not required to have an integrable derivative … What is the meaning of  u in the regularization term? Intuitively: norm of gradient |  u| is replaced with variation | Du|

Total variation Theorem (informally): if u  BV (  ) then

Hausdorff measure area > 0 area = 0 How can we measure zero-measure sets?

Hausdorff measure 1) cover with balls of diameter   2) sum up diameters for optimal cover (do not waste balls) 3) refine:   0

Hausdorff measure Formally: For A  R N k -dimensional Hausdorff measure of A up to normalization factor; covers are countable H N is just the Lebesgue measure curve in image: its length = H 1 in R 2

Total variation Theorem (more formally): if u  BV (  ) then u+u+ u-u- u(x)u(x) xx0x0 u +, u - - approximate upper and lower limits S u = {x  ; u + > u - } the jump set

Energy functional data term regularization for background image regularization for background masks

Total variation: example = perimeter = 4 Divide each side into n parts

Edge-preserving smoothness Space of Bounded Variations Small total variation (= sum of perimeters) Large total variation (= sum of perimeters)

Edge-preserving smoothness Space of Bounded Variations Small total variation Large total variation

Edge-preserving smoothness Space of Bounded Variations BV informally: functions with discontinuities on curves Edges are preserved, texture is not preserved: original sequence temporal median energy minimization in BV

Energy functional Time-discretized problem: Find minimum of E subject to:

Existence of solution Under usual assumptions  1,2 : R +  R + strictly convex, nondecreasing, with linear growth at infinity minimum of E exists in BV(B,C 1,…,C T )

(non-)Uniqueness is not convex w.r.t. ( B,C 1,…,C T ) ! Solution may not be unique.

Uniqueness But if  c  3  range 2 ( N t, t=1,…,T, x   ), then the functional is strictly convex, and solution is unique. Interpretation: if we are allowed to say that everything is foreground, background image is not well-defined

Finding solution BV is a difficult space: you cannot write Euler- Lagrange equations, cannot work numerically with function in BV. Strategy: construct approximating functionals admitting solution in a more regular space solve minimization problem for these functionals find solution as limit of the approximate solutions

Approximating functionals Recall:  1,2 ( s ) = s 2 gives smooth solutions Idea: replace  i with  i,  which are quadratic at s  0 and s  

Approximating functionals

Approximating problems has unique solution in the space  – convergence of functionals: if E   -converge to E then approximate solutions of min E  converge to min E

More results: Sweden

More results: Highway

More results: INRIA_1

More results: INRIA_1 Sequence restoration

More results: INRIA_2 Sequence restoration