1. Background vs. foreground segmentation of video sequences = +

2. The Problem Separate video into two layers: –stationary background –moving foreground Sequence is very noisy; reference image (background) is not given

3. Simple approach (1) temporal mean background temporal median

4. Simple approach (2) threshold

5. Simple approach: noise can spoil everything

6. Variational approach Find the background and foreground simultaneously by minimizing energy functional Bonus: remove noise

7. Notations  [0,t max ] N(x,t) original noisy sequence B(x) background image C(x,t) background mask (1 on background, 0 on foreground) given need to find

8. Energy functional: data term BN B - NC

9. Energy functional: data term Degeneracy: can be trivially minimized by C  0 (everything is foreground) B  N (take original image as background)

10. Energy functional: data term C1

11. there should be enough of background original images should be close to the restored background image in the background areas

12. Energy functional: smoothness For background image B For background mask C

13. Energy functional

14. Edge-preserving smoothness Regularization term Quadratic regularization [Tikhonov, Arsenin 1977] ELE: Known to produce very strong isotropic smoothing

15. Edge-preserving smoothness Regularization term Change regularization ELE:

16. Edge-preserving smoothness Regularization term ELE:

17. Edge-preserving smoothness Regularization term ELE: n  Change the coordinate system: across the edge along the edge Compare:

18. Edge-preserving smoothness Regularization term Weak edge (s  0) Conditions on  Isotropic smoothing  s) is quadratic at zero  (s) s

19. Edge-preserving smoothness Regularization term Strong edge (s   ) Conditions on  no smoothing across the edge: more smoothing along the edge: Anisotropic smoothing  s) does not grow too fast at infinity  (s) s

20. Edge-preserving smoothness Regularization term Conclusion Using regularization term of the form: we can achieve both isotropic smoothness in uniform regions and anisotropic smoothness on edges with one function 

21. Edge-preserving smoothness Regularization term Example of an edge-preserving function:

22. Edge-preserving smoothness Space of Bounded Variations Even if we have an edge-preserving functional: if the space of solutions {u} contains only smooth functions, we may not achieve the desired minimum:

23. Edge-preserving smoothness Space of Bounded Variations which one is “better”?

24. Bounded Variation – ND case bounded open subset, function Variation of over where φ

25. Edge-preserving smoothness Space of Bounded Variations integrable (absolute value) and with bounded variation Functions are not required to have an integrable derivative … What is the meaning of  u in the regularization term? Intuitively: norm of gradient |  u| is replaced with variation | Du|

26. Total variation Theorem (informally): if u  BV (  ) then

27. Hausdorff measure area > 0 area = 0 How can we measure zero-measure sets?

28. Hausdorff measure 1) cover with balls of diameter   2) sum up diameters for optimal cover (do not waste balls) 3) refine:   0

29. Hausdorff measure Formally: For A  R N k -dimensional Hausdorff measure of A up to normalization factor; covers are countable H N is just the Lebesgue measure curve in image: its length = H 1 in R 2

30. Total variation Theorem (more formally): if u  BV (  ) then u+u+ u-u- u(x)u(x) xx0x0 u +, u - - approximate upper and lower limits S u = {x  ; u + > u - } the jump set

31. Energy functional data term regularization for background image regularization for background masks

32. Total variation: example = perimeter = 4 Divide each side into n parts

33. Edge-preserving smoothness Space of Bounded Variations Small total variation (= sum of perimeters) Large total variation (= sum of perimeters)

34. Edge-preserving smoothness Space of Bounded Variations Small total variation Large total variation

35. Edge-preserving smoothness Space of Bounded Variations BV informally: functions with discontinuities on curves Edges are preserved, texture is not preserved: original sequence temporal median energy minimization in BV

36. Energy functional Time-discretized problem: Find minimum of E subject to:

37. Existence of solution Under usual assumptions  1,2 : R +  R + strictly convex, nondecreasing, with linear growth at infinity minimum of E exists in BV(B,C 1,…,C T )

38. (non-)Uniqueness is not convex w.r.t. ( B,C 1,…,C T ) ! Solution may not be unique.

39. Uniqueness But if  c  3  range 2 ( N t, t=1,…,T, x   ), then the functional is strictly convex, and solution is unique. Interpretation: if we are allowed to say that everything is foreground, background image is not well-defined

40. Finding solution BV is a difficult space: you cannot write Euler- Lagrange equations, cannot work numerically with function in BV. Strategy: construct approximating functionals admitting solution in a more regular space solve minimization problem for these functionals find solution as limit of the approximate solutions

41. Approximating functionals Recall:  1,2 ( s ) = s 2 gives smooth solutions Idea: replace  i with  i,  which are quadratic at s  0 and s  

42. Approximating functionals

43. Approximating problems has unique solution in the space  – convergence of functionals: if E   -converge to E then approximate solutions of min E  converge to min E

44. More results: Sweden

45. More results: Highway

46. More results: INRIA_1

47. More results: INRIA_1 Sequence restoration

48. More results: INRIA_2 Sequence restoration