Optical Flow Estimation using Variational Techniques Darya Frolova

Part I Mathematical Preliminaries

Agenda Minimization problem: Investigate the existence of a solution with respect to reasonable assumptions on functional F and space A - conditions for A and F, - main theorem : existence and uniqueness of minimum Optical Flow - what is this? - different approaches - formulation of minimization problem for Optical Flow - existence of solution (using main theorem )

minimization problem: Theorem A is a compact set, function F is continuous on A. Then such that What happens when A is not a compact? What happens when functional F is not continuous? Does the minimum exist? a b x0x0 F (x)

New Theorem Theorem A is a compact set Then such that function F is continuous on A A = BV (Ω) – space of functions of bounded variation functional F is coercive and lower semicontinuous on A

Bounded Variation -1D case f has bounded variation over if exists const M such that for all Consider a x1x1 x2x2 x n -1 b …… Definition

Bounded Variation Definition The space of functions of bounded variation on Ω is denoted BV ( Ω ) Total variation of function f can be represented as follows: φ where - function φ is compactly supported (is zero outside of a compact set), -

If f is differentiable, then Supremum for and 1 f(x) sign(df/dx) Then variation of function f = Function f does not need to be differentiable

Bounded Variation – ND case bounded open subset, function Variation of over where φ

Bounded Variation – example 1) functions under integral are bounded on [0,1] 2) - problem at x = 0, 2 nd term is not bounded

Definitions Definition Banach space – complete, normed linear space Definition Space X is said to be complete, when any Cauchy sequence {x n } from X converges Definition Consider sequence {x n } from X. If natural, such that for any m, n > N, then {x n } - Cauchy sequence Space of functions of bounded variation BV(Ω) is a Banach space

Definition Functional f is coercive if Definition Consider sequence {x n }. Denote Then lower limit of sequence {x n } is a limit of sequence {X k }: ++ ++ x f (x)

Dual Space Let X denote a real Banach space. Dual space of X : X * – space of linear bounded operators X * = { f : X → R } Definition Linear operator: f (αx + βy) = α f (x) + β f (y) Bounded operator: bounded set converts to bounded set

Dual space Definition X is called reflexive if ( X *) * ReflexiveNot reflexive e.g. finite-dimensional (normed) spaces, Hilbert spaces e.g. the space of sequences ℓ ∞ ℓ∞ = {x n } : ( if bidual space of X is equal to X )

Topologies on X Definition Sequence {x n } from X strongly converges to point x if If sequence strongly converges, then it weakly converges Definition Sequence {x n } from X weakly converges to point x if for every functional f from X* ( if converges sequence of real numbers f (x n ) for any linear bounded f )

Topologies on X* Definition Sequence { f n } from X* strongly converges to f if Definition Sequence { f n } from X weakly converges to f if for every g from ( X* ) * (bidual space of X) Definition Sequence { f n } from X weakly* converges to f if for every x from X

Direct Method Problem f : X → R, where X is a Banach space inf f (x) xXxX Does the solution exist? The proof consists of three steps, which is called Direct Method of Variational Calculus

Theorem If functional f is coercive and lower semicontinuous on X, X is Banach and reflexive space Then functional f has its minimum on X :

Direct Method Step A Construct minimizing sequence {x n } for functional f : Step B a) If f is coercive, then minimizing sequence is bounded Step C If f is lower semicontinuous at point x 0 then x 0 is a point of minimum: b) If X is reflexive, then there exists weakly convergent subsequence of minimizing sequence:

Step A There exists an infimum of the set { f (x), x  X }  R and exists a sequence  R, which converges to this infimum { f (x), x  X }  R

Step B (a) If f is coercive, then minimizing sequence is bounded : Proof Consider minimizing sequence {x n }: Assume that it is not bounded: (equivalently, for any ball with radius C = 1,2… there exists an element outside this ball) If we form a subsequence of these, it will satisfy: f is coercive    But is subsequence of f (x n ), that is why So, unbounded { x n } cannot be minimizing sequence. To prove

Step B (b) we can conclude that exists x 0  X and exists subsequence such that: We proved that minimizing sequence is bounded. If X is reflexive ( ( X * )* = X ) then using theorem about weak sequential compactness, which states that: for any bounded sequence in reflexive Banach space there exists weakly convergent subsequence If X is reflexive, then there exists weakly convergent subsequence of minimizing sequence To prove Proof

Step C If f is lower semicontinuous at x 0  x 0 is a minimum point of f We proved the existence of such that: If there exists limit of f (x n ) then there exists a limit of its subsequence and these limits are equal: On the other hand, from the lower semicontinuity it follows that Hence This means that To prove Proof

Part II Optical Flow

Image Sequence Sequence of images contains information about the scene, We want to estimate motion (using variational formulation)

2D motion field Optical center 2D motion field Projection on the image plane of the 3D velocity of the scene 3D motion field Image intensity I1I1 I2I2 Motion vector - ?

Optical Flow What we are able to perceive is just an apparent motion, called Optical flow (motion, observable only through intensity variations) Intensity remains constant – no motion is perceived No object motion, moving light source produces intensity variations

Optical flow-methods Correlation-based techniques - compare parts of the first image with parts of the second in terms of the similarity in brightness patterns in order to determine the motion vectors Feature-based methods - compute and analyze Optical Flow at small number of well-defined image features Gradient-based methods - use spatiotemporal partial derivatives to estimate flow at each point

Brightness constancy Intensity of a point keeps constant along its trajectory ( reasonable for small displacements ) intensity of the pixelat time Start from point x 0 at time t 0. Trajectory → ( t, x (t) ) Differentiating → We will search the Optical Flow as the velocity field:

Brightness constancy Given sequence I (t, x) and time t 0 Find the velocity v (x) such that: We need to find the velocity field - 2 components We have 1 scalar equation Can find only “normal flow” Component in the direction of gradient I Optical Flow constraint (OFC)

Aperture problem

Solving the aperture problem Second order derivative constraint : conservation of along trajectories Rigid deformations are not considered (object moves locally in one direction) Sensitive to noise

Weighted least squares Velocities are constant in small window (spatial neighborhood) w (x) is a window function ( gives more influence to the constraint at the center of the neighborhood than at the periphery) Too local, no global regularity

Regularizing the velocity field smoothing term A (v) S (v) Velocity should change slowly in spatial domain (in image plain) Optical Flow constraint Horn and Schunck

Discontinuities But smoothing term does not allow to save discontinuities Discontinuities near edges are lost Synthetic example (method of Horn and Schunck)

Discontinuity-preserving approach Where function η permits noise removal and edge conservation [Black et.al, Cohen, Kumar, Balas, Tannenbaum, Blanc-Feraud] [Suter, Gupta and Prince, Guichard and Rudin]

Discontinuity-preserving approach summary Given : sequence I (t, x) Find: velocity field v that minimizes the energy functional E S (v) Smoothing term, we need to find conditions on η for saving discontinuities H (v) Is related to homogeneous regions A (v) L 1 norm of the Optical Flow constraint

Smoothing term Function η : R + → R + is strictly convex, nondecreasing η (0) = 0 (without loss of generality) η ( s )

Homogeneous term Idea : if there is no texture (there is no gradient), then there is no possibility to correctly estimate the flow field So, we may force it to be zero Without loss of generality:

Existence of solution Theorem Under the following hypotheses: η : R + → R + is strictly convex, nondecreasing, η (0) = 0 The minimization problem has a unique solution in BV( Ω )

Existence of a solution Proof Using direct method of Variational Calculus: Step A Construct minimizing sequence {v n } for functional E : Step B a) If E is coercive, then minimizing sequence is bounded b) If BV (Ω) is reflexive, then there exists weakly convergent subsequence of minimizing sequence: Step C If E is lower semicontinuous at point x 0 then x 0 is a point of minimum:

Existence of a solution +  c is bounded +  So, functional E is coercive

Existence of a solution Proof Using direct method of Variational Calculus: Step A Construct minimizing sequence {v n } for functional E : Step B a) If E is coercive, then minimizing sequence is bounded b) If BV (Ω) is reflexive, then there exists weakly convergent subsequence of minimizing sequence: Step C If E is lower semicontinuous at point x 0 then x 0 is a point of minimum:

Existence of a solution Space of functions of bounded variation is not reflexive: ( BV *)* ≠ BV But it has such a property that every bounded sequence I j from BV ( Ω ) has a subsequence that weakly* converges to some element I from BV ( Ω )

Existence of a solution Proof Using direct method of Variational Calculus: Step A Construct minimizing sequence {v n } for functional E : Step B a) If E is coercive, then minimizing sequence is bounded b) If BV (Ω) is reflexive, then there exists weakly convergent subsequence of minimizing sequence: Step C If E is lower semicontinuous at point x 0 then x 0 is a point of minimum:

The End