Computer Science Department Distribution Fields A Unifying Representation for Low-Level Vision Problems Erik Learned-Miller with Laura Sevilla Lara, Manju Narayana, Ben Mears

2 Distribution Fields Unifying Low-Level Vision  Tracking  Optical Flow  Affine invariant matching  Stereo  Backgrounding  Registration  Image Stitching What kind of representation serves all these problems well?

3 Distribution Fields Unifying Descriptors and Representations  HOG, SIFT, Shape Contexts  Geometric blur  Multi-scale methods  Mixture of Gaussian backgrounding  Bilateral filter  Joint alignment (congealing)

4 Distribution Fields Tracking

5 Distribution Fields Basics of Tracking Frame TFrame T+d

6 Distribution Fields Basics of Tracking Frame TFrame T+d

7 Distribution Fields Basics of Tracking Frame TFrame T+d

8 Distribution Fields Basics of Tracking Frame TFrame T+d

9 Distribution Fields The Core Matching Problem patch I image J Find best match of patch I to image J, for some set of transformations.

10 Distribution Fields Key issues  Large “basin of attraction” in gradient descent  Robustness to appearance changes of target

11 Distribution Fields Local optimum problem in alignment Unaligned Stuck in a local optimum

12 Distribution Fields Common solution to gradient descent matching  Blur images? “Spreads information” Also destroys information through averaging

13 Distribution Fields Exploding an image One plane for each feature value.

14 Distribution Fields Spatial Blur: 3d convolution with 2d Gaussian

15 Distribution Fields Spatial Blur: 3d convolution with 2d Gaussian KEY PROPERTY: doesn't destroy information through averaging

16 Distribution Fields Properties of Distribution Field Representation  Much larger basin of attraction than other descriptors.  A useful probability model from a single image.  Inherently multi-scale.  Extension and unification of many popular descriptors.  State of the art performance from extremely simple algorithms.

17 Distribution Fields Properties of Distribution Field Representation  Much larger basin of attraction than other descriptors.  A useful probability model from a single image.  Inherently multi-scale.  Extension and unification of many popular descriptors.  State of the art performance with extremely simple algorithms.  THANKS!

18 Distribution Fields The likelihood match

19 Distribution Fields Sharpening match

20 Distribution Fields Understanding the sharpening match What standard deviation maximizes the likelihood of a given point under a zero-mean Gaussian?

21 Distribution Fields Intuition behind sharpening match  Increase standard deviation until it matches “average distance” to matching points.

22 Distribution Fields Properties of the sharpening match  A patch has probability of 1.0 under its own distribution field.  Probability of an image patch degrades gracefully as it is translated away from best position.  Optimum “sigma” value gives a very intuitive notion of the quality of the image match.

23 Distribution Fields Basin of attraction studies

24 Distribution Fields Basin of attraction studies GIVEN A RANDOM PATCH...

25 Distribution Fields Basin of attraction studies AND A RANDOM DISPLACEMENT...

26 Distribution Fields Basin of attraction studies

27 Distribution Fields Basin of attraction studies

28 Distribution Fields Basin of attraction results

29 Distribution Fields Tracking results  State of the art results on tracking with standard sequences Very simple code Trivial motion model Simple memory model

30 Distribution Fields Closely Related work  Mixture of Gaussian backgrounding (Stauffer...)  Shape contexts (Belongie and Malik)  Congealing (me)  Bilateral filter  SIFT (Lowe), HOG (Dalal and Triggs)  Geometric Blur (Berg)  Rectified flow techniques (Efros, Mori)  Mean-shift tracking  Kernel tracking  and many others...