1. Motion Detail Preserving Optical Flow Estimation Li Xu 1, Jiaya Jia 1, Yasuyuki Matsushita 2 1 The Chinese University of Hong Kong 2 Microsoft Research Asia

2. Conventional Optical Flow Middlebury Benchmark [Baker et al. 07] Dominant Scheme: Coarse-to-Fine Warping

3. Large Displacement Optical Flow Region Matching [Brox et al. 09, 10] Discrete Local Search [Steinbrucker et al. 09]

4. Both Large and Small Motion Exist Capture large motion Preserve sub-pixel accuracy

5. Our Work Framework – Extended coarse-to-fine motion estimation for both large and small displacement optical flow Model – A new data term to selectively combine constraints Solver – Efficient numerical solver for discrete-continuous optimization

6. Outline Framework – Extended coarse-to-fine motion estimation for both large and small displacement optical flow Model – A new data term to selectively combine constraints Solver – Efficient numerical solver for discrete-continuous optimization

7. The Multi-scale Problem

8. Ground truth

9. The Multi-scale Problem Ground truth

10. … Estimate Ground truth

11. The Multi-scale Problem Large discrepancy between initial values and optimal motion vectors Our solution – Improve flow initialization to reduce the reliance on the initialization from coarser levels

12. Extended Flow Initialization Sparse feature matching for each level

13. Extended Flow Initialization Identify missing motion vectors

14. Extended Flow Initialization Identify missing motion vectors

15. Extended Flow Initialization …

16. Fuse

17. Outline Framework: extended initialization for coarse- to-fine motion estimation Model: selective data term Efficient numerical solver

18. Data Constraints Average Gradient constancy Color constancy

19. Pixels moving out of shadow Problems Color constancy is violated Average: : ground truth motion of p 1 Gradient constancy holds

20. Pixels undergoing rotational motion Problems Color constancy holds Gradient constancy is violated : ground truth motion of p 2 Average:

21. Our Proposal Selectively combine the constraints where

22. Comparisons

23. Outline Framework: extended initialization for coarse to fine motion estimation Model: selective data term Efficient numerical solver

24. Energy Functions and Solver Total energy Probability of a particular state of the system

25. Mean Field Approximation Partition function Sum over all possible values of α... The effective potential E eff (u) [Geiger & Girosi, 1989]

26. Optimal condition (Euler-Lagrange equations) It decomposes to {

27. {

28. Algorithm Skeleton For each level Extended Flow Initialization (QPBO) Continuous Minimization (Iterative reweight) – Update – Compute flow field (Variable Splitting) {

29. Results Averaging constraints Ours Difference

30. Middlebury Dataset EPE=0.74

31. Results from Different Steps Coarse-to-fine Extended coarse-to-fine

32. EPE=0.15 rank =1 EPE=0.24 rank =1

33. Large Displacement Overlaid Input

34. Large Displacement Motion Estimates Coarse-to-fineOur ResultWarping Result

35. Comparison Motion Magnitude Maps LDOP [Brox et al. 09 ][Steinbrucker et al. 09]Ours

36. More Results Overlaid Input

37. Conventional Coarse-to-fine Our Result

38. More Results Overlaid Input

39. Coarse-to-fine Our Result

40. Conclusion Extended initialization (Framework) Selective data term (Model) Efficient numerical scheme (Solver) Limitations – Featureless motion details – Large occlusions

41. Thank you!

42. More Results Overlaid Input

43. Coarse-to-fine Our Results