1. The University of Ontario How to fit a surface to a point cloud? or optimization of surface functionals in computer vision Yuri Boykov TRICS seminar Computer Science Department

2. The University of Ontario Optimization of surface functionals in computer vision n Computer vs. human vision n Model fitting in computer vision templates, pictorial structures, trees, deformable models, contours/snakes, meshes, surfaces, complexes, graphs, weak-membrane model, Mumford-Shah, Potts model,…… n Optimization in computer vision dynamic programming, gradient descent, PDEs, shortest paths, min. spanning trees, linear and quadratic programming, primal-dual schema, network flow algorithms, QPBO,... n Applications segmentation, stereo, multi-view reconstruction, optical flows surface fitting

3. The University of Ontario Contours

4. The University of Ontario +Shading M.C. Escher Drawing hands 3D shape understanding

5. The University of Ontario +Color Da Vinci Madonna Litta Recognition

6. The University of Ontario +Texture Magritte Souvenir de Voyage recognizing material

7. The University of Ontario +Texture The New Yorker Album of Drawings, The Viking Press, NY, 1975 recognizing 3D perspective

8. The University of Ontario What do humans get by ‘looking’? J. Vermeer The Guitar Player n Contours n Shading n Color n Texture n … basic image cues:

9. The University of Ontario What do humans get by ‘looking’? n Contours n Shading n Color n Texture n … basic image cues: n Segmentation n Motion n 3D shape perception n 3D scene geometry n Detection/Recognition n … higher-level perception:

10. The University of Ontario What do computers get by ‘looking’? x y 3D plot of image intensity I(x,y) x y I(x,y) x y

11. The University of Ontario What do computers get by ‘looking’? P. Picasso The Guitar Player n Intensity discontinuities (contours) n Intensity gradients (shading) n Multi-valued intensities (color) n Filtering (e.g. texture) n … basic image cues: higher-level grouping?

12. The University of Ontario model Bayesian approach Prior + Data Low-level cues (local info) high-level knowledge (global picture) Fit some prior model into data

13. The University of Ontario Rigid Template Matching In matching we estimate “position” of a rigid template in the image “Position” includes global location parameters of a rigid template: - translation, rotation, scale,… Face template image translation, rotation, scaling

14. The University of Ontario Non-rigid (parametric) matching 1. Pick one image (red) 2. Warp the other images to match it (homographic transform) 3. Blend panorama mosaicing

15. The University of Ontario e.g.… using homographies

16. The University of Ontario e.g…. using flexible templates In flexible template matching we estimate “position” of each rigid component of a template For tree-structured models, efficient global optimization is possible via DP (Felzenswalb&Huttenlocher 2002)

17. The University of Ontario tracking parameters => activity recognition Bottom-up tracker

18. The University of Ontario 5-18 deformable contours (“snakes”) n 2D curve which matches to image data n Initialized near target, iteratively refined n Can restore missing data initial intermediate final Optimization gets harder when a loop is introduced. DP does not apply. One solution: gradient descent Kass, Witkin, Terzopoulos 1987

19. The University of Ontario 6-19 Cremers, Tischhäuser, Weickert, Schnörr, “Diffusion Snakes”, IJCV '02 local minima, fixed contour topology

20. The University of Ontario 6-20 A contour may be approximated from u(x,y) with near sub- pixel accuracy C -0.8 0.2 0.5 0.7 0.3 0.6 -0.2 -1.7 -0.6 -0.8 -0.4 -0.5 Level set function u(x,y) is normally discretized/stored over image pixels Values of u(p) can be interpreted as distances or heights of image pixels Implicit representation of contours Osher&Sethian 1989

21. The University of Ontario 6-21 [Visualization is courtesy of O. Juan] Simple evolution Morphological Operation: Erosion

22. The University of Ontario 6-22 Visualization is courtesy of O. Juan Example of gradient descent evolution Gradient descent w.r.t. Euclidean length

23. The University of Ontario 6-23 Example of gradient descent evolution Laplacian Osher&Sethian 1989 Gradient descent w.r.t. Euclidean length

24. The University of Ontario 6-24 [example from Goldenberg, Kimmel, Rivlin, Rudzsky, IEEE TIP ’01] Geodesic Active Contours via Level-sets

25. The University of Ontario 6-25 Other geometric energy functionals besides length [courtesy of Ron Kimmel] weighted length Functional E( C ) gradient descent evolution weighted area alignment (flux)  Geometric measures commonly used in segmentation

26. The University of Ontario in 3D… deformable meshes, level-sets, … Estimation of position for mesh points Many loops. optimization - gradient descent GOALS: global optima (?) “right” functional (?) Typical problems: - local minima (clutter, outliers) -over-smoothing

27. The University of Ontario Global Optimization and Surface Functionals

28. The University of Ontario More generally... Estimate labels for graph nodes I p L along one scan line in the image observed noisy image I image labeling L (restored intensities) NOTE: similar to robust regression model estimation

29. The University of Ontario (simple example) Piece-wise smooth restoration n Markov Random Fields (MRF) approach weak membrane model (Geman&Geman’84, Blake&Zisserman’83,87) discontinuity preserving prior optimizing E(L) is NP hard! (Continuous analogue: Mamford-Shah functional, 1989)

30. The University of Ontario I p L observed noisy image I image labeling L (restored intensities) (simple example) Piece-wise constant restoration along one scan line in the image

31. The University of Ontario (simple example) Piece-wise constant restoration Potts model Boykov Veksler Zabih ’01 Greig et al.’89 (for 2 labels) global optimization is still NP hard, but there are fast provably good combinatorial approximation algorithms, linear and quadratic programming, QPBO, primal-dual schema “perceptual grouping”

32. The University of Ontario Perceptual grouping from stereo (Birchfield &Tomasi’99) constant label = plane

33. The University of Ontario Binary labeling (binary image restoration) original binary image I optimal binary labeling L Greig Porteous Seheult ’89 Globally optimal solution is possible using combinatorial graph cut algorithms pseudo-boolean optimization Hammer’65, Picard&Ratlif’75

34. The University of Ontario Binary labeling (object extraction) object segmentation left ventricle of heart

35. The University of Ontario Binary labeling (object extraction) C Globally optimal solution is possible using graph cut algorithms pseudo-boolean optimization (Hammer’65, Picard&Ratlif’75) surface extraction Boykov&Jolly’01 left ventricle of heart

36. The University of Ontario Implicit surface representation via graph-cuts Any contour (or surface in 3D) satisfying labeling of exterior/interior points (pixel centers) is acceptable if some explicit surface has to be output. 0 1 1 1 1 1 0 0 0 0 0 0

37. The University of Ontario Geometric length any convex, symmetric metric (e.g. Riemannian) Flux any vector field v Regional bias any scalar function f “edge alignment” Tight characterization for geometric functionals of contour C that can be globally optimized by graph cut algorithms (Kolmogorov&Boykov’05) disclaimer: for pairwise interactions only Global optimization of geometric surface functionals

38. The University of Ontario Globally optimal surfaces in 3D Volumetric segmentation (BJ01,BK’03,KB’05)

39. The University of Ontario Binary labeling (object extraction) Blake et al.’04, Rother et al.’04 iteratively re-estimate color models e.g. using mixture of Gaussians

40. The University of Ontario Segmentation for Image Blending

41. The University of Ontario Segmentation for Image Blending

42. The University of Ontario Optimal surfaces in 3D 3D reconstruction Vogiatzis, Torr, Cippola’05 Local cues: voxel’s photoconsistency Prior: smoothness, projective geometry constraints

43. The University of Ontario Globally optimal surfaces in 3D Lempitsky&Boykov, 2006 from a cheap digital camera

44. The University of Ontario 3D model multi-view reconstruction set up Furukawa&Ponce 2006 Globally optimal surfaces in 3D (texture mapped)

45. The University of Ontario Surface fitting to point cloud a cloud of 3D points (e.g. from a laser scanner) 3D model : surface fitting :

46. The University of Ontario Surface fitting to point cloud

47. The University of Ontario Surface fitting functional Fitting a surface into a cloud of oriented points ( Lempitsky&Boykov, 2007) data fit prior

48. The University of Ontario Optimal surfaces in 3D Fitting a surface into a cloud of oriented points ( Lempitsky&Boykov, 2007) From 10 views No initialization is needed

49. The University of Ontario Global vs. local optimization regional potentials Fitting a surface into a cloud of oriented points ( Lempitsky&Boykov, 2007) initial solution local minima global minima

50. The University of Ontario Fitting to sparse data

51. The University of Ontario Fitting to sparse data

52. The University of Ontario Fitting to sparse data

53. The University of Ontario Summary Global optimization -Your solution is as good as your functional -No need to worry about initial guess or convergence issues -Polynomial algorithms, but many practical issues (efficient data structures, memory limitations, parallelization, dynamic applications,…) - Many useful functionals are NP hard (lots of approximation methods are developed) - New approaches allowing global optimization are introduced (including new version of level-sets)