1. The University of Ontario University of Bonn July 2008 Optimization of surface functionals using graph cut algorithms Yuri Boykov presenting joint work with V.Kolmogorov, O.Veksler, D.Cremers, V.Lempitsky, O.Juan, A.Delong

2. The University of Ontario Optimization of surface functionals using graph cut algorithms n optimization for image segmentation (overview) energy models in vision ( weak membrane, MRF, Mumford-Shah, etc.) energies for contours and surfaces n surfaces and binary labelling of grids geometric surface functionals and submodular binary energies –optimization via graph cut algorithms –metrication errors global vs. local optimization computational issues n applications, extensions

3. The University of Ontario Example of image labeling: piece-wise smooth image restoration I p L How to compute L from I ? observed noisy image I image labeling L (restored intensities)

4. The University of Ontario Piece-wise smooth labeling (image restoration) n discrete MRF approach weak membrane model (Geman&Geman’84, Blake&Zisserman’83,87) line process, Geman&Geman’84 discontinuity preserving potentials Blake&Zisserman’83,87

5. The University of Ontario Piece-wise smooth labeling (image restoration) n continuous approach Mumford-Shah model (Mumford&Shah 85,89)

6. The University of Ontario Piece-wise constant labeling (image restoration) I p L observed noisy image I image labeling L (restored intensities)

7. The University of Ontario Piece-wise constant labeling (image restoration) n Potts model BVZ ‘98 Greig et al.’89 for 2 labels n Mumford-Shah Chan-Vese ’02 for 2 labels Continuous: Discrete:

8. The University of Ontario Piece-wise constant labeling (frontal-parallel stereo) a pair of “stereo” images (left and right eyes views) depth map (label = depth layer)

9. The University of Ontario Piece-wise constant labeling (frontal-parallel stereo) n Potts model BVZ ‘98 n Mumford-Shah Continuous: Discrete: Data penalty function. In stereo it describes photoconsistency of pixel p when it is assigned to each specific depth layer (label)

10. The University of Ontario Binary labeling (binary image restoration) original binary image I optimal binary labeling L Greig Porteous Seheult ’89 Globally optimal solution is possible using combinatorial graph cut algorithms pseudo-boolean optimization Hammer’65, Picard&Ratlif’75

11. The University of Ontario Binary labeling (object extraction)

12. The University of Ontario Binary labeling (object extraction) C Boykov&Jolly’01

13. The University of Ontario Binary labeling (object extraction) n-links s t a cut Where would penalties come from? Example 1: hard constraints p q

14. The University of Ontario Graph cuts like “region growing”

15. The University of Ontario Graph cuts like “region growing”

16. The University of Ontario Graph cuts like “region growing”

17. The University of Ontario Graph cuts like “region growing”

18. The University of Ontario Graph cuts

19. The University of Ontario Graph cuts 2

20. The University of Ontario Graph cuts 2 Any paths would work, but shorter paths give faster algorithms (in theory and practice)

21. The University of Ontario Graph cuts 3

22. The University of Ontario Graph cuts 3 Finds the strongest boundary (least number of holes)

23. The University of Ontario Binary labeling (object extraction) Globally optimal cut can be computed in polynomial time

24. The University of Ontario push-relabel vs. augmenting paths alternatively: move flow excesses locally - opportunistic strategy assuming they all can reach the other terminals

25. The University of Ontario push-relabel vs. augmenting paths motivation: - path sharing - parallelization opportunities (e.g. GPU cuts, region push-relabel, Delong&Boykov08)

26. The University of Ontario Binary labeling (object extraction) s t Example 2: known color distributions for object and background

27. The University of Ontario Binary labeling (object extraction) s t a cut Example 2: known color distributions for object and background

28. The University of Ontario Binary labeling (object extraction) Blake et al.’04, Rother et al.’04 Example 3: iteratively re-estimate color models e.g. using mixture of Gaussians

29. The University of Ontario Binary labeling (object extraction) n Potts model BJ’01, BK’03-05 Continuous: Discrete: ? n Geodesic Active Contours Caselles et al. 93-95, Tenenbaum et al. 95

30. The University of Ontario Geometric properties of contour C and energy of binary labeling L(p) n Properties of the interior n Properties of the boundary ?

31. The University of Ontario Integral geometry C a set of all lines L a subset of lines L intersecting contour C Euclidean length of C : the number of times line L intersects C Cauchy-Crofton formula probability that a “randomly drown” line intersects C

32. The University of Ontario Graph cuts and integral geometry Boykov&Kolmogorov’03 C Euclidean length graph cut cost for edge weights: the number of edges of family k intersecting C Edges of any regular neighborhood system generate families of lines {,,, } Graph nodes are imbedded in R2 in a grid-like fashion

33. The University of Ontario Metrication errors “standard” 4-neighborhoods (Manhattan metric) larger-neighborhoods8-neighborhoods Euclidean metric Riemannian metric

34. The University of Ontario Removing metrication artifacts 4-neighborhood 8-neighborhood

35. The University of Ontario What geometric functionals can be globally optimized via graph cuts? Geometric length any convex, symmetric metric (e.g. Riemannian) Flux any vector field v Regional bias any scalar function f (“edge alignment”) Tight characterization for geometric functionals of contour C that can be globally optimized by graph cut algorithms (Kolmogorov&Boykov’05) disclaimer: for pairwise interactions only submodularity of energy implies

36. The University of Ontario Globally optimal surface in 3D Volumetric segmentation: metric g is based on image gradient

37. The University of Ontario Globally optimal surface in 3D Vogiatzis, Torr, Cippola’05 Multiview reconstruction: metric g is based on photoconsistency

38. The University of Ontario Globally optimal surface in 3D Fitting a surface into a cloud of oriented points ( Lempitsky&Boykov, 2007)

39. The University of Ontario Globally optimal surface in 3D Fitting a surface into a cloud of oriented points ( Lempitsky&Boykov, 2007) From 10 views No initialization is needed

40. The University of Ontario Global vs. local optimization regional potentials Fitting a surface into a cloud of oriented points ( Lempitsky&Boykov, 2007) initial solution local minima global minima

41. The University of Ontario Computing min s/t cuts n Augmenting paths [Ford & Fulkerson, 1962] n Push-relabel [Goldberg-Tarjan, 1986] n Pseudoflows [Hochbaum, 1997] n Poor control of locality n Computing global minima requires whole graph to fit into memory (RAM) problems of standard algorithms

42. The University of Ontario Computing min s/t cuts n Better control of locality?

43. The University of Ontario Computing min s/t cuts region size =1 (local relabeling) region size=n (global relabeling) region size =16region size=49 region push-relabel [Delong&Boykov’08]

44. The University of Ontario Computing min s/t cuts Theoretical worst case running time r=1 region size r (log scale) r=n 1 CPU accounting for parallelization opportunities 4 CPU region push-relabel [Delong&Boykov’08]

45. The University of Ontario Computing min s/t cuts region push-relabel [Delong&Boykov’08]

46. The University of Ontario Computing min s/t cuts region push-relabel [Delong&Boykov’08] Scales well to large graphs that do not fit into available memory

47. The University of Ontario GENERALIZATIONS OF S/T GRAPH CUTS

48. The University of Ontario using parametric max-flow methods n optimization of ratio functionals in N-D using Dinkelbach’s method (Kolmogorov, Boykov, Rother 2007) in 2D can also use DP (Cox et al’96, Jermyn&Ishikawa’01)

49. The University of Ontario Related to isoperimetric problem => bias to circles using parametric max-flow methods

50. The University of Ontario other labels a Extending to multiple labels a-expansion [Boykov,Veksler,Zabih’98] Basic idea: break multi-way cut computation into a sequence of binary s-t cuts

51. The University of Ontario Multi-way graph cuts Multi-object Extraction

52. The University of Ontario Multi-way graph cuts Stereo/Motion with slanted surfaces (Birchfield &Tomasi 1999) Labels = parameterized surfaces EM based: E step = compute surface boundaries M step = re-estimate surface parameters

53. The University of Ontario Multi-way graph cuts stereo vision original pair of “stereo” images depth map ground truth BVZ 1998 KZ 2002

54. The University of Ontario Multi-way graph cuts Graph-cut textures (Kwatra, Schodl, Essa, Bobick 2003) similar to “image-quilting” (Efros & Freeman, 2001) A B C D E F G H I J A B G D C F H I J E

55. The University of Ontario normalized correlation, start for annealing, 24.7% err simulated annealing, 19 hours, 20.3% err a-expansions (BVZ 89,01) 90 seconds, 5.8% err a-expansions vs. simulated annealing