1. S I E M E N S C O R P O R A T E R E S E A R C H 1 1 Computing Exact Discrete Minimal Surfaces: Extending and Solving the Shortest Path Problem in 3D with Application to Segmentation Leo Grady Department of Imaging and Visualization Siemens Corporate Research

2. S I E M E N S C O R P O R A T E R E S E A R C H 2 2 Outline Introduction Extending the shortest path problem to 3D Method for computation Results Conclusion

3. S I E M E N S C O R P O R A T E R E S E A R C H 3 3 A shortest path is easy to find in 3D, but it is no longer a boundary Introduction – Shortest path in 3D

4. S I E M E N S C O R P O R A T E R E S E A R C H 4 4 Introduction – Minimal boundary segmentation Minimal segmentation specified two ways Cutting edges/specifying normals Graph Cuts Max-flow/Min-cut Specifying boundary Intelligent scissors/Live wire Dijkstra’s algorithm 4  4 image4  4 weighted graph abstraction

5. S I E M E N S C O R P O R A T E R E S E A R C H 5 5 Specifying boundaryShortest path on dual graph = Sometimes called “cracks” or “edgels” Introduction – Minimal boundary segmentation

6. S I E M E N S C O R P O R A T E R E S E A R C H 6 6 4  4  4 volume Primal - Cuts Max-flow/Min-cut Dual - Surfaces ??? Introduction – Minimal boundary segmentation

7. S I E M E N S C O R P O R A T E R E S E A R C H 7 7 Q: Why bother if you have max-flow/min-cut? A: Different inputs, different outputs Max-flow/min-cut Input: Source (inside) and terminal (outside) Output: Closed contour/surface Shortest path/min surface Input: A piece of the boundary Output: Open contour/surface Different algorithmic complexity Both algorithms persist in 2D, but max-flow/min-cut only option in 3D Introduction – Minimal boundary segmentation

8. S I E M E N S C O R P O R A T E R E S E A R C H 8 8 Outline Introduction Extending the shortest path problem to 3D Method for computation Results Conclusion

9. S I E M E N S C O R P O R A T E R E S E A R C H 9 9 Extending the Shortest Path Problem Input: Two points (0D boundary) Output: Minimum 1D path having that boundary How to extend problem to 3D? Path is shortest relative to weighting (metric) 0-dimensional boundaryminimal 1-dimensional object

10. S I E M E N S C O R P O R A T E R E S E A R C H 10 Extending the Shortest Path Problem Input: Closed contour (1D boundary) Output: Minimum 2D surface having that boundary minimal 2-dimensional object 1-dimensional boundary Surface is minimal relative to weighting (metric)

11. S I E M E N S C O R P O R A T E R E S E A R C H 11 Extending the Shortest Path Problem 2D intelligent scissors/live wire ubiquitous segmentation method

12. S I E M E N S C O R P O R A T E R E S E A R C H 12 Red – Initial (boundary) contours Yellow – Computed minimal surface contours Extending the Shortest Path Problem

13. S I E M E N S C O R P O R A T E R E S E A R C H 13 Outline Introduction Extending the shortest path problem to 3D Method for computation Results Conclusion

14. S I E M E N S C O R P O R A T E R E S E A R C H 14 Method How to compute? Use boundary operator to enforce constraint : 1D 0D: 2D 1D

15. S I E M E N S C O R P O R A T E R E S E A R C H 15 Method Minimize path/surface 2D - Binary vector indicating presence/absence of edge in path - Weights of each edge in the graph - Vector indicating nodes in boundary - Node-edge incidence matrix 3D - Weights of each face in the complex - Binary vector indicating presence/absence of face in surface Use boundary operator as constraint Subject to: - Vector indicating edges in boundary Since boundary operator is linear, representable by a matrix Subject to: - Edge-face incidence matrix

16. S I E M E N S C O R P O R A T E R E S E A R C H 16 Continuum interpretation - I Use generalized Stokes Theorem: : 1D 0D: 2D 1D Fundamental Theorem of calculus:“Standard” Stokes Theorem Careful! Instead of bivectors, formulated in “primal” space Method - Digression

17. S I E M E N S C O R P O R A T E R E S E A R C H 17 Continuum interpretation - II Use boundary as constraint Method - Digression Subject to: RHS consists of a unit closed contour 3D - Vector field taking nonzeros on the normals of the surface - Ambient vector field (e.g., derived from image gradients) 2D - Vector field taking nonzeros along minimal path RHS consists of two delta functions at endpoints - Ambient vector field (e.g., derived from image gradients)

18. S I E M E N S C O R P O R A T E R E S E A R C H 18 Method Minimal surface problem Subject to: Integer programming problem – Bad! However: Sometimes we can apply linear programming to an integer programming problem and guarantee an integer solution.

19. S I E M E N S C O R P O R A T E R E S E A R C H 19 Method Minimal surface problem Subject to: If constraints are feasible, then For some integer. Therefore, For the matrix spanning the nullspace of So, we can rewrite the problem in terms of Joint work with Vladimir Kolmogorov

20. S I E M E N S C O R P O R A T E R E S E A R C H 20 Method Original minimal surface problem Subject to: New minimal surface problem Subject to: Guaranteed to give an integer solution for an integer iff is totally unimodular. A matrix is totally unimodular if every square submatrix has a determinant equal to one of the set

21. S I E M E N S C O R P O R A T E R E S E A R C H 21 Method New minimal surface problem Subject to: Big question: What is ? The volume-face boundary operator : 3D 2D Followed by the : 2D 1D, gives zero May also be stated as: “The boundary of the boundary is zero”

22. S I E M E N S C O R P O R A T E R E S E A R C H 22 Method Volume-face incidence in dual lattice is node-edge incidence in primal lattice DualPrimal

23. S I E M E N S C O R P O R A T E R E S E A R C H 23 Method New minimal surface problem Subject to: Since all node-edge incidence matrices are totally unimodular is guaranteed to be integer, and therefore is guaranteed to be integer.

24. S I E M E N S C O R P O R A T E R E S E A R C H 24 Method Solvable using generic linear programming code! Conclusion: Minimal surface problem Subject to: (and so is the other formulation) Subject to:

25. S I E M E N S C O R P O R A T E R E S E A R C H 25 Outline Introduction Extending the shortest path problem to 3D Method for computation Results Conclusion

26. S I E M E N S C O R P O R A T E R E S E A R C H 26 Results - Correctness

27. S I E M E N S C O R P O R A T E R E S E A R C H 27 Red – Initial (boundary) contours Yellow – Computed minimal surface contours Results – 3D Segmentation

28. S I E M E N S C O R P O R A T E R E S E A R C H 28 Outline Introduction Extending the shortest path problem to 3D Method for computation Results Conclusion

29. S I E M E N S C O R P O R A T E R E S E A R C H 29 Conclusion 1. Natural extension of shortest path given two points is minimal surface given a closed contour 2. Minimal surface problem solvable with generic linear programming code 3. There are, in fact, two integral linear programming problems that could be solved to achieve the solution

30. S I E M E N S C O R P O R A T E R E S E A R C H 30 More Information My webpage: http://cns.bu.edu/~lgrady MATLAB toolbox for graph theoretic image processing at: http://eslab.bu.edu/software/graphanalysis/ Writings and code Combinatorial minimal surface MATLAB code: http://cns.bu.edu/~lgrady/minimal_surface_matlab_code.zip Acknowledgements Marie-Pierre Jolly – Posing the problem Gareth Funka-Lea – Support and enthusiasm for the work Yuri Boykov – Enthusiasm and encouragement of the topic Chenyang Xu – Extensive comments on the paper Vladimir Kolmogorov – Technical analysis of LP problem