Minimal Surfaces for Stereo Chris Buehler, Steven J. Gortler, Michael F. Cohen, Leonard McMillan MIT, Harvard Microsoft Research, MIT

Motivation Optimization based stereo over greed based –No early commitment –Enforce interactions: each pixel sees unique item –Penalize interactions: non-smoothness

Stereo by Optimization Early algorithms: dynamic programming –(Baker ‘81, Belumeur & Mumford ‘92…) –Don’t generalize beyond 2 camera, single scanline

Stereo by Optimization Recent Algorithms: iterative  expansion –(… Kolmogorov & Zabih ‘01) –very general –NP-Complete Local opt found quickly in practice Recent algorithms: MIN-CUT –(Roy & Cox ‘96, Ishikawa & Geiger ‘98) –Polynomial time global optimum –New interpretation to such methods

Contributions Stereo as a discrete minimal surface problem Algorithms: Polynomial time globally optimal surface –Using MIN-CUT (Sullivan ‘90) –Build from shortest path Applications to stereo vision –Rederive previous MIN-CUT stereo approaches –New 3-camera stereo formulation (Ayache ‘88)

Planar Graph Shortest Path Given: an embedded planar graph –faces, edges, vertices

Planar Graph Shortest Path A non negative cost on each edge 57

Planar Graph Shortest Path Two boundary points on the exterior of the complex

Planar Graph Shortest Path Find minimal curve: (collection of edges) with given boundary

Planar Graph For stereo Camera LeftCamera Right Selected Match Selected Occlusion

Algorithms Classic: Dijkstra’s –Works even for non-planar graphs Wacky: use duality –But this will generalize to higher dimension

Duality

face vertex edge cross edge - same cost 57

Duality Split exterior

Source Sink Source Duality Add source and sink

Cuts Source Sink Cuts of dual graph = partitions of dual verts Cost = sum of dual edges spanning the partition MIN-CUT can be found in polynomial time

Source Sink Cuts Claim: Primalization of MIN-CUT will be shortest path

Sink Source Sink Why this works Cuts of dual graph = partitions of dual verts

Sink Source Sink Why this works Partition of dual verts = partition of primal faces

Sink Source Sink Source Sink Why this works Partition of primal faces = primal path

Sink Source Sink Source Sink Why this works Cuts in dual correspond to paths in primal MIN-CUT in dual corresponds to shortest path in primal

Same idea works for surfaces!

Increasing the dimension Planar graph: verts, edges, faces cost on edges boundary: 2 points on exterior sol: min path Spacial compex: verts, edges, faces, cells cost on faces boundary: loop on exterior sol: min surface

Increasing the dimension Planar graph: verts, edges, faces boundary: 2 points on exterior sol: min path Spacial compex: verts, edges, faces, cells cost on faces boundary: loop on exterior sol: min surface

Increasing the dimension Planar graph: verts, edges, faces boundary: 2 points on exterior sol: min path Spacial compex: verts, edges, faces, cells cost on faces boundary: loop on exterior sol: min surface

Dual construction for min surf face vertex edge cross edge Sink Source cell vertex face cross edge MIN-CUT primalizes to min surf

Checkpoint Solve for minimal paths and surfaces –MIN-CUT on dual graph Apply these algorithms to stereo vision

Flatland Stereo Camera LeftCamera Right Geometric interpretation of Cox et al. 96 pixel

Flatland Stereo Camera LeftCamera Right Geometric interpretation of Cox et al. 96 pixel

Flatland Stereo Camera LeftCamera Right Cost: unmatched/discontinuity, β

Flatland Stereo Camera LeftCamera Right Cost: correspondence quality

Flatland Stereo Camera LeftCamera Right

Camera LeftCamera Right Match Unmatched Uniqueness & monotonicity solution is directed path Flatland Stereo

Camera LeftCamera Right Match Occlusion, discontinuity Note: unmatched pixels also function as discontinuities Flatland Stereo

Flatland to Fatland Camera LeftCamera Right

Flatland to Fatland Camera LeftCamera Right

2 cameras, 3d

One Cuboid Among Many Solve for minimal surface

Geometric interpretation IG98

Three Camera Rectification (Ayache ‘88)

Three Camera

One cuboid

Dual graph of one cuboid

One Cuboid Among Many Solve for minimal surface

More divisions of middle cell

More expressive decomposition

Complexity Vertices and edges: 20 n d –n: pixels per image –d: max disparity Time complexity O((nd) 2 log(nd)) About 1 min

Results LL image RC KZ01 MS

LL image RC KZ01 MS

Future Application of MS to n cameras –Monotonicity/oriented manifold enforces more than uniqueness –see Kolmogorov & Zabih (today 11:00am) Other applications of MS