1. Discontinuity Preserving Stereo with Small Baseline Multi-Flash Illumination Rogerio Feris 1, Ramesh Raskar 2, Longbin Chen 1, Karhan Tan 3 and Matthew Turk 1 1 University of California, Santa Barbara 2 Mitsubishi Electric Research Labs 3 Epson Palo Alto Lab

2. Introduction Correspondence Problem Stereo Near Depth Discontinuities: - Occlusion Problem - Perspective Distortions - Violation of Smoothness Constraints Passive Versus Active Methods

3. Introduction Our Approach: Small Baseline Multi-Flash Illumination - Simple, Inexpensive - Compact, Self-Contained - Discontinuity Preserving

4. Depth Edges with Multi-Flash Raskar, Tan, Feris, Yu, Turk – ACM SIGGRAPH 2004

9. Bottom Flash Top Flash Left Flash Right Flash Ratio images and directions of epipolar traversal Shadow-Free Depth Edges Shadow-FreeDepth Edges

10. Qualitative Depth Map

11. Qualitative Depth Sign of Depth Edge - Indicates which side is the foreground and which side is the background Shadow Width - Encodes object relative distances

12. Sign of Depth Edge + - + - (+) Foreground (-) Background Original Ratio Left Ratio Right Signed Edges

13. Shadow Width Bottom Flash Image Ratio Image Plot Along Scanline

14. Shadow Width Bottom Flash Image Ratio Image Shadow Width Estimation: Meanshift Segmentation algorithm applied on the ratio image

15. Imaging Geometry Object Flash Shadow Camera B z1z1 z2z2 f Shadow Width d

16. Qualitative Depth Working on this Equation … Log Depth Difference Shadow Width Gradient-Domain Problem!

17. Qualitative Depth 1) Compute Sharp Depth Gradient G = (G h,G v ) Log Depth Difference Sign of depth edge 2) Compute Q by integrating G (Poisson Equation) 3) Qualitative depth map Q = exp(Q)

18. Qualitative Depth Useful Prior Information for Stereo !

19. Occlusion Map

20. Partial Occlusion Problem Object Camera Occlusion AB (Seen by A but not by B)

21. Occlusion Bounded by Shadows Object CameraAB Flash Occlusion (Seen by A but not by B)

22. Occlusion Bounded by Shadows Object CameraAB Flash Lower Bound Shadow

23. Occlusion Bounded by Shadows Object CameraAB Flash Upper Bound Shadow

24. Occlusion Bounded by Shadows Object Camera Occlusion AB Average of Upper/Lower Shadow widths Flash

25. Occlusion Bounded by Shadows Occlusion Map Left ViewRight View

26. Discontinuity Preserving Stereo Matching

27. Local Stereo Problem: Shape and size of correlation window - Small Window Ambiguities / Noise - Large Window Problems at Depth Discontinuities Depth Edge Preserving Local Stereo

28. Local Stereo Smooth Disparity Delimited by depth edges + Occlusions Correlation Window Problem: Shape and size of correlation window - Small Window Ambiguities / Noise - Large Window Problems at Depth Discontinuities Depth Edge Preserving Local Stereo

29. Local Stereo Left View Depth Edges + OcclusionGround Truth Challenging Scene: - Ambiguous patterns, textureless regions, geometrically complex object, thin structures

30. Local Stereo Conventional 9x9 Conventional 31x31 Our Approach 31x31 --- Conventional Stereo Our Approach

31. Global Stereo Global Optimization – Markov Random Field (MAP-MRF) X = {x s } Disparity of each pixel (Hidden) Y = {y s } Matching cost at each disparity (Observed) X3X3 X1X1 X2X2 X7X7 X4X4 X6X6 X5X5 X8X8 y1y1 y2y2

32. Global Stereo Global Optimization – Markov Random Field (MAP-MRF) X = {x s } Disparity of each pixel (Hidden) Y = {y s } Matching cost at each disparity (Observed) Data Term Smoothness Term Inference by Belief Propagation [Jian Sun et al, 2003]

33. Global Stereo Qualitative Depth Map as Evidence - Used to set the smoothness term - Information propagation is stopped at depth edges - Encourage disparities for neighboring pixels according to depth difference in qualitative map Occlusion Penalty

34. Global Stereo Conventional Belief Propagation Our Approach RMS: 0.9589 RMS: 0.4590

35. Conclusions Contributions - Stereo with small baseline illumination - Useful Feature Maps (Qualitative Depth + Occlusion Map) - Enhanced Local and Global Stereo Algorithms Pros / Cons - Robust, Simple, Inexpensive and Compact - Limited to handle outdoor scenes and motion Website (datasets, source code) - http://www.cs.ucsb.edu/~rferis/multi-flash-stereo

36. Thank you ! Multi-Flash Stereo Webpage http://www.cs.ucsb.edu/~rferis/multi-flash-stereo Four Eyes Lab, UCSB http://ilab.cs.ucsb.edu

38. Occlusion Bounded by Shadows Occlusion Detection by averaging length of shadows Images taken with light sources surrounding the other camera