1. Presented by: Ali Agha March 02, 2009

2. Outline Sterevision overview Motivation & Contribution Structured light & method overview Related work Disparity computation Results Conclusion Future work

3. STEREO VISION When 3D information of a scene is needed

4. Depth from Disparity (x R -x L )/f = b/z disparity=d RL =(x R -x L )

5. Motivation of the presented paper “A taxonomy and evaluation of dense two-frame stereo correspondence algorithms”. Intl. J. Comp. Vis., 2002. http://www.middlebury.edu/stereo/. Tsukuba Venus

6. need for more challenging scenes accurate ground truth information?? Motivation of the presented paper

7. Contributions of this work A method for acquiring high-complexity stereo image pairs with pixel-accurate correspondence information. Does not require the calibration of the light sources High resolution in comparison with range sensors

8. Process Pipeline This method uses structured light and consists of the following stages: Acquire all desired views under all illuminations. Rectify the images Decode the light patterns at each pixel to compute correspondences. Compute the view and illumination disparity and combine them

9. Structured light Structured-light techniques rely on projecting one or more special light patterns onto a scene, usually in order to directly acquire a range map of the scene http://en.wikipedia.org/wiki/File:1-stripesx7.svg

10. Structured light A pair of cameras and one or more light projectors are used http://en.wikipedia.org/wiki/File:1-stripesx7.svg

11. Related Work in Decoding light patterns Overview of Techniques Posdame et al Carrihill et al Boyer Kak LeMoigne et al Morita et al Vuylsteke et al Tajima et al Wust Capson Griffin et al Maruyama Abe J. Batlle, E. Mouaddib, and J. Salvi. Recent progress in coded structured light as a technique to solve the correspondence problem: a survey. Pat. Recog., 31(7):963–982, 1998.

12. Related work -CODED STRUCTURED LIGHT TECHNIQUES Posdamer-Daltschuler 1981-82-87

13. Related work -CODED STRUCTURED LIGHT TECHNIQUES Inokuchi, Sato and Matsuda 1984 8 bits temporally Gray-coded pattern projection 8 bits temporally binary-coded pattern projection

14. Gray Code Using such binary images requires log 2 (n) patterns to distinguish among n locations.

15. Decoding the light patterns Using average of all-white and all-black In practice, the only reliable way is to project both the code pattern and its inverse. In surfaces with widely varying reflection properties, use two different exposure times (0.5 and 0.1 sec.). If this largest difference is still below a threshold, the pixel is labeled “unknown”

16. Disparity computation View disparities Illumination disparities Definition: views – the images taken by the cameras Illuminations – the structured light patterns projected onto the scene.

17. View disparities Assuming rectified views leads simple 1D search Practical issues: Occlusion Unknown code values (due to shadows or reflections). A perfect matching code value may not exist (interpolation errors) Several perfect matching code values may exist (limited resolution)

18. View disparities The first problem (partial occlusion) is unavoidable The number of unknown code values can be reduced by using more than one illumination source As a final consistency check, we establish disparities d LR and d RL independently and cross-check for consistency.

19. View disparities scene under illumination view disparities

20. Illumination disparities disparity between the cameras and the illumination sources. The difference in our case is that we can register these illumination disparities with our rectified view disparities d LR without the need to explicitly calibrate the illumination sources (video projectors).

21. Illumination disparities Relationship between the left view L and illum. source 0. Each pixel whose view disparity has been established can be considered a (homogeneous) 3D scene point S=[x,y,d,1] with projective depth d = d LR (x, y). The pixel’s illumination disparity (u 0L, v 0L ) P = M 0L Sin which P = [u 0L v 0L 1]

22. Practical Issues A small number of pixels with large disparity errors can strongly affect the least-squares fit. Outlier detection by iterating the above process. Only those pixels with low residual errors are selected as input to the next iteration.

23. Illumination disparities Given the projection matrix M 0L, we can now solve equation for d LR at all pixels Note that these disparities are available for all points illuminated by source 0, even those that are not visible from the right camera.

24. Combining the disparity estimates Remaining task is to combine the 2N + 2 disparity maps. Create combined maps for each of L and R separately Whenever there is a majority of values within close range, we use the average otherwise, the pixel is labeled unknown. L and R maps are checked for consistency, for unoccluded pixels, d LR (x, y) = − d RL (x + d LR (x, y), y),

25. Combined disparity Most stereo implementations work with much smaller image sizes. So, we downsample the images and disparity maps to quarter size (460 × 384). Note that for the downsampled images, we now have disparities with quarter-pixel accuracy.

26. Unknown Disparities A remaining issue is that of holes, i.e., unknown disparity values Small holes can be filled by interpolation Large holes may remain in areas where no illumination codes were available to begin with. Two main sources: surfaces that show very low reflection areas that are shadowed under all illuminations.

27. Results Two different scenes, Cones and Teddy.

28. Experiments In experimental setup, a single digital camera (Canon G1) translating on a linear stage, and one or two light projectors illuminating the scene from different directions.

29. Results

30. Verification To verify that stereo data sets are useful for evaluating stereo matching algorithms, several of the algorithms from the Middlebury Stereo Page has been ran on our new images.

31. Conclusion a new methodology to acquire highly precise and reliable ground truth disparity measurements camera-projector disparities, which can be used as an auxiliary source of information to increase the reliability of correspondences and to fill in missing data.

32. Considerations for Future work Exploiting in navigation Field of view is limited by the range of light projector Investigate the number of projected patterns which directly affect the speed of the method In daylight or dark places Invisible lights

33. Thank you Questions??

36. Related work -CODED STRUCTURED LIGHT TECHNIQUES Posdamer-Daltschuler 1981-82-87

37. Related work -CODED STRUCTURED LIGHT TECHNIQUES Inokuchi, Sato and Matsuda 1984 8 bits temporally Gray-coded pattern projection 8 bits temporally binary-coded pattern projection

38. Related work -CODED STRUCTURED LIGHT TECHNIQUES Sato, Yamamoto and Inokuchi 1986-87 proposed to use a Liquid Crystal Device which allows an increased number of columns to be projected with a high accuracy. The system also improves the coded speed, against a slide projector, so the LCD can be electronically controlled.

39. If an object has a high textural contrast or any high reßected surface regions, then, some pattern segmentation errors can be produced. – Solution? The problem of a light projector is sometimes a result of heat irradiation onto the scene

40. Related work -CODED STRUCTURED LIGHT TECHNIQUES Hattori and Sato 1995 replace the light projector with a semiconductor laser, which gives a high power illumination with low heat irradiation. The proposed system, named Cubiscope The Cubiscope system

41. Related work – Carrihill-Hummelk Look at the notes

42. Related work – Boyer-Kak Colour

43. Related work – Le Moigne-Waxman not-coded grid patterns

44. Related work – Morita-Yakima-Sakata

45. Related work – Vuylsteke-Oosterlinck