1. Virtual Control of Optical Axis of the 3DTV Camera for Reducing Visual Fatigue in Stereoscopic 3DTV Presenter: Yi Shi & Saul Rodriguez March 26, 2008

2. Outline ► Motivation ► Stereo Matching ► Virtual View Images Synthesis ► Experimental Results ► Discussion

3. Motivation ► Trade-off between visual comfort and 3D impact with respect to the baseline-stretch of a 3DTV camera. ► Optimal baseline-stretch is around 65 mm but even the diameter of one commercial TV camera could be greater than 65 mm. Figure 1. Baseline-stretch: Human Vs. 3DTV Camera

4. Motivation (Cont.) ► Stereoscopic images taken by a 3DTV camera with wide baseline-stretch give uncomfortable feeling and cause undesirable effects such as Puppet Theater Effect: Human beings reproduced in reduced size appear as if they were animated puppets. Figure 2. Puppet Theater Effect (Chinese Pi Ying Xi)

5. Motivation (Cont.) ► Approach: Virtually ► Approach: Virtually control the positions of optical axis of a 3DTV camera by synthesizing the virtual views between two real cameras. Figure 3. Virtual control of the optical axis of a 3DTV camera

6. Stereo Matching ► Stereo matching is a process to find corresponding points between reference and target image. ► Disparity is defined as the position difference. ► Cameras are assumed to be parallel, and if not, can be made parallel by certain rectification algorithm. Figure 4. Disparity diagram

7. Stereo Matching (cont.) ► Block matching  For each epipolar line in two images, fixing one block associated to a certain pixel in the 1st image and linearly scan the blocks in the 2nd image.  For each pair of blocks, the mean squared error is calculated for each of the three RGB components and a block in the 2nd image with the minimum value is extracted as a matched block.  The disparity of the corresponding pixel is then calculated.

8. Stereo Matching (cont.) ► Drawback of conventional stereo matching: Disparity estimation around occlusion boundaries tends to be erroneous. ► Need to exclude occlusion areas adaptively in the process of stereo matching. ► Use three kinds of shifted matching window: left- shifted, centered, and right-shifted. ► Disparity calculation: First calculate the sum of the square difference for three types of shifted matching window along the epipolar line. Then regard the displacement that gives the smallest sum of the square difference as the disparity of the pixel. Figure 5. 3 kinds of matching windows

9. Stereo Matching (cont.) ► Edge adaptive shifted window technique  Observation: A region without vertical edges hardly includes the occlusion area in the usual stereo setup.  Thus applying only the centered window if there is no edge in a window is more preferable.  Edge detection can be done by Sobel operator which requires negligible computation.  Computational complexity of non-edge areas can be reduced up to about 1/S of those using non-adaptive shifted windows, where S is the number of shifted matching windows.

10. Stereo Matching (cont.) ► Coarse-to-fine hierarchical matching scheme is used which attempts to maximize the qualities of stereo matching results while maintaining the computational complexity to an acceptable level. Figure 6. Matching Parymid

11. Stereo Matching (cont.) ► Occlusion area detection  Two disparity maps are obtained for both left and right views.  Then compare the disparity of a pixel in one image with that of the corresponding pixel in the other image over its entirety.  If the two disparities differ more than one pixel, the pixel is considered as an occluded one, otherwise, not.

12. Virtual View Images Synthesis ► First determine the target position of virtual view image ► Apply forward mapping ► Guess disparities for uncovered area, which shows up due to difference of the disparities. ► Color uncovered area using the guessed disparities. ► If target point does not belong to a detected occlusion area originally, interpolate the color shown in (d) Figure 7. Color synthesis

13. Experimental Results Figure 8. Results on a city view (Univ. of Tsukuba)

14. Experimental Results (cont.) Figure 9. Detected occlusion area

15. Experimental Results (cont.) ► Quantified error rate measurement  Percentage of badly matched pixels is defined as:

16. Experimental Results (cont.) Figure10. Errors of disparity map compared with the ground truth map

17. Experimental Results (cont.) Table 1. Percentage of badly matched pixels ► Percentage of badly matched pixels

18. Experimental Results (cont.) Figure 11. Simulation results for the “ Dog and Flower ” indoor scene

19. Experimental Results (cont.) Figure 12. Simulation results for the outdoor scene.

20. Experimental Results (cont.) Chart 1. The average scores of subjective evaluation for the indoor stereoscopic scene

21. Experimental Results (cont.) Chart 2. The average scores of our subjective evaluation for the outdoor HD stereoscopic scene

22. Discussion ► ► The virtual control of the optical axis is a first step toward a versatile framework of post-processing for providing comfortable stereoscopic contents. ► ► It would be interesting as a future work to analyze various mutually affecting factors on visual comfortableness and 3D impact in stereoscopic images, such as focal length, convergence angle, resolution, and the distance to main objects ► ► Use a more rigorous subjective assessment with a sufficient number of subjects evaluating a large set of stereoscopic images.

23. References ► [1] J. Park, G. M. Um, C. Ahn, and C. Ahn, Virtual control of optical axis of the 3DTV camera for reducing visual fatigue in stereoscopic 3DTV, ETRI Journal, Volume 26, Number 6, December 2004. ► [2] M. Accame, F. G.B. De Natale, and D. D. Giusto, Hierarchical block matching for disparity estimation in stereo sequences, International Conference on Image Processing, Volume 2, page(s):374-377, October 1995. ► [3] J. Park and S. Inoue, Arbitrary View Generation Using Multiple Cameras, Proc. IEEE ICIP'97, volume 1, page(s): 149-153. Santa Barbara, USA, Oct. 1997.

24. Thank you!