Virtual Control of Optical Axis of the 3DTV Camera for Reducing Visual Fatigue in Stereoscopic 3DTV Presenter: Yi Shi & Saul Rodriguez March 26, 2008

Outline ► Motivation ► Stereo Matching ► Virtual View Images Synthesis ► Experimental Results ► Discussion

Motivation ► Trade-off between visual comfort and 3D impact with respect to the baseline-stretch of a 3DTV camera. ► Optimal baseline-stretch is around 65 mm but even the diameter of one commercial TV camera could be greater than 65 mm. Figure 1. Baseline-stretch: Human Vs. 3DTV Camera

Motivation (Cont.) ► Stereoscopic images taken by a 3DTV camera with wide baseline-stretch give uncomfortable feeling and cause undesirable effects such as Puppet Theater Effect: Human beings reproduced in reduced size appear as if they were animated puppets. Figure 2. Puppet Theater Effect (Chinese Pi Ying Xi)

Motivation (Cont.) ► Approach: Virtually ► Approach: Virtually control the positions of optical axis of a 3DTV camera by synthesizing the virtual views between two real cameras. Figure 3. Virtual control of the optical axis of a 3DTV camera

Stereo Matching ► Stereo matching is a process to find corresponding points between reference and target image. ► Disparity is defined as the position difference. ► Cameras are assumed to be parallel, and if not, can be made parallel by certain rectification algorithm. Figure 4. Disparity diagram

Stereo Matching (cont.) ► Block matching  For each epipolar line in two images, fixing one block associated to a certain pixel in the 1st image and linearly scan the blocks in the 2nd image.  For each pair of blocks, the mean squared error is calculated for each of the three RGB components and a block in the 2nd image with the minimum value is extracted as a matched block.  The disparity of the corresponding pixel is then calculated.

Stereo Matching (cont.) ► Drawback of conventional stereo matching: Disparity estimation around occlusion boundaries tends to be erroneous. ► Need to exclude occlusion areas adaptively in the process of stereo matching. ► Use three kinds of shifted matching window: left- shifted, centered, and right-shifted. ► Disparity calculation: First calculate the sum of the square difference for three types of shifted matching window along the epipolar line. Then regard the displacement that gives the smallest sum of the square difference as the disparity of the pixel. Figure 5. 3 kinds of matching windows

Stereo Matching (cont.) ► Edge adaptive shifted window technique  Observation: A region without vertical edges hardly includes the occlusion area in the usual stereo setup.  Thus applying only the centered window if there is no edge in a window is more preferable.  Edge detection can be done by Sobel operator which requires negligible computation.  Computational complexity of non-edge areas can be reduced up to about 1/S of those using non-adaptive shifted windows, where S is the number of shifted matching windows.

Stereo Matching (cont.) ► Coarse-to-fine hierarchical matching scheme is used which attempts to maximize the qualities of stereo matching results while maintaining the computational complexity to an acceptable level. Figure 6. Matching Parymid

Stereo Matching (cont.) ► Occlusion area detection  Two disparity maps are obtained for both left and right views.  Then compare the disparity of a pixel in one image with that of the corresponding pixel in the other image over its entirety.  If the two disparities differ more than one pixel, the pixel is considered as an occluded one, otherwise, not.

Virtual View Images Synthesis ► First determine the target position of virtual view image ► Apply forward mapping ► Guess disparities for uncovered area, which shows up due to difference of the disparities. ► Color uncovered area using the guessed disparities. ► If target point does not belong to a detected occlusion area originally, interpolate the color shown in (d) Figure 7. Color synthesis

Experimental Results Figure 8. Results on a city view (Univ. of Tsukuba)

Experimental Results (cont.) Figure 9. Detected occlusion area

Experimental Results (cont.) ► Quantified error rate measurement  Percentage of badly matched pixels is defined as:

Experimental Results (cont.) Figure10. Errors of disparity map compared with the ground truth map

Experimental Results (cont.) Table 1. Percentage of badly matched pixels ► Percentage of badly matched pixels

Experimental Results (cont.) Figure 11. Simulation results for the “ Dog and Flower ” indoor scene

Experimental Results (cont.) Figure 12. Simulation results for the outdoor scene.

Experimental Results (cont.) Chart 1. The average scores of subjective evaluation for the indoor stereoscopic scene

Experimental Results (cont.) Chart 2. The average scores of our subjective evaluation for the outdoor HD stereoscopic scene

Discussion ► ► The virtual control of the optical axis is a first step toward a versatile framework of post-processing for providing comfortable stereoscopic contents. ► ► It would be interesting as a future work to analyze various mutually affecting factors on visual comfortableness and 3D impact in stereoscopic images, such as focal length, convergence angle, resolution, and the distance to main objects ► ► Use a more rigorous subjective assessment with a sufficient number of subjects evaluating a large set of stereoscopic images.

References ► [1] J. Park, G. M. Um, C. Ahn, and C. Ahn, Virtual control of optical axis of the 3DTV camera for reducing visual fatigue in stereoscopic 3DTV, ETRI Journal, Volume 26, Number 6, December ► [2] M. Accame, F. G.B. De Natale, and D. D. Giusto, Hierarchical block matching for disparity estimation in stereo sequences, International Conference on Image Processing, Volume 2, page(s): , October ► [3] J. Park and S. Inoue, Arbitrary View Generation Using Multiple Cameras, Proc. IEEE ICIP'97, volume 1, page(s): Santa Barbara, USA, Oct

Thank you!