1. WIEN Stereo-based Image and Video Analysis for Multimedia Applications Application: “Computer-generated Stereoscopic Paintings“ M. Gelautz, E. Stavrakis, and M. Bleyer Interactive Media Systems Group Technical University Vienna, A-1040 Vienna, Austria Email: gelautz@ims.tuwien.ac.at

2. ISPRS 2004 WIEN Stereo Analysis for Multimedia Applications Artistic Rendering of Stereo Views n Our goal is to provide stereo views of real scenes a hand-painted appearance. n The work combines photogrammetric techniques (stereo) with computer graphic algorithms. n Available painterly rendering algorithms have been designed for single images – painterly rendering of stereo views presents a new topic of research!

3. ISPRS 2004 WIEN Stereo Analysis for Multimedia Applications Traditional Stereoscopic Painting Fig1. Example of a stereoscopic painting by Salvador Dalí. Fig1. Example of a stereoscopic painting by Salvador Dalí (“The sleeping smoker”, 1972/73).

4. ISPRS 2004 WIEN Stereo Analysis for Multimedia Applications Motivation for Computer- generated Stereo Painting n The manual creation of stereo paintings is a labor intensive task. n The artist needs to reproduce the same composition twice from different viewpoints. n Some painters used stereo photography to base their compositions on. n The excessive effort associated with the manual creation of stereoscopic paintings can be reduced by computer vision techniques.

5. ISPRS 2004 WIEN Stereo Analysis for Multimedia Applications Single Image Painterly Rendering Hertzmann, A., 1998: “Painterly rendering with curved brush strokes of multiple sizes” (a) Original image (b) Rendered image (impressionist style) paint spilling

6. ISPRS 2004 WIEN Stereo Analysis for Multimedia Applications Requirements of Stereoscopic Painterly Rendering n Observation: Individual painting of the left and right stereo image will usually not produce satisfactory results. n Requirements: u To preserve coherence between the brush strokes of the two images. u To preserve the depth discontinuities in order to prevent „paint spilling“. u To deal with occlusions. n Solution: We incorporate a stereo-derived depth map into the painting algorithm.

7. ISPRS 2004 WIEN Stereo Analysis for Multimedia Applications Stereo Matching Algorithm n “A Layered Stereo Matching Algorithm Using Color Segmentation and Global Visibility Constraints“ n Key features: u Color segmentation of the reference image. u Color segments are clustered into more robust depth layers (mean shift clustering). u Iterative refinement of the solution based on a cost function (greedy search algorithm). n Results: 2nd rank (among 30 algorithms) on the Middlebury Stereo Evaluation Website (http://www.middlebury.edu/stereo)

8. ISPRS 2004 WIEN Stereo Analysis for Multimedia Applications Clustering of Color Segments into Layers Fitting of planar models (depth layers) to segments (top) and layers (bottom) -> layer fitting is more robust than segment fitting!

9. ISPRS 2004 WIEN Stereo Analysis for Multimedia Applications Stereo Matching Results Evaluation: 0.2% „bad“ pixels (i.e., unoccluded pixels whose absolute disparity error is greater than 1) Sawtooth stereo pair (Middlebury website) Left image Right image Stereo disparities Ground truth

10. ISPRS 2004 WIEN Stereo Analysis for Multimedia Applications Stereo Painting Algorithm (1) n Modify Hertzmann‘s single image painting algorithm to prevent brush strokes from crossing depth discontinuities. n Paint the reference image using the modified single image painting algorithm. n Paint (dilated) occluded regions of the second image. n Project paint from the reference image to non- occluded regions of the second image. As a consequence, we avoid repainting large parts of the second image.

11. ISPRS 2004 WIEN Stereo Analysis for Multimedia Applications Stereo Painting Algorithm (2) Original image Stereo disparities Occlusion map Occlusion paint

12. ISPRS 2004 WIEN Stereo Analysis for Multimedia Applications Stereo Painting Result (1) Single image painting results Stereo image painting results Left image Right image

13. ISPRS 2004 WIEN Stereo Analysis for Multimedia Applications Stereo Painting Result (2) Original stereo pair Painted stereo pair

14. ISPRS 2004 WIEN Stereo Analysis for Multimedia Applications Stereo Painting Result (3) Creating the paintings…

15. ISPRS 2004 WIEN Stereo Analysis for Multimedia Applications Summary and Outlook n We have presented u a new stereo matching algorithm that relies on clustering color-segmented regions into depth layers. u an algorithm for stereoscopic painterly rendering based on real image pairs. n We plan to use the stereo-derived depth maps to enrich the computer-generated paintings with further depth cues (e.g., different levels of detail).

16. ISPRS 2004 WIEN Stereo Analysis for Multimedia Applications Acknowledgements n This work was supported by the Austrian Science Fund (FWF) under project P15663. n We wish to thank “Verwertungsgesell- schaft Bildender Künstler (VBK)“, Austria, for permission to reproduce „The sleeping smoker“.