Technische Universität Berlin Communication Systems Group Director: Prof. Thomas Sikora Sebastian Knorr 21/08/2007 Super-Resolution Stereo- and Multi-View Synthesis from Monocular Video Sequences S. Knorr, M. Kunter, and T. Sikora

Communication Systems GroupS. Knorr: From 2D- to Stereo- to Multi-View Video Outline  Motivation  Overview of the conversion system  Stereo- and multi-view synthesis  Super-resolution approach  Simulation results  Subjective Evaluation  Limitations, conclusion and future work 2

Communication Systems GroupS. Knorr: From 2D- to Stereo- to Multi-View Video Motivation  Conversion of existing 2D video material automaticaly into 3D (stereoscopic video)  High quality stereoscopic video for 3D devices (e.g. shutter glasses, auto-stereoscopic displays, etc.)  Multi-View video for multi-user 3D displays 3 2D TV-display auto-stereoscopic display shutter-glasses anaglyph

Communication Systems GroupS. Knorr: From 2D- to Stereo- to Multi-View Video 4 Overview of the Conversion System

Communication Systems GroupS. Knorr: From 2D- to Stereo- to Multi-View Video 5 Stereo- and Multi-view Synthesis Multi-view camera setup (red cameras) original camera path (grey cameras) 3D point cloud current camera (blue camera)

Communication Systems GroupS. Knorr: From 2D- to Stereo- to Multi-View Video 6

Communication Systems GroupS. Knorr: From 2D- to Stereo- to Multi-View Video Principles of Stereo- and Multi-view Synthesis  Define a virtual camera for each frame of the the sequence (at least one for stereo)  Project 3D points into the virtual camera:  Estimate global homography between the virtual view and the closest view of the original camera path 7

Communication Systems GroupS. Knorr: From 2D- to Stereo- to Multi-View Video Stereo- and Multi-view Synthesis  Estimate global homography between the virtual view and the closest view of the original camera path (e.g. minimizing a cost function of the transfer errors)  Warp complete image into the virtual view 8 original left view virtual right view

Communication Systems GroupS. Knorr: From 2D- to Stereo- to Multi-View Video Super-Resolution Stereo View Synthesis 99

Communication Systems GroupS. Knorr: From 2D- to Stereo- to Multi-View Video Super-Resolution Stereo View Results 10

Communication Systems GroupS. Knorr: From 2D- to Stereo- to Multi-View Video Simulation Results 11 Original view and 8 virtual views for multi-user auto-stereoscopic displays Anaglyph stereo pair of "Statue" Anaglyph stereo pair of "Medusa"

Communication Systems GroupS. Knorr: From 2D- to Stereo- to Multi-View Video 12 Simulation Results [Source: BBC-documentation "Planet Earth", Series "Ice Worlds"]

Communication Systems GroupS. Knorr: From 2D- to Stereo- to Multi-View Video Subjective Evaluation (MOS) 13

Communication Systems GroupS. Knorr: From 2D- to Stereo- to Multi-View Video Subjective Evaluation (DMOS) 14

Communication Systems GroupS. Knorr: From 2D- to Stereo- to Multi-View Video  Scene has to be (almost) static  Restrictions on the camera movement  just forward- or backward camera movement is not allowed  up- and down movement can be handled by transposing the frames by 90°  Virtual cameras too far from the camera path result in large errors when estimating homographies Limitations 15

Communication Systems GroupS. Knorr: From 2D- to Stereo- to Multi-View Video Conclusion:  the presented algorithm is highly attractive as a tool for user-assisted 2D-3D conversion and 3D production systems  software tool is highly attractive for amateur film-maker Future work:  applying local homographies  extension for dynamic scenes Conclusion and Future Work 16

Communication Systems GroupS. Knorr: From 2D- to Stereo- to Multi-View Video Homepage: Contact: Sebastian Knorr Thank you for your attention! 17

Communication Systems GroupS. Knorr: From 2D- to Stereo- to Multi-View Video  Determine the positions of the virtual stereo cameras  Normalize the coordinates of the cameras  Define positions of the virtual cameras (parallax is a multiple of 64 mm)  Determine projection matrices of the virtual stereo cameras 18 Stereo- and Multi-view Synthesis user interaction!