Introduction Michael Bleyer LVA Stereo Vision

VU Stereo Vision (3.0 ECTS/2.0 WS)  Anrechenbarkeit: Wahlfach im Masterstudium “Computergraphik & Digitale Bildverarbeitung” Wahlfach im Masterstudium “Medieninformatik”  Webseite der LVA:

VU Stereo Vision (3.0 ECTS/2.0 WS)  Vorlesungstermine (9 Einheiten): Fr 4. März ( ) Fr 11. März ( ) Fr 18. März ( ) Fr 25. März ( ) Fr 01. April ( ) Fr 08. April ( ) Fr 15. April ( ) Fr 06. Mai ( ) Fr 13. Mai ( ) Mündliche Prüfung nach Vereinbarung  Ort: Seminarraum 188/2

Topics Covered in the Lecture (1)  Session 1 - Introduction 3D perception What is disparity? Applications  Session 2 – Basics 3D geometry Challenges in stereo matching Assumptions  Session 3 – Local methods Principle Adaptive windows  Section 4 – Global methods Stereo as energy minimization problem Dynamic programming

Topics Covered in the Lecture (2)  Session 5 - Graph ‐ Cuts Alpha-expansions Fusion-moves  Session 6 – Smoothness Term Belief propagation Different smoothness terms  Session 7 – Data Term Different match measures Role of color  Session 8 - Segmentation ‐ Based Stereo Occlusion handling An example algorithm Stereo and matting  Session 9 – Surface Stereo One of our recent paper Demo - Autostereoscopic display

Homework  Implement a block matching algorithm You will get more details in session 3. The algorithm is very simple. I do not care about programming languages. You have to present your algorithm as part of the oral exam.

My Promise  After attending the lecture you should: Know the basics of stereo visions Be able to understand the current state-of-the art Have understood several principles that you can also use for other vision problems:  Optical flow  Segmentation  Matting  Inpainting  Image Restoration  …

Michael Bleyer LVA Stereo Vision 3D Perception

Human-Eye Separation(~6.5cm) Left 2D ImageRight 2D Image Brain 3D View

3D Perception Human-Eye Separation(~6.5cm) Left 2D ImageRight 2D Image Brain 3D View If we ensure that the left eye sees a 2D image and the right eye sees another one, our brain will try to overlay the images to generate a 3D impression. How can we use this for watching 3D movies?

Anaglyphs  Two images of complementary color are overlaid to generate one image.  Glasses required (e.g. red/green)  Red filter cancels out red image component, green filter cancels out green component  Each eye gets one image => 3D impression  Current 3D cinemas use this principle. However, polarization filters are used instead of color filters. (Anaglyph Image) (Red/Green Glasses)

Shutter Glasses  Display flickers between left and right image (i.e. each even frame shows left image, each odd frame shows right image)  When left frame is shown, shutter glasses close right eye and vice versa.  Requires new displays of high frame rate (120Hz).  Currently pushed by Nvidea to address gaming market. (Shutter Glasses and 120 Hz Display) (Nvidea Artwork)

Autostereoscopic Displays  No glasses required!  Matrix of many transparent lenses put on the display.  Lenses distort pixels so that left eye gets a left image and right eye gets a right image (if you are standing in a sweet spot) => 3D impression  Novel viewpoint capability: You can walk in front of the display and get a perceptively correct depth impression depending on your current viewpoint.  You will get a demo soon (Philips Wowvx Display)

Free Viewing (No glasses required, but some practice)  The way how you usually look at the display (no 3D):

Free Viewing (No glasses required, but some practice)  Parallel Viewing: Left Image Right Image

Free Viewing (No glasses required, but some practice)  Cross Eye Viewing:  Most likely the simpler method. Right Image Left Image

Learning Cross Eye Viewing  Take a pencil and hold it in the middle of your eyes.  Look at the pencil and slowly change its distance to your eyes  If you found the right distance, you see a third image inbetween left and right images.  This third image is in 3D  Practise, it is worth the effort. Right 2D Image Left 2D Image

3D on YouTube

Michael Bleyer LVA Stereo Vision Computational Stereo

Left 2D ImageRight 2D Image Brain 3D View Replace human eyes with a pair of slightly displaced cameras.

Computational Stereo Left 2D ImageRight 2D Image Brain 3D View Displacement (Stereo Baseline) Replace human eyes with a pair of slightly displaced cameras.

Displacement (Stereo Baseline) Computational Stereo Left 2D ImageRight 2D Image Brain 3D View

Displacement (Stereo Baseline) Computational Stereo Left 2D ImageRight 2D Image Computer 3D View

Displacement (Stereo Baseline) Computational Stereo Left 2D ImageRight 2D Image Computer 3D View

Displacement (Stereo Baseline) Computational Stereo Left 2D ImageRight 2D Image Computer 3D View How can we accomplish a fully automatic 2D to 3D conversion?

What is Disparity?  The amount to which a single pixel is displaced in the two images is called disparity.  A pixel’s disparity is inversely proportional to its depth in the scene.

What is Disparity?  The amount to which a single pixel is displaced in the two images is called disparity.  A pixel’s disparity is inversely proportional to its depth in the scene. Background Disparity (Small)

What is Disparity?  The amount to which a single pixel is displaced in the two images is called disparity.  A pixel’s disparity is inversely proportional to its depth in the scene. Foreground Disparity (Large)

Disparity Encoding  The disparity of each pixel is encoded by a grey value.  High grey values represent high disparities (and low gray values small disparities).  The resulting image is called disparity map.

Disparity and Depth  The disparity map contains sufficient information for generating a 3D model.

Disparity and Depth  The disparity map contains sufficient information for generating a 3D model. The challenging part is to compute the disparity map. This task is known as the stereo matching problem. Stereo matching will be the topic of this lecture!!!

Applications (just a few examples)

3D Reconstruction from aerial images  Stereo cameras are mounted on an airplane to obtain a terrain map.  Images taken from

3D Reconstruction of Cities  City of Dubrovnik reconstructed from images taken from Flickr in a fully automatic way.  [S. Agarwal, N. Snavely, I. Simon, S. Seitz and R. Szeliski “Building Rome in a Day”, ICCV, 2009]

Driver Assistance / Autonomous driving cars  For example, use stereo to measure distance to other cars.  DARPA Grand Challenge  Image taken from

The Mars Rover  Reconstruct the surface of Mars using stereo vision

Human Motion Capture  Fit a 3D model of the human body to the computed point cloud.  [R. Plänkers and P. Fua, “Articulated Soft Objects for Multiview Shape and Motion Capture”, PAMI, 2003]

Bilayer Segmentation – Z-Keying  Goal: Divide image into a foreground and a background region.  Simple background subtraction will fail if there is motion in the background.  Solution: Compute depth map If the depth of a pixel is larger than a predefined threshold, pixels belongs to the foreground  [A. Criminisi, G. Cross, A. Blake and V. Kolmogorov, “Bilayer Segmentation of Live Video”, CVPR, 2006]

Novel View Generation  Given a 3D model of the scene, one can use a virtual camera to record new views from arbitrary viewpoints.  For example: Freeze frame effect from the movie Matrix.  [L. Zitnick, S. Kang, M. Uyttendaele, S. Winder, and R. Szeliski, "High-quality video view interpolation using a layered representation", SIGGRAPH, 2004] Left View (recorded) Right View (recorded) Virtual Interpolated View (not recorded)

MS Kinect

Understanding Human Vision  If we can teach the computer to see in 3D, we can also learn more about the way how the human perceives depth.

Summary  3D Perception Principle of human 3D vision Ways for watching movies in 3D  Computational Stereo Stereo Matching Problem  Applications