3D/Multview Video

Outline Introduction 3D Perception and HVS 3D Displays 3D Video Representation Compression

Introduction By 2015 – over 20 million homes be watching 3D TV – 3D mobile devices >11% market share

3D Perception and the Human Visual System (HVS) Depth Cues – Monocular (Pictorial) Cues relative size, motion parallax, occlusion (interposition), light and shade, texture gradient, haze, and perspective – Binocular Cues Rely on binocular stereopsis (looking at a 3D scene, each eye sees a unique image from slightly different angles) – Oculomotor cues Based on the ability to sense the position of our eyes and the tension in the eye muscles

3D Perception and the Human Visual System (HVS) Depth Information OculomotorAccommodationConvergenceMyosisVisualBinocularMonocularStaticDynamic Eye FocusEye Rotation AnglePupil SizeRetinal Disparity Classic Pictorial Cues Motion Parallax and Dynamic Occlusion

Binocular Cues Binocular Disparity – provides cues about relative distance of objects and their depth structures, as well as absolute depths

Oculomotor Cues – Convergence positioning the eyes so that the difference between the information projected in both retinae is minimized – Accommodation the ability of the eye to change the optical power of its lens in order to focus on objects at various distances

3D Displays Aided-View Displays – Rely on special user-worn devices (e.g. stereo glasses) – Different multiplexing methods direct optimal signal to appropriate eye color multiplexing, polarization multiplexing, time multiplexing Free-View Displays – Also known as auto-stereoscopic displays – Relief the viewer from the discomfort of wearing specialized glasses

Auto-stereoscopic Displays Auto-stereoscopic displays – No need for glasses – Divides viewing space into a finite number of viewing slots (only one image/view is visible) – Each eye sees a different image, and images change as viewer moves/changes head position Two-views – Viewer perceives correct stereoscopic image only if standing at ideal distance and correct position Multi-view – Allows viewers to perceive a 3D image when eyes are anywhere within viewing zone – Supports multiple users

Auto-stereoscopic Displays Head tracking

Auto-stereoscopic Displays – parallax barriers – lenticular arrays

3D Video Representation Explicit – Encode each video separately (simulcast coding) – Jointly encode all views (e.g. using multi-view coding) Implicit – Scene geometry information, such as depth maps, along with texture data (video-plus-depth (V+D) representation)

Conventional Stereo Video Simplest 3D video data representation – Two views -> two eyes Multiplexed spatially (passive) or temporally (active) – Temporal multiplexing maintains full resolution of each view Disadvantage: – Hardware dependency (acquisition process is tailored to a specific type of displays, baseline distance between the two cameras is fixed)

Conventional Stereo Video Time Multiplexing (double the frame rate) Spatial Multiplexing (half the resolution)

Video Plus Depth 2D video signal along with geometry information of the scene texture depth map

View Synthesis Holes resulting from 3D warping

Multi-View Plus Depth (MVD) Example: 2-views plus depth

Layered Depth Video (LDV) The occlusion layer in the LDV representation is used to fill any holes during the view synthesis process

Coding 3D Videos Conventional Stereo – Fidelity range extensions (FRExt) in H.264/MPEG-4 AVC define Special stereo video supplemental enhancement information (SEI) – Two views interlaced field coding mode of H.264/MPEG-4 AVC

Coding 3D Videos Video-Plus-Depth – Existing video codecs optimized for encoding texture information which are viewed by the user – Can use same codecs for depth coding but compression artifacts have more severe effect) – ISO/IEC MPEG-C Part 3 standard representation format for depth maps

Coding 3D Videos Multi-View Video – H.264/MPEG-4 AVC standard MVC extension view switching random access, view scalability, and backward compatibility – Inter-view prediction – Independent baseview for backward compatibility

MVC Prediction Structure

Thank You!