1 Michael M. Bronstein New dimensions of media 1 November 2007 New dimensions of media Michael M. Bronstein Department of Computer Science Technion – Israel Institute of Technology cs.technion.ac.il/~mbron MMSN, San Jose 1 November 2007

2 Michael M. Bronstein New dimensions of media 1 November 2007 Dimensions of media RadioColor televisionBlack-and-white television 3D video

3 Michael M. Bronstein New dimensions of media 1 November 2007 Conventional 2D TV Video: Bullettime You see exactly what the camera shot

4 Michael M. Bronstein New dimensions of media 1 November 2007 Video: Bullettime Interactive selection of viewpoints Free viewpoint TV (FTV)

5 Michael M. Bronstein New dimensions of media 1 November 2007 Augmented reality Place 3D generated objects in the scene Video: Japanese TV

6 Michael M. Bronstein New dimensions of media 1 November 2007 Emerging computer vision applications Gesture recognition (user interface) Face recognition (biometrics) Analysis of 3D non-rigid objects Bronstein et al, DV systems

7 Michael M. Bronstein New dimensions of media 1 November D/Stereoscopic TV Three-dimensional depth perception of the scene

8 Michael M. Bronstein New dimensions of media 1 November 2007 Evolution 1839 Talbot invents photographic process 1838 Wheatson explains binocular vision 1851 Brewster streoscope sold in London 1891 Anaglyph is invented 1928 First television broadcast in USA 1922 First 3D movie premiere 1952 First color 3D movie premiere 1995 First IMAX 3D fiction movie Today 3D content available User-grade 3D displays 3DTV broadcast Early 2000s Commercial 3D monitors appear ?

9 Michael M. Bronstein New dimensions of media 1 November 2007 Ecosystem 3D video acquisition 3D scene representation Coding & Transmission Rendering & Display Computer graphics

10 Michael M. Bronstein New dimensions of media 1 November 2007 Our 3D perception Binocular vision Each eye sees a slightly different picture Depth perception (stereopsis) achieved by merging two pictures

11 Michael M. Bronstein New dimensions of media 1 November 2007 Shape from stereo Left eyeRight eye Parallax Closer objects have larger parallax Depth recovered from parallax (disparity) between corresponding points Correspondence problem Can be generalized to multiple views

12 Michael M. Bronstein New dimensions of media 1 November 2007 Stereo cameras NASA Mars rover IMAX 3D camera Multiview camera array (Carnegy Mellon)

13 Michael M. Bronstein New dimensions of media 1 November 2007 Structured light Active sensor 3D shape extracted from deformation of the projected light pattern Typically slow and unsuitable for fast moving objects Camera Moving projector

14 Michael M. Bronstein New dimensions of media 1 November D on your desk Can be literally done using just a pencil and a lamp! Jean-Yves Bouguet, 1998

15 Michael M. Bronstein New dimensions of media 1 November 2007 Coded light Sequence of black/white patterns projected onto the object Patterns form a binary code encoding the angle relative to the projector Depth recovered by triangulation Active sensor Light patternAngle Camera Projector

16 Michael M. Bronstein New dimensions of media 1 November 2007 Time-of-flight Principle of a laser range finder Distance to surface measured by timing the travel of a pulse of light Requires nanosecond gating Active sensor Object Transmitted pulse of light Object Reflected pulse of light Nanosecond gate Sensor

17 Michael M. Bronstein New dimensions of media 1 November D scene representation Stereo pairDepth mapMeshLayered depth map (2.5D) No 3D reconstruction Allows view synthesis Natural for stereo cameras Explicit 3D model Requires rendering Used for synthetic CG content IMAGE-BASEDMODEL-BASED

18 Michael M. Bronstein New dimensions of media 1 November 2007 Stereo pair LeftRight Left and right views are stored side-by-side Can use existing video compression and distribution standards Large redundancy: special compression algorithms are desirable

19 Michael M. Bronstein New dimensions of media 1 November 2007 Stereo compression CONVENTIONAL VIDEO Motion-compensated prediction (MCP) IPPPIPPPI Motion-compensated prediction IPPPIPPPI IPPPIPPPI Disparity- compensated prediction (DCP) Motion-compensated prediction STEREOSCOPIC VIDEO

20 Michael M. Bronstein New dimensions of media 1 November D+Z Conventional video (2D)Depth (Z) 2D video and corresponding depth map are stored side-by-size Uses existing video compression and distribution standards Z-component increases bandwidth only by 5-20% Used by Philips (WOW vx)

21 Michael M. Bronstein New dimensions of media 1 November 2007 Geometry compression 3D video Motion analysis Mesh compression Trajectory compression Motion trajectories Mesh Common representation & coding MPEG spirit Compress mesh and motion separately Active research field

22 Michael M. Bronstein New dimensions of media 1 November 2007 Correspondence again Mesh motion = displacement of corresponding points Non-rigid shapes are the biggest challenge Intrinsic geometric correspondence Methods borrowed from data mining (generalized MDS) Bronstein et al, 2007

23 Michael M. Bronstein New dimensions of media 1 November 2007 Mesh compression Progressive mesh compression Both similar and dissimilar to image compression Laplace-Beltrami (spectral methods) Successive approximation (progressive meshes) Connectivity compression Hoppe, 1996

24 Michael M. Bronstein New dimensions of media 1 November 2007 Viewpoint selection Capture the scene with multiple cameras closely located Transition between different views If the cameras are dense enough, the transition will be smooth Used in QuickTime VR Only existing viewpoints Object

25 Michael M. Bronstein New dimensions of media 1 November 2007 Panorama Produce a wide-angle or 360 degrees panorama Change the viewpoint by selecting a subset of the panorama Used in QuickTime VR Limited range of viewpoints

26 Michael M. Bronstein New dimensions of media 1 November 2007 Image-based rendering Rendering based on existing views instead of geometric model Light field: 2D collection of images (4D array) + time New viewpoint = 2D slice of the 4D array, produced by interpolation Requires multiple views (tens) Problems with shadows, specularities, etc. Visualization of light field Existing views New view

27 Michael M. Bronstein New dimensions of media 1 November D display How to create three-dimensional depth perception of the scene? Display to each eye the image it would see Stereo camera = eyes Multiview: synthesize the views the eyes would see IMAGE-BASEDMODEL-BASED Render the views the eyes would see Computer graphics methods Head mounted display (VR glasses)

28 Michael M. Bronstein New dimensions of media 1 November D on a big screen Block unwanted rays Screen

29 Michael M. Bronstein New dimensions of media 1 November 2007 Anaglyph Compatible with any color display Requires color glasses Unnatural colors + = LeftRightAnaglyph Color glasses Screen

30 Michael M. Bronstein New dimensions of media 1 November 2007 Polarized glasses Screen Polarized light Used in IMAX 3D Requires special projection Requires polarized glasses Polarized projector Polarized projector

31 Michael M. Bronstein New dimensions of media 1 November 2007 Shutter glasses Left and right image displayed interleaved in time (double framerate) Shutter glasses keep only one of the eyes open at a time Requires screen and glasses sync

32 Michael M. Bronstein New dimensions of media 1 November 2007 Do we need glasses?

33 Michael M. Bronstein New dimensions of media 1 November 2007 Lenticular display Lenticular lens sending parts of the image to different eyes No glasses needed (autostereoscopic) Can be attached to a legacy monitor User-grade commercial products available from Sharp, Philips, etc. Viewing angle is a challenge Screen Lenticular lens

34 Michael M. Bronstein New dimensions of media 1 November 2007 Volumetric display Display a periodically time-varying 2D image on a rotating mirror Illusion of 3D object due to visual persistence 360 degrees view Perspecta display Projector Rotating screen

35 Michael M. Bronstein New dimensions of media 1 November 2007 Summary: acquisition Stereo Time-of-flight PASSIVEACTIVE Structured light Coded light

36 Michael M. Bronstein New dimensions of media 1 November 2007 Summary: representation & compression Conventional (MCP-based) Mesh compression PIXEL-TYPE DATAGEOMETRIC DATA 2D+Z Stereoscopic (DCP-based)

37 Michael M. Bronstein New dimensions of media 1 November 2007 Summary: display VR glassesColor glassesPolarized glasses HEAD-MOUNTEDSTEREOSCOPIC GLASSES Shutter glasses VOLUMETRICAUTOSTEREOSCOPIC Lenticular

38 Michael M. Bronstein New dimensions of media 1 November 2007 Conclusion 3D applications are an active research and development field Another hype? Looks like 3D is going to become a commodity Still many challenges Wait 5-10 years and see