1. Three-Dimensional Video Postproduction and Processing Ibraheem Alhashim - July 10 th 2013 CMPT 880

2. Overview 1  History + Fundamentals  3D display technologies  Basic processing  View synthesis  2D to 3D conversion  3D media for mobile devices  Outlook

3. Overview 2  History + Fundamentals  3D display technologies  Basic processing  View synthesis  2D to 3D conversion  3D media for mobile devices  Outlook

4. A bit of history 3  Imaging technology James Maxwell 1855 JC d'Almeida 1858 Joseph Niépce 1826 Stereoscope 1860 Underwood 1901 The Power of Love (1922 film) The Jazz Singer 1927

5. A bit of history 4 Becky Sharp (1935)

6. A bit of history 5  Timeline of 3D movies 1950 s 1980 s 2010 s Worldwide: $2,782,275,172

7. 3D Feature Films 6  Highest-grossing films 2012: nine of the top 15 were in 3D [businessinsider.com]  Industry forecast > 20% of TVs by 2015 Year3D Films 20054 20067 20088 200920 201030 201138 201238

8. Other uses 7  WW2 maps  Virtual reality  3D Electron Microscopy  Video games + virtual cinema 1968 1945 1980 2013

9. 3D Video 8  Usually marketed as objects popping off screen

10. 3D Video 9  In reality.. It’s the same old concept Present slightly different image per eye The brain combines them and perceives depth  Trick human visual system  Stereo 3D content production: technical, psychological, and creative skills

11. 3D Video - Issues 10  Not as straightforward as 2D production  Several considerations for a good 3D experience  Balance between 3D effect and overall experience  Minimize viewing discomfort Stereoscopic comfort zone Scene depth adaptation Control of global and local disparity Video composition

12. Stereoscopic Comfort Zone 11  Comfort zone  Stereographer  “bring the whole real world inside this virtual space”

13. Control of Absolute Disparity 12  Convergence is controlled by shifting the views

14. Scene Depth Adaptation 13  Keep in mind disparity range (screen size and resolution)  Carefully plan a scene’s 3D effect  Consider transitions and provide resting periods  Post-production depth adaptation Manual changes per display

15. 3D Display Adaptation 14  From cinema to TV  Depth composition has to modified (stereographer)  Depth information allows for virtual view interpolation

16. Local Disparity Adaptation 15  Objects should be within stereoscopic window  Intentional depth changes Physical / multiple camera rigs Synthetic / depth-editing  Objects at the border Can cause retinal rivalry Should be cropped by virtually shifting screen plane closer However, not applicable to live broadcast

17. Live auto-correction 16  Automatic correction & manipulation of stereo live broadcast  Live sport events (big player)  Close objects could abruptly appear  Open research problem  Some kind of novel view synthesis

18. Video composition 17  Mixing and Composition of 3D Material, Real and Animated Content  Graphics overlay / subtitles  Cannot be simply pasted over other footage  Leverage knowledge about depth range of footage

19. 3D Video - Issues 18  Summary  3D Production has an “art” component  Different medium requires different parameters  Content makers / directors need to think about 3D issues  Stereographer try to balance 3D effect with overall viewing experience

20. 3D video 19  Fundamentals  3D display technologies  Basic processing  View synthesis  2D to 3D conversion  3D media for mobile devices  Outlook

21. 3D Display Technologies 20  “Offer immersive experience”  3D Glasses (cinema + TV)  Head-mounted displays  Volumetric and holographic displays  Autostereoscopic displays

22. Autostereoscopic displays 21  Best choice for mobile devices  Backward compatible & closer to viewer expectation  Most common Parallax barrier Lenticular sheet

23. Autostereoscopic displays 22  Crosstalk (ghosting) is most important parameter  “information meant for one eye intrudes into the other eye’s view”

24. Autostereoscopic displays 23  Issues  Generally less available depth range  More ghosting artifacts  Also, depth information is essential Synthesize different views

25. 3D video 24  Fundamentals  3D display technologies  Basic processing  View synthesis  2D to 3D conversion  3D media for mobile devices  Outlook

26. Basic processing 25  Signal processing to avoid visual artifacts  Any small visual discrepancy could cause discomfort  Three main categories 1. Correction of Geometrical Distortions 2. Color Matching 3. Adjustment of Stereo Geometry

27. Correction of Geometrical Distortions 26  Camera rigs might not be perfectly aligned  Real lenses impose radial distortions by nature  Other lens parameters might not sync  E.g. geometrical lens distortions or chromatic aberration

28. Color Matching 27  Color discrepancy can lead to eyestrain and visual fatigue  Manual calibration need to be done on cameras  Automatic methods exist (histogram filtering)

29. Color Matching 28  Modern professional postproduction tools incorporate stereo color matching and grading

30. Adjustment of Stereo Geometry 29  Convergence need to be selected and balanced carefully to achieve good stereo content  Usually by shifting images horizontally in contrary directions, however, cropping & scaling is needed “shift-crop-scale” Demo http://stereo.nypl.org/createhttp://stereo.nypl.org/create

31. Adjustment of Stereo Geometry 30  Stereo baseline is fixed during shooting  Several hardware solutions help camera team analyze the disparity range  Also help visualize result of shift-crop-scale

32. 3D video 31  Fundamentals  3D display technologies  Basic processing  View synthesis  2D to 3D conversion  3D media for mobile devices  Outlook

33. 3D Depth information 32  Depth information is needed for  novel output images in post-production  adjusting the view parallax (different screens)  many different uses in postproduction

34. Depth info 33  Extracting depth information  (time of flight camera / SfS)  Structure-from Stereo (SfS)  Advanced computer vision problem  Stereo matching Local – block matching, optical flow est. Global – graph cuts, simulated annealing

35. Structure-from Stereo 34  Global methods are more accurate Slow + don’t work well on video / motion  Local methods are more widely used Window-based methods Some system are in real-time Blocky output

36. Example of Depth-based method 35  Apply hybrid recursive matching (HRM)  Follow by cross-trilateral median filtering (ACTMF)  Semi-automatic

37. Post processing depth 36  Align depth discontinues to object borders  Remove noise and mismatches  Fill occlusions  Approaches  Use image segmentation  Neighborhood filtering

38. View synthesis 37  Synthesize new virtual stereo views by image- based rendering Input – depth + color Output – image with new view  Two types depth-based + warping-based

39. Depth-based 38  Computer vision techniques Image-based rendering (IBR) Depth-based rendering (DIBR) Layered-depth images (LDI) Intermediate view reconstruction (IVR)

40. Depth-Image-Based Rendering 39  Need pixel-by-pixel depth maps  Recent focus  Handle depth discontinues  Better depth boundaries

41. Warping-based 40  Methods that deform the image content directly  Compress or stretch by nonlinear warp function  Do not need camera calibration, segmentation, fill holes  Worst case, visible wobbling artifact

42. View synthesis 41  Summary Depth maps are computed using computer vision techniques (still active) Generate new views by image-based rendering or warping Warping methods can potentially have less visual problems DEMO

43. 3D video 42  Fundamentals  3D display technologies  Basic processing  View synthesis  2D to 3D conversion  3D media for mobile devices  Outlook

44. 2D to 3D 43  3D to 2D is trivial  Hot topic for 3DTV and 3D cinema  Methods so far are Manual (computer assisted) Automatic

45. Depth cues 44  Human visual system Monocular cues Binocular cues

46. Depth cues 45  Monocular depth cues - things can be seen by one eye (2D Camera) Focus / defocus, perspective, relative size Light and shading and texture Motion parallax  Binocular – 2 eyes or 3D camera rigs accommodation, convergence, and binocular discrepancy

47. Manual 2D to 3D 46  Applicable to prerecorded video  Utilize depth cues to generate a stereoscopic view for each frame  Time consuming and costly  Cost vs. quality

48. Manual 2D to 3D 47  Three major steps Rotoscoping / segmentation Depth assignment Inpainting  Few companies provide process as a service

49. Depth Assignment 48  Shifting different parts of scene to simulate 3D  To avoid cardboard effect hire a “3D compositor” Create displacement maps for each pixel Use 3D primitives, spheres or cubes Use DIBR or 3D warping to synthesize view cannot handle transparencies well

50. Occlusion filling 49  Also known as “in-painting”  One of the most challenging parts in 2D/3D conversion

51. Automatic conversion 50  Automatic systems Extract depth information + synthesize stereoscopic images

52. Automatic segmentation 51  Automatic segmentation of background Background subtraction techniques (ML) Optical-flow-based (uses motion as cue)  Objects in scene segmentation Contour-based (color, edges, texture) “SnapCut” state of the art algorithm in After Effects

53. Automatic depth estimation 52  3D Structure recovery (Shape From X)  Depth from focus/defocus  Depth from geometric cues Relative sizes, gradients  Depth from color and intensity Lights and shadow  Depth from motion Motion parallax

54. Automatic in-painting 53  Relay on - texture synthesis or border continuation  Some work is done on spatio-temporal inpainting Enforce global temporal consistency for patches  Object motion could reveal background Not viable in practice

55. Real-time and Offline conversion 54  Real-time Approximate using motion parallax Could use color + intensity information Height-depth cues Hybrid approaches are better  Offline Apply structure from motion Create virtual scene using different views Slow cloudification?

56. 2D to 3D 55  Summary  Hire specialist to rebuild depth and frame objects  Relay on computer vision methods to build depth info  Fill background using in-painting  Still an active research topic segmentation + depth estimation + synthesis

57. 3D video 56  Fundamentals  3D display technologies  Basic processing  View synthesis  2D to 3D conversion  3D media for mobile devices  Outlook

58. 3D media for mobile devices 57  Consider size of display  Computing power + Network constraints  Encoding 3D Video coding (V+D) Multiview Video Coding (MVC)

59. Issues 58  Stereoscopic images have different disparity  Coding artifacts Dealing with different channels  Transmission artifacts Image distortion on loss Temporal mismatch  Display artifacts autostereoscopic display problems (ghosting)

60. 3D media for mobile devices 59  Not only mobile devices, Internet connected devices  Standards for 3D video encoding are still developing Challenges for 2D video on mobile and more (depth)  3D Video conferencing demand real-time processing  Display technologies are evolving Resolution, need an adaptive and future ready format

61. 3D video 60  Fundamentals  3D display technologies  Basic processing  View synthesis  2D to 3D conversion  3D media for mobile devices  Outlook

62. Outlook 61  Today, 3D video technology are becoming practical  Content creation process have matured still a lot of room for improvement and extension  There is a huge commercial incentive in this field  A lot of areas to work on…

63. Outlook 62

64. Commentary 63  3D has been historically about generating revenue using mind tricks  Tricks are getting much better!  Immersive experience ≠ quality

65. Commentary 64  Future of 3D Video Consumers decide it Good for short media If all devices support it  Cinema products are profitable!

66. Commentary 65  Similar strategies could apply on different technology  Light-field camera – Lytro first product 2011 (2006)  Nokia / Pelican Imaging 16-lens array camera 2014

67. Thank you! 66

68. Video demo 67  http://www.youtube.com/watch?v=pEXSkRpL0dI http://www.youtube.com/watch?v=pEXSkRpL0dI