1. Advanced Computer Technology II FTV and 3DV KyungHee Univ. Master Course 2009311004 Kim Kyung Yong 10/10/2015

2. INDEX Introduction Definition of FTV Overview of FTV Technologies FTV Applications and Services Standardization Items for FTV Requirements of FTV video 3DV Introduction, Motivation Problem statement Proposed Technical Solution Reference 10/10/20152

3. Introduction Evolution of image systems 10/10/20153 Digital camera Digital cinema Digital TV FTV Photography Cinema TV Pixel-based system Ray-based system pastpresentfuture Digitalize More views More pixels  FTV : Free viewpoint TV

4. Definition of FTV FTV (Free viewpoint TV) –A new and very broad type of audio-visual system that allows each user to view the real 3D space from different user viewpoints.

5. Overview of FTV Technologies FTV System and FTV data format  DVB : Digital Video Broadcasting

6. Overview of FTV Technologies Configuration of FTV System 10/10/20156 AcquisitionCorrectionEncoding Transmission / Storage Decoding View Generation Display

7. Overview of FTV Technologies 1. Acquisition Linear arrangement 1D-arc arrangement Circular arrangement 2D-array arrangement

8. Overview of FTV Technologies 2. Correction Before After 1) Correction of Misalignment2) Color Correction Camera A Camera B - Before - After Camera A Camera B

9. Overview of FTV Technologies 3. Encoding and Decoding –The standardization is progressing with MVC in JVT. –This is work to do us.

10. Overview of FTV Technologies 4. View Generation –For the linear camera arrangement –For the circular camera arrangement

11. Overview of FTV Technologies 4. View Generation [Cont.] –Example of generated FTV images at various times and viewpoints

12. Overview of FTV Technologies 5. Display –2D display with head tracking

13. Overview of FTV Technologies 5. Display [Cont.] –3D display with and without head tracking

14. Overview of FTV Technologies 5. Display [Cont.] –3D displays with various numbers of views

15. Overview of FTV Technologies 5. Display [Cont.] –360 degree ray-reproducing 3D display (SeeLINDER)

16. FTV Applications and Services Applications of FTV 10/10/201516 FieldExamples communication/broadcastvery realistic, user-centered Entertainmentgame, sports, event, exhibition, Expo Advertisementnew type of presentation nature observationbird watching, safari, undersea park sightseeingbarrier-free museumnew type of exhibition art/contentcreation of new media art and digital content video productioncapturing and production can be separated unrealistic camera motions can be achieved archivespatio-temporal archive, living national treasures, traditional entertainment educationdistant education, educational tool medicineexamination, operation, education security, surveillancestore, factory, building, street, public facilities transportationtraffic monitoring at intersection, ITS

17. FTV Applications and Services System configurations of FTV Services 10/10/201517

18. Standardization Items for FTV Position of depth search and interpolation 10/10/201518

19. Standardization Items for FTV FTV reference model with items considered for standardization –FTV Data Format Multi-view video, camera parameters, depth data –Decoder –Interpolation 10/10/201519

20. Requirements of FTV video Requirements for Data Format on Visual information –Video data Video and audio data shall be supported in the data format. Video data may be corrected from captured images by real multiple cameras, so that misalignment of camera geometry and colors can be removed. –Depth maps Depth maps shall be supported in the data format. Depth maps may be generated by conversion from captured images by real multiple cameras or through special depth cameras or by other means. Types of depth information should be defined that are able to derive the original object distance. –Camera parameter Camera parameters, both extrinsic and intrinsic parameters, shall be supported in the data format. –Additional data Additional data could be necessary to make the system more efficient. E.g. data to represent transparency or specular reflection. 10/10/201520

21. Requirements of FTV video –Low complexity for editing The data format should allow for editing with low complexity. –Applicability The data format should be applicable for both natural and synthetic scenes. 10/10/201521

22. Requirements of FTV video Requirements for Compression on Depth –Compression efficiency Depth maps should be coded efficiently. –Synthesis accuracy The impact on the visual distortion on synthesized views should be minimized. 10/10/201522

23. Requirements of FTV video Requirements for Interpolation on Visual information –Control of rendering images The range of free viewpoint should be controlled. –Specification of rendering parameters Rendering control by specifying parameters in bit-stream should be supported. –Low complexity Conversion from FTV data to rendered data should be low complexity. –Display types Various capabilities of displays, e.g. display types, number of views should be supported. –Scalability Scalable rendering should be supported for real-time processing. 10/10/201523

24. Introduction, Motivation 3D Video –3D video and free viewpoint video are advanced types of visual media that expand user sensation. –The functionality of 3D video shall denote depth perception of the observed scenery, also known as stereo. –Free viewpoint video provides the user the functionality selecting an arbitrary viewpoint onto the observed scenery within practical limits. 10/10/201524

25. Problem statement It illustrates 3DV and FVV systems and applications that are not efficiently supported yet by available standards Conventional stereo displays only provide an unnatural 3D impression, since head motion parallax viewing can not be supported with 2 fixed views. 10/10/201525

26. Problem statement Auto-stereoscopic display –This is only possible, if the available input data support rendering a continuum of output views. 10/10/201526

27. Problem statement Holografika’s HV RC640 wide angle 3D display –The more views are used the wider the effect. 10/10/201527

28. Proposed Technical Solution Multi-view video plus depth (MVD) format[Cont.] –System 1. Very limited free viewpoint functionality is supported as view selection. 10/10/201528

29. Proposed Technical Solution Multi-view video plus depth (MVD) format[Cont.] –System 2. However, quality of virtual views decreases with distance from the available camera, due to dis-occlusion and other problems. Therefore single video plus depth is not suitable to support wide range free viewpoint navigation and wide angle 3D displays that require more than 2 views displayed at the same time. 10/10/201529

30. Proposed Technical Solution Multi-view video plus depth (MVD) format[Cont.] –Proposed System Multi-view video plus depth is a combination of the 2 formats illustrated in Fig. 3 and Fig. 4. 10/10/201530

31. Proposed Technical Solution Generation of MVD typically involves a number of sophisticated image processing algorithms. (M > N) 10/10/201531

32. Proposed Technical Solution Multi-view video plus depth (MVD) format[Cont.] 10/10/201532

33. Reference MPEG N9466 –Applications and Requirements on FTV MPEG M14417 –Proposal on Requirements for FTV JVT-W100 –Multi-view video plus depth (MVD) format for advanced 3D video systems 10/10/201533