1. Communication & Multimedia C. -Y. Tsai 2006/4/20 1 Multiview Video Compression Student: Chia-Yang Tsai Advisor: Prof. Hsueh-Ming Hang Institute of Electronics, NCTU

2. Communication & Multimedia C. -Y. Tsai 2006/4/20 2 Outline Introduction Introduction Coding methods Coding methods Performance Performance Conclusions Conclusions

3. Communication & Multimedia C. -Y. Tsai 2006/4/20 3 References A. Smolic, P. Kauff, “ Interactive 3-D video representation and coding technologies ”, Proceedings of the IEEE, vol. 93, no. 1, pp 98-110, Jan. 2005. ISO/IEC JTC1/SC29/WG11, “ Submissions received in CfP on Multiview Video Coding ”, MPEG Document M12969, Bangkok, Thailand, January 2006. ISO/IEC JTC1/SC29/WG11, “ Multiview video compression using V frames ”, MPEG Document M12828, Bangkok, Thailand, January 2006.

4. Communication & Multimedia C. -Y. Tsai 2006/4/20 4 Introduction

5. Communication & Multimedia C. -Y. Tsai 2006/4/20 5 Multivew= Multiple Viewpoints Applications of multiview Applications of multiview 3D video 3D video Free viewpoints selection Free viewpoints selection

6. Communication & Multimedia C. -Y. Tsai 2006/4/20 6 Multivew= Multiple Viewpoints Reasons for multiview compression Reasons for multiview compression PC is powerful enough PC is powerful enough Increasing network bandwidth Increasing network bandwidth Future 3D video Future 3D video

7. Communication & Multimedia C. -Y. Tsai 2006/4/20 7 MPEG Standarization Call for proposal (N7567, Oct. 2005) Call for proposal (N7567, Oct. 2005) Proposal competition (M12969, Jan. 2006) Proposal competition (M12969, Jan. 2006) NTT and Nagoya University NTT and Nagoya University Thomson and University of Southern California Thomson and University of Southern California KDDI Corp. KDDI Corp. ETRI and Sejong University (=M12871) ETRI and Sejong University (=M12871) MERL (=M12828) MERL (=M12828) KBS and Yonsei University (=M12874) KBS and Yonsei University (=M12874) Fraunhofer-HHI (=M12945) Fraunhofer-HHI (=M12945) Technical University of Berlin Technical University of Berlin

8. Communication & Multimedia C. -Y. Tsai 2006/4/20 8 Coding Methods Disparity compensated view prediction (DCVP) View synthesis prediction (VSP) Hierarchical B frames

9. Communication & Multimedia C. -Y. Tsai 2006/4/20 9 Multiview Frame Structure 1234567...... ….. time view

10. Communication & Multimedia C. -Y. Tsai 2006/4/20 10 Block diagram Predictions based on H.264/AVC JM95 Predictions based on H.264/AVC JM95

11. Communication & Multimedia C. -Y. Tsai 2006/4/20 11 Block diagram Decoder Decoder

12. Communication & Multimedia C. -Y. Tsai 2006/4/20 12 DCVP DCVP= Disparity Compensated View Prediction DCVP= Disparity Compensated View Prediction Problems Problems High spatial correlations between different viewpoints High spatial correlations between different viewpoints Solution Solution Prediction between viewpoints Prediction between viewpoints

13. Communication & Multimedia C. -Y. Tsai 2006/4/20 13 DCVP DCVP= Disparity Compensated View Prediction DCVP= Disparity Compensated View Prediction ….. IBBPBBIBBPBBI IBBPBBIBBPBBI IBBPBBIBBPBBI IBBPBBIBBPBBI IBBPBBIBBPBBI P B B

14. Communication & Multimedia C. -Y. Tsai 2006/4/20 14 VSP VSP= View Synthesis Prediction VSP= View Synthesis Prediction Problems Problems Different viewpoints have different depth Different viewpoints have different depth Rotation, translation speed Rotation, translation speed Solution Solution Synthesis virtual images before real prediction Synthesis virtual images before real prediction

15. Communication & Multimedia C. -Y. Tsai 2006/4/20 15 VSP time view View Synthesis Via View Interpolation View Synthesis Via View Warping R: Rotation matrix D: Depth information T: Translation matrix A: Intrinsic matrix C C’C’

16. Communication & Multimedia C. -Y. Tsai 2006/4/20 16 How to get depths? From camera record From camera record From well-known computer vision algorithms From well-known computer vision algorithms Block-based depth search Block-based depth search where || I [c,t,x,y] – I [c ’,t,x ’,y ’ ] || denotes the average error between the block at (x,y) in camera c at time t

17. Communication & Multimedia C. -Y. Tsai 2006/4/20 17 How to get depths? Depths map: Depths map: Left: computer vision algorithm Left: computer vision algorithm Right: block based depth search Right: block based depth search Compression result: Compression result: Depth information: 5-10% total bitrates Depth information: 5-10% total bitrates Left and right have equal performance Left and right have equal performance

18. Communication & Multimedia C. -Y. Tsai 2006/4/20 18 Prediction structure

19. Communication & Multimedia C. -Y. Tsai 2006/4/20 19 Hierarchical B pictures Hierarchical B pictures Hierarchical B pictures Fully compatible to AVC Main profile Fully compatible to AVC Main profile Non-dyadic decomposition is available Non-dyadic decomposition is available

20. Communication & Multimedia C. -Y. Tsai 2006/4/20 20 Hierarchical B pictures

21. Communication & Multimedia C. -Y. Tsai 2006/4/20 21 Performance

22. Communication & Multimedia C. -Y. Tsai 2006/4/20 22 Experiments

23. Communication & Multimedia C. -Y. Tsai 2006/4/20 23 Experiments

24. Communication & Multimedia C. -Y. Tsai 2006/4/20 24 Conclussion

25. Communication & Multimedia C. -Y. Tsai 2006/4/20 25 Conclussion DCVP & VSP can reduce the correlation between viewpoints DCVP & VSP can reduce the correlation between viewpoints Future work Future work Depth search algorithms Depth search algorithms Motion synthesis Motion synthesis MCTF MCTF Correlation between temporal and viewpoints axis Correlation between temporal and viewpoints axis Rate-control Rate-control Scalability Scalability Error protection Error protection Low delay issue Low delay issue

26. Communication & Multimedia C. -Y. Tsai 2006/4/20 26 Thanks for your attention! Any questions?