1. Distributed Video System realized on mobile device with efficient Feedback channel 分散式影像編碼在手機上的實現與有效率 的回饋通道 1 Chen, chun-yuan 陳群元 Advisor: Prof. Wu, Ja-Ling 吳家麟 教授 2012/7/9

2. Outline  Motivation and introduction  DVC architecture overview  DVC architecture with mobile device  Proposed algorithm for efficient feedback channel  Experimental Results  Conclusion  Future work 2

3. Motivation 3  Video communication is essential in Mobile device  Decrease the coding time and battery consuming on mobile.

4. Introduction  For Mobile device video codec  Conventional video coding (ex:MPEG-4 H.264)  Heavy weight encoder, light weight decoder.  Distributed Video Coding  Light weight encoder, heavy weight decoder. 4

5. Conventional video codec 5

6. Tse-Chung Su, Yun-Chung Shen, and Ja-Ling Wu, “Real-time decoding for LDPC based distributed video coding,” MM '11 Proceedings of the 19th ACM international conference on Multimedia,, 2011. DVC to H.264 Transcoder 6 Clouding server: DVC to H.264 transcoder DVC encoder H.264 decoder

7.  DVC codec baseline:  DISPAC+DVC transcoder 7 Tse-Chung Su. Yun-Chung Shen. and Ja-Ling Wu. 2011. Real-time Decoding for LDPC Based Distributed Video Coding. National Taiwan University

8. Time complexity analysis without feedback channel  DVC Decoding without Feedback channel  Side-Information generation  LDPCA 8 Tse-Chung Su, Yun-Chung Shen, and Ja-Ling Wu, “Real-time decoding for LDPC based distributed video coding,” MM '11 Proceedings of the 19th ACM international conference on Multimedia,, 2011.

9. Time complexity analysis with feedback channel  DVC Decoding  Side-Information generation  Send-Request at Feedback channel  LDPCA 9 Bottle neck! T. V. Lakshman and U. Madhow, “The performance of TCP/IP for networks with high bandwidth-delay products and random loss, IEEE/ACM Trans. Networking, vol. 6, pp. 336–350, June 1997.

10. Outline  Motivation and introduction  DVC architecture Overview  DVC architecture with mobile device  Proposed algorithm for efficient feedback channel  Experimental Results  Conclusion  Future work 10

11. Distributed Video Codec 11 DCT WZ Frames LDPCA Encoder Buffer LDPCA Decoder Key Frames Conventional Intraframe Encoder Conventional Intraframe Decoder SI generation& refinement DCT SI Reconstruction Mode selection CRC CRC check CRC-8 WZ bitstream Correlation Noise Modeling IDCT& Deblocking filter Decoded WZ Frames Decoded Key Frames WZ Frame WZ Frame WZ Frame GOP size 4 Key Frame Key Frame

12. LDPC decoding 101 syndrome bits Kschischang, F.R., Frey, B.J., and Loeliger, H.-A. 2001. Factor graphs and the sum-product algorithm. IEEE Trans. Inform. Theory a bc 1 2 34 56 7 Vertical processing Horizontal processing a b c d e f g a 1 b 1 c 1 d 1 e 1 f 1 g 1 a 25 b 25 c 25 d 25 e 25 f 25 g 25 Side Information (real number) +  0 -  1 decode output hard decision

13. LDPC Accumulate (LDPCA) codes 13 Rate adaptivity D. Varodayan et al., "Rate-adaptive codes for distributed source coding," EURASIP Signal Processing Journal, Special Section on Distributed Source Coding, 2006

14. 65 LDPC codes

15. Outline  Motivation and introduction  DVC architecture overview  DVC architecture with mobile device  Proposed algorithm for efficient feedback channel  Experimental Results  Conclusion  Future work 15

16. Distributed Video Codec with mobile 16 DCT WZ Frames LDPCA Encoder Buffer LDPCA Decoder Key Frames Conventional Intraframe Encoder Conventional Intraframe Decoder SI generation& refinement DCT SI Reconstruction Mode selection CRC CRC check CRC-8 WZ bitstream Correlation Noise Modeling IDCT& Deblocking filter Decoded WZ Frames Decoded Key Frames Transcode Decoded Video sequence With specific form Video deliver

17. Distributed Video Codec with mobile 17 DCT WZ Frames LDPCA Encoder Buffer LDPCA Decoder Key Frames Conventional Intraframe Encoder Conventional Intraframe Decoder SI generation& refinement DCT SI Reconstruction Mode selection CRC CRC check CRC-8 WZ bitstream Correlation Noise Modeling IDCT& Deblocking filter Decoded WZ Frames Decoded Key Frames Transcode Decoded Video sequence With specific form Video deliver

18. Key frame encoding on Mobile  Previous ver. use JM9.5  Transcoder ref x264 on PC 18

19. Distributed Video Codec with mobile 19 DCT WZ Frames LDPCA Encoder Buffer LDPCA Decoder Key Frames Conventional Intraframe Encoder Conventional Intraframe Decoder SI generation& refinement DCT SI Reconstruction Mode selection CRC CRC check CRC-8 WZ bitstream Correlation Noise Modeling IDCT& Deblocking filter Decoded WZ Frames Decoded Key Frames Transcode Decoded Video sequence With specific form Video deliver

20. WZ coding with NDK 20

21. Distributed Video Codec with mobile 21 DCT WZ Frames LDPCA Encoder Buffer LDPCA Decoder Key Frames Conventional Intraframe Encoder Conventional Intraframe Decoder SI generation& refinement DCT SI Reconstruction Mode selection CRC CRC check CRC-8 WZ bitstream Correlation Noise Modeling IDCT& Deblocking filter Decoded WZ Frames Decoded Key Frames Transcode Decoded Video sequence With specific form Video deliver

22. Network connection  Create network connection to connect remote server.  TCP socket in C 22 Clouding server: DVC to H.264 transcoder

23. ACK at Feedback channel 23 … syndromes

24. Decoding complexity on Feedback channel  Packet header occupy large part of network bandwidth.  48 bits syndromes vs. 24 bytes header  Network latency in communication at Feedback channel  About 20 thousands times requests for 10 sec sequence. 24

25. Outline  Motivation and introduction  DVC architecture with mobile device  DVC architecture overview  Proposed algorithm for efficient feedback channel  Experimental Results  Conclusion  Future work 25

26. Efficient feedback channel  we propose two methods to decrease time consuming at feedback channel.  Estimate the syndrome size per bitplane  Estimate the syndrome size per WZ frame 26

27. Number of requests per bitplane per band DC AC1AC2 AC3 AC4AC5 AC6AC7AC8 AC9AC10 AC11 AC12 AC13 AC14 Bitplane Number

28. Number of requests per bitplane per band DC AC1AC2 AC3 AC4AC5 AC6AC7AC8 AC9AC10 AC11 AC12 AC13 AC14 Bitplane Number

29. Number of requests per bitplane per band DC AC1AC2 AC3 AC4AC5 AC6AC7AC8 AC9AC10 AC11 AC12 AC13 AC14 Bitplane Number

30. Efficient feedback channel  we propose two methods to decrease time consuming at feedback channel.  Estimate the syndrome size per bitplane  Estimate the syndrome size per WZ frame 30

31. Average number of requests per bitplane DC AC1AC2 AC3 AC4AC5 AC6AC7AC8 AC9AC10 AC11 AC12 AC13 AC14 Bitplane Number

32. Estimate the syndromes size per WZ frame  Estimate syndroms size by correspond WZ frame in previous GOP  Ex. GOP 4 32 Key frame WZ frame WZ frame WZ frame First GOP Key frame Key frame WZ frame WZ frame WZ frame Second GOP

33. Outline  Motivation and introduction  DVC architecture overview  DVC architecture with mobile device  Proposed algorithm for efficient feedback channel  Experimental Results  Conclusion  Future work 33

34. Test condition  12 CPU, 24 processor  Intel(R) Xeon(R) CPU X5650 @ 2.67GHz  GPU: Tesla M2050  Mobile device: HTC sensation

35.  Test sequences :  QCIF, 15Hz, all frames  GOP Size 2, 4 and 8  Only luminance component is used Test materials SoccerForemanCoastguardHall Monitor Motion: High Low

36. Decoding time Method\videoforemansoccercoastguardhall With feedback channel 446.83 sec419.77 sec482.40 sec417.29 sec Estimate per bitplane 237.56 sec 271.61 sec 305.15 sec Estimate per WZ frame 104.66 sec73.28 sec109.27 sec78.96 sec 36

37. Request amount 37

38. Speed up ratio 38

39. Bitrate 39

40. RD(Foreman) 40

41. RD(hall) 41

42. Outline  Motivation and introduction  DVC architecture overview  DVC architecture with mobile device  Proposed algorithm for efficient feedback channel  Experimental Results  Conclusion  Future work 42

43. Conclusion  We build a realistic DVC system based on mobile and cloud.  Propose algorithm for Feedback channel which the previous work doesn’t discuss or improve.  Transcode the decoded sequence to mobile codec. 43

44. Outline  Motivation and introduction  DVC architecture overview  DVC architecture with mobile device  Proposed algorithm for efficient feedback channel  Experimental Results  Conclusion  Future work 44

45. Future Work  Take a more precise estimation of syndrome size.  Use motion information to estimate the syndrome size.  Build a reliable DVC codec which is loss tolerant. 45

46. 46 Thank You

47. streaming 47 WZ Frame WZ Frame WZ Frame GOP size 4 Key Frame Key Frame WZ Frame WZ Frame WZ Frame GOP size 4 Key Frame Key Frame

48. 48 WZ Frame WZ Frame WZ Frame GOP size 4 Key Frame Key Frame WZ Frame WZ Frame WZ Frame GOP size 4 Key Frame Key Frame

49. 統計關係圖

50. To do  Real time decoder  Efficient intra coding  Print MSG on Mobile 50

51. DVC decoding time  Side Info Creation: 0.85 sec  Overcomplete Transform: 2.35 sec  Correlation Noise Modeling: 2.33 sec  Conditional Bit Prob Compute: 0.60 sec  Update Side Info: 7.14 sec  Ldpca Decode: 123.34 sec  Motion Learning: 1.79 sec  Others: 1.80 sec 51

52. Distributed Video Coding D. Varodayan, A. Aaron, and B. Girod, “Rate-Adaptive Codes for Distributed Source Coding,”EURASIP Signal Processing Journal, Special Issue on Distributed Source Coding,,November 2006. Channel Encoder Channel Decoder LDPC Encoder LDPC Decoder

53. Distributed Video Codec 53 DCT WZ Frames Extract bit-plane bit-plane 1 LDPCA Encoder Buffer bit-plane 2 LDPCA Decoder Request bits Key Frames Conventional Intraframe Encoder Conventional Intraframe Encoder Interpolation/ Extrapolation DCT Reconstruction

54. Distributed Video Codec 54 DCT WZ Frames LDPCA Encoder Buffer LDPCA Decoder Request bits Key Frames Conventional Intraframe Encoder Conventional Intraframe Encoder Interpolation/ Extrapolation DCT Reconstruction

55. Distributed Video Codec 55 DCT WZ Frames LDPCA Encoder Buffer LDPCA Decoder Key Frames Conventional Intraframe Encoder Conventional Intraframe Encoder SI generation& refinement DCT SI Reconstruction Mode selection CRC CRC check CRC-8 WZ bitstream Correlation Noise Modeling IDCT& Deblocking filter Decoded WZ Frames Decoded Key Frames

56. Distributed Video Codec with mobile 56 DCT WZ Frames LDPCA Encoder Buffer LDPCA Decoder Key Frames Conventional Intraframe Encoder Conventional Intraframe Encoder SI generation& refinement DCT SI Reconstruction Mode selection CRC CRC check CRC-8 WZ bitstream Correlation Noise Modeling IDCT& Deblocking filter Decoded WZ Frames Decoded Key Frames Transcode Decoded Video sequence With specific form Video deliver

57. 57 Video recode With mobile camera

58. ACK at Feedback channel 58 … LDPCA Encoder &CRC LDPCA Decoder &CRC Syndromes for one bitplane

59. ACK at Feedback channel 59 … LDPCA Encoder &CRC LDPCA Decoder &CRC Syndromes for one bitplane Decoding one bitplane with received syndromes And chack CRC

60. ACK at Feedback channel 60 … LDPCA Encoder &CRC LDPCA Decoder &CRC Syndromes for one bitplane

61. 61

62. 我最少需 要 48bits 的 parity bits 1584bits 48 24 LdpcaEncode 我還需要 更多的 parity Bits… 我解完了 一個 bitplane 了 ! Buffer 依序解完 所有 bitplanes 24

63. DVC to H.264 Transcoder 63 Clouding server: DVC to H.264 transcoder DVC encoder H.264 decoder

64.  接著講 header overhead  每次都傳一點，等於傳的大部分都是 header(20Bytes)  傳輸太多次，會造成大量的 delay 64

65. Contribution  Realize DVC encoding part on Mobile device  Implement Feedback channel  Implement encoder’s component with x264  Propose an efficient Feedback channel  Transcode the decoded sequence to mobile codec 65

66. Outline  Motivation and introduction  DVC architecture with mobile device  Propose an efficient feedback channel  Transcoding  Experiment Result  Conclusion  Future work 66

67. Time without feedback channel  Total decoding time: 17.10 sec  Average WZ decoding speed: 7.60 fps  Time information of each component:  Side Info Creation: 0.47 sec  Overcomplete Transform: 0.78 sec  Correlation Noise Modeling: 4.20 sec  Conditional Bit Prob Compute: 0.82 sec  Update Side Info: 3.78 sec  Motion Learning: 1.25 sec  Others: 0.42 sec  Ldpca Decode: 5.36 sec  SI:5.5 67

68. Time with feedback channel  Total decoding time: 330.51 sec  Average WZ decoding speed: 0.39 fps  Time information of each component:  Side Info Creation: 0.57 sec  Overcomplete Transform: 1.03 sec  Correlation Noise Modeling: 2.78 sec  Conditional Bit Prob Compute: 0.75 sec  Update Side Info: 2.54 sec  Motion Learning: 1.01 sec  Others: 0.70 sec  Ldpca Decode: 321.13 sec 68

69. Time without feedback channel  Total decoding time: 17.10 sec  Ldpca Decode: 5.36 sec  SI:5.5 69

70. Time with feedback channel  Total decoding time: 330.51 sec  Ldpca Decode(+network): 321.13 sec  Ldpca Decode: 5.36 sec  SI:5.5  Etc : 3.88 70

71. Quantization Q8 2^7 2^6 2^5 2^4 2^3 2^2 63bits for one block Q5 2^52^4 2^3 2^2 2^32^2 2^32^2 37bits for one block

72. Distributed Video Codec 72 DCT WZ Frames LDPCA Encoder Buffer LDPCA Decoder Key Frames Conventional Intraframe Encoder Conventional Intraframe Encoder SI generation& refinement DCT SI Reconstruction Mode selection CRC CRC check CRC-8 WZ bitstream Correlation Noise Modeling IDCT& Deblocking filter Decoded WZ Frames Decoded Key Frames

73. 73

74. Distributed Video Codec with mobile 74 DCT WZ Frames LDPCA Encoder Buffer LDPCA Decoder Key Frames Conventional Intraframe Encoder Conventional Intraframe Encoder SI generation& refinement DCT SI Reconstruction Mode selection CRC CRC check CRC-8 WZ bitstream Correlation Noise Modeling IDCT& Deblocking filter Decoded WZ Frames Decoded Key Frames Transcode Decoded Video sequence With specific form Video deliver

75. Distributed Video Codec with mobile 75 DCT WZ Frames LDPCA Encoder Buffer LDPCA Decoder Key Frames Conventional Intraframe Encoder Conventional Intraframe Encoder SI generation& refinement DCT SI Reconstruction Mode selection CRC CRC check CRC-8 WZ bitstream Correlation Noise Modeling IDCT& Deblocking filter Decoded WZ Frames Decoded Key Frames Transcode Decoded Video sequence With specific form Video deliver

76. DVC Encoder Realization  Implement DVC encoder on Mobile  Media recording  Key frame encoding  WZ frame encoding  Feedback channel transmission  Network connection between mobile and server  Transcode the result sequence 76

77. methodSyndromes need (bits) Request number Cover rate(bitplanes num=130*63= 8190) Total decoding time(speed up) LDPCA time(speed up) BitRate PSNR naive25756082575608(+13 0*63* 不算 server 來的 request 只算 mobile 傳送 syndrome) 13hr=4680013hr(with feedback channel) 450.32 kbps 39.29 rearrangeme nt 257560823983406.20 sec386.90 sec450.32 kbps estimator 371534436086734/8190=0.8 22 87.99 sec(4.61) 70.36 sec(5.498) 564.29 kbps parallel 2825256 15736( 可以不 用再送，只靠 pre-send 的 bitplane 數 =3249) 3249/8190=0.3 96 237.56 sec(1.709) 219.14 sec(1.765) 475.29 kbps

78. methodSyndromes need (bits) Request number Cover rate(bitplanes num=130*63= 8190) Total decoding time(speed up) LDPCA time(speed up) BitRate PSNR naive25756082575608(+13 0*63* 不算 server 來的 request 只算 mobile 傳送 syndrome) 13hr=4680013hr(with feedback channel) 450.32 kbps 39.06 rearrangeme nt 192237620416448.61 sec413.36 sec485.96 kbps estimator 390225624197291/8190=0.8 9 100.20 sec72.27 sec(5.719) 683.95 kbps parallel 2825256 ?( 卡在 frame2) 15736( 可以不 用再送，只靠 pre-send 的 bitplane 數 =3249) 3249/8190=0.3 96 237.56 sec 219.14 sec(1.886) 475.29 kbps

79. methodSyndromes need (bits) Request number Cover rate(bitplanes num=130*63= 8190) Total decoding time(speed up) LDPCA time(speed up) BitRate PSNR naive25756082575608(+13 0*63* 不算 server 來的 request 只算 mobile 傳送 syndrome) 13hr=4680013hr(with feedback channel) 450.32 kbps rearrangeme nt 358665628721431.45 sec395.96 sec479.39 kbps estimator 3715344 ?(60 後死掉..) 36086734/8190=0.8 22 87.99 sec 70.36 sec(5.627) 564.29 kbps parallel3719712 21512( 可以不 用再送，只靠 pre-send 的 bitplane 數 =3249) 2572/8190=0.3 14 271.61 sec 244.18 sec(1.621) 492.70 kbps

80. methodSyndromes need (bits) Request number Cover rate(bitplanes num=142*63= 8946) Total decoding time(speed up) LDPCA time(speed up) BitRate PSNR naive25756082575608(+13 0*63* 不算 server 來的 request 只算 mobile 傳送 syndrome) 13hr=4680013hr(with feedback channel) 450.32 kbps 39.47 rearrangeme nt 92469617440505.84 sec477.45 sec153.45 kbps estimator 138573618527823/8946=0.8 74 109.75 sec93.24 sec(5.12) 195.62 kbps parallel971376 10568( 可以不 用再送，只靠 pre-send 的 bitplane 數 =3249) 3367/8946=0.3 76 305.15 sec 285.59 sec(1.671) 157.72 kbps

81. formula  (ES bt ) WZn =(S bt ) WZ(n-GOPsize)  WZn means the WZ frame’s index  n-GOPsize means the correspond WZ frame in the previous GOP.  bt is the bitplane index.  (ES bt ) WZn means the estimated syndromes bit for the bt’th bitplane of WZn frame.  (S bt ) WZ(n-GOPsize) means the syndromes bit for the bt’th bitplane of WZ(n-GOPsize) frame

82. formula  (ES acn bt ) WZn = ( S dc bt-1 ) WZn  WZn means the n’th WZ frame.  acn means the n’th AC band.  bt means the bitplane index of this AC band.  Which ES acn bt mean the estimated syndrome size for the bt’th bitplane in n’th AC band.  dc means the DC band.  S dc bt-1 means the syndromes bits for bt-1’th bitplane in DC band.

83. Time complexity 83

84. 84 Yu-Shan Pai, Han-Ping Cheng, Yun-Chung Shen,and Ja-Ling Wu, “Fast decoding for LDPC based distributed video coding,” MM '11 Proceedings of the 19th ACM international conference on Multimedia,, 2010.

85. Transcode to mobile codec  Because DVC only encode luminance part in YUV, so we got only Y for output.  Our mobile device can only read mp4 and 3gp. Y UV R G B Frame 1 Frame 2 Frame 3 Frame4 Frame 5 Frame 6 Frame 7 Frame n.mp4

86. Correlation with LDPCA 86

87. 統計關係圖