1. Speed up in feedback channel for a LDPCA base distributed video coding system on mobile device 在手機裝置上對低密度奇偶校驗碼為 基礎之分散式編碼中的回饋通道加速 Chen,chun-yuan 陳群元 Advisor: Prof. Wu, Ja-Ling 吳家麟 教授 2012/5/18

2. outline  Motivation and introduction  Mobile video trans  Traditional video codec  Distributed video codec  DVC architecture  Channel coding  Ldpca  The video communication system on Mobile device  Speed up DVC decoding time in feedback channel  Experiment Result  Conclusion

3. outline  Motivation and introduction  Mobile video trans  Traditional video codec  Distributed video codec  DVC architecture  Channel coding  Ldpca  The video communication system on Mobile device  Speed up DVC decoding time in feedback channel  Experiment Result  Conclusion

4. Digital Video Service  Video compression is an essential component of broadcast and entertainment media  Multimedia video everywhere! mobile video conference mobile cameras phone Video surveillance Wireless sensor network

6. Emerging application mobile cameras phone Wireless sensor network Video surveillance mobile video conference  Requiring low complexity and power-efficient encoder…

7. Emerging application  Conventional video coding (e.g. H.264/AVC, MPEG-2) - I nherent high complexity encoder, low complexity decoder  Requiring low complexity and power-efficient encoder…  Distributed video coding (DVC) - New video coding paradigm shifts complexity from encoder to decoder

8. outline  Motivation and introduction  Mobile video trans  Traditional video codec  Distributed video codec  DVC architecture  Channel coding  Ldpca  The video communication system on Mobile device  Speed up DVC decoding time in feedback channel  Experiment Result  Conclusion

9. Conventional Video Codec  MPEG-2, H.264, HEVC(H.265) ENCODER DECODERLightweight Heavyweight

10. Distributed Video Coding (DVC)  A new paradigm for video compression ENCODER DECODER Lightweight Heavyweight

11. Application of DVC  Video conferencing with mobile devices DVC to H.264 Transcoder Cloud Computational Resource DVC encoder (Low Complexity) H.264 decoder (Low Complexity) DVC encoded bitstream H.264 encoded bitstream

12. outline  Motivation and introduction  Mobile video trans  Traditional video codec  Distributed video codec  DVC architecture  Channel coding  Ldpca  The video communication system on Mobile device  Speed up DVC decoding time in feedback channel  Experiment Result  Conclusion

13. Distributed Video Coding R Y =H (Y) Source encoder Correlation is exploited by motion estimation R Y =H (X|Y) Source encoder H (X|Y)H (Y) -------Distributed-------  Slepian-Wolf Theorem (1973) Wyner-Ziv Theorem (1976) Channel encoder Source decoder Side Information creation Channel decoder X Y Virtual channel H (X, Y)

14. Distributed Video Coding D. Varodayan, A. Aaron, and B. Girod, “Rate-Adaptive Codes for Distributed Source Coding,”EURASIP Signal Processing Journal, Special Issue on Distributed Source Coding,,November 2006. Channel Encoder Channel Decoder LDPC Encoder LDPC Decoder

15. Distributed Video Coding D. Varodayan, A. Aaron, and B. Girod, “Rate-Adaptive Codes for Distributed Source Coding,”EURASIP Signal Processing Journal, Special Issue on Distributed Source Coding,,November 2006. Channel Encoder Channel Decoder LDPC Encoder LDPC Decoder Key frame Key frame WZ frame WZ frame WZ frame GOP size 4 Key frame Key frame WZ frame GOP size 2

16. Quantization  DCT coefficients bands DCT coefficient band b1 : { S 1 1, S 2 1, S 3 1, … S N 1 } DCT coefficient band b2 : { S 1 2, S 2 2, S 3 2, … S N 2 } DCT coefficient band b16 : { S 1 16, S 2 16, S 3 16, … S N 16 } … DC band AC bands Block1 S11S11 S12S12 S16S16 S17S17 S13S13 S15S15 S18S18 S 1 13 S14S14 S19S19 S 1 12 S 1 14 S 1 10 S 1 11 S 1 15 S 1 16 Block2 S21S21 S22S22 S26S26 S27S27 S23S23 S25S25 S28S28 S 2 13 S24S24 S29S29 S 2 12 S 2 14 S 2 10 S 2 11 S 2 15 S 2 16 Block3 S31S31 S32S32 S36S36 S37S37 S33S33 S35S35 S38S38 S 3 13 S34S34 S39S39 S 3 12 S 3 14 S 3 10 S 3 11 S 3 15 S 3 16 …

17. Distributed Video Coding D. Varodayan, A. Aaron, and B. Girod, “Rate-Adaptive Codes for Distributed Source Coding,”EURASIP Signal Processing Journal, Special Issue on Distributed Source Coding,,November 2006. Channel Encoder Channel Decoder LDPC Encoder LDPC Decoder

18. Quantization Q8 2^7 2^6 2^5 2^4 2^3 2^2 63bits for one block Q8 2^52^4 2^3 2^2 2^32^2 2^32^2 37bits for one block

19. Bit plane Extraction 0010000001 00000 11110 Bit planes of DC band: Bit plane 1: Bit plane 2: Bit plane 3: Bit plane 4: Bit plane 5: Independently Channel Encode (LDPCA) 46 7 06 3 1 7 7 30 1 5  For each DCT coefficient band… MSB LSB Zig-zag order

20. outline  Motivation and introduction  Mobile video trans  Traditional video codec  Distributed video codec  DVC architecture  Channel coding  Ldpca  The video communication system on Mobile device  Speed up DVC decoding time in feedback channel  Experiment Result  Conclusion

21. LDPC decoding Sum-Product Algorithm (Message Passing) Side Information (real number) +  0 -  1 467 甲乙丙 3521 decode output hard decision a b c d e f g a 25 b 25 c 25 d 25 e 25 f 25 g 25 Vertical processing Horizontal processing a 1 b 1 c 1 d 1 e 1 f 1 g 1 1 2 3 4 5 6 7 011011 011 From DVC encoder (syndrome bits) a b c d e f g Kschischang, F.R., Frey, B.J., and Loeliger, H.-A. 2001. Factor graphs and the sum-product algorithm. IEEE Trans. Inform. Theory

22. LDPC decoding Sum-Product Algorithm (Message Passing) Side Information (real number) +  0 -  1 467 甲乙丙 3521 decode output hard decision a b c d e f g a 25 b 25 c 25 d 25 e 25 f 25 g 25 Vertical processing Horizontal processing a 1 b 1 c 1 d 1 e 1 f 1 g 1 011 From DVC encoder (syndrome bits) Kschischang, F.R., Frey, B.J., and Loeliger, H.-A. 2001. Factor graphs and the sum-product algorithm. IEEE Trans. Inform. Theory

23. Sum-Product Algorithm Vertical Processing ABCDE F GIHJ KLOMN 011011 Z P a b c de f g P = K + F + a Z = F + P + a

24. Sum-Product Algorithm Horizontal Processing 011011 PQRST UVXWY ZADBC H a b c d e f g K

25. LDPC Accumulate (LDPCA) codes Rate adaptivity D. Varodayan et al., "Rate-adaptive codes for distributed source coding," EURASIP Signal Processing Journal, Special Section on Distributed Source Coding, 2006

26. 65 LDPC codes 1584 (Side information) 48~1584 (from encoder) 4752 edges 1584 3 99

27. 65 LDPC codes 3

28. 我最少需要 48bits 的 parity bits 1584bits 48 24 LdpcaEncode 我還需要更 多的 parity Bits… 我解完了一 個 bitplane 了 ! Buffer 依序解完所 有 bitplanes 24

29. outline  Motivation and introduction  Mobile video trans  Traditional video codec  Distributed video codec  DVC architecture  Channel coding  Ldpca  The video communication system on Mobile device  Speed up DVC decoding time in feedback channel  Experiment Result  Conclusion

30. Distributed Video Coding D. Varodayan, A. Aaron, and B. Girod, “Rate-Adaptive Codes for Distributed Source Coding,”EURASIP Signal Processing Journal, Special Issue on Distributed Source Coding,,November 2006. Channel Encoder Channel Decoder LDPC Encoder LDPC Decoder

31. Distributed Video Coding D. Varodayan, A. Aaron, and B. Girod, “Rate-Adaptive Codes for Distributed Source Coding,”EURASIP Signal Processing Journal, Special Issue on Distributed Source Coding,,November 2006. Channel Encoder Channel Decoder LDPC Encoder LDPC Decoder

32. Distributed Video Coding D. Varodayan, A. Aaron, and B. Girod, “Rate-Adaptive Codes for Distributed Source Coding,”EURASIP Signal Processing Journal, Special Issue on Distributed Source Coding,,November 2006. Channel Decoder LDPC Decoder Key frame request WZ Bit stream 1’th bitstream&CRC 63’th bitstream&CRC

33. Distributed Video Coding D. Varodayan, A. Aaron, and B. Girod, “Rate-Adaptive Codes for Distributed Source Coding,”EURASIP Signal Processing Journal, Special Issue on Distributed Source Coding,,November 2006. Channel Decoder LDPC Decoder Key frame request WZ Bit stream 1’th bitstream&CRC 63’th bitstream&CRC

34. outline  Motivation and introduction  Mobile video trans  Traditional video codec  Distributed video codec  DVC architecture  Channel coding  Ldpca  The video communication system on Mobile device  Speed up DVC decoding time in feedback channel  Experiment Result  Conclusion

35. Sequence nameSI genLDPCA DecFeedback channelEtc. foreman9.3123.3283.098.88 soccer6.0495.31313.678.68 coastguard14.7598.06310.5915.46 hall3.87109.89348.565.25

36. Time ratio

37. Amdahl's law

38.  Maximum speedup can be reached by improving the most critical part of the system  Feedback channel in the DVC coding. 29% 66.6%

39. Amdahl's law  Maximum speedup can be reached by improving the most critical part of the system  Feedback channel in the DVC coding. 95.6%

40. Amdahl's law  Maximum speedup can be reached by improving the most critical part of the system  Feedback channel in the DVC coding.  estimate syndromes size per bitplane 92.8% 1.7 X

41. Amdahl's law  Maximum speedup can be reached by improving the most critical part of the system  Feedback channel in the DVC coding.  estimate syndromes size per WZ frame 82.8% 4.6 X

42. Amdahl's law  Maximum speedup can be reached by improving the most critical part of the system  LDPC decoding in the DVC decoder. 86%~94% 29%~36% QCIF

43. Methods to speed up at feedback channel  So we propose two methods to decrease time consuming at feedback channel.  Estimate the syndromes size per WZ frame  Estimate the syndromes size per bitplane

44. Number of requests per bitplane per band DC AC1AC2 AC3 AC4AC5 AC6AC7AC8 AC9AC10 AC11 AC12 AC13 AC14 Bitplane Number

45. Number of requests per bitplane per band DC AC1AC2 AC3 AC4AC5 AC6AC7AC8 AC9AC10 AC11 AC12 AC13 AC14 Bitplane Number

46.  The number of errors increases as the bitplanes significance becomes lower since  the correlation between the corresponding bitplanes of corresponding DCT bands of the side information and the WZ frames becomes weaker.

47.  Moreover, the lower is the correlation, the higher  the amount of parity bits necessary for successful  decoding. For example, for the Soccer sequence,  the number of decoder requests is higher when  compared to the Coast Guard sequence since each  bitplane of the side information has a higher  number of errors when compared to the Soccer  sequence due to the lower quality side information.

48.  The highest number of requests for the highest  number of errors happens for the last bitplane of the  DC band in the Soccer sequence, where about 20  requests are needed to correct about 500 errors  (32% of errors). This is expected since the correla-  tion between the original frames and the correspond-  ing side information is lower for the least significant  bitplanes of the lower frequency coefficients; note  that these coefficients have more bitplanes to code  due to the smaller quantization bin size.

49. Estimate by previous bands  1.formula  2.time ratio

50. Number of requests per bitplane per band DC AC1AC2 AC3 AC4AC5 AC6AC7AC8 AC9AC10 AC11 AC12 AC13 AC14 Bitplane Number

51. Number of requests per bitplane per band DC AC1AC2 AC3 AC4AC5 AC6AC7AC8 AC9AC10 AC11 AC12 AC13 AC14 Bitplane Number

52. Number of requests per bitplane per band DC AC1AC2 AC3 AC4AC5 AC6AC7AC8 AC9AC10 AC11 AC12 AC13 AC14 Bitplane Number

53. 3.5 DC AC1AC2 AC3 AC4AC5AC6 AC7AC8AC9 AC10AC11AC12AC13AC14 Bitplane Number

55. formula  (ES acn bt ) WZn = ( S dc bt-1 ) WZn  WZn means the n’th WZ frame.  acn means the n’th AC band.  bt means the bitplane index of this AC band.  Which ES acn bt mean the estimated syndrome size for the bt’th bitplane in n’th AC band.  dc means the DC band.  S dc bt-1 means the syndromes bits for bt-1’th bitplane in DC band.

56. formula  (ES acn bt ) WZn = ( S dc bt-1 ) WZn –P(acn)  WZn means the n’th WZ frame.  acn means the n’th AC band.  bt means the bitplane index of this AC band.  Which ES acn bt mean the estimated syndrome size for the bt’th bitplane in n’th AC band.  dc means the DC band.  S dc bt-1 means the syndromes bits for bt-1’th bitplane in DC band.  P(acn) is the penalty function depends on correlation between DC band and the correspond AC band

57. Speed up ratio  406.39->237.87=1.708 times

58. Amdahl's law  Maximum speedup can be reached by improving the most critical part of the system  Feedback channel in the DVC coding. 95.6%

59. Amdahl's law  Maximum speedup can be reached by improving the most critical part of the system  Feedback channel in the DVC coding. 92.8% 1.7 X

60. Estimate by reference frame  1.formula  2.time ratio

61. formula  (ES acn bt ) WZn = ( S dc bt-1 ) WZn  WZn means the n’th WZ frame.  acn means the n’th AC band.  bt means the bitplane index of this AC band.  Which ES acn bt mean the estimated syndrome size for the bt’th bitplane in n’th AC band.  dc means the DC band.  S dc bt-1 means the syndromes bits for bt-1’th bitplane in DC band.

62. formula  (ES bt ) WZn =(S bt ) WZ(n-GOPsize)  WZn means the WZ frame’s index  n-GOPsize means the correspond WZ frame in the previous GOP.  bt is the bitplane index.  (ES bt ) WZn means the estimated syndromes bit for the bt’th bitplane of WZn frame.  (S bt ) WZ(n-GOPsize) means the syndromes

63. formula  (ES bt ) WZn =(S bt ) WZ(n-GOPsize) -P(Residual GOPnum, Residual GOPnum-1 )  WZn means the WZ frame’s index  n-GOPsize means the correspond WZ frame in the previous GOP.  bt is the bitplane index.  (ES bt ) WZn means the estimated syndromes bit for the bt’th bitplane of WZn frame.  (S bt ) WZ(n-GOPsize) means the syndromes

64. Formula(cont.)  P(Residual GOPnum, Residual GOPnum-1 )  Mean the penalty function depends on the residual of this and previous GOP, which Residual is evaluated by the residual value of the middle frame of one GOP.

65. Time ratio  406.39->70.36=5.77 times

66. Amdahl's law  Maximum speedup can be reached by improving the most critical part of the system  Feedback channel in the DVC coding. 95.6%

67. Amdahl's law  Maximum speedup can be reached by improving the most critical part of the system  Feedback channel in the DVC coding. 82.8% 4.6 X

68. outline  Motivation and introduction  Mobile video trans  Traditional video codec  Distributed video codec  DVC architecture  Channel coding  Ldpca  The video communication system on Mobile device  Speed up DVC decoding time in feedback channel  Experiment Result  Conclusion

69. Test condition  12 CPU, 24 processor  Intel(R) Xeon(R) CPU X5650 @ 2.67GHz  GPU: Tesla M2050  Mobile device: HTC sensation

70. DVC encoder on mobile  14 s for foreman sequence, GOP8, intra mode on.  4 s for foreman sequence, GOP8, intra mode off.

71.  Test sequences :  QCIF, 15Hz, all frames  GOP Size 2, 4 and 8  Only luminance component is used Test materials SoccerForemanCoastguardHall Monitor Motion: High Low

72. Speed up

73. Bitrate

74. RD curve  Foreman with LRSS,GOP8

75. RD curve  Hall monitor with LRSS,GOP8

76. Ref motion  Adjust the estimated syndromes bit depends on motion between KEY frames.  It will not cause a lot computing time and network loading.

77. 統計關係圖

78. outline  Motivation and introduction  Mobile video trans  Traditional video codec  Distributed video codec  DVC architecture  Channel coding  Ldpca  The video communication system on Mobile device  Speed up DVC decoding time in feedback channel  Experiment Result  Conclusion

79. conclusion  A practical DVC coding system is built on mobile devices.  DVC encoder and H.264 decoder keep the mobiles light weight.  We propose two method to reduce transmission time spent in feedback channel.  Estimate the syndromes size per WZ frame  Estimate the syndromes size per bitplane

80. 失敗 + 統計關係圖

81. 4 DC AC1AC2 AC3 AC4AC5AC6 AC7AC8AC9 AC10AC11AC12AC13AC14 Bitplane Number

83. Thank you