Distributed Video System realized on mobile device with efficient Feedback channel 分散式影像編碼在手機上的實現與有效率 的回饋通道 1 Chen, chun-yuan 陳群元 Advisor: Prof. Wu, Ja-Ling 吳家麟 教授 2012/6/28

Outline  Motivation and introduction  DVC architecture overview  DVC architecture with mobile device  Propose an efficient feedback channel  Transcoding  Experiment Result  Conclusion  Future work 2

Motivation 3  Video communication is essential in Mobile device  Decrease the coding time and battery consuming on mobile.

Introduction  For Mobile device video codec  Conventional video coding (ex:MPEG-4 H.264)  Heavy weight encoder, light weight decoder.  Distributed Video Coding  Light weight encoder, heavy weight decoder. 4

Conventional video codec 5

DVC to H.264 Transcoder 6 Clouding server: DVC to H.264 transcoder DVC encoder H.264 decoder

 DVC codec baseline:  DISPAC+DVC transcoder 7 Tse-Chung Su. Yun-Chung Shen. and Ja-Ling Wu Real-time Decoding for LDPC Based Distributed Video Coding. National Taiwan University

Time complexity analysis without feedback channel  DVC Decoding without Feedback channel  Side-Information generation  LDPCA 8

Time complexity analysis with feedback channel  DVC Decoding  Side-Information generation  Send-Request at Feedback channel  LDPCA 9 Bottle neck!

Outline  Motivation and introduction  DVC architecture Overview  DVC architecture with mobile device  Propose an efficient feedback channel  Transcoding  Experiment Result  Conclusion  Future work 10

Distributed Video Codec 11 DCT WZ Frames LDPCA Encoder Buffer LDPCA Decoder Key Frames Conventional Intraframe Encoder Conventional Intraframe Encoder SI generation& refinement DCT SI Reconstruction Mode selection CRC CRC check CRC-8 WZ bitstream Correlation Noise Modeling IDCT& Deblocking filter Decoded WZ Frames Decoded Key Frames WZ Frame WZ Frame WZ Frame GOP size 4 Key Frame Key Frame

Outline  Motivation and introduction  DVC architecture overview  DVC architecture with mobile device  Propose an efficient feedback channel  Transcoding  Experiment Result  Conclusion  Future work 12

Distributed Video Codec with mobile 13 DCT WZ Frames LDPCA Encoder Buffer LDPCA Decoder Key Frames Conventional Intraframe Encoder Conventional Intraframe Encoder SI generation& refinement DCT SI Reconstruction Mode selection CRC CRC check CRC-8 WZ bitstream Correlation Noise Modeling IDCT& Deblocking filter Decoded WZ Frames Decoded Key Frames Transcode Decoded Video sequence With specific form Video deliver

DVC Encoder Realization  Implement DVC encoder on Mobile  Key frame encoding  WZ frame encoding  Efficient intra mode realization  Feedback channel transmission  Network connection between mobile and server  Determine payload of a packet with LDPCA protocal 14

Key frame encoding on Mobile  Previous ver. use JM9.5  Transcoder ref x.264 on PC 15

DVC Encoder Realization  Implement DVC encoder on Mobile  Key frame encoding  WZ frame encoding  Efficient intra mode realization  Feedback channel transmission  Network connection between mobile and server  Determine payload of a packet with LDPCA protocal 16

WZ coding with NDK 17

DVC Encoder Realization  Implement DVC encoder on Mobile  Key frame encoding  WZ frame encoding  Efficient intra mode realization  Feedback channel transmission  Network connection between mobile and server  Determine payload of a packet with LDPCA protocal 18

Efficient intra block coding  Previous ver. Transcoder Use JM code.  Intra block encoding consumes lots of time in encoding. 19 Clouding server: DVC to H.264 transcoder

DVC Encoder Realization  Implement DVC encoder on Mobile  Key frame encoding  WZ frame encoding  Efficient intra mode realization  Feedback channel transmission  Network connection between mobile and server  Determine payload of a packet with LDPCA protocal 20

Network connection  Create network connection to connect remote server.  TCP socket in C 21 Clouding server: DVC to H.264 transcoder

DVC Encoder Realization  Implement DVC encoder on Mobile  Key frame encoding  WZ frame encoding  Efficient intra mode realization  Feedback channel transmission  Network connection between mobile and server  Determine payload of a packet with LDPCA protocal 22

ACK at Feedback channel 23 … syndromes

Consuming time at Feedback channel  Packet header occupy large part of network bandwidth.  Network latency in communication at Feedback channel 24

Outline  Motivation and introduction  DVC architecture with mobile device  DVC architecture overview  Propose an efficient feedback channel  Transcoding  Experiment Result  Conclusion  Future work 25

Efficient feedback channel  we propose two methods to decrease time consuming at feedback channel.  Estimate the syndromes size per WZ frame  Estimate the syndromes size per bitplane 26

Average number of requests per bitplane DC AC1AC2 AC3 AC4AC5 AC6AC7AC8 AC9AC10 AC11 AC12 AC13 AC14 Bitplane Number

Estimate the syndromes size per WZ frame  Estimate syndromes size by correspond WZ frame in previous GOP  Ex. GOP 4 28 Key frame Key frame WZ frame WZ frame WZ frame First GOP Key frame Key frame WZ frame WZ frame WZ frame Second GOP

formula  (ES bt ) WZn =(S bt ) WZ(n-GOPsize)  WZn means the WZ frame’s index  n-GOPsize means the correspond WZ frame in the previous GOP.  bt is the bitplane index.  (ES bt ) WZn means the estimated syndromes bit for the bt’th bitplane of WZn frame.  (S bt ) WZ(n-GOPsize) means the syndromes bit for the bt’th bitplane of WZ(n-GOPsize) frame

Efficient feedback channel  we propose two methods to decrease time consuming at feedback channel.  Estimate the syndromes size per WZ frame  Estimate the syndromes size per bitplane 30

Number of requests per bitplane per band DC AC1AC2 AC3 AC4AC5 AC6AC7AC8 AC9AC10 AC11 AC12 AC13 AC14 Bitplane Number

Number of requests per bitplane per band DC AC1AC2 AC3 AC4AC5 AC6AC7AC8 AC9AC10 AC11 AC12 AC13 AC14 Bitplane Number

Estimate by previous bands  1.formula  2.time ratio

Number of requests per bitplane per band DC AC1AC2 AC3 AC4AC5 AC6AC7AC8 AC9AC10 AC11 AC12 AC13 AC14 Bitplane Number

formula  (ES acn bt ) WZn = ( S dc bt-1 ) WZn  WZn means the n’th WZ frame.  acn means the n’th AC band.  bt means the bitplane index of this AC band.  Which ES acn bt mean the estimated syndrome size for the bt’th bitplane in n’th AC band.  dc means the DC band.  S dc bt-1 means the syndromes bits for bt-1’th bitplane in DC band.

Outline  Motivation and introduction  DVC architecture overview  DVC architecture with mobile device  Propose an efficient feedback channel  Transcoding  Experiment Result  Conclusion  Future work 36

Transcode to mobile codec  Because DVC only encode luminance part in YUV, so we got only Y for output.  Our mobile device can only read mp4 and 3gp. Y UV R G B Frame 1 Frame 2 Frame 3 Frame4 Frame 5 Frame 6 Frame 7 Frame n.mp4

Outline  Motivation and introduction  DVC architecture overview  DVC architecture with mobile device  Propose an efficient feedback channel  Transcoding  Experiment Result  Conclusion  Future work 38

Experiment Result  規格  實驗數據 :  時間 (&decode) 、 packet 數量 ( 傳幾次 )+ 39

Test condition  12 CPU, 24 processor  Intel(R) Xeon(R) CPU 2.67GHz  GPU: Tesla M2050  Mobile device: HTC sensation

 Test sequences :  QCIF, 15Hz, all frames  GOP Size 2, 4 and 8  Only luminance component is used Test materials SoccerForemanCoastguardHall Monitor Motion: High Low

Speed up

Bitrate

RD curve  Foreman with LRSS,GOP8

RD curve  Hall monitor with LRSS,GOP8

Outline  Motivation and introduction  DVC architecture overview  DVC architecture with mobile device  Propose an efficient feedback channel  Transcoding  Experiment Result  Conclusion  Future work 46

Conclusion  We combine DISPAC and transcoder, then port it’s encoder on mobile.  Implement and propose Feedback channel.  Record video with mobile device’s camera.  Transcode the decoded sequence to mobile codec. 47

Outline  Motivation and introduction  DVC architecture overview  DVC architecture with mobile device  Propose an efficient feedback channel  Transcoding  Experiment Result  Conclusion  Future work 48

Future Work  Take a more precise estimation of syndrome size.  Use motion information to estimate the syndrome size.  Build a reliable DVC codec which is loss tolerant. 49

50 Thank You

統計關係圖

To do  Real time decoder  Efficient intra coding  Print MSG on Mobile 52

DVC decoding time  Side Info Creation: 0.85 sec  Overcomplete Transform: 2.35 sec  Correlation Noise Modeling: 2.33 sec  Conditional Bit Prob Compute: 0.60 sec  Update Side Info: 7.14 sec  Ldpca Decode: sec  Motion Learning: 1.79 sec  Others: 1.80 sec 53

Distributed Video Coding D. Varodayan, A. Aaron, and B. Girod, “Rate-Adaptive Codes for Distributed Source Coding,”EURASIP Signal Processing Journal, Special Issue on Distributed Source Coding,,November Channel Encoder Channel Decoder LDPC Encoder LDPC Decoder

Distributed Video Codec 55 DCT WZ Frames Extract bit-plane bit-plane 1 LDPCA Encoder Buffer bit-plane 2 LDPCA Decoder Request bits Key Frames Conventional Intraframe Encoder Conventional Intraframe Encoder Interpolation/ Extrapolation DCT Reconstruction

Distributed Video Codec 56 DCT WZ Frames LDPCA Encoder Buffer LDPCA Decoder Request bits Key Frames Conventional Intraframe Encoder Conventional Intraframe Encoder Interpolation/ Extrapolation DCT Reconstruction

Distributed Video Codec 57 DCT WZ Frames LDPCA Encoder Buffer LDPCA Decoder Key Frames Conventional Intraframe Encoder Conventional Intraframe Encoder SI generation& refinement DCT SI Reconstruction Mode selection CRC CRC check CRC-8 WZ bitstream Correlation Noise Modeling IDCT& Deblocking filter Decoded WZ Frames Decoded Key Frames

Distributed Video Codec with mobile 58 DCT WZ Frames LDPCA Encoder Buffer LDPCA Decoder Key Frames Conventional Intraframe Encoder Conventional Intraframe Encoder SI generation& refinement DCT SI Reconstruction Mode selection CRC CRC check CRC-8 WZ bitstream Correlation Noise Modeling IDCT& Deblocking filter Decoded WZ Frames Decoded Key Frames Transcode Decoded Video sequence With specific form Video deliver

59 Video recode With mobile camera

ACK at Feedback channel 60 … LDPCA Encoder &CRC LDPCA Decoder &CRC Syndromes for one bitplane

ACK at Feedback channel 61 … LDPCA Encoder &CRC LDPCA Decoder &CRC Syndromes for one bitplane Decoding one bitplane with received syndromes And chack CRC

ACK at Feedback channel 62 … LDPCA Encoder &CRC LDPCA Decoder &CRC Syndromes for one bitplane

63

我最少需 要 48bits 的 parity bits 1584bits LdpcaEncode 我還需要 更多的 parity Bits… 我解完了 一個 bitplane 了 ! Buffer 依序解完 所有 bitplanes 24

DVC to H.264 Transcoder 65 Clouding server: DVC to H.264 transcoder DVC encoder H.264 decoder

 接著講 header overhead  每次都傳一點，等於傳的大部分都是 header(20Bytes)  傳輸太多次，會造成大量的 delay 66

Contribution  Realize DVC encoding part on Mobile device  Implement Feedback channel  Implement encoder’s component with x264  Propose an efficient Feedback channel  Transcode the decoded sequence to mobile codec 67

Outline  Motivation and introduction  DVC architecture with mobile device  Propose an efficient feedback channel  Transcoding  Experiment Result  Conclusion  Future work 68

Time without feedback channel  Total decoding time: sec  Average WZ decoding speed: 7.60 fps  Time information of each component:  Side Info Creation: 0.47 sec  Overcomplete Transform: 0.78 sec  Correlation Noise Modeling: 4.20 sec  Conditional Bit Prob Compute: 0.82 sec  Update Side Info: 3.78 sec  Motion Learning: 1.25 sec  Others: 0.42 sec  Ldpca Decode: 5.36 sec  SI:5.5 69

Time with feedback channel  Total decoding time: sec  Average WZ decoding speed: 0.39 fps  Time information of each component:  Side Info Creation: 0.57 sec  Overcomplete Transform: 1.03 sec  Correlation Noise Modeling: 2.78 sec  Conditional Bit Prob Compute: 0.75 sec  Update Side Info: 2.54 sec  Motion Learning: 1.01 sec  Others: 0.70 sec  Ldpca Decode: sec 70

Time without feedback channel  Total decoding time: sec  Ldpca Decode: 5.36 sec  SI:5.5 71

Time with feedback channel  Total decoding time: sec  Ldpca Decode(+network): sec  Ldpca Decode: 5.36 sec  SI:5.5  Etc :