Distributed Video System realized on mobile device with efficient Feedback channel 分散式影像編碼在手機上的實現與有效率 的回饋通道 1 Chen,chun-yuan 陳群元 Advisor:Prof. Wu,Ja-Ling 吳家麟 2012/6/6
Outline Motivation and introduction DVC architecture with mobile device Propose an efficient feedback channel Transcoding Experiment Result Conclusion Future work 2
Motivation Video communication is essential in Mobile device Decrease the coding time and battery consuming on mobile. 3
Motivation 4
Introduction For Mobile device video codec Conventional video coding (ex:MPEG-4 H.264) Heavy weight encoder, light weight decoder. Distributed Video Coding Light weight encoder, heavy weight decoder. 5
Conventional video codec 6
DVC to H.264 Transcoder 7 Clouding server: DVC to H.264 transcoder DVC encoder H.264 decoder
Channel Coding R Y =H (Y) Source encoder Correlation is exploited by motion estimation R Y =H (X|Y) Source encoder H (X|Y)H (Y) Distributed Slepian-Wolf Theorem (1973) Wyner-Ziv Theorem (1976) Channel encoder Source decoder Side Information creation Channel decoder X Y Virtual channel H (X, Y)
Distributed Video Codec 9 DCT WZ Frames LDPCA Encoder Buffer LDPCA Decoder Key Frames Conventional Intraframe Encoder Conventional Intraframe Encoder SI generation& refinement DCT SI Reconstruction Mode selection CRC CRC check CRC-8 WZ bitstream Correlation Noise Modeling IDCT& Deblocking filter Decoded WZ Frames Decoded Key Frames
Our DVC codec: DVC transcoder 10
DVC to H.264 Transcoder 11 Clouding server: DVC to H.264 transcoder DVC encoder H.264 decoder
DVC Decoding Process DVC Decoding without Feedback channel Side-Information generation LDPCA 12
DVC Decoding Process DVC Decoding Side-Information generation Send-Request at Feedback channel LDPCA 13
Outline Motivation and introduction DVC architecture with mobile device Propose an efficient feedback channel Transcoding Experiment Result Conclusion Future work 14
DVC Encoder Realization Implement DVC encoder on Mobile Key frame encoding WZ frame encoding Efficient intra mode realization Feedback channel transmission Network connection between mobile and server Determine payload of a packet with LDPCA protocal 15
Key frame encoding on Mobile Previous ver. use JM9.5 Transcoder ref x.264 on PC 16
DVC Encoder Realization Implement DVC encoder on Mobile Key frame encoding WZ frame encoding Efficient intra mode realization Feedback channel transmission Network connection between mobile and server Determine payload of a packet with LDPCA protocal 17
WZ coding with NDK 18
DVC Encoder Realization Implement DVC encoder on Mobile Key frame encoding WZ frame encoding Efficient intra mode realization Feedback channel transmission Network connection between mobile and server Determine payload of a packet with LDPCA protocal 19
Efficient intra block coding Previous ver. Transcoder Use JM code. Intra block encoding consumes lots of time in encoding. 20 Clouding server: DVC to H.264 transcoder
DVC Encoder Realization Implement DVC encoder on Mobile Key frame encoding WZ frame encoding Efficient intra mode realization Feedback channel transmission Network connection between mobile and server Determine payload of a packet with LDPCA protocal 21
Network connection Create network connection to connect remote server. TCP socket in C 22 Clouding server: DVC to H.264 transcoder
DVC Encoder Realization Implement DVC encoder on Mobile Key frame encoding WZ frame encoding Efficient intra mode realization Feedback channel transmission Network connection between mobile and server Determine payload of a packet with LDPCA protocal 23
ACK at Feedback channel 24 … syndromes
Time consuming at Feedback channel Packet header occupy large part of network bandwidth. Network latency in communication at Feedback channel 25
Outline Motivation and introduction DVC architecture with mobile device Propose an efficient feedback channel Transcoding Experiment Result Conclusion Future work 26
Efficient feedback channel we propose two methods to decrease time consuming at feedback channel. Estimate the syndromes size per WZ frame Estimate the syndromes size per bitplane 27
Number of requests per bitplane per band DC AC1AC2 AC3 AC4AC5 AC6AC7AC8 AC9AC10 AC11 AC12 AC13 AC14 Bitplane Number
Estimate the syndromes size per WZ frame Estimate syndromes size by correspond WZ frame in previous GOP Ex. GOP 4 29 Key frame Key frame WZ frame WZ frame WZ frame First GOP Key frame Key frame WZ frame WZ frame WZ frame Second GOP
formula (ES bt ) WZn =(S bt ) WZ(n-GOPsize) WZn means the WZ frame’s index n-GOPsize means the correspond WZ frame in the previous GOP. bt is the bitplane index. (ES bt ) WZn means the estimated syndromes bit for the bt’th bitplane of WZn frame. (S bt ) WZ(n-GOPsize) means the syndromes
Efficient feedback channel we propose two methods to decrease time consuming at feedback channel. Estimate the syndromes size per WZ frame Estimate the syndromes size per bitplane 31
Number of requests per bitplane per band DC AC1AC2 AC3 AC4AC5 AC6AC7AC8 AC9AC10 AC11 AC12 AC13 AC14 Bitplane Number
Number of requests per bitplane per band DC AC1AC2 AC3 AC4AC5 AC6AC7AC8 AC9AC10 AC11 AC12 AC13 AC14 Bitplane Number
Estimate by previous bands 1.formula 2.time ratio
Number of requests per bitplane per band DC AC1AC2 AC3 AC4AC5 AC6AC7AC8 AC9AC10 AC11 AC12 AC13 AC14 Bitplane Number
formula (ES acn bt ) WZn = ( S dc bt-1 ) WZn WZn means the n’th WZ frame. acn means the n’th AC band. bt means the bitplane index of this AC band. Which ES acn bt mean the estimated syndrome size for the bt’th bitplane in n’th AC band. dc means the DC band. S dc bt-1 means the syndromes bits for bt-1’th bitplane in DC band.
Outline Motivation and introduction DVC architecture with mobile device Propose an efficient feedback channel Transcoding Experiment Result Conclusion Future work 37
Transcode to mobile codec Because DVC only encode luminance part in YUV, so we got only Y for output. Our mobile device can only read mp4 and 3gp. Y UV R G B Frame 1 Frame 2 Frame 3 Frame4 Frame 5 Frame 6 Frame 7 Frame n.mp4
Outline Motivation and introduction DVC architecture with mobile device Propose an efficient feedback channel Transcoding Experiment Result Conclusion Future work 39
Experiment Result 規格 實驗數據 : 時間 (encode&decode) 、 packet 數量 ( 傳幾次 )+ RD 使用 x264 對 quality 的影響 (intra 改 quality 應該 會變,但變快 ) 40
Outline Motivation and introduction DVC architecture with mobile device Propose an efficient feedback channel Transcoding Experiment Result Conclusion Future work 41
Contribution Realize DVC encoding part on Mobile device Implement Feedback channel Implement encoder’s component with x264 Propose an efficient Feedback channel Transcode the decoded sequence to mobile codec 42
Outline Motivation and introduction DVC architecture with mobile device Propose an efficient feedback channel Transcoding Experiment Result Conclusion Future work 43
Future Work Take a more precise estimation of syndrome size. Use motion information to estimate the syndrome size. Build a reliable DVC codec which is loss tolerate 44
45 Thank You
To do Real time decoder Efficient intra coding Print MSG on Mobile 46
DVC decoding time Side Info Creation: 0.85 sec Overcomplete Transform: 2.35 sec Correlation Noise Modeling: 2.33 sec Conditional Bit Prob Compute: 0.60 sec Update Side Info: 7.14 sec Ldpca Decode: sec Motion Learning: 1.79 sec Others: 1.80 sec 47
Distributed Video Coding D. Varodayan, A. Aaron, and B. Girod, “Rate-Adaptive Codes for Distributed Source Coding,”EURASIP Signal Processing Journal, Special Issue on Distributed Source Coding,,November Channel Encoder Channel Decoder LDPC Encoder LDPC Decoder
Distributed Video Codec 49 DCT WZ Frames Extract bit-plane bit-plane 1 LDPCA Encoder Buffer bit-plane 2 LDPCA Decoder Request bits Key Frames Conventional Intraframe Encoder Conventional Intraframe Encoder Interpolation/ Extrapolation DCT Reconstruction
Distributed Video Codec 50 DCT WZ Frames LDPCA Encoder Buffer LDPCA Decoder Request bits Key Frames Conventional Intraframe Encoder Conventional Intraframe Encoder Interpolation/ Extrapolation DCT Reconstruction
Distributed Video Codec 51 DCT WZ Frames LDPCA Encoder Buffer LDPCA Decoder Key Frames Conventional Intraframe Encoder Conventional Intraframe Encoder SI generation& refinement DCT SI Reconstruction Mode selection CRC CRC check CRC-8 WZ bitstream Correlation Noise Modeling IDCT& Deblocking filter Decoded WZ Frames Decoded Key Frames
52
我最少需 要 48bits 的 parity bits 1584bits LdpcaEncode 我還需要 更多的 parity Bits… 我解完了 一個 bitplane 了 ! Buffer 依序解完 所有 bitplanes 24
DVC to H.264 Transcoder 54 Clouding server: DVC to H.264 transcoder DVC encoder H.264 decoder
接著講 header overhead 每次都傳一點,等於傳的大部分都是 header(20Bytes) 傳輸太多次,會造成大量的 delay 55
Outline Motivation and introduction DVC architecture with mobile device Propose an efficient feedback channel Transcoding Experiment Result Conclusion Future work 56