Scalable Joint Source- Network Coding of Video Yufeng Shan Supervisors: Prof. Shivkumar Kalyanaraman Prof. John W. Woods April 2007

2 Outline  Problem statement and motivations  Our proposed techniques  Fine grain adaptive FEC (FGA-FEC)  Generalized FGA-FEC for wireless networks  Overly multi-hop FEC scheme  Distributed FGA-FEC over multihop network  Conclusion and future work

3 Problem statement and challenges  Problem:  Simultaneously streaming video to diverse users, such as powerful PCs, laptops and handset devices, over heterogeneous networks.  Challenges:  Different users may have different video frame rate /resolution /quality preferences.  May be in different networks, such as high speed wired network, multi-hop ad hoc wireless network or cellular network.  Neither the network nor the video application can provide quality assurances working independent of each other.

4 Conventional Approaches  Windows media, Real player  Send separate copies of the same bitstream to different users, bandwidth utilization is inefficient (unicast to each user).  Layered multicast ( by Steven McCanne )  Send different video layer to different multicast groups, limited by number of video layers, no efficient error correction approach.  Proxy-based streaming  Caches video content at local proxy disk and transcodes it for different users, delay and computation inefficiency.

5 Our Techniques Router Video server DSN Encoding Scalable joint source-network coding:  Scalable video + Scalable FEC + in network adaptation FGA-FEC Generalized FGA- FEC OM-FEC Distributed FGA- FEC  Header CRC/FEC  Cross-layer FEC Adaptatio n Video Performance Video Performance G-FGA- FEC FGA-FEC De/re-code FGA-FEC De/re-code

6 Outline  Problem statement and motivations  Our proposed techniques  Fine grain adaptive FEC (FGA-FEC);  Generalized FGA-FEC for wireless networks;  Overly multi-hop FEC scheme;  Distributed FGA-FEC over multihop network  Conclusion and future work

7 Background -- scalable video We treat three types of scalability:  SNR (aka quality, bitrate) scalability  Spatial (resolution) scalability  Temporal (frame-rate) scalability

8 SNR (quality) scalability Decoding more data corresponds to better quality Embedded compressed file

9 Resolution scalability Decoding more data corresponds to larger picture size Embedded compressed file

10 Frame-rate scalability One GOP - 16 frames Original video Half frame rate – 8 frames ¼ frame rate – 4 frames 1/8 frame rate – 2 frames Decoding more data corresponds to displaying frames at higher rate Embedded compressed file Begin End

11 A(0,0,0) A(0,1,0)A(0,2,0)A(0,3,0)A(0,4,0) A(0,0,0) A(1,0,0)A(1,1,0)A(1,2,0)A(1,3,0)A(1,4,0) A(0,0,0) A(2,0,0)A(2,1,0)A(2,2,0)A(2,3,0)A(2,4,0) A(0,0,0) A(3,0,0)A(3,1,0)A(3,2,0)A(3,3,0)A(3,4,0) A(0,0,0) A(4,0,0)A(4,1,0)A(4,2,0)A(4,3,0)A(4,4,0) A(4,0.1) A(4,1,1) A(4,2,1)A(4,3,1) A(4,4,1) A(4,0,2)A(4,1,2)A(4,2,2)A(4,3,2)A(4,4,2) Resolution Quality Frame Rate Digital items in view of three forms of scalability (quality / resolution / frame-rate). 3-D representation of bitstream Adaptation of the bitstream is achieved by choosing a subset of these items along user’s preferred adaptation order: For example: SNR -> Temporal -> Spatial

12 Next, we need scalable FEC To protect the scalable video bitstream, we have these basic requirements for channel coding:  High adaptivity: If part of the video bitstream is actively dropped (scaled), FEC protecting that part of data should also be removed  Accuracy: near that of robust non-scalable video  Efficiency: Low computational cost

13 Fine Grain Adaptive FEC Goal:  To encode a video to facilitate efficient and precise adaptation of the encoded bitstream at intermediate overlay nodes for diverse users. Main idea: Extend existing approaches (PET, MD-FEC) to work with scalable video and perform the adaptation in the network.

14 Multiple Description FEC (MD-FEC) If any i out of N packets are received, the decoder can decode up to R i. Our work extends this concept to provide fine-grained scalability of FEC, applied to scalable video by Puri and Ramachantran MSB LSB

15 FGA-FEC overview Server DSN 30/C/3M A B 30/C/2M C 30/C/1M D 15/Q/1M E F 30/C/1M G 15/C/384k 15/Q/384k A-G : users and their requirements (frame rate/resolution/bitrate) DSN : Data service node, which aggregates video requirements to server and performs data adaptation for users, without transcoding. FGA-FEC uses overlay infrastructure to assist video streaming to heterogeneous users simultaneously by providing light weight support at intermediate overlay nodes. Encoding Adaptatio n

16 FGA-FEC Encoding  Bitstream is divided into N sections from MSB to LSB  Each section is further split into small blocks  RS(N,i) codes are applied at block level to section S i, vertically  Each block column is independently accessible  one description = one network packet S 1...S i S N R 1 R i-1 R i R N-1 R N R 0 B 1 B i B 2i B mi Enlarged view of Section S i... S 1 (1)S 2 (1)S i (1) S N (1) FECS 2 (2)S i (2)...S N (2) FEC S i (3)...S N (3) FEC S i (i)...S N (i) FEC...S N (i+1) FEC...S N (N) Sect.1.2.i Description1 2 3 Description i +1 Description N... Sect.N...

17 FGA-FEC – optimal rate allocation FEC assignment is optimized by minimizing the expected distortion over the channel with packet loss probability p, available bandwidth B. Subject to: where q i is the probability of receiving i out of N packets, R total is the total bandwidth occupied by both FEC and video data. Also r i is the source rate of section S i. N is the number of descriptions encoded for each GOP The result of the optimization is the video source-rate breakpoints R i

18 FGA-FEC adaptation at intermediate nodes  Direct truncation: only shorten each packet in a GOP by removing unwanted blocks.  FGA-FEC adaptation: adapt the encoded GOP by a combined shortening and/or dropping packets.  An algorithm is proposed to near optimally adapt the encoded GOP for available bandwidth and user preference  No video or FEC transcoding, only a packet shorting/dropping Two adaptation methods are used in FGA-FEC : At DSN Adapted description One description one GOP Shorten & drop

19 Simulation-- FGA-FEC adaptation vs. optimal decoding/recoding  Comparison of FGA-FEC, optimal decoding/recoding solution, and direct truncation at different available bandwidths;  In FGA-FEC, MC-EZBC encoded GOP #7, Foreman CIF is adapted from 1100 Kbps to satisfy different users;  FGA-FEC optimization is based on 15% loss rate at source, N=

20 ns-2 Simulation-- FGA-FEC vs. MD-FEC transcoding MD-FEC transcoding: FEC decode/recode at each intermediate node FGA-FEC adaptation: Encode at server, adaptation in the intermediate nodes Want to see the received video quality at User 4 and User 12, Link loss probability is 0.01, N=64, Foreman, CIF

21 ns-2 Simulation-- FGA-FEC vs. MD-FEC transcoding Receiver 4 FGA-FEC is 0.01dB lower The quality loss is due to the adaptation precision. Receiver 12 FGA-FEC is 0.4 dB lower

22 ns-2 Simulation-- FGA-FEC vs. Layered multicast Layered multicast: Base layer bit rate 0.6, and enhancement layers 0.32, 0.42, 0.32 Mbps, respectively FGA-FEC adaptation: Encode at server, adaptation in the intermediate nodes Want to see the received video quality at User 5 and User 7, Link loss probability is 0.01, N=64, Foreman, CIF

23 ns-2 Simulation-- FGA-FEC vs. Layered multicast Receiver 5 Layered multicast can subscribe up to two layers The quality loss of layered multicast is due to coarser available layers. Receiver 7 Layered multicast can subscribe up to three layers

24 Outline  Problem statement and motivations  Our proposed techniques  Fine grain adaptive FEC (FGA-FEC);  Generalized FGA-FEC for wireless networks;  Overly multi-hop FEC scheme;  Distributed FGA-FEC over multihop network  Conclusion and future work

25 Generalized FGA-FEC for wireless  channel bit errors due to channel fading and noise;  large bandwidth fluctuations, and  intermediate node computational capability constraint  limited maximum transmission unit (MTU) size. In addition to congestion related packet losses, in wireless must deal with:

26 Generalized FGA-FEC for wireless A kind of Product codes ( extend Sherwood and Zeger’s codes) :  Column codes: Reed Solomon for packet loss  Row codes: CRC + BCH for bit errors S 1 (1)S 2 (1)...S k (1) S N (1) FECS 2 (2)...S k (2) S N (2) FEC...S k (3) S N (3) FEC...S k (k) S N (k) FEC...FEC...S N (k+1) FEC...FEC...S N (N) Sect.1.2.k... Description1 2 3 Description k +1 Description N... S 1 S k S N R 1 R k-1 R k R N-1 R N R 0 Sect.N R S e n c o d e d i n b l o c k l e v e l Embedded bitstream Generalized FGA-FEC = FGA-FEC Bit level protection + A BCH code is represented as: BCH(n,k,t): n=2 m -1, n-k<=mt

27 Optimal product code assignment Find a concatenated column RS code assignment c c and row BCH code assignment c r from a set of RS codes C RS and BCH codes C BCH BCH(n,k,t), such that the end-to-end video distortion over a lossy channel is minimized. Subject to channel rate constraint: where R s, R RS, R CRC and R BCH are rates allocated to the video source, RS parity bits, CRC and BCH check bits, respectively. Here B denotes the maximum available channel bandwidth

28 Fast optimization method Given a BCH code BCH(n,k,t) Exhaustive search results Probability of correctly decoding BCH code Near optimal points, search starts here, it can find the optimal solution within a few iterations GOP #7 of Foreman, CIF, N= 64, p drop =0.05, BER pb varies n=8191, m=13

29 Multi-cluster descriptions coding  Multi-cluster descriptions coding: one GOP  Multiple of N descriptions due to:  Limited network packet size;  Easier adaptation, such as encode each temporal layer as one cluster as below. Example of encoding one GOP into 3 clusters, each cluster is coded with product codes  Single-cluster description coding: one GOP  N descriptions One GOP of scalable bitstream Low frame rate Mid frame rate High frame rate

30 Simulation – FGA-FEC vs. MD-FEC in wireless Channel changes over time between node 2 and 3 Consider two adaptation orders: SNR->Temporal, SNR-> Spatial Frame rate scaling Resolution scaling p drop =0.05 at node2

31 Outline  Problem statement and motivations  Our proposed techniques  Fine grain adaptive FEC (FGA-FEC);  Generalized FGA-FEC for wireless networks;  Overly multi-hop FEC scheme;  Distributed FGA-FEC over multihop network  Conclusion and future work

32 Motivations -- an example Hop B (i) 660K625K615K700K900K1100K P (i) 2.5%3%3.5%2.5%1.5%1% Sender Receiver N1 N5 N2N3 (B 6, P 6, RTT 6 ) (B 1, P 1, RTT 1 ) Given an overlay path: The parameter of each hop FEC method End to end Hop- by-hop OM-FEC data rate (goodput) 529K594 K Path loss rate 14% OM-FEC vs. Traditional FEC Segment 2 Segment 3 Segment 1 N4

33 OM-FEC algorithm i L M N1 Nn NhNj Ni ………… OM-FEC partitions a given overlay path into segments: By solving the following: minimize(N segement ) min{B start, …, B i }  (B data + B FEC (start  >i)) min{B start, …, B i+1 } (i+1)) FEC that should be added to the Segment is B FEC (start ->i), Subject to: The i-th node is boundary node, if the two inequalities hold:

34 Video Simulations Topology: loss rate on each hop (L1–L10) random within [0.4%-1.2%]  H.263+ encoded Foreman CIF, at bitrate 1530 Kbps;  Available bandwidth for each hop: 1656 Kbps Average number of segments partitioned is only 2 vs. 10 in hop-by-hop FEC

35 PlanetLab Experiments  Video: Foreman QCIF, H.263+, 512 Kbps, 30 fps, I frame every second  Packet-loss rate from Utah to CMU is set to 5%, other links are set to 1%.  The available bandwidth from Utah to CMU is also upper bounded to 550 Kbps.  Two paths: 1): with nodes shown at map 2): Add Node 4: planet1.ecse.rpi.edu at last hop with 1% loss rate planetlab1.flux.utah.edu planetlab-1.cmcl.cs.cmu.edu planetlab1.cs.cornell.edu video.testbed.ecse.rpi.edu nima.eecs.berkeley.edu

36 OM-FEC experimental results Video PSNR of OM-FEC scheme vs. e2e scheme (5 virtual links) Video PSNR of OM-FEC scheme vs. e2e scheme (4 virtual links) Results show: OM-FEC outperforms e2e FEC scheme As more congested links are encountered, gain due to OM-FEC increases Large PSNR loss of the e2e scheme is due to the dependency of bitstream, one packet drop could cause later packets of the same GOP useless.

37 Outline  Problem statement and motivations  Our proposed techniques  Fine grain adaptive FEC (FGA-FEC);  Generalized FGA-FEC for wireless networks;  Overly multi-hop FEC scheme;  Distributed FGA-FEC over multihop network  Conclusion and future work

38 Problem statement and our solution  In FGA-FEC We assume no congestion between DSNs  Remaining problem: –What if there is congestion between DSNs? For example, in a rate constrained multihop ad hoc wireless network.  Our solution – distributed FGA-FEC: - FEC coding and optimization algorithm will be run at DSNs in a distributed way to serve the diverse users.

39 The optimization algorithm Minimize the e2e distortion for each user: Subject to: where Problem : Solution : Use Lagrange multiplier method: Given one value of λ, corresponds to the point on the D(R) curve with slope equal to Solution can be found by searching over λ

40 The Distributed FGA-FEC Algorithm To reduce the overall computation while maintain the best possible video quality, we propose two approaches:  A coordination method between optimization processes running at adjacent nodes to reduce the optimization computation.  Full search, search with previous GOP, search with neighbor  Applied the idea of OM-FEC to reduce the number of FGA-FEC decode/recode nodes, do FEC computation at selected DSNs

41 Simulation - reduce optimization computation Test number of λ iterations need to reach optimization stop points, Foreman, CIF. The proposed coordination between adjacent nodes can reduce the optimization computation requirement.

42 Simulation– distributed FGA-FEC vs. Hop-by-hop FEC decode/recode Hop-by-hop FGA-FEC decode/recode: FGA-FEC decode/recode at each intermediate node Distributed FGA-FEC: Identify the congested links and select appropriate nodes to do FGA-FEC decode/recode. Want to see the received video quality at User 5 and User 12 Link loss probability is 0.01, N=64, Foreman, CIF

43 Simulation– distributed FGA-FEC vs. Hop- by-hop FGA-FEC decode/recode Receiver 5 The two schemes deliver similar video quality, but distributed FGA-FEC uses fewer FEC computation nodes 3 vs. 6 at FEC decode/recode Receiver 12

44 Measured CPU time We measure the CPU time using FGA-FEC adaptation and FGA-FEC decode/recode in a Dell PC with 1.6 GHZ CPU, 256 M memory, running Red Hat linux 8.2, N=64. Scheme performed at DSNCPU time (ms) FGA-FEC decode/recode52.6 FGA-FEC adaptation2.9 FGA-FEC direct truncation10 -2 Distributed FGA-FEC can greatly reduce the computation burden, while can deliver near optimal video quality

45 Conclusions  Fine grain adaptive FEC (FGA-FEC)  to encode and adapt a scalable video  Generalized FGA-FEC for wireless networks  to do encoding and adaptation over wireless networks  Overly multi-hop FEC scheme  to efficiently utilize one congested path  Distributed FGA-FEC over multihop network  to stream video over a congested heterogeneous networks  Header error correction  To improve the effective throughput of wireless networks  Cross-layer FEC  To joint optimize protocol with application layer FEC

46 Future work  Evaluate FGA-FEC over DCT-based standard coders such as H.264/AVC and SVC  Extend the distributed FGA-FEC to work with Spatial and Temporal scalability  Extend FGA-FEC idea to multi-point video conferencing

47 [1] Yufeng Shan, Ivan Bajic, John W. Woods and Shivkumar Kalyanaranman “ Scalable video streaming with fine grain adaptive forward error correction” submitted to IEEE trans. CSVT, 2006 [2] Su Yi, Yufeng Shan, Shivkumar Kalyanaraman and Babak Azimi-Sadjadi, "Video streaming over Ad Hoc wireless ntworks with header error protection", submitted to Ad Hoc Networks, 2006 [3] Yufeng Shan, Ivan V. Bajic, Shivkumar Kalyanaraman and John W. Woods, “Overlay multi-hop FEC scheme for video streaming,” Signal Processing: Image Communications Vol. 20/8, 2005 [4] Yufeng Shan, John. W. Woods and Shivkumar Kalyanaraman “Fine grain adaptive FEC over wireless networks”, submitted to ICIP 2007 [5] Su Yi, Yufeng Shan, Shivkumar Kalyanaraman and Babak Azimi-Sadjadi, "Header error protection for multimedia data transmission in wireless AdHoc networks", ICIP, 2006 [6] Yufeng Shan, Ivan Bajic, Shivkumar Kalyanaraman, and John W. Woods, "Joint source-network error control coding for ccalable overlay streaming," ICIP, 2005 [7] Yufeng Shan, Su Yi, Shivkumar Kalyanaraman and John.W. Woods, "Two-Stage FEC scheme for scalable video transmission over wireless networks" SPIE Communications/ITCom, Multimedia Systems and Applications, Oct [8] Yufeng Shan, Ivan Bajic, Shivkumar Kalyanaraman, and John W. Woods, "Overlay multi-hop FEC scheme for video streaming over peer-to-peer networks," ICIP, 2004 [9] Yufeng Shan and Shivkumar Kalyanaraman “Hybrid video downloading/streaming over peer-to-peer networks,” ICME, 2003 Publication and submissions

48 Thanks