Real-time smoothing for network adaptive video streaming Kui Gao, Wen Gao, Simin He, Yuan Zhang J. Vis. Commun. Image R. 16 (2005)

Outline Motivation The architecture of network-adaptive streaming system A Real-time dynamic smoothing algorithm How to select and schedule packets How to retransmit the lost packet Simulation results Conclusion

Motivation Problems: Variation of network bandwidth Packet may lost The current Internet Provide best-effort services Do not provide QoS Goals: Provide a smoothing quality at a client end (real-time smoothing) Maximizing the utilization of the variable network bandwidth (adaptive video streaming) Propose a dynamic real-time smoothing algorithm to select and scheduling the packet –Use FGS streaming to maintaining a very flexible and simple video- coding structure

The architecture of network-adaptive streaming system ServerClient Use a real-time smoothing algorithm Feedback information includes: Packet loss rate, RTT(round-trip time)

How to select and schedule the packet Considerations Playback distortion (lost, damaged) Network bandwidth Packet Deadline (arrive before the playback time) Packet dependency Summary Different portions of video bitstream have different contributions to the video quality. The important packets (with large distortions) can be transmitted earlier. The packets of base layer are the most important. Feedback information

When packet loss, how to recover Use retransmission scheme to recover lost packet But not all lost packet can be recovered Due to the available network bandwidth and the delivery deadline constraint Summary: The lower layer has more chances to retransmit its packets.

The characteristics of the transmission buffer (1/2) The packet in transmission buffer has two states: In the ready states: Can be scheduled for transmission In the blocked states: Can not be scheduled until packet loss or its timeout RTO ≦ t out How to select a packet to transmit at t cur from the transmission buffer S buf is the set of all the ready packets S buf = {p i,j |a i,j ≦ t cur and t cur +c i,j ≦ d i and in ready state} p i,j : the packet of the jth layer in frame F i a i,j : release time (the earliest time the packet in the buffer) c i,j : process time of p i,j d i : deadline (the latest time p i,j should be sent to client)

The characteristics of the transmission buffer (2/2) How to select First step Server selects the ready packets of base layer with earliest deadline firstly (EDF) Second step If no base layer packets in ready state, the serer select and schedule the ready packets of enhancement layers In enhancement layer –According to rate-distortion decision –If the packets have same distortion, packet with earliest deadline are served first

The transmission flow Base 0 Layer 1 Layer 2 Layer 3 Base 0 Layer 1 Layer 2 Layer 3 Base 0 Layer 1 Layer 2 Layer 3 Base 0 Layer 1 Layer 2 Layer 3 … (Frames) (1) According to feedback to decide a i (release time) (2) Put packets into buffer according to a i Transmit buffer a i =4 a i =3 a i =2 a i =1 Ready state Block state client Base 0 Layer 1 Layer 2 Layer 3 Feedback information & (ACK or NAK) (3) Base layer transmit first Enhancement layer transmit according to the distortion a i : the earliest time the packet in the buffer Feedback includes: packet loss rate & RTT(round-trip time) ACK: packet unloss NAK: packet loss

A Real-time smoothing algorithm n frames, a scalable media stream F = {F 0,F 1, …,F n-1 } m: the number of layers p i,j (0 ≦ i ≦ n-1, 0 ≦ j ≦ m): the packet of the jth layer in frame F i a i : release-time (the earliest time p i,j in the transmission buffer) d i : deadline (the latest time p i,j should be sent to client) assume the packets in the same frame have the same a i and d i p loss (t): packet loss rate RTT(t): round-trip-time RTO(t): retransmission timeout The processing time of p i,j : c i,j = b i,j /X(t) b i,j : the size of the packet p i,j X(t): the available network bandwidth at time t The fulfill-time of p i,j : f i,j = s i,j + c i,j s i,j : the scheduler time of p i,j (fulfill-time = scheduler time + processing time) feedback information

How to decide the release time (a i ) MPEG-4 video can accept an error rate of or lower How to decide the release time (the earliest time p i,j into the transmission buffer) If the release time is too early The transmission buffer must be very large If the release time is too late There is no chance to retransmission It is important to select the retransmission times K for lossy packets of the base layer Let be the total loss probability of the p i,0 through K times retransmission, we get the smallest K that ≦ a i = d i – K * RTO (K & RTO decided by feedback) d i : deadline (the latest time p i,j should be sent to client)

Constraint of playback (In enhancement layer) (1/2) It can be Arbitrarily truncated (decided by the available channel bandwidth) Tolerate the channel errors A packet p i,j can be selected and scheduled if satisfied The current time is later than release time (a i ≦ t cur ) The fulfill-time is earlier than deadline (a i ≦ t cur and t cur + c i,j ≦ d i ) p.s: c i,j = process time of p i,j According to rate-distortion optimized decision

Constraint of playback (In enhancement layer) (2/2) Assume the quantization of a frame is Q if the maximum number bit-plane of a frame is m, and the last bit-plane is z, then Q=2 m-z A distortion model built for uniform quantizer (UQ) is The distortion of packet p i,j is The total quality is when J is maximize, the total quality is the best

Simulation results (1/3) Use TFRC protocol to decide the sending rate s: the packet size, R is RTT (round-trip time), t RTO is retransmission timeout value, and p is the pack loss ratio. The sequences include Foreman, Coastguard and Akiyo Encoded with 30 frames per second There are 7 enhancement layer The playback frame rate is 30Hz

Simulation results (2/3)

Simulation results (3/3)

Conclusion Propose a real-time smoothing algorithm According to feedback information to select and schedule the packet –under a rate-distortion optimized The algorithm improves the utility of the bandwidth smoothes the playback quality

The FGS framework

Scalable Video Coding