Video Streaming Concepts Reading: John G. Apostolopoulos, Wai- tian Tan, Susie J. Wee, “Video Streaming: Concepts, Algorithms, and Systems”, HP Laboratories Palo Alto,

Classification of video apps  Point2point vs multicast vs broadcast  Is there a “reverse channel”?  Pros and cons of reverse channel  Real-time vs pre-encorded(stored) video  What about “almost real-time”?  Interactive vs non-interactive video  Static vs dynamic channels  Bandwidth, delay, and loss are static or dynamic  CBR vs VBR channels  Do not confuse with CBR/VBR encoding  Packet-switched vs circuit-switched channels  QoS support?

Video compression standards  What do the standards specify?  Encoder and decoder implementations?  Bit stream syntax?  Decoding process?

Video streaming challenges  Video delivery via file download vs streaming ?  Characteristics of Internet are unknown or time- varying  Bandwidth  rate control  Delay jitter  playout buffer  Loss  error control

Transport and Rate Control for overcoming Time-varying BW  Objectives  How to estimate the appropriate transmission rates dynamically at the time of streaming ? – available BW estimation  How media coding has evolved to support such dynamic changes in transmission rates ? – coding rate control  Rate control for video streaming over TCP  TCP is a window (not rate) based transport protocol  TCP adjusts the window size based on AIMD congestion-control algorithm  varying throughput (saw-tooth pattern)  Retransmission  enlarge delay jitter  Nevertheless, TCP is often used in video streaming, especially when receiver window is appropriately sized  Stable and scalable TCP rate control  TCP guarantees delivery  How would you calculate the receiver window size to set the streaming rate at a certain value?

Transport and Rate Control for overcoming Time-varying BW  Rate control for video streaming over UDP  Separation of error control and rate control  TCP-friendly rate control  Mimic average throughput of TCP = f(RTT, p)  Smooth the instantaneous fluctuations of TCP’s AIMD algorithm  Rate  1/RTT : video streaming 에 적합한가 ?  Some media streaming system do not perform rate control  Multicasting: Identical stream is transmitted to all recipients via channels of different levels of congestion  No feedback channel  Receiver-driven rate control  Layered multicast of scalable or layered compressed video

Meeting Transmission BW Constraints  If channel transmission rate < media bit rate, then ??  Transcoding  Decode and re-encode to the desired bit rate  Include bit rate reduction, spatial downsampling, frame rate reduction, changing compression formats  Requires application-layer gateways  Multiple file switching  Multiple copies of the same content at different bit-rates. Client chooses the appropriate media rate.  Multi-rate switching: enables dynamic switching between different rates within a session  No recompression penalty, but BW waste for multiple copies of the same media  Scalable compression (layered coding) 7

Evolving Approaches in Internet  IntServ model  QoS guarantees BW, packet loss rate, delay on per flow basis  Explicit resource reservation via RSVP  DiffServ model  Classify and differentiate among classes based on a tag(code-point) in each packet 8

Playout Buffer for Overcoming Jitter  Solution: Playback buffering  Key question: how long should the playback buffer (or playback delay) be?  Playback buffers have additional advantages:  Error recovery through ReTx  Error resilience through interleaving  Smooth throughput variations (e.g., due to TCP)  Adaptive Media Playout

Error Control for Overcoming Channel Losses  Bit error vs packet loss  In wired network, almost packet loss due to congestion  In wireless network, bit errors or burst errors may cause  To be passed up to the appl. Layer  Or, discarded (packet loss)  Solutions?  Retransmissions  Commonly used, but constrained by delay budget and playback delay  Requires back-channel  Forward Error Correction  Send N packets (K data packets + K-N redundant packets). If received packets >= K correctlry, data packets are recovered.  Introduce bandwidth overhead (and potentially delay)  Or, cannot recover.  Loss/error concealment  Loss/error resilient video coding

Loss/Error Concealment  Pixels, MBs, slices, and entire frame may be lost.  Exploits the correlation along  Spatial interpolation  Temporal extrapolation (freeze frame)  Motion-compensated temporal extrapolation  Motion-compensated block + estimation of lost MV from ??  Error concealment is performed at the decoder 11

Error Resilience Video Coding: Overcoming Loss of Bitstream Synch.  Resynch Marker  Put the Resync markers (placed where?)  Reversible VLCs (MPEG-4)  Less efficient than VLS  Data Partitioning (MPEG-4)  Place most important data just after markers  Application Level Framing (ALF)  Design the packet(frame) payload to minimize the effect of loss 12

Error Resilience Video Coding: Overcoming Incorrect State and Error Propagation  Overcome error propagation  Use periodic I-pictures (GOP)  Use periodic intra-coding of MBs  In point-to-point comm. with back channel (short RTT)  Both the encoder and decoder store multiple previously coded frames  Decoder notifies the packet error to the encoder  The encode tells which picture should be used as the reference for the next prediction (Reference Picture Selection in H.263) 13

Scalable (Layered) Video Coding for Lossy Networks  SVC implicitly assumes broadcast or multicast of video steams (including overlay multicasting)  Base layer and several enhancement layers need different treatments  unequal error protection (UEP),  prioritized transmission  Useful if network provides several classes of service or priorities (DiffServ, IntServ)  Under the best effort Internet ?

Multiple Description Video Coding  Multiple description coding (MDC) video  Several “descriptions” of the same video  The more descriptions you receive, the better  Significant redundancy among descriptions  What if all descriptions are subject to simultaneous losses?  MDC video with path diversity

Media Streaming Protocols and Standards  Protocols for video streaming over the Internet  Media delivery: RTP/RCTP  RTP does not provide real-time delivery, neither support any QoS  Only provides time stamps, sequence numbering, video source identification, payload specification  RTCP provides QoS feedback in terms of # of los packets, delay, inter-arrival jitter, etc.  Media Session Control  RTSP (Realtime Streaming Protocol)  SIP (Session Initiation Protocol)  Media Description and Announcement  SDP (Session Description Protocol)  SAP (Session Announcement Protocol) 16

Additional Video Streaming Topics  Multicast  Should solve heterogeneity problem  Network heterogeneity: different channel conditions (BW, error protection)  Receiver heterogeneity  Approaches  Different multicasts for different ranges of intended bit-rates: e.g) simulcast  Different multicasts can contain incremental information (SVC): e.g) layered multicast  End-to-end security and Transcoding  Streaming over wired and wireless Links  Loss differentialtion:  Loss due to congestion or noise ?  Streaming media CDN 17

Streaming Media CDN  Conventional CDN vs streaming media CDN  Issues  Application-layer multicast (overlay multicast)  Seamless hand-off  MD-CDN  Achieve path diversity from the infrastruture of CDN 18