1. Streaming Video Over the Internet Andreas Panteli 03/05/2012 ECE 654 Advanced Computer Networks 03 May 2012 Streaming Video Over The Internet 1

2. Introduction  Video principles  When an image appears on the retina it is retained for some millisecond before decaying  If a sequence of images is drawn line by line at 50 images/sec the eye does not notice that it is looking at discrete images  Streaming Video  Video content need not be downloaded in full  It is being played out while parts of the content are being received and decoded  Real-time multimedia have timing constraints  Audio and video data must be played out continuously  Bandwidth, delay and loss requirements  No QoS guarantees to streaming video, from the current best-effort Internet! 03 May 2012 Streaming Video Over The Internet 2

3. An Architecture for Video Streaming 03 May 2012 Streaming Video Over The Internet 3 Raw Video Compressed Audio Application-layer QoS Control Compressed Video Transport Protocol Video Decoder Audio Decoder Raw Audio Video Compression Audio Compression Application-layer QoS Control Transport Protocol Media Synchronization Storage Device Streaming ServerClient/Receiver Internet (Continuous media distribution services)

4. Key Areas  Video Compression  Application-Layer QoS Control  Continuous Media Distribution Services  Media Synchronization Mechanisms  Protocols for Streaming Media 03 May 2012 Streaming Video Over The Internet 4

5. Key Areas  Video Compression  JPEG standard  MPEG standard  Application-Layer QoS Control  Continuous Media Distribution Services  Media Synchronization Mechanisms  Protocols for Streaming Media 03 May 2012 Streaming Video Over The Internet 5

6. Digital Video  Digital video is represented as a sequent of frames, each consisting of a rectangular grid of picture elements (pixels)  For color video, 8 bits for each of the RGB colors are being used  24 bits/pixel => 16 million colors  To produce smooth motion, digital video, we must display at least 25 frames/sec, but to avoid flickering we need to repaint each frame on the screen twice  Smoothness and flickering affects the bandwidth requirements  Consider the lower video configuration for computer monitor 1024x768 with 24 bits/pixel and 25 frames/sec  Need to be fed at 472 Mbps  We need video compression! 03 May 2012 Streaming Video Over The Internet 6

7. Video Compression  Two algorithms are needed  Encoding and Decoding  Asymmetries of the algorithms  Acceptable for the encoding algorithm to be slow and require expensive HW, if the decoding algorithm is fast  Only for video on demand  For real-time multimedia like video conferencing, slow encoding is unacceptable  Acceptable to have a slightly different video signal than the original, after encoding and decoding (Lossy system)  Lossy systems are important  Accepting a small amount of information loss can give a huge payoff in terms of the compression ratio possible 03 May 2012 Streaming Video Over The Internet 7

8. The JPEG Standard  Compression algorithm for still images  Could be applied to each image in succession to achieve video compression 03 May 2012 Streaming Video Over The Internet 8  Compute the luminance Y and the two chrominances I and Q  Y = 0.30R + 0.59G + 0.11B  I = 0.60R – 0.28G - 0.32B  Q = 0.21R – 0.52G + 0.31B  Compress the two chrominances twice than the luminance  Lossy reduction, but the eye barely notices it since it responds to luminance more than to chrominance  JPEG often produces 20:1 compression or better  Decoding a JPEG image requires running the algorithm backwards

9. The MPEG Standard (1/2)  Main algorithms for video compression and international standards since 1993  Can compress both audio and video 03 May 2012 Streaming Video Over The Internet 9  Three parts in MPEG-1  Audio  Video  System (integrates the other two)

10. The MPEG Standard (2/2)  Two kinds of redundancies in videos  Spatial redundancy  Coding each frame separately with JPEG  Temporal redundancy  Consecutive frames are often almost identical  MPEG-1 output consists of four kinds of frames  I (Intracoded) frames: Self-contained JPEG-encoded still pictures  P (Predictive) frames: Block-by-Block difference with the last frame  B (Bidirectional) frames: Differences between the last and the next frame  D (DC-coded) frames: Block averages used for fast-forward  MPEG-1 can achieve a compression factor of 40  Decoding has about the same complexity as encoding 03 May 2012 Streaming Video Over The Internet 10

11. Key Areas  Video Compression  Application-Layer QoS Control  Congestion control  Error control  Continuous Media Distribution Services  Media Synchronization Mechanisms  Protocols for Streaming Media 03 May 2012 Streaming Video Over The Internet 11

12. Application-Layer QoS Control Congestion Control  Rate Control  Source-Based Rate Control  Receiver-Based Rate Control  Hybrid-Rate Control  Rate Shaping  Frame-dropping filter  Layer-dropping filter  Frequency filter  Re quantization filter Error Control  FEC  Delay-Constrained Retransmission  Error-Resilient Encoding  Error-Concealment 03 May 2012 Streaming Video Over The Internet 12  Avoids congestion and maximizes video quality in the presence to packet loss

13. Rate Control (1/2)  Technique to determine the sending rate of video traffic based on the estimated available bandwidth of the network  Source-based rate control  The sender is responsible for adapting the video transmission rate based on feedback from the receivers  Probe-based approach  Sending rate adjustment to maintain the packet loss ratio below a certain threshold  Model-based approach 03 May 2012 Streaming Video Over The Internet 13

14. Rate Control (2/2)  Receiver-based rate control  The receivers regulate the receiving rate of video streams by adding/dropping channels  Probe-based approach  When no congestion is detected, join a channel  => increase of it’s receiving rate  When congestion is detected, drop a channel  => reduction of it’s receiving rate  Model-based approach  Explicit estimation for the available network bandwidth  Also based on Equation (1)  Hybrid rate Control  Combination of the above mentioned techniques 03 May 2012 Streaming Video Over The Internet 14

15. Rate Shaping  Matches the rate of a pre-compressed video bit stream to the target rate constraint  A rate shaper (or filter) is required for source based rate control 03 May 2012 Streaming Video Over The Internet 15  Types of filters  Codec filter  Performs transcoding between different compression schemes  Frame-dropping filter  I, P,B, D frames in MPEG  Frequency filter  Operates in the frequency domain (DCT coefficients)  Re-quantization filter  Re-quantizes the DCT coefficients with a larger quantization step

16. Error Control (1/2)  Forward Error Correction (FEC)  Add redundant information so that original message can be reconstructed in the presence of packet loss  Channel coding  Stream is chopped into segments  Each segment is packetized into k packets  A block code generates an n-packet block, where n>k  User only needs to receive any k packets out of n of the block  Source coding  N-th group of blocks contains redundant information of (n-1)th group of blocks  Joint source/channel coding  Optimal rate allocation between Channel and Source coding 03 May 2012 Streaming Video Over The Internet 16

17. Error Control (2/2)  Delay-constraint Retransmission 03 May 2012 Streaming Video Over The Internet 17  Error-resilient Encoding  Enhances robustness of compressed video to packet loss  A video is compressed into multiple streams (descriptions)  Each description provides acceptable visual quality  Combined descriptions provide a better visual quality  Error Concealment  Performed by the receiver when packet loss has already occurred  Spatial interpolation  Missing pixel values are reconstructed using neighboring pixels  Temporal interpolation  Loss data is reconstructed using data from the previous frame When the receiver detects the loss of packet N if (T c + RTT + D s < T d (N)) send the request for packet N to the sender Where T c current time RTTestimated round-trip time D s a slack term (e.g receiver’s decoding delay) T d (N)time when packet N is scheduled for display

18. Key Areas  Video Compression  Application-Layer QoS Control  Continuous Media Distribution Services  Network Filtering  Application-level Multicast  Content Replication  Media Synchronization Mechanisms  Protocols for Streaming Media 03 May 2012 Streaming Video Over The Internet 18

19. Network Filtering  Adequate support from the network is critical in order to provide quality multimedia presentations 03 May 2012 Streaming Video Over The Internet 19  Routers have no knowledge of the format of the media streams  Randomly discard packets  Filters receive the client’s requests and adapt the stream sent by the server accordingly  Typically, frame-dropping filters are used  Drop packets in a way that gracefully degrades the stream’s quality  Increase bandwidth efficiency by discarding late frames R: Router

20. Application-level Multicast  Internet’s original design fails to effectively support streaming-media multicast  IP multicast has barriers  Scalability, network management, support for higher layer functionality (e.g. error, flow and congestion control)  Application-level multicast to the rescue  Builds a multicast service on top of the Internet  Media bridge  Routing in the application layer  Each Media Bridge is interconnected with one or more neighboring Media- Bridges through explicit configuration  Collectively employ a distributed application-level multicast routing algorthm 03 May 2012 Streaming Video Over The Internet 20 UnicastMulticast

21. Content Replication Mirroring  Scatter copies of the original multimedia files in different locations around the Internet  Clients retrieve data from nearest duplicate server Caching  Exploits temporal locality  Cache retrieves data from server  Clients retrieve data from cache if available  Cache sharing and cache hierarchies 03 May 2012 Streaming Video Over The Internet 21  Advantages  Increased network’s bandwidth efficiency  Reduced load on streaming servers  Reduced latency for clients  Increased availability

22. Key Areas  Video Compression  Application-Layer QoS Control  Continuous Media Distribution Services  Media Synchronization Mechanisms  Intra-stream Synchronization  Inter-stream Synchronization  Inter-object Synchronization  Protocols for Streaming Media 03 May 2012 Streaming Video Over The Internet 22

23. Media Synchronization  Multimedia applications  Various integrated media streams that must be presented in a synchronized fashion  Media streams may lose synchronization after moving from the server to the client  Three layers of synchronization  Intra-stream synchronization  Maintain continuity of logical data units  Inter-stream synchronization  Maintain temporal relationships among different continuous media  e.g. movements of the lips of a speaker do not correspond to the presented audio  Inter-object synchronization  Relation with time-independent data, such as text, still image and streams  e.g. during a presentation, audio is commenting one slide while another slide is presented 03 May 2012 Streaming Video Over The Internet 23

24. Media Synchronization Mechanisms  Delay in the network is unpredictable due to the best-effort nature of the Internet  Axes-based specifications, or time-stamping  At the source, a stream is time-stamped to keep temporal information within the stream and with respect to other streams  Mechanisms implemented on the end systems  Preventive mechanisms  Minimize latencies and jitters  i.e. disk-reading scheduling algorithms, network transport protocols, operating systems and synchronization schedulers  Corrective mechanisms  Recover synchronization in the presence of synchronization errors  i.e. Stream synchronization protocol (SSP)  Concept of ”intentional delay” 03 May 2012 Streaming Video Over The Internet 24

25. Key Areas  Video Compression  Application-Layer QoS Control  Continuous Media Distribution Services  Media Synchronization Mechanisms  Protocols for Streaming Media  Transport Protocols  Session Control Protocols 03 May 2012 Streaming Video Over The Internet 25

26. Protocols for Streaming Video  Data Plane  Compressed video/audio is packetized at the real-time transport protocol (RTP) layer  RTP packetized streams are then passed to the UDP/TCP layer and IP layer  IP packets are transported over the Internet 03 May 2012 Streaming Video Over The Internet 26  Control Plane  Real-time control protocol (RTCP) and real-time streaming protocol (RTSP) packets are multiplexed at the UDP/TCP layer  IP packets are transported over the Internet

27. Transport Protocols  UDP and TCP provide basic transport functions  UDP and TCP  Multiplexing, Error control (e.g. checksum)  TCP only:  Retransmission, Congestion control, Flow control  UDP is typically employed as the transport protocol for video streams  No packet delivery guarrantees  Receiver needs to rely in upper layer to detect packet loss  RTP and RTCP run on top of UDP/TCP 03 May 2012 Streaming Video Over The Internet 27  RTP : does not guarantee QoS or reliable delivery  Time stamping  Sequence numbering  Payload type identification  Source identification  RTCP : designed to work in conjunction with RTP  QoS feedback  Participant identification  Control packets scaling  Inter-media synchronization  Minimal session control information

28. Session Control Protocols  RTSP (Real-Time Streaming Protocol)  Support VCR-like control operations  (stop, pause/resume, fast forward, fast backward)  Choosing delivery channel (UDT, multicast UDT, TCP)  Establish and control streams of continuous audio and video media between the media servers and the clients  Media retrieval: ask server to setup a session to send the requested media data  Adding media to an existing session: server or client can notify each other about any additional media becoming available to the established session  SIP (Session Initiation Protocol)  Similar to RTSP  Support user mobility by proxying and redirecting request 03 May 2012 Streaming Video Over The Internet 28

29. References  A. S. Tanenbaum, Computer Networks, 4th ed. New Jersey: Prentice Hall, 2002.  Dapeng Wu, Hou, Y.T., Wenwu Zhu, Ya-Qin Zhang, Peha, J.M., "Streaming video over the Internet: approaches and directions“, Circuits and Systems for Video Technology, IEEE Transactions on, vol.11, no.3, pp.282-300, Mar 2001  Dapeng Wu, Yiwei Thoms Hou, Ya-Qin Zhang, "Transporting real-time video over the Internet: challenges and approaches," Proceedings of the IEEE, vol.88, no.12, pp.1855-1877, Dec 2000  Dapeng Wu and Y. Thomas Hou and Jason Yao and Y. Thomas and Hou Jason Yao and H. Jonathan Chao, “Real- time Video over the Internet: A Big Picture”, IEEE NetWorld+Interop, 2000 03 May 2012 Streaming Video Over The Internet 29

30. Questions? Thank you! 03 May 2012 Streaming Video Over The Internet 30