Concepts of Multimedia Processing and Transmission IT 481, Lecture #8 Dennis McCaughey, Ph.D. 30 October, 2006

08/28/2006 IT 481, Fall Broadcast Environment

08/28/2006 IT 481, Fall Video Transmission System Example

08/28/2006 IT 481, Fall Rate Control Video applications involve real-time display of the decoded sequence. Transmission across a Constant Bit Rate (CBR) channel requires a constant end-to- end delay between the time that the encoder processes a frame and the time at which that same frame is available to the decoder A buffer is require to match the Variable Bit Rate (VBR) encoder with the CBR channel

08/28/2006 IT 481, Fall Admission Control Decision if a new connection with a given set of QoS parameters can be allowed into the network Criteria: the new connection will not degrade the QoS of other ongoing connections Need models for predicting the expected bit rates of video sources especially packet switched networks Simpler for circuit switched networks where the transmission resources are constant over the duration of the connection

08/28/2006 IT 481, Fall Usage Parameter Control Prevent sources from exceeding the traffic parameters negotiated at call setup –Maliciously or unintentionally Example: tracking cell rate in an ATM network and verifies that the source remains within that rate Example: “Leaky Bucket” –a cell counter incremented with each cell arrival –Decremented at fixed intervals –If counter exceeds a threshold cells are dropped or marked for deletion

08/28/2006 IT 481, Fall Multi-Resolution Encoding Separation of the information into two or more layers or resolutions Coarse Resolution –Reduce the spatial or temporal resolution of the sequence or by having images of lower quality –Transmitted by high priority packets Detailed Resolution –Transmitted by lower priority packets than can be discarded first Multi-Resolution Encoding enables efficient error concealment techniques –e.g. interpolate lost information from packets that were not lost

08/28/2006 IT 481, Fall Layered Video Coding Provide multiple image quality levels simultaneously across multiple network channels –e.g. Standard Resolution and High Definition TV Each receiver individually tunes its reception rate by adjusting the number of layers that it receives Two problems to solve –Layered Compression Multiple image quality levels –Layered Transmission Problem Selective delivery of layer subsets to individual receivers

08/28/2006 IT 481, Fall Structure of a Simulcast Coder

08/28/2006 IT 481, Fall Simulcast Coder Produces a multirate set of signals that are independent of one another Each layer provides improved quality, independent of sublayers Single layer (nonscalable) decoder can decode any layer Total available bandwidth is partitioned dependent on the desired quality for each independent sublayer

08/28/2006 IT 481, Fall Structure of a Layered Coder

08/28/2006 IT 481, Fall Layered Compression Input signal is compressed into a number of discrete layers, arranged in a hierarchy that provides progressive refinement –If first layer is received, decoder will produce the lowest quality version of the signal –If the decoder receives two layers, it will combine the second layer with the first layer to produce improved quality –Overall, the quality improves with the number of layers that are received and decoded Layered compression and layered transmission must be combined to be effective Slide courtesy of Hung Nguyuen

08/28/2006 IT 481, Fall Simulcast vs. Layered Approach Distortion measures quality degradation between reconstructed and original signals –D I (R): an ideal coder –D R (R): a real coder –D L (R): a layered coder –D S (R): a simulcast coder

08/28/2006 IT 481, Fall Layered Transmission Each layer is transmitted on a different network channel. Network only forwards the number of layers that each physical link can support. Each user receives the best quality signal that the network can deliver. The network must be able to drop layers selectively at each bottleneck link

08/28/2006 IT 481, Fall Video Communication System

08/28/2006 IT 481, Fall Error Resilience Redundancy is added to the compressed bitstream to allow the detection and correction of errors –Can be added in either the source or channel encoder Shannon Information Theory: –Separately design the source and channel coders to achieve error-free transmission so long as the source is represented by a rate below the channel capacity Source should compress the source as much as possible Channel coder, via Forward Error Correction (FEC) adds redundancy bits to enable error detection and correction

08/28/2006 IT 481, Fall Binary Symmetric Channel

08/28/2006 IT 481, Fall Shannon’s Capacity Theorem If the Rate (R) of a code R = log2(m)/L is less than channel capacity C, there exists a combination of source and channel encoders such that the source can be communicated over the channel with fidelity arbitrarily close to perfect –m = Number of message words –L = Number of code word bits

08/28/2006 IT 481, Fall Types of Codes Block Codes –Hamming Codes –Bose-Chaudhuri-Hocquenhem BCH Codes –Reed-Solomon Codes Convolutional Codes

08/28/2006 IT 481, Fall Example: (n,k) Hamming Code (n,k) = (2 m -1,2 m -1-m), for any m

08/28/2006 IT 481, Fall (7,4) Hamming Code Corrects all single errors

08/28/2006 IT 481, Fall Parity Check Matrix

08/28/2006 IT 481, Fall Decoding

08/28/2006 IT 481, Fall (7,4) Messages and Codes (Systematic Code) MessagesCode Words

08/28/2006 IT 481, Fall Received Code Words Received Words errors Transmitted Words

08/28/2006 IT 481, Fall Syndromes BinaryDecimal Syndrome Table Rows in Syndrome Table indicate error locations Syndromes point to rows

08/28/2006 IT 481, Fall Performance Bit Error Probability p=10 -6 M = 100 Bits Probability that message is received correctly, P c =( ) 100 =.367 Use a hamming (7,4) code –25 7-Bit code words with 4 info bits per word –Probability that a word is correctly decoded P w = probability that one or fewer bits are decoded incorrectly =( ) + 7 (10 -2 ) =.998 –Probability that message is received correctly =(.998) 25 =.95 A substantial improvement

08/28/2006 IT 481, Fall Video Streaming Architecture

08/28/2006 IT 481, Fall Scalable and Non-Scalable Coder/Decoder Scalable Encoder/Decoder A nonscalable video encoder generates one compressed bit-stream. Scalable video encoder compresses a raw video sequence into multiple sub-streams

08/28/2006 IT 481, Fall Video Stream Architecture Building Blocks Video Compression Application Layer QoS Control Continuous Media Distribution Services Streaming Servers Media Synchronization Protocols for Multimedia

08/28/2006 IT 481, Fall Video Compression Video compression schemes can be classified into two categories: scalable and nonscalable Scalable video is capable of gracefully coping with the bandwidth fluc­tuations in the Internet. Primarily concerned with scalable video coding techniques. We will also discuss the requirements imposed by streaming applications on the video encoder and decoder.

08/28/2006 IT 481, Fall Application Layer QoS Control Congestion control takes the form of rate control. Three kinds of rate control: 1) Source-based, –The source-based rate control is suitable for unicast; –Most recent studies on source-based rate control have been focused on TCP-friendly adaptation –A number of TCP-friendly adaptation schemes have been proposed and demonstrated to achieve certain degree of fairness among competing connections, including TCP connections. –However, strictly TCP-like rate control may result in sharp reductions in the transmission rate, and possibly unpleasant visual quality [66]. –Needs further investigation on how to trade off responsiveness in detecting and reacting to congestion with smooth fluctuation in visual quality. 2) Receiver-based, and 3)Hybrid rate-control. –Both are suitable for multicast since both can achieve good tradeoff between bandwidth efficiency and service flexibility for multicast video.

08/28/2006 IT 481, Fall Source Based Rate Control

08/28/2006 IT 481, Fall Continuous Media Distribution Services Continuous media distribution services are built on top of the best-effort Internet with the aim of achieving QoS and efficiency for streaming video. A major topic of active research is how to build a scalable, efficient, cost-effective and incremental deployable infrastructure for continuous media distribution.

08/28/2006 IT 481, Fall Streaming Servers Essential in providing streaming services. Current research efforts include: –1) Efficient support VCR-like interac­tive control; –2) The design of efficient and reliable storage and retrieval of multimedia objects on disk arrays; –3) The design of highly scalable multimedia servers in a variety of environments ranging from video-on-demand servers to integrated multimedia file system –4) The design of fault-tolerant storage systems Desirable features of both parity and mirroring –Trade off the parity group size with the number of disks across which original data of a single disk is replicated for mirroring.

08/28/2006 IT 481, Fall Media Synchronization Intra-stream synchronization: The lowest layer of con­tinuous media or time-dependent data (such as video and audio) is the media layer. –Maintains the continuity of logical data units. –Prevents pauses and gaps. Inter-stream synchronization: The second layer of time-dependent data is the stream layer. –Maintains temporal relationships among different continuous media. –Prevents skew between the streams may become intolerable. –eg, users could be annoyed if they notice that the movements of the lips of a speaker do not correspond to the presented audio. Inter-object synchronization: The highest layer of a mul­timedia document is the object layer, which integrates streams and time- independent data such as text and still images. –Synchronization at this layer is referred to as inter-object synchronization. –The objective of inter-ob­ject synchronization is to start and stop the presentation of the time-independent data within a tolerable time interval, – e.g., the audience of a slide show could be annoyed if the audio is commenting one slide while another slide is being presented

08/28/2006 IT 481, Fall Protocols for Multimedia Network Layer Protocol Transport Protocol Session Control Protocol –UDP –TCP –RTP –RTCP

08/28/2006 IT 481, Fall Standardized Protocols Several protocols have been standardized for communication between clients and streaming servers. Future research topics on design of protocols include: –1) How to take caches into account (e.g., how to communicate with continuous media caches and how to control continuous media caches); –2) How to efficiently support pause/resume operations in caches (since the pause/resume operations interfere with the sharing of a multimedia stream among different viewers); and –3) How to provide security in the protocols

08/28/2006 IT 481, Fall Unicast and Multicast Unicast video distribution using multiple point-to-point connections. Multicast video distribution using point-to-multipoint transmission.

08/28/2006 IT 481, Fall Streaming Server Components Communicator: –A communicator involves the application layer and transport protocols implemented on the server. –Through a communicator, the clients can communicate with a server and retrieve multimedia contents in a continuous and synchronous manner. Operating system: –Different from traditional operating systems, –An operating system for streaming services needs to satisfy real-time requirements for streaming applica­tions. Storage system: A storage system for streaming services

08/28/2006 IT 481, Fall Protocol Stack for Multimedia

08/28/2006 IT 481, Fall RTP RTP does not guarantee QoS or reliable delivery, but rather, provides the following functions in support of media streaming: Time-stamping: RTP provides time-stamping to synchronize different media streams. Sequence numbering: RTP employs sequence numbering to place the incoming RTP packets in the correct order. Since packets arriving at the receiver may be out of sequence (UDP does not deliver packets in sequence),. Payload type identification: The type of the payload contained in an RTP packet is indicated by an RTP-header field called payload type identifier. –The receiver interprets the content of the packet based on the payload type iden­tifier. –Certain common payload types such as MPEG-audio and video have been assigned payload type numbers –For other payloads, this assignment can be done with session control protocols. Source identification: The source of each RTP packet is identified by an RTP-header field called Synchronization Source identifier (SSRC), which provides a means for the receiver to distinguish different sources.

08/28/2006 IT 481, Fall RTCP QoS feedback: This is the primary function of RTCP. –RTCP provides feedback to an application regarding the quality of data distribution. –The feedback is in the form of sender reports (sent by the source) and receiver re-ports (sent by the receiver). –The reports can contain in-formation on the quality of reception such as: –1) Fraction of the lost RTP packets, since the last report; –2) Cumulative number of lost packets, since the beginning of re­ ception; –3) Packet interarrival jitter; and –4) Delay since receiving the last sender’s report. Participant identification: A source can be identified by the SSRC field in the RTP header. – RTCP provides a human-friendly mechanism for source identification. –RTCP SDES (source description) packets contain textual information called canonical names as globally unique identifiers of the session participants. –It may include a user’s name, telephone number, address, and other information.

08/28/2006 IT 481, Fall RTCP Control packets scaling: To scale the RTCP control packet transmission with the number of participants, a control mechanism is designed as follows. – The control mechanism keeps the total control packets to 5% of the total session bandwidth. –Among the control packets, 25% are allocated to the sender reports and 75% to the receiver reports. –To prevent control packet starvation, at least one control packet is sent within 5 s at the sender or receiver. Inter-media synchronization: RTCP sender reports contain an indication of real time and the corresponding RTP timestamp. This can be used in inter-media synchroniza­tion like lip synchronization in video. Minimal session control information. This optional functionality can be used for transporting session information such as names of the participants.

08/28/2006 IT 481, Fall Session Control Protocols RTSP functions –Support VCR-like control operations such as stop, pause/re­sume, fast forward, and fast backward. –Provides a means for choosing delivery channels (e.g., UDP, mul­ticast UDP, or TCP), and delivery mechanisms based upon RTP. –RTSP works for multicast as well as unicast. –Also establishes control streams of continuous audio and video media between the media servers and the clients. Specifically, RTSP provides the following operations. –Media retrieval: The client can request a presentation description, and ask the server to setup a session to send the requested media data; –Adding media to an existing session: The server or the client can notify each other about any additional media becoming available to the established session

08/28/2006 IT 481, Fall Session Control Protocols Similar to RTSP, SIP can also create and terminate sessions with one or more par­ ticipants. Unlike RTSP, SIP supports user mobility by proxying and redirecting requests to the user’s current location

08/28/2006 IT 481, Fall References D. Wu et. Al. “Streaming Video over the Internet: Approaches and Directions”, IEEE Transactions on Circuits and Systems for Video Technology, Vol. 11, No.3, March 2001