CMPT365 Multimedia Systems 1 Multimedia Networking/Communications Spring 2015 CMPT 365 Multimedia Systems

2 Outline  Delay and loss r RTSP r Multicast/broadcast

CMPT365 Multimedia Systems 3 Multimedia Communication/Networking: What’s it ? Key issue: Media distribution over network: r Media (audio/video) at source r transmitted to client r streaming: client playout begins before all data has arrived

CMPT365 Multimedia Systems 4 Streaming Multimedia: What’s it ? Cumulative data streaming: at this time, client playing out early part of video, while server still sending later part of video time

CMPT365 Multimedia Systems 5 MM Networking Applications Fundamental characteristics: r Typically delay sensitive m end-to-end delay m delay jitter r But loss tolerant: infrequent losses cause minor glitches r Opposite to data, which are loss intolerant but delay tolerant. Classes of MM applications: 1) Streaming stored audio and video 2) Streaming live audio and video 3) Real-time interactive audio and video Jitter is the variability of packet delays within the same packet stream

CMPT365 Multimedia Systems 6 (1) Streaming Stored Multimedia 1. video recorded 2. video sent 3. video received, played out at client Cumulative data streaming: at this time, client playing out early part of video, while server still sending later part of video network delay time

CMPT365 Multimedia Systems 7 (1) Streaming Stored Multimedia: Interactivity r VCR-like functionality: client can pause, rewind, FF, push slider bar m 10 sec initial delay OK m 1-2 sec until command effect OK m RTSP often used (more later) r timing constraint for still-to-be transmitted data: in time for playout

CMPT365 Multimedia Systems 8 (2) Streaming Live Multimedia Examples: r Internet radio talk show r Live sporting event Streaming r playback buffer r playback can lag tens of seconds after transmission r still have timing constraint Interactivity r fast forward impossible r rewind, pause possible!

CMPT365 Multimedia Systems 9 (3) Interactive, Real-Time Multimedia r end-end delay requirements: m audio: < 150 msec good, < 400 msec OK includes application-level (packetization) and network delays higher delays noticeable, impair interactivity r session initialization m how does callee advertise its IP address, port number, encoding algorithms? r applications: IP telephony, video conference, distributed interactive worlds

CMPT365 Multimedia Systems 10 When media meet Internet … Multimedia applications: network audio and video (“continuous media”) network provides application with Quality-of-Service needed for application to function. QoS

CMPT365 Multimedia Systems 11 Internet multimedia: simplest approach audio, video not streamed: r no, “pipelining,” long delays until playout! r audio or video stored in file r files transferred as HTTP object m received in entirety at client m then passed to player

CMPT365 Multimedia Systems 12 Internet multimedia: naïve streaming approach r browser GETs metafile r browser launches player, passing metafile r player contacts server (HTTP) r server streams audio/video to player (HTTP/TCP)

CMPT365 Multimedia Systems 13 Streaming Multimedia: UDP or TCP? TCP r rate fluctuates due to TCP congestion control m Saw-tooth behavior r larger playout delay m smooth TCP delivery rate/retransmission r HTTP/TCP passes more easily through firewalls UDP r short playout delay (2-5 seconds) r lower overhead r smoothed rate r error recovery m Not 100%, but real-time

CMPT365 Multimedia Systems 14 Beyond Congestion Control: Client-side operations r jitter removal r decompression r error concealment r graphical user interface w/ controls for interactivity Media Player

CMPT365 Multimedia Systems 15 constant bit rate video transmission Cumulative data time variable network delay client video reception constant bit rate video playout at client client playout delay buffered video Jitter Removal: Client Buffering

CMPT365 Multimedia Systems 16 Client Buffering r Client-side buffering, playout delay compensate for network-added delay, delay jitter r Q: How to determine the playout delay ? buffered video variable fill rate, x(t) constant drain rate, d

CMPT365 Multimedia Systems 17 Case Study: Real-time interactive applications r Voice over IP (VoIP) r Videoconference with Webcams - Internet phone, Skype, GoogleTalk … Going to now look at Internet phone example in detail

CMPT365 Multimedia Systems 18 Interactive Multimedia: Internet Phone Introduce Internet Phone by way of an example r speaker’s audio: alternating talk spurts, silent periods. m 64 kbps during talk spurt r pkts generated only during talk spurts m 20 msec chunks at 8 Kbytes/sec: 160 bytes data r application-layer header added to each chunk. r Chunk+header encapsulated into UDP segment. r application sends UDP segment into socket every 20 msec during talkspurt.

CMPT365 Multimedia Systems 19 Internet Phone: Packet Loss and Delay r network loss: IP datagram lost due to network congestion (router buffer overflow) r delay loss: IP datagram arrives too late for playout at receiver m delays: processing, queueing in network; end-system (sender, receiver) delays m typical maximum tolerable delay: 400 ms r loss tolerance: depending on voice encoding, losses concealed, packet loss rates between 1% and 10% can be tolerated.

CMPT365 Multimedia Systems 20 constant bit rate transmission Cumulative data time variable network delay (jitter) client reception constant bit rate playout at client client playout delay buffered data Delay Jitter r Consider the end-to-end delays of two consecutive packets: difference can be more or less than 20 msec

CMPT365 Multimedia Systems 21 Internet Phone: Fixed Playout Delay r Receiver attempts to playout each chunk exactly q msecs after chunk was generated. m chunk has time stamp t: play out chunk at t+q. m chunk arrives after t+q: data arrives too late for playout, data “lost” r Tradeoff for q: m large q: less packet loss m small q: better interactive experience

CMPT365 Multimedia Systems 22 Fixed Playout Delay Sender generates packets every 20 msec during talk spurt. First packet received at time r First playout schedule: begins at p Second playout schedule: begins at p’

CMPT365 Multimedia Systems 23 Internet Phone: Fixed Playout Delay r Tradeoff for q: m large q: less packet loss m small q: better interactive experience

CMPT365 Multimedia Systems 24 Adaptive Playout Delay, I Dynamic estimate of average delay at receiver: where u is a fixed constant (e.g., u =.01). r Goal: minimize playout delay, keeping late loss rate low r Approach: adaptive playout delay adjustment: m Estimate network delay, adjust playout delay at beginning of each talk spurt. m Silent periods compressed and elongated. m Chunks still played out every 20 msec during talk spurt.

CMPT365 Multimedia Systems 25 Adaptive playout delay II Also useful to estimate the average deviation of the delay, v i : The estimates d i and v i are calculated for every received packet, although they are only used at the beginning of a talk spurt. For first packet in talk spurt, playout time is: where K is a positive constant. Remaining packets in talkspurt are played out periodically

CMPT365 Multimedia Systems 26 How to deal with packet loss ? r Network loss r Delay loss r Feekback ? m Non-realtime

CMPT365 Multimedia Systems 27 Recovery from packet loss – FEC (1) forward error correction (FEC): simple scheme r for every group of n chunks create a redundant chunk by exclusive OR-ing the n original chunks r send out n+1 chunks, increasing the bandwidth by factor 1/n. r can reconstruct the original n chunks if there is at most one lost chunk from the n+1 chunks

CMPT365 Multimedia Systems 28 Recovery from packet loss – FEC (2) r Playout delay needs to be fixed to the time to receive all n+1 packets r Tradeoff: m increase n, less bandwidth waste m increase n, longer playout delay m increase n, higher probability that 2 or more chunks will be lost

CMPT365 Multimedia Systems 29 Recovery from packet loss – FEC (3) 2nd FEC scheme “piggyback lower quality stream” send lower resolution audio stream as the redundant information for example, nominal stream PCM at 64 kbps and redundant stream GSM at 13 kbps. Whenever there is non-consecutive loss, the receiver can conceal the loss. Can also append (n-1)st and (n-2)nd low-bit rate chunk

CMPT365 Multimedia Systems 30 Recovery from packet loss - Interleaving Interleaving r chunks are broken up into smaller units r for example, 4 5 msec units per chunk r Packet contains small units from different chunks r if packet is lost, still have most of every chunk r has no redundancy overhead r but adds to playout delay

CMPT365 Multimedia Systems 31 Outline r Delay and loss r RTSP r Multicast/broadcast

CMPT365 Multimedia Systems 32 User Control of Streaming Media: RTSP HTTP r Does not target multimedia content r No commands for fast forward, etc. RTSP: RFC 2326 r Client-server application layer protocol. r For user to control display: rewind, fast forward, pause, resume, repositioning, etc… Real-time Streaming Protocol

CMPT365 Multimedia Systems 33 RTSP Example Scenario: r metafile communicated to web browser r browser launches player r player sets up an RTSP control connection, data connection to streaming server

CMPT365 Multimedia Systems 34 Metafile Example Twister <track type=audio e="PCMU/8000/1" src = "rtsp://audio.example.com/twister/audio.en/lofi"> <track type=audio e="DVI4/16000/2" pt="90 DVI4/8000/1" src="rtsp://audio.example.com/twister/audio.en/hifi"> <track type="video/jpeg" src="rtsp://video.example.com/twister/video">

CMPT365 Multimedia Systems 35 RTSP Operation

CMPT365 Multimedia Systems 36 RTSP Exchange Example C: SETUP rtsp://audio.example.com/twister/audio RTSP/1.0 Transport: rtp/udp; compression; port=3056; mode=PLAY S: RTSP/ OK Session 4231 C: PLAY rtsp://audio.example.com/twister/audio.en/lofi RTSP/1.0 Session: 4231 Range: npt=0-  normal playback time C: PAUSE rtsp://audio.example.com/twister/audio.en/lofi RTSP/1.0 Session: 4231 Range: npt=37 C: TEARDOWN rtsp://audio.example.com/twister/audio.en/lofi RTSP/1.0 Session: 4231 S: OK

CMPT365 Multimedia Systems 37 Outline r Delay and loss r RTSP r Multicast/Broadcast

CMPT365 Multimedia Systems 38 Challenges: Heterogeneity r Receivers m Bandwidth M ADSL,10M/100M Ethernet, 384K wireless … m Computation power - Tablet, Laptop, Workstation … r Sessions m Uneven session populations

CMPT365 Multimedia Systems 39 Not an Atypical Network

CMPT365 Multimedia Systems 40 Solution 1: Simulcast r Replicated streams m Same video program m Different rates r Each stream for a cluster of receivers m Same access technology m Same bottleneck 1.5 Mbps encoding 28.8 Kbps encoding

CMPT365 Multimedia Systems 41 Solution 1: Simulcast r Pros m Simplicity m Compatibility r Cons m Redundancy Number of Streams Redundancy Mismatch Single-rate MulticastSimulcastMultiple-unicast

CMPT365 Multimedia Systems 42 Solution 2: Layered Multicast r Cumulative layered coding (Scalable coding) m Base layer - Most important feature - Low rate, low quality m Enhancement layers - Progressively refine the quality r Receiver-driven Adaptation m Layer  Multicast group m Congestion ? -No: Join group -Yes: Leave group

CMPT365 Multimedia Systems 43 Solution 2: Layered Multicast r Pros m No replication redundancy m End-to-end control r Cons m Incompatibility m Overhead for scalability -Computation time, bandwidth Base Layer Enhancement Layer I B P B P B P Base Layer Enhancement Layer P B B B P P PI Base Layer Enhancement Layer I P P P P B B B Temporal scalability Spatial scalability SNR scalability.

CMPT365 Multimedia Systems 44 Solution 3: Proxy-based Video Filtering r Multiple video proxies (agents, gateways,..) r Each proxy for a confined region m Bandwidth monitoring m Video filtering - Frame dropping, transcoding … r Pros m Fine-grained rate control m Work with no IP multicast r Cons m Expensive m High computation cost Sender ReceiverRouter Proxy

CMPT365 Multimedia Systems 45 Further exploration r Texbook Chapter 15