1. Fundamental Abstractions Multimedia Over IP and Wireless Networks Section 14.3 Presented by Scott Kristjanson CMPT-820 Multimedia Systems Instructor: Dr. Mohamed Hefeeda Nov 9 th 2010

2. Scott Kristjanson – CMPT 820 - SFU 2 Slide 2 Presentation Overview  Context for Fundamental Abstractions  Buffering and Leaky Bucket Models  Compound Streams  MBR and Scalable Streams  Temporal Coordinate Systems and Timestamps  Conclusions  Questions  References

3. Scott Kristjanson – CMPT 820 - SFU 3 Slide 3 Context for Fundamental Abstractions Some Context:  The internet is not reliable as a communication channel  But for the purposes of this talk, we will assume it is reliable  Focus is on buffering and transmission of encoded signals  Encoding and Decoding assumed to be instantaneous Some Goals:  Understand fundamental abstractions of streaming media on demand  Understand Leaky Bucket models for idealized bit streams  Understand when it is safe for a client to begin playback  Understand terms: CBR vs VBR Streams, Preroll Delay, Timestamps

4. Scott Kristjanson – CMPT 820 - SFU 4 Slide 4 Buffering and Leaky Bucket Models  Communication Pipeline  Constant Bit Rate (CBR) Transmission  Variable Bit Rate (VBR) Transmission  Buffer Tubes  Leaky Buckets

5. Scott Kristjanson – CMPT 820 - SFU 5 Slide 5 Communications Pipeline Point A – captured frames instantaneously encoded and put into the encoder buffer Point B – when bits enter the communication channel Point C – when bits leave the communication channel Point D – when frames removed from decoder buffer and instantaneously decoded Simple On-Line Streaming Model to Start discussion:  Isochronous (synchronous) noiseless communication channel  Encoder constant bit-rate transmission rate over the channel  Instantaneous source coding rate Recall that to match encode/decode rates to transmission rate:  Encoding Buffer required between Encoder and the Channel  Decoding Buffer required between Channel and the Decoder MULTIMEDIA OVER IP AND WIRELESS NETWORKS, pg 465 FIGURE 14.3

6. Scott Kristjanson – CMPT 820 - SFU 6 Slide 6 Constant Bit Rate (CBR) Media Streams Used by On-Line Media on Demand Streaming when: Constrained by maximum transmission bit rate Consistent bit-rate more important than Quality Encoder uses Quantization Control to control bit rate of Encoded stream Quantization Control

7. Scott Kristjanson – CMPT 820 - SFU 7 Slide 7 Coded Bit Stream Schedules Coding Schedule: Sequence of times at which successive bits in the coded bit stream pass a given point in the communications pipeline Schedule A – Staircase function which increases by b(n) bits at time T(n) Schedule B – when bits enter the communication channel with slope R bits/second Schedule C – when bits leave the communication channel with slope R bits/second Schedule D – when frames removed from decoder buffer and instantaneously decoded MULTIMEDIA OVER IP AND WIRELESS NETWORKS, pg 465 FIGURE 14.4

8. Scott Kristjanson – CMPT 820 - SFU 8 Slide 8 Coded Bit Stream Schedules MULTIMEDIA OVER IP AND WIRELESS NETWORKS, pg 465 FIGURE 14.4 More details: Gap between A and B represents, at any point in time, the number of bits in encoder buffer Gap between B and C is transmission delay Gap between C and D represents, at any point in time, the number of bits in decoder buffer Line A and D are the same shaped line Line B is the lower bound to schedule A (if A goes slower than B, buffer will empty) Line C is upper bound to schedule D (D can’t go any faster than C)

9. Scott Kristjanson – CMPT 820 - SFU 9 Slide 9 Buffer Tube contains a Coding Schedule MULTIMEDIA OVER IP AND WIRELESS NETWORKS, pg 466 FIGURE 14.5 Buffer Tube: Contains the source coding schedule (either line A or D) Encoder uses a “rate control” algorithm to assign number of bits b(n) to each frame to maintain buffer within Tube Resultant Bit Stream is called a CBR bit stream with average rate R Used by On-Line source encoder to ensure output will not cause decoder to overflow/under-run buffer Buffer Tube characterized by three parameters (R, B, F d ) Slope R = transmission speed in bits/second Height B = Capacity in bits of each buffer (end-to-end delay = B/R seconds) F d = Number of bits in decoder buffer before it starts presenting the data (F d = B – F e ) (delay D d = F d /R) F e = Number of bits in encoder buffer before it starts transmitting the data (delay D e = F e /R)

10. Scott Kristjanson – CMPT 820 - SFU 10 Slide 10 Selecting the best Buffer Tube Infinite number of buffer tubes can contain a given finite-duration bit stream Slope R, width B, and offset F e are not unique Lower channel bit rate R can be accommodated with more buffering B Selecting the best Buffer Tube, select slope R: with tightest fit to stream’s schedule {(b(n), T(n))} with average bit rate = (Total Number of Bits)/(Duration of Bit Stream) MULTIMEDIA OVER IP AND WIRELESS NETWORKS, pg 467 FIGURE 14.6

11. Scott Kristjanson – CMPT 820 - SFU 11 Slide 11 Preroll Delay Decoder buffer delay D d also referred to as Preroll Delay Typically largest component of start up delay for Streaming Media on Demand For off-line source encoding, encoder delay not important Focus is on decoder buffer delay D d = F d /R only trade-off between reduced preroll delay and initial frame quality See 14.4 for more details

12. Scott Kristjanson – CMPT 820 - SFU 12 Slide 12 Variable Bit Rate (VBR) Streams VBR Jordi Ribas-Corbera [2], pg 675 FIGURE 2 CBR Jordi Ribas-Corbera [2], pg 674 FIGURE 1 VBR Used for producing Media Stream with constant distortion for consistent quality Can not maintain constant bit rate using QC to control encoded frame size b(n) A VBR steam model is a generalization of the CBR stream model: Both CBR and VBR have schedules with finite sequences of frame sizes over time Both can be contained in Buffer Tubes with the appropriate (R, B, F d ) VBR requires wider Buffer Tube than for equivalent CBR stream VBR requires larger preroll delays than CBR due to variability in bit rate - allows the decoder to have more bits in its buffer to prevent decoder buffer under-run

13. Scott Kristjanson – CMPT 820 - SFU 13 Slide 13 Leaky Bucket Extend model to case where encoder does not use channel continuously  Peak transmission rate R higher than average bit rate of stream  When encoder has bits to send, it sends them at rate R, idle otherwise  More realistic model for packet networks Modelled as a Leaky Bucket: (R, B,F e )  Encoder dumps b(n) bits into leaky bucket at time T(n)  Bits leak out of the bucket and into the channel at rate R  Bucket has capacity B and initial fullness of F e bits  B e (n): Encoder buffer fullness after frame n added to buffer  F e (n): Encoder buffer fullness before frame n added to buffer Given initial Leaky Bucket Fullness F e before bit transmission starts, The system can be modelled as:

14. Scott Kristjanson – CMPT 820 - SFU 14 Slide 14 The Tightest Leaky Bucket A leaky bucket (R, B, F e ) is said to contain a bit stream with schedule {(b(n), τ(n))} if the bucket does not overflow, that is, B e (n) ≤ B for all n = 0, 1,..., N For a given stream, want the Tightest Leaky Bucket which contains it. Define minimum bucket capacity given leak rate R and initial fullness F e as: Define the corresponding initial decoder buffer fullness as: Denote the minimum of each of these over F e as: Each is minimized by the same value of F e which is equal to: Denote the minimum of each of these over F e as: Define the corresponding initial decoder buffer fullness as:

15. Scott Kristjanson – CMPT 820 - SFU 15 Slide 15 Leaky Bucket Parameters as a function of R Buffer size B min depends on Bucket Leak Rate R For R < Rc, larger and larger buffers are required For R >> Rc, buffers seldom filled and few required For off-line encoding of media on demand, Value of R used by client often differs from source coding rate R c Encoder pre-computes required buffer size and preroll delay for various values of R Pre-computed values for various R can be stored in header as (R, B min, F d ) triplets MULTIMEDIA OVER IP AND WIRELESS NETWORKS, pg 470 FIGURE 14.7 R c = 600Kbps

16. Scott Kristjanson – CMPT 820 - SFU 16 Slide 16 Compound Streams Streaming Media files often contain multiple independent streams Audio Stream Video Streams Leaky bucket for aggregate stream: Derived as sum of leaky buckets containing each component stream Will contain the combined stream, but not strictly the tightest possible Given: Audio stream with leaky bucket (R a, B a, F a ) Video stream with leaky bucket (R v, B v, F v ) Compute aggregate stream leaky bucket as follows:

17. Scott Kristjanson – CMPT 820 - SFU 17 Slide 17 Component Streams with Multiple Leaky Buckets Given L a audio and L v video leaky buckets: There are (L a x L v ) possible pairs but only (L a + L v ) pairs are of interest. They lie on lower convex hull of set used to compute B e min (R) & F d min (R) Indices for pairs easily found by minimizing the Lagrangian for some positive Lagrange multiplier λ>0 : Chose L a audio leaky buckets by minimizing the Lagrangian: Chose L v video leaky buckets by minimizing the Lagrangian: Pair up Audio and Video leaky buckets with same λ

18. Scott Kristjanson – CMPT 820 - SFU 18 Slide 18 MBR and Scalable Streams MBR adds choice of streaming from set of mutually exclusive media streams, each with a different coding rate. Scalable Streams similar to compound streams except must chose sets of layers. Lots of combinations! Most are not useful. Choose based on selecting a desired distortion level versus bit rate. For each combined stream (i,j), define an over-all distortion and source rate: Select a “desirable” subset of audio/video combinations (i,j) such that: D i,j ≤ D i’,j’ for all (i’,j’) such that R i’,j’ ≤ R i,j As with compound streams, use Lagrangian method to find desirable pairs.

19. Scott Kristjanson – CMPT 820 - SFU 19 Slide 19 Temporal Coordinate Systems Media time refers to clock running on capture device Monotonically increasing timestamps of the original content One second of media time elapses in one second of real time Client time t refers to playback time at client Time of media playback at client Content may be played back at different values of V V < 1 for Slow-Motion V = 1 for PLAY V > 1 for Fast-Forward Conversion from media time to client time:

20. Scott Kristjanson – CMPT 820 - SFU 20 Slide 20 Timestamps Decoder timestamp (DTS) arrival deadline for decoding the frame at the client Provides decoding dependency order for P and B Frames Presentation timestamp (PTS) Tells renderer when to render and present the frames Useful for synchronizing concurrent streams Decoder and Presentation order can differ Presentation Order: I 0 B 1 B 2 P 3 B 4 B 5 P 6 Decoding Order:I 0 P 3 B 1 B 2 P 6 B 4 B 5

21. Scott Kristjanson – CMPT 820 - SFU 21 Slide 21 Summary  Media on Demand streams modelled by Encoder using Buffer Tubes  Leaky Bucket model more appropriate for transmission over packet networks  Leaky Bucket parameters can be summed for aggregate streams  Can use Langrage method to determine best set of Leaky Bucket pairs to use  Preroll Delay a function of Buffer requirements and Transmission Rate R  Decoder Timestamps help decoder sort and decode frames in decoder buffer  Presentation Timestamps describe the order in which to present the frames

22. Scott Kristjanson – CMPT 820 - SFU 22 Slide 22 Questions? Tight bounds on Leaky Buckets Does not seem realistic Need to take into account packet loss and random network delays Are these theoretical bounds useful in practice? Start-up delays and buffer sizes not normally a concern for users Calculating Preroll based on B min and R seems unrealistic assumes Rate R is known, but it is not constant in practice depends on many factors such as Server and Network Load varies over time and is not known when streaming starts Need more adaptive buffering strategy Summing Leaky Bucket parameters for Compound Streams As text points out, these are rough approximations only

23. Scott Kristjanson – CMPT 820 - SFU 23 Slide 23 Time for a Quiz! Q1: When is a Leaky Bucket model more appropriate than Buffer Tube? Answer: When using a shared channel such as packet network Q2: What is Preroll and why is it important? Answer: Delay between Decoder receiving first frames and displaying them. Important to prevent buffer under-flow which would stall the display. Q3: According to the Author, how do you minimize Preroll? Answer: Use Tightest possible Leaky Bucket and/or trade-off initial frame quality

24. Scott Kristjanson – CMPT 820 - SFU 24 Slide 24 References: [1] M[1] Multimedia Over IP and Wireless Networks Mihaela Van Der Schaar, Philip A. Chou Academic Press, 2007 [2] A Generalized Hypothetical Reference Decoder for H.264/AVC Jordi Ribas-Corbera, Philip A. Chou, Shankar L. Regunathan IEEE Transactions on Circuits and Systems for Video Technology, 13(7), July 2003

25. Scott Kristjanson – CMPT 820 - SFU 25 Slide 25 Thanks!