CS :: Fall 2003 Media Scaling / Content Adaptation Ketan Mayer-Patel

CS :: Fall 2003 Media Scaling Overview Change encoding of the media to accommodate changes in available bandwidth. Components of the problem: –Estimating bandwidth. –What can be adjusted. –How to adjust. –Cross media issues. –Transcoding pre-encoded media.

CS :: Fall 2003 Estimating Bandwidth Requires feedback. –How often? –In what form? TCP-friendliness –Current thought: act like TCP. –Padhye’s equation:

CS :: Fall 2003 Feedback How often? –Design of TFRC says you want to limit this to once per RTT. –Otherwise, rate of feedback is a design decision. Tradeoff: responsiveness vs. feedback bandwidth Should be related to when the source can use the info.

CS :: Fall 2003 Feedback In what form? –Media independent information: Loss rate and delay. –Media specific information: Quality of reconstructed media. Not necessarily well related to loss and delay. User preferences –For example: Regions of interest. Media specific feedback very application dependent.

CS :: Fall 2003 Scaling Dimensions Different media can be scaled along different dimensions. Two major considerations: –Encoding support. The media representation often restricts the range and type of scalability. –Perceptual quality. Non-linear subjective response to changes. Think of frame size changes.

CS :: Fall 2003 Video Temporal scaling –Reduce the resolution of the stream by reducing the frame rate Spatial scaling –Reduce the number of pixels in an image Frequency scaling –Reduce the number of DCT coefficients used in compression Amplitude scaling –Reduce the color depth of each pixel in the image Color space scaling –Reduce the number of colors available for displaying the image

CS :: Fall 2003 Perceptual Impact and MPEG Temporal Scaling Spatial Scaling Frequency Scaling Amplitude Scaling Color Scaling Medium High Medium Low Perceptual Impact Highly Very Not very Effectiveness Possible Unsupported Supported Unsupported Representable

CS :: Fall 2003 Scaling Dynamics Given some adjustment parameter p: –How can we adjust p given target rate r? Depends on how p is related to r. 3 basis situations: –Well known in advance. –Well known for future. –Indeterminate

CS :: Fall 2003 Scaling Dynamics Two types of “well known” relationships. –Immediate relationship. Given target bitrate r and media lifespan, parameter can be perfectly adjusted to give desired number of bits. This is rare. –Future relationship. Parameter is adjusted after the fact to conform to target rate next time media is produced. Example: frame rate.

CS :: Fall 2003 Scaling Dynamics If relationship is indeterminate, what can we do? –Incremental adjustments in appropriate direction. –Basic algorithm: At time t i Encode using p i Calculate rate mismatch d i p i+1 = adjustment(p i, d i ) –Issues: Accounting for errors between target and achieved rates. Designing a good adjustment function.

CS :: Fall 2003 Bitrate Accounting Cumulative accounting. –Spectrum of accounting choices: No carry. –Deficit/surplus simply informs the adjustment function. Indefinite carry. –In the limit, overall average rate matches target exactly. In between, a variety of decay and window-based possibilities.

CS :: Fall 2003 Adjustment Functions Basic issues: –Shape –Stability Lots of control theory here. Informal rules of thumb: –Damp small changes. –Keep history (feedback) Take advantage of domain-specific knowledge.

CS :: Fall 2003 Quality Example Quality/rate tradeoff for video generally looks like this: Quality Rate The problem is that if we don’t know what the curve is. Useful quality adjustment range

CS :: Fall 2003 Quality Example Possible strategies? –Multipass solutions. Generally take too long. –Content analysis Complex and only work in restricted domains. –Model the tradeoff curve. Simple Only works well is model is close to reality. Also known as a “rate-distortion framework”

CS :: Fall 2003 Rate-Distortion Ideal: –bits vs. quality as a closed form, invertible formula Strategies –Choose a proxy for quality SNR measures Playback characteristics –Example: recovered frames per second User feedback –Choose a proxy for rate Q Factor Number of non-zero coefficients Framerate

CS :: Fall 2003 Rate-Distortion cont’d –Fit curve Choice of function family based on representative sampling of content domain. Derive parameters –Static for content domain. –Adaptively based on recent ADU’s. –Analysis of sub-sampled ADU.

CS :: Fall 2003 Cross Media Issues Aggregate behavior of multiple related streams may require different per stream behavior. –Example: 64 kbs audio with 300 kbs video. –Suppose available bandwidth drops to 182 kbs 32 kbs audio, 150 kbs video 64 kbs audio, 118 kbs video. Integrate the adaptation process Cross inform the adaptation process

CS :: Fall 2003 Transcoding Have assumed that media encoded on the fly so far. What about pre-recorded material? Transcoding issues: –Computational complexity Compressed domain processing. –Scalability and availability of the service. More common solution: –Preset, multiple representations.

CS :: Fall 2003 InfoPyramid Holistic approach to multimedia documents. Simultaneous adaptation of all related media. Utility curves defined for each media type along its adaptive dimension. Utility curve for document is summation of components. Document adaptation done as a resource allocation problem.