Royalty Cost Based Optimization for Video Compression Emrah Akyol, Onur G. Guleryuz, and M. Reha Civanlar DoCoMo USA Labs, Palo Alto, CA USA

2 Outline Setup and motivation Setup and motivation Problem definition Problem definition Our solution with some interesting simulation results. Our solution with some interesting simulation results. Conclusion Conclusion

3 Setup-1: Diverse set of terminals in media delivery Cell phones PDAs … HDTVs Media Encoding 1 Encoding 2 Encoding K … Required quality/ effective bandwidth … Increasing media quality/effective bandwidth Example terminals decoding the media content licensing cost 1 content licensing cost 2 content licensing cost K

4 Setup-2: Diverse set of tools Tool 1 Tool 2 Tool 3 Tool 4 … Tool T Media data Decoded Media Tool j = video motion compensation Integer-pixel accuracy motion compensation 1/2-pixel accuracy motion compensation 1/4-pixel accuracy motion compensation 1/8-pixel accuracy motion compensation Increasing quality Compression tools, error correction tools, transport tools, … Tools have different royalty/licensing costs. (Media Consumer)

5 Thought Experiment Many media delivery technologies available. One can transport media through a variety of networks, using a vast range of tools that correspond to vast ranges in efficiency in end to end delivery. Rather than restricting to rigid toolsets, standard profiles, etc., can one be flexible and allow all tools to contribute? Standardization process mostly allows a coarse set of options. It mostly caters to the average good. Many good tools and technologies get cut out because they are not general enough. Selection almost always involves compromises. Standardization royalties may force simple tools and sophisticated tools equal share of the revenue. Impetus for alternative avenues for tool deployment. There is movement in this direction (software decoders, MPEG RVC, …). MPEG RVC (reconfigurable video coding): put many video compression tools (potentially overlapping functionality) in one big library, language syntax to specify which tools are needed in decoding a given video stream. Why?

6 Media Quality Effective Bandwidth Royalty Cost Royalty cost of delivering media at a particular quality and bandwidth. Example surface defined by achievable [quality, bandwidth, royalty cost] triplets. All triplets below the surface are achievable. Media delivery cost surface (content licensing + tool licensing)

7 Cuts from the surface Media Quality Effective Bandwidth Q Media Quality Effective Bandwidth C Royalty Cost Media Quality Effective Bandwidth B Royalty Cost “MPEG-1” “h.264/AVC”

8 Complicated Royalty Costs My content costs $5 for cell phone terminals, $15 for HDTV terminals. My compression tool is free of charge. My compression tool costs $0.001 per use. My tool costs $0.10 per movie, $0.15 for sports,.... My tool costs $0.10 except when combined with all free tools, in which case it too becomes free. My tool is free for not-for-profit use. …

9 System Level: Media Server I will mostly talk about compression related tools

10 Media ServerRegistry User Certificate can be used to verify with information from the user site (for each media segment or periodically) to ensure the legitimacy of the media. Time instant : certificate Time instant : Encoded media and associated certificate System Level: Media Delivery : quality : bandwidth : tool list … Time instant :

11 Rate - Distortion - Royalty Cost Optimization Setup Problem Definition: For each of the M segments, find the set of tools to use such that distortion is minimized under total rate and total royalty cost constraints, i.e., s.t., This is a simplification. Optimization can get elaborate. No reason to pay for the latest/greatest tools if plenty of bandwidth if simple/easy content if cheaper tools are available …

12 Optimization Example Using Compression Tools Table II: Rate and distortion changes with different tools. The utilized tools are subpixel accurate ME, loop filter, advanced entropy coding, and multiple reference frames respectively. The rate reduction is shown as percentage with respect to the baseline along with PSNR gain at QP=25. Clip-1 foreman Clip-2 akiyo Clip-3 coast. Clip-4 mobile Clip-5 bus Tool-1 Rate 56.51%55.93%14.93%51.56%50.05% ΔPSNR Tool-2 Rate 1.05%0.37%0.76%0.02%0.45% ΔPSNR Tool-3 Rate 7.21%4.69%10.85%7.81%4.85% ΔPSNR Tool-4 Rate 15.28%2.45%0.65%19.30%5.87% ΔPSNR Table I: Assigned cost of using each tool for two different cases Case-1Case-2 Tool-1 (Subpixel MV) 10 Tool-2 (Loop filter) Tool-3 (CABAC) Tool-4 (Multiple Ref.) Case 1: “Flat Rate”, similar to today’s licensing. Case 2: “Fair Rate”, each tool gets paid based on its contribution

13 “Flat Rate” Constant toolset: Optimized tools for the entire duration of the content. Adaptive toolset: Optimized tools for each GOP (toolset can change every GOP = media segment) concatenation of 10 clips Significant reduction in royalty costs with small loss in efficiency Inefficient tools get cut out (lobbying for tools is useless). Adaptive ~ constant Quality

14 “Fair Rate” Adaptive better than constant. Significant reductions are difficult. Quality

15 Tool Usage Adaptive allows more tools to contribute (tools that are good in niche situations get used). Tool-1Tool-2Tool-3Tool-4 C=75% Adaptive 97%48%98%57% C=75% Constant 100%0%100%0% C=25% Adaptive 69%0%20%10% C=25% Constant 100%0% C=75% Adaptive 76%72%98%67% C=75% Constant 100% 0% C=25% Adaptive 29%41%2%18% C=25% Constant 0% 100% “Flat Rate” “Fair Rate”

16Conclusion A system that allows practical deployment of royalty cost optimized media delivery. A system that allows practical deployment of royalty cost optimized media delivery. Very interesting optimization problem with sophisticated royalty costs. Very interesting optimization problem with sophisticated royalty costs. Content adaptive. Content adaptive. Increases efficiency. Allows non-standard tools to contribute. Increases efficiency. Allows non-standard tools to contribute. Much reduced royalties when plenty of bandwidth/resources. Much reduced royalties when plenty of bandwidth/resources. Flat rate: significant reductions in royalty cost possible. Flat rate: significant reductions in royalty cost possible. Fair rate: significant reductions difficult. Fair rate: significant reductions difficult. Adaptive optimization allows each tool to contribute when its niche comes. Adaptive optimization allows each tool to contribute when its niche comes. Can allow other resources relevant to media delivery into optimization (power consumption, memory usage, etc.) Can allow other resources relevant to media delivery into optimization (power consumption, memory usage, etc.) Going forward: Going forward: Optimization issues. Optimization issues. More tools. More tools. Dependencies among tools. Dependencies among tools. Addition of communication/networking related tools. Addition of communication/networking related tools. Fair rate Allow segment based licensing

17 Combines game theory and economic concepts with rate-distortion Combines game theory and economic concepts with rate-distortion How much are customers willing to pay to get quality Q at bandwidth B? (Utilization curves). How much are customers willing to pay to get quality Q at bandwidth B? (Utilization curves). Optimal costs based on utilization curves and game theory. Optimal costs based on utilization curves and game theory. What does today’s licensing look like? What does today’s licensing look like? Games Games Going forward: How should tools be priced? Distortion Rate T1 + T2: cost3(R) T2: cost2(R) T1: cost1(R) Free R free region D,R not achievable