1 MobiUS: Enable Together-Viewing Video Experience across Two Mobile Devices Guobin Shen, Yanlin Li, Yongguang Zhang Microsoft Research Asia

2 Contents Introduction Collaborative Half-frame Decoding System Architecture and Implementation Experimental Results and Evaluation Discussion and conclusion

3 Introduction Motivation: A new better-together mobile application paradigm when multiple mobile devices are placed together

4 A specific together-viewing video application A higher resolution video is played back across screens of two mobile devices placed side by side

5 Assumptions One device has a higher resolution video whose size is about twice of its screen size while the other not Two devices can communicate effectively and directly via high-speed local wireless networks Two devices are homogeneous: same/similar software and hardware capabilities

6 Requirements Real-time synchronous playback At least 15 frames per second (fps) Same frame rendered at two screens simultaneously Energy efficiency Work in resource-constrained environment Limited processing power, memory, battery life … Dynamic adaption Expand the video on to two devices with another coming Shrink it on to one screen with another leaving

7 Possible Solutions Full-frame Decoding-based Approaches: Thin client model Thick client model Half-frame Decoding-based Approaches: Whole-bitstream transmission (WTHD) Partial-bitstream transmission (PTHD)

8 Thin Client Model Computation of Mb not utilized Huge bandwidth demand Unbalanced energy consumption Short operating lifetime Ma: Decode whole frame Mb Decoded right half-frame Display left half-frameDisplay right half-frame

9 Thick Client Model Computation power of both devices utilized Less bandwidth requirement Balanced energy consumption Abuse more computation power than necessary MaMb Decode whole frame Display the left half-frame Decode whole frame Display the right half-frame Whole bitstream

10 Whole-bitstream transmission (WTHD) Computation power of both devices utilized Less bandwidth requirement Balanced energy consumption Abuse more bandwidth than necessary MaMb Decode the left half-frame Display the left half-frame Decode the right half-frame Display the right half-frame Whole bitstream

11 Partial-bitstream transmission (PTHD) Computation power of both devices utilized Less bandwidth requirement Balanced energy consumption Implementation complexity MaMb Decode the left half-frame Display the left half-frame Decode the right half-frame Display the right half-frame Right half-bitstream

12 Comparison Which method is the best? schemeComput. complexity BW efficiency Impl. complexity Feasibility Thin/CHigh/LowWorstSimpleNo Thick/CHighBadSimpleNo WTHDLowBadComplexPossible PTHDLowGoodComplexPreferred

13 However, there is no free lunch.

14 Background on Video Coding Properties of video sequences:  Strong spatial correlation: each frame is an image  Strong temporal correlation: capturing instant of neighboring frames close to each other Basic logic of video coding: Maximally strip off spatial and temporal correlations

15 Motion Compensated Prediction MCP creates recursive temporal frame dependency Challenges arise from motion, but is worsened by recursive temporal dependency Ma Mb Cross-boundary reference effect

16 How to perform efficient half-frame decoding? Cross-device collaboration (CDC) transmit the missing reference to each other

17 Fundamental Facts Markovian effect of MCP a later frame only depends on a previous reference frame, no matter how the reference frame is obtained Highly skewed MV distribution the motion vector is centered at the origin (0,0) more than 80% of motion vectors are smaller than 8

18 Push-based Cross-device Delivery Scheme Record positions of blocks needing cross-device reference and associated motion vectors Perform a light-weight pre-scanning process and motion analysis Before decoding nth frame, look ahead by one frame

19 Collaborative half-frame decoding Push-based CDC scheme Real-time playback

20 Can it be better in energy efficiency? Percentage of boundary blocks that require cross-device collaboration and their corresponding bandwidth requirement SequenceCHDec CD ReferenceBW Requirement Bestcap22.8%253 kbps SmallTrap26.2%192 kbps Liquid20.7%231 kbps

21 Cumulative distribution functions of horizontal component of motion vectors for the whole frames and the boundary columns

22 The bandwidth requirement of the helping traffic is relatively high, reaching half of the bandwidth required for sending the half bitstream To make best use of multiple radio interfaces, the streaming data should be low enough for the Bluetooth’s throughput to be capable of More than 90% motion vectors are smaller than 16, the width of a macroblock

23 Guardband-based collaborative half-frame decoding scheme Each device decodes one more column of macroblocks

24 Is it a good idea? SequenceCHDecGB-CHDec CD RefBW ReqCD RefBW Req BestCap22.8%253 kbps3.4%76.9 kbps SmallTrap26.2%192 kbps1.3%30.6 kbps Liquid20.7%231 kbps2.5%53.2 kbps

25 76% associated CDC traffic savings 7% extra computational cost Each device decodes one more column of macroblocks

26 How about larger extended half-frame?

27 Additional 10% CDC traffic reduction Another 7% computation overhead A two-macroblock-wide guardband Larger guardband is not so beneficial

28 Argument Shall we need CDC traffic for decoding the boundary blocks in the guardband? Yes, only if we need to decode the whole guardband correctly. However, we do not have to ensure the guardband to be correctly and completely decoded.

29 Different decoding schemes for guardband blocks Not decoded at all Not referenced at all Best-effort decoded without CDC traffic and insurance of correctness Referenced by the guardband blocks of the next frame Correctly decoded, resorting to CDC traffic when necessary Referenced by the half-frame blocks of the next frame

30 System Archtecture

31 automatically set up a network between two mobile devices

32 a simple radio signal strength based strategy Ensure a close proximity setting

33 Check capability of a newly added device and inform the content host about the arrival or departure of the other device

34 Application level synchronization strategy RTT-based synchronization procedure

35 RTT-based Synchronization Scheme Ma Mb Estimate RTT Display the next frame Wait half RTT Display next frame

36 Decoded frames Half-bitstreams for local device Half-bitstreams for the other device Hold and send/receive the cross- device collaboration data to the other device

37 Parse the original bitsream into two half bitstreams and extract the motion vectors Independent full- frame based fast DCT- domain down-scaling decoding module The guardband- based collaborative half-frame decoding module

38 Configuration of Two Devices HP iPAQ rw6828Dopod 838 ProcessorIntel Xscale 416 MHzOMAP MHz OSMicrosoft Windows Mobile Version 5.0, Phone Edition Wireless ConnectionWiFi, Bluetooth ScreenQVGA resolution (320*240) RAM64 MB

39 Experimental Results

40 Benchmark of Mobile Devices Mobile devices are cost-effectively designed, Just able to meet the real-time playback requirement for videos at the same resolution of the screen

41 Decoding Speed

42 Decoding Speed Both collaborative half-frame decoding schemes significantly improve the decoding speed. The guardband-based scheme is only slightly slower than the half-frame decoding case.

43 Synchronization

44 Synchronization Due to periodical synchronization

45 Synchronization Due to a large scene change with very high motion. It is tolerable because the human visual system is far less sensitive for such slight asynchronism, especially when the motion is large.

46 Energy Efficiency Collaborative half-frame decoding scheme leads to significant energy savings. Decoding Scheme WiFi Lifetime (seconds) Full-frame OFF16438 ON7482 Half-frame OFF23736 ON8375

47 Discussions Further optimization opportunities Service provisioning User study Assumption on homogeneity

48 Further Optimization Opportunities Computing saving the color space conversion consumes 30% of the overall time Collaborative traffic reduction simple compression; error concealment technique Energy consumption reduction save screen backlight energy consumption through the gamma adjustment make use of dynamic voltage scaling capability

49 Service Provisioning New encoder profiles to generate completely self-constrained substreams each substream corresponds to half-frame Efficient arbitrary resizing transcoding to generate video content with suitable resolution

50 User Study

51 Assumption on Homogeneity Only technical constraints:  Ability to play back video on its own  Networking capabilities  Same pixel resolution

52 Conclusion Fulfill requirements under assumptions:  Real-time synchronous playback  Energy efficiency  Adaptive Weak assumptions: only one device has video file the video resolution is twice of the screen size Future work:  How to automatically achieve load balancing  How to expand to more screens......

53 Questions?