1. Practical and Scalable Transmission of Segmented Video Sequences to Multiple Players using H.264 Fabian Di Fiore, Panagiotis Issaris Expertise Centre for Digital Media Hasselt University, Belgium {fabian.difiore, takis.issaris}@uhasselt.be

2. MIG09 November, 23 2009 2/30 Introduction  The integration of multimedia streams of various nature is an important feature of many of today's games  Should not be limited to inter-person communication only  Many other interesting applications are in the queue  omnidirectional video  you control the camera and, hence, the viewport  the replacement of traditional computer-generated background images by video sequences  viewport depends on your position in a platform game  geospatial data visualization applications (battlefield, Google Earth) Introduction Approach Measurement Results Conclusions motivation need for distributing personal viewports on the same video sequence to large amounts of players

3. MIG09 November, 23 2009 3/30 Introduction Introduction Approach Measurement Results Conclusions contribution Video consists of sequences of frames

4. MIG09 November, 23 2009 4/30 Introduction Introduction Approach Measurement Results Conclusions contribution Let’s focus on one such frame

5. MIG09 November, 23 2009 5/30 Introduction Introduction Approach Measurement Results Conclusions contribution We'd like to have multiple viewports...

6. MIG09 November, 23 2009 6/30 Introduction Introduction Approach Measurement Results Conclusions contribution … of possibly different sizes

7. MIG09 November, 23 2009 7/30 Introduction Introduction Approach Measurement Results Conclusions contribution … and possibly moving

8. MIG09 November, 23 2009 8/30 Introduction Introduction Approach Measurement Results Conclusions contribution … and then distribute it (to heterogeneous clients).

9. MIG09 November, 23 2009 9/30 How can we accomplish this?  System serving video 1) - Decode - For each client  crop the required viewing area  encode it  send it in compressed form to the client 2) - Encode once (H.264) - For each client  cut out the required viewing area  send itto the client Introduction Approach Measurement Results Conclusions overview

10. MIG09 November, 23 2009 10/30 Encode for each client Introduction Approach Measurement Results Conclusions encode for each client  Extremely CPU-intensive and thus not scalable  Possible loss of quality (if source was already compressed)

11. MIG09 November, 23 2009 11/30 How can we accomplish this?  System serving video 1) - Decode - For each client  take the required viewing area  encode it  send it in compressed form to the client 2) - Encode once (H.264) - For each client  cut out the required viewing area  send itto the client Introduction Approach Measurement Results Conclusions overview

12. MIG09 November, 23 2009 12/30 H.264  Video encoder  input: a sequence of frames  output: smaller bitstream conforming to a specification  Block-based video codecs  divide each input frame into a grid structure  each individual area is called a macroblock  resulting output bitstream consists of a highly compressed representation of these macroblocks  the representation of each macroblock in the compressed form is of variable length: a bit error in for example the first macroblock leads to the failure to decode any macroblock in the frame Introduction Approach Measurement Results Conclusions H.264: general

13. MIG09 November, 23 2009 13/30 H.264  H.264  aka MPEG-4/AVC  most recent ITU & MPEG video coding standard widely used (Blu-ray, DVB-S2, QuickTime,... )  provides a feature called a ‘slice’ which allows macroblocks to be grouped  each slice becomes individually en/decodable  coded video bitstream can have a very simple structure using one slice per frame, however, having multiple slices is advantageous for parallel processing and network transmission Introduction Approach Measurement Results Conclusions H.264 errors in a slice will not prevent the correct decoding of the other slices

14. MIG09 November, 23 2009 14/30 How can we accomplish this?  System serving video 1) - Decode - For each client  take the required viewing area  encode it  send it in compressed form to the client 2) - Encode once (H.264) - For each client  cut out the required viewing area  send itto the client Introduction Approach Measurement Results Conclusions overview

15. MIG09 November, 23 2009 15/30 Cut out the selected area  Cut out the video... Introduction Approach Measurement Results Conclusions regular slice subdivision

16. MIG09 November, 23 2009 16/30 Cut out the selected area  … by regarding the frame as being composed of many independent areas…  use regular slice subdivision over all frames Introduction Approach Measurement Results Conclusions regular slice subdivision

17. MIG09 November, 23 2009 17/30 Cut out the selected area  … and then for each client, select the areas enclosing the desired viewport. Introduction Approach Measurement Results Conclusions regular slice subdivision

18. MIG09 November, 23 2009 18/30 Cut out the selected area  Removing unnecessary slices Introduction Approach Measurement Results Conclusions removing slices

19. MIG09 November, 23 2009 19/30 Cut out the selected area  Removing unnecessary slices  deleting one slice affects all others  renumbering of macroblock numbers: slices need to be altered  dangerous due to variable bit length encoding: implies bit shifts  moving view frustrum  the renumbering of macroblocks due to the cropping, is different before and after the movement of the cropping window. So, the motion compensation process in the decoder will start using the wrong image data to reconstruct the image  video frame size could change: decoder issues Introduction Approach Measurement Results Conclusions removing slices

20. MIG09 November, 23 2009 20/30 Cut out the selected area  Replacing unnecessary slices  replace by an artificially generated slice of minimal size  no renumbering of macroblocks  no issues of bit shifting  no header modifications needed  larger decoded picture buffer size  generated on the fly  intra-frames (~ I frames): gray blocks  inter-frames (~ P frames): only skip-bits  macroblocks of subsequent ‘empty’ slices are grouped together into a single empty slice to prevent overhead of the slices’ headers Introduction Approach Measurement Results Conclusions replacing slices

21. MIG09 November, 23 2009 21/30 Cut out the selected area Introduction Approach Measurement Results Conclusions replacing slices Replacing slices empties part of a frameGrouping empty slices

22. MIG09 November, 23 2009 22/30 Concerns  Frame dependency  motion vectors can point to a replaced slice (i.e. gray area) in a previous frame  solution: restrict motion vectors to current slice area Introduction Approach Measurement Results Conclusions concerns

23. MIG09 November, 23 2009 23/30 Concerns  Grid structure  H.264 forces grouping of macroblocks to be consecutive in raster scan order  slices cannot span more than one row of macroblocks  macroblock: 16 x 16  rectangular slices of 16 x.... (e.g., 16 x 192)  FMO  Flexible Macroblock Ordering  not supported yet by real-time decoders Introduction Approach Measurement Results Conclusions concerns

24. MIG09 November, 23 2009 24/30 Concerns  Prediction artifacts  moving the view frustum might cause motion compensation to use data we cropped out in previous frames  sudden large movements  we receive predicted slices for which we do not have the basis of the prediction (only gray artificial slices)  solution: send out more slices, located around the viewport Introduction Approach Measurement Results Conclusions concerns

25. MIG09 November, 23 2009 25/30 Measurement results and comparison  Evaluation of replacing unneeded slices by empty ones  Comparisons are made against the official reference encoder (JM)  Our implementation is built on the reference encoder as it is the only encoder fully implementing H.264  No specific decoder needed Introduction Approach Measurement Results Conclusions settings

26. MIG09 November, 23 2009 26/30 Measurement results and comparison  panoramic video sequence  input stream  2 clients  actual videostream sent to third client

27. MIG09 November, 23 2009 27/30 Measurement results and comparison  Grid structure overhead caused by using 512 slices per frame Introduction Approach Measurement Results Conclusions grid structure overhead

28. MIG09 November, 23 2009 28/30 Measurement results and comparison  Bitrate reduction using slice replacement Introduction Approach Measurement Results Conclusions bit rate reduction  purple: one slice per frame (i.e. full frame)  green: 512 slices  red: slices replaced Using FMO and optimized encoder will have even bigger impact

29. MIG09 November, 23 2009 29/30 Measurement results and comparison  Bitrate reduction using slice replacement (low vs hi quality) Introduction Approach Measurement Results Conclusions bit rate reduction  green: unaltered  blue: inter slices replaced  orange/yellow: all slices replaced  red: replaced and grouped

30. MIG09 November, 23 2009 30/30 Conclusion and future work  Practical way to distribute parts of a video sequence to large amounts of users  Conforming to H.264 standard  Frame extraction directly from compressed bitstream  Eliminating need for additional decoding and encoding  The overhead associated with the slice structure is mitigated by the bandwidth reduction in transmitting high- quality video streams Conclusion and future work Additional benefit will be gained when - FMO is practically usable (encoder and decoder) - using an optimized encoder (instead of reference encoder) Additional benefit will be gained when - FMO is practically usable (encoder and decoder) - using an optimized encoder (instead of reference encoder)