Quantitative Comparison of H.264/AVC and VP8 Video Codecs for Video Conferencing Scott Kristjanson CMPT-820 Multimedia Systems Instructor: Dr. Mohamed Hefeeda Dec 2 nd 2010

Scott Kristjanson – CMPT SFU 2 Slide 2 Presentation Overview  Project Overview  Methods  Results  Conclusions  Future Work  References

Scott Kristjanson – CMPT SFU 3 Slide 3 Project Overview Compare Video Codecs for use with Video Conferencing: VP8 – by Google’s WebM project H.264/AVC – used x256 Open Source Codec

Scott Kristjanson – CMPT SFU 4 Slide 4 VP8 Overview Video Codec owned by Google and made open Source WebM Project: Matroska container with VP8 video and Vorbis audio VP8 - Earlier versions VP7 (Skype), VP6 (Flash Video), VP3 (Theora) Key Features: Adaptive Loop Filtering – Identifies Fast/Slow/Static regions and adapts encoding to match for improved quality/performance Golden Frames – for storing reference point to background layer. Can be used for reference without display them. Faster encoding time by avoiding use of Arithemetic VLE (CABAC). Dynamically scales resolution. No need to involve application, encoder decides. No I-frame required as a transition point. Structure: Frame same as H.264, Segment like a Slice Partitions used to separate critical data from video data I-Frame, P-Frames, Golden-Frames, no B-Frames due to Patents For more information, see Google’s “Intro to VP8” slides at:

Scott Kristjanson – CMPT SFU 5 Slide 5 Methods Quantitative Comparison: Invoke encoders/decoders with various presets: Manually from DOS, then use YUV_Compare to compute Y-PSNR ViCoS Codec Comparison utility from MSU – gives ranking only Used sequences foreman and salesman to simulate video conference calls Encode with bit rates 100, 800, 1500 kbps to produce bit rate curves Extract Quantitative Data Y-PSNR – Signal to Noise Ratio for Luminosity Y-SSIM– Structural Simularity Data for Luminosity Encoding Time Frame Rate Bit Rate Qualitative Comparison View original YUV file and encoded/decoded files using YUVviewer Test VP8 in Video Conferencing System to verify Quantitative Data

Scott Kristjanson – CMPT SFU 6 Slide 6 Reference Video Sequences Salesman Ideal choice for simulating video conference Not used since not available in CIF format Tried to convert from.avi to CIF format but ran out of time Foreman Good choice, but with some background movement Used in detailed quantitative analysis Used with Vicos for codec over-all evaluation Mobile Slow motion with detailed cartoon-like background Used with Vicos for codec over-all evaluation Stefan Lots of motion and lots of background detail Used with Vicos for codec over-all evaluation

Scott Kristjanson – CMPT SFU 7 Slide 7 DOS Command Line Testing x264 encode/decode: vp8 encode/decode: Visually compare original video to encoded video with YUV_Viewer

Scott Kristjanson – CMPT SFU 8 Slide 8 Computing Y-PSNR with AviSynth AVS script to read and compare two YUV files: Encoded/Decoded YUV video from DOS command line testing Original YUV video yuv_compare_foreman.avs foreman_yuv_compare.log loadplugin("C:\megui\tools\avisynth_plugin\rawsource.dll") decodedClip = RawSource("C:\foreman_352x288_decoded_vp8.yuv",352, 288, "I420") originalClip = RawSource("C:\foreman_352x288.yuv", 352, 288, "I420") Compare(decodedClip, originalClip, "YUV", "foreman_yuv_compare.log") Mean Max Max Absolute Mean Pos. Neg. Frame Dev. Dev. Dev. Dev. PSNR (dB) …… … … …… Total frames processed: 300 Minimum Average Maximum Mean Absolute Deviation: Mean Deviation: PSNR: Overall PSNR: PSNR

Scott Kristjanson – CMPT SFU 9 Slide 9 MSU’s Video Codecs Scoring System (ViCoS) Allows custom Codecs to be bound in and tested bind in encoders and decoders (x264 and vp8) define encoding and decoding parameters and presets to be tested encodes/decodes test sequences automatically and computes distortion generates relative ranking and relative scores for PSNR and SSIM only Versions: Demo version used – provides ranking scores only Professional version – creates PNSR/SSIM charts automatically

Scott Kristjanson – CMPT SFU 10 Slide 10 Video Quality – Peak Signal to Noise Ratio (Y-PSNR) Bit-Rate/Quality Trade-Off: 100kbps – Poor Quality 800kbps – Acceptible to good 1500kbps – Very Good to Excellent PSNR Ranges: 38+Excellent Very good 34-35Good 26-33Poor 25 or less Garbage

Scott Kristjanson – CMPT SFU 11 Slide 11 Video Quality – Structural Simularity (SSIM) SSIM score under 0.95 is quite noticible: 100kbps – Poor Quality for all codecs (especially x264 Fast Zero-Latency & Ultrafast) 800kbps – most codecs Acceptible to good (except VP8 RT CBR and x264 Ultrafast) 1500kbps – all Very Good (except x264 Ultrafast)

Scott Kristjanson – CMPT SFU 12 Slide 12 Encoder Frame Rates x264 Encoder significantly faster than VP8 for similar Quality

Scott Kristjanson – CMPT SFU 13 Slide 13 Good Codec Presets for Video Conferencing Need min of 15 fps for Video Conferencing, prefer 30 fps Want decent quality – Y-PSNR of 34 or better Low Latency: under 150ms required 4 Presets met these requirements: Codec Preset

Scott Kristjanson – CMPT SFU 14 Slide 14 Conclusion VP8 and x264 provide similar video quality For higher bit rates, x264 PSNR better by 2% For lower rates, VP8 wins by almost 10% x264 provides better performance in some cases x264 is between 36% and up to 2x faster than VP8 A big improvement! VP8 used to be 5-20 times slower!

Scott Kristjanson – CMPT SFU 15 Slide 15 References: [1] Image Quality Assessment: From Error Visibility to Structural Similarity Zhou Wang, Alan Conrad Bovik, Hamid, Rahim Sheikh, and Eero P. Simoncelli IEEE Transactions on Image Processing, Vol 13, No. 4, April [2] [2] 6 th Annual MPEG-4 AVC H.264 video codecs comparison D. Vatolin, D. Kulikov, and A. Parshin, April

Scott Kristjanson – CMPT SFU 16 Slide 16 Backup Slides

Scott Kristjanson – CMPT SFU 17 Slide 17 H.264/AVC Presets Used Codecs tested with various presets and tunings Looking for good balance of quality and speed for video conferencing Each tested with 3 target Encoding bit rates: 100, 800, 1500 kbps x264 Presets: Placebo 2-Pass Slower Encoding Speed, higher quality Fast (tested with 2 tunings) 2-Pass Faster Encoding Speed, good quality 2-Pass tuned with zero latency specified Default 1-pass with default settings Baseline 1-pass fast encoding but poor quality at lower bit rates Ultrafast When speed is all that matters!

Scott Kristjanson – CMPT SFU 18 Slide 18 VP8 Presets Used Best 2-Pass encoding Maximimizes Quality but slower encode times Good 2-Pass 2-Pass encoding, less quality but faster encode times measured with 1, 2, and 4 cpu cores specified Good 1-Pass 1-Pass encoding, less quality but faster encode times Fast VBR 1-Pass variable bit-rate fast encoding RealTime CBR Encoder Frame buffer size 6 with and without deadline specified

Scott Kristjanson – CMPT SFU 19 Slide 19 VP8 Support for MultiCore Processors

Scott Kristjanson – CMPT SFU 20 Slide 20 Encoding Bit Rate vs Target Bit Rate Both Codecs appear to over-run target slightly VP8 tracks to target bit-rate slightly better

Scott Kristjanson – CMPT SFU 21 Slide 21 Video Quality – Structural Simularity (SSIM) SSIM score under 0.95 is quite noticible: 100kbps – Poor Quality for all codecs (especially x264 Fast 0-Latency and Ultrafast) 800kbps – most codecs Acceptible to good (except VP8 RT CBR and x264 Ultrafast) 1500kbps – all Very Good