1. doc.: IEEE 802.11-06/0965r1 Submission July 2006 Royce Fernald - Intel CorporationSlide 1 Video Performance Notice: This document has been prepared to assist IEEE 802.11. It is offered as a basis for discussion and is not binding on the contributing individual(s) or organization(s). The material in this document is subject to change in form and content after further study. The contributor(s) reserve(s) the right to add, amend or withdraw material contained herein. Release: The contributor grants a free, irrevocable license to the IEEE to incorporate material contained in this contribution, and any modifications thereof, in the creation of an IEEE Standards publication; to copyright in the IEEE’s name any IEEE Standards publication even though it may include portions of this contribution; and at the IEEE’s sole discretion to permit others to reproduce in whole or in part the resulting IEEE Standards publication. The contributor also acknowledges and accepts that this contribution may be made public by IEEE 802.11. Patent Policy and Procedures: The contributor is familiar with the IEEE 802 Patent Policy and Procedures, including the statement "IEEE standards may include the known use of patent(s), including patent applications, provided the IEEE receives assurance from the patent holder or applicant with respect to patents essential for compliance with both mandatory and optional portions of the standard." Early disclosure to the Working Group of patent information that might be relevant to the standard is essential to reduce the possibility for delays in the development process and increase the likelihood that the draft publication will be approved for publication. Please notify the Chair as early as possible, in written or electronic form, if patented technology (or technology under patent application) might be incorporated into a draft standard being developed within the IEEE 802.11 Working Group. If you have questions, contact the IEEE Patent Committee Administrator at.http:// ieee802.org/guides/bylaws/sb-bylaws.pdfstuart.kerry@philips.compatcom@ieee.org Authors:

2. doc.: IEEE 802.11-06/0965r1 Submission July 2006 Royce Fernald - Intel CorporationSlide 2 Abstract This presentation defines the primary metrics for video performance as well as key secondary metrics This presentation corresponds to the draft text submission 802.11-06/0964r1

3. doc.: IEEE 802.11-06/0965r1 Submission July 2006 Royce Fernald - Intel CorporationSlide 3 Agenda  Background –Proposal Framework –Test Output –Video Delivery and Video Quality –Measuring Video Delivery and Video Quality –The Media Delivery Index –Video Delivery, Video Quality and Wireless Performance –References Test Description (doc.: 802.11-06/0964r1) –Introduction and Purpose –Resource Requirements –Test Setup –Video Content Preparation and Analysis –Calibration –Baseline Configuration –Modifiers –Measurement Procedures –Reported Results –Sample Data Motion

4. doc.: IEEE 802.11-06/0965r1 Submission July 2006 Royce Fernald - Intel CorporationSlide 4 Proposal Framework Usage case: Video streaming applications Test Environments: –Conducted Environment –OTA Environments Primary Metrics –Video Delivery Video Delivery Error Rate (VDER) –Video Quality Perceived Video Quality Mean Opinion Score (MOS) Secondary Metrics –Media Delivery Index (MDI) Media Loss Rate (MLR) Delay Factor (DF)

5. doc.: IEEE 802.11-06/0965r1 Submission July 2006 Royce Fernald - Intel CorporationSlide 5 Test Output Video Delivery and Video Quality metrics are the primary output of the test MDI quantifies the impact of the network on the test outcome –Helps isolate root causes of poor performance

6. doc.: IEEE 802.11-06/0965r1 Submission July 2006 Royce Fernald - Intel CorporationSlide 6 Video Delivery and Video Quality There are two complementary ways to characterize video performance – video delivery and video quality Video delivery metrics quantify the impact of dropped, repeated or out-of-sequence video frames –Video delivery errors are caused by large-scale packet losses or delay on the link Video quality metrics quantify the impact of video artifacts on end user perception –Video artifacts are caused by packet loss –Different codecs / content types translate levels of packet loss to different levels of video quality

7. doc.: IEEE 802.11-06/0965r1 Submission July 2006 Royce Fernald - Intel CorporationSlide 7 Measuring Video Delivery and Video Quality Video Delivery Measurements –Video delivery measurements are made with video playback analysis applications (VPAA) –VPAAs provide quantitative metrics that reflect the performance of the underlying transport medium –A correlation can be established between video delivery measurements and MOS of end user perception Video Quality Measurements –Video quality analysis applications (VQAA) output mean opinion scores (MOS) that represent the expected end user perception of video quality –VQAAs are based on psychovisual models (such as VQM) which are designed to characterize the quality of a perceived video clip relative to a source file

8. doc.: IEEE 802.11-06/0965r1 Submission July 2006 Royce Fernald - Intel CorporationSlide 8 The Media Delivery Index MDI defines media-weighted secondary metrics that quantify the impact of wireless link performance on observed video quality The media loss rate (MLR) reflects the portion of the video stream that was undelivered due to packet loss –The MLR is the packet loss rate expressed as a percentage of the video stream –High MLR corresponds to video delivery errors (client buffer starvation) or noticeable display artifacts from undelivered packets The delay factor (DF) quantifies the amount of buffering required to maintain an uninterrupted media stream –The DF is the required client buffer depth divided by the bitrate of the video stream –High DF corresponds to non-fluid playback and loss of interactivity for real-time applications such as video conferencing

9. doc.: IEEE 802.11-06/0965r1 Submission July 2006 Royce Fernald - Intel CorporationSlide 9 Video Delivery, Video Quality and Wireless Performance – Best Effort Traffic (i.e. UDP/RTP)

10. doc.: IEEE 802.11-06/0965r1 Submission July 2006 Royce Fernald - Intel CorporationSlide 10 Video Delivery, Video Quality and Wireless Performance – Guaranteed Traffic (i.e. TCP)

11. doc.: IEEE 802.11-06/0965r1 Submission July 2006 Royce Fernald - Intel CorporationSlide 11 References IEEE 802.11-06/0964r1 “Video Performance” IEEE 802.11-06/0651r1 “Video Delivery versus Attenuation in a Conducted Environment” IEEE 802.11-06/0321r0 “Video Gross Error Detector Video over Wireless Methodology” IEEE 802.11-06/0144r1 “Video over Wireless Methodology” IEEE 802.11-05/1194r0, “Video Testing Methodology” IEEE 802.11-05/0887r0, “Video Testing Strategy”

12. doc.: IEEE 802.11-06/0965r1 Submission July 2006 Royce Fernald - Intel CorporationSlide 12 Agenda Background –Proposal Framework –Test Output –Video Delivery and Video Quality –Measuring Video Delivery and Video Quality –The Media Delivery Index –Video Delivery, Video Quality and Wireless Performance –References  Test Description (doc.: 802.11-06/0964r1) –Introduction and Purpose –Resource Requirements –Test Setup –Video Content Preparation and Analysis –Calibration –Baseline Configuration –Modifiers –Measurement Procedures –Reported Results –Sample Data Motion

13. doc.: IEEE 802.11-06/0965r1 Submission July 2006 Royce Fernald - Intel CorporationSlide 13 Introduction and Purpose This test measures the video streaming performance of the link between a DUT and a WLCP –Video traffic is streamed to the DUT over the RF interface –The DUT display data is captured and analyzed for video delivery errors and video quality defects –Traffic measured by the 802.11 traffic analyzer is used to calculate the MDI (media loss rate and delay factor)

14. doc.: IEEE 802.11-06/0965r1 Submission July 2006 Royce Fernald - Intel CorporationSlide 14 Hardware Resource Requirements The basic test configuration is defined in section 5 In addition, the following items are required: –A traffic generator capable of generating video traffic The video traffic generator can be a general purpose PC or dedicated device that streams video content to the DUT –Video capture device, a PC or dedicated device capable of connecting to the DUT display port and saving video data to non- volatile storage –The DUT may also act as the video capture device if it has the ability to save or process media streams in real-time (not progressive download)

15. doc.: IEEE 802.11-06/0965r1 Submission July 2006 Royce Fernald - Intel CorporationSlide 15 Software Resource Requirements A video playback analysis application capable of instrumenting video content for the detection of dropped, repeated and out-of-sequence video frames A video quality analysis application capable of analyzing video frames for image defects and rendering a quality score based on a perceptual model of the video format under test A video encoder application for encoding test media in the required format A video capture application for capturing display data from the DUT

16. doc.: IEEE 802.11-06/0965r1 Submission July 2006 Royce Fernald - Intel CorporationSlide 16 Test Setup

17. doc.: IEEE 802.11-06/0965r1 Submission July 2006 Royce Fernald - Intel CorporationSlide 17 Video Content Preparation Video content must be instrumented by the video playback analysis application before it can be used for this test Instrumentation ensures content-independent VDER results Uncompressed source content is first instrumented and then encoded in the desired test format before being loaded on the video traffic generator Preparation can be performed in real-time if supported by the video traffic generator

18. doc.: IEEE 802.11-06/0965r1 Submission July 2006 Royce Fernald - Intel CorporationSlide 18 Video Content Preparation Procedure

19. doc.: IEEE 802.11-06/0965r1 Submission July 2006 Royce Fernald - Intel CorporationSlide 19 Video Content Analysis After passing through the RF data connection, video content is captured from the DUT display and saved to a file (typically uncompressed) The instrumented video capture file is examined for video delivery error and video quality defects by the video playback analysis application and the video quality analysis application Analysis can be performed in real-time if supported by the video capture device

20. doc.: IEEE 802.11-06/0965r1 Submission July 2006 Royce Fernald - Intel CorporationSlide 20 Video Content Analysis Procedure

21. doc.: IEEE 802.11-06/0965r1 Submission July 2006 Royce Fernald - Intel CorporationSlide 21 Calibration Video performance tests require that the DUT support the video format, bitrate, network transport and streaming protocol used by the video traffic generator Video format compatibility must be verified before the test begins Video performance measurements can be affected by the capture device, so capture device performance must be verified with local video playback from the DUT If there are too many errors detected during calibration, the capture device should not be used for video performance tests

22. doc.: IEEE 802.11-06/0965r1 Submission July 2006 Royce Fernald - Intel CorporationSlide 22 Baseline Configuration Maximum transmit power setting QoS enabled with video traffic at high priority (no background traffic) No security No power management Video traffic configured to model desired application (i.e. 8 Mbps MPEG2 for DVD-quality video) Default DUT buffer size No content encryption

23. doc.: IEEE 802.11-06/0965r1 Submission July 2006 Royce Fernald - Intel CorporationSlide 23 Modifiers QoS disabled QoS enabled with background traffic at low priority MAC security Power management enabled, different profiles Alternate video bitrates and/or network transports DUT buffer size Video content encryption

24. doc.: IEEE 802.11-06/0965r1 Submission July 2006 Royce Fernald - Intel CorporationSlide 24 Measurement Procedure Load video content on the video traffic generator Associate DUT with WLCP Repeat the following steps for each attenuation or range increment: Start the video capture operation on the capture device and then start the video stream to the DUT After the specified test duration, stop the capture operation and then stop the video stream Analyze the resulting video capture file with the video playback analysis application and the video quality analysis application to measure the video delivery error rate and the video quality score Calculate MDI using metrics from the 802.11 traffic analyzer

25. doc.: IEEE 802.11-06/0965r1 Submission July 2006 Royce Fernald - Intel CorporationSlide 25 Reported Results The reported results should include the following items –Calibrated VDER, including the capture device baseline –Video Quality MOS and DUT MOS upper bound –MDI MLR and DF –Detailed information about the test configuration including the video format, network transport and video capture device Attenuation (dB) or Range (m) Video Delivery Error Rate (Errors/minute) Video Quality (MOS) MDI – MLRMDI – DF Sample Report:

26. doc.: IEEE 802.11-06/0965r1 Submission July 2006 Royce Fernald - Intel CorporationSlide 26 Sample Data Compares video streaming performance of wireless adapters “A” and “B” Conducted environment Standard definition video (720x480 pixels) MPEG2 transport stream –12 Mbps MPEG2 video –Constant bitrate –No audio UDP video transport

27. doc.: IEEE 802.11-06/0965r1 Submission July 2006 Royce Fernald - Intel CorporationSlide 27 Sample Data – Adapter A

28. doc.: IEEE 802.11-06/0965r1 Submission July 2006 Royce Fernald - Intel CorporationSlide 28 Sample Data – Adapter B

29. doc.: IEEE 802.11-06/0965r1 Submission July 2006 Royce Fernald - Intel CorporationSlide 29 Agenda Background –Proposal Framework –Test Output –Video Delivery and Video Quality –Measuring Video Delivery and Video Quality –The Media Delivery Index –Video Delivery, Video Quality and Wireless Performance –References Test Description (doc.: 802.11-06/0964r1) –Introduction and Purpose –Resource Requirements –Test Setup –Video Content Preparation and Analysis –Calibration –Baseline Configuration –Modifiers –Measurement Procedures –Reported Results –Sample Data  Motion

30. doc.: IEEE 802.11-06/0965r1 Submission July 2006 Royce Fernald - Intel CorporationSlide 30 Motion Move to instruct the editor to incorporate the contents of document 802.11-06/0964r1 into the P802.11.2 draft. Technical (75%) Y: N: A: