doc.: IEEE /0661r0 Submission July 2005 Alexander Tolpin, IntelSlide 1 TGT Conductive Test Environment and Metrics Notice: This document has been prepared to assist IEEE It is offered as a basis for discussion and is not binding on the contributing individual(s) or organization(s). The material in this document is subject to change in form and content after further study. The contributor(s) reserve(s) the right to add, amend or withdraw material contained herein. Release: The contributor grants a free, irrevocable license to the IEEE to incorporate material contained in this contribution, and any modifications thereof, in the creation of an IEEE Standards publication; to copyright in the IEEE’s name any IEEE Standards publication even though it may include portions of this contribution; and at the IEEE’s sole discretion to permit others to reproduce in whole or in part the resulting IEEE Standards publication. The contributor also acknowledges and accepts that this contribution may be made public by IEEE Patent Policy and Procedures: The contributor is familiar with the IEEE 802 Patent Policy and Procedures, including the statement "IEEE standards may include the known use of patent(s), including patent applications, provided the IEEE receives assurance from the patent holder or applicant with respect to patents essential for compliance with both mandatory and optional portions of the standard." Early disclosure to the Working Group of patent information that might be relevant to the standard is essential to reduce the possibility for delays in the development process and increase the likelihood that the draft publication will be approved for publication. Please notify the Chair as early as possible, in written or electronic form, if patented technology (or technology under patent application) might be incorporated into a draft standard being developed within the IEEE Working Group. If you have questions, contact the IEEE Patent Committee Administrator at. Date: July 2005 Authors:

doc.: IEEE /0661r0 Submission July 2005 Alexander Tolpin, IntelSlide 2 Abstract This document introduces the description of Conductive Test Environment and few metrics (Throughput vs. Attenuation, TX Rate Adaptation and Antenna Diversity) for performance testing of wireless LAN devices This presentations corresponds to the document t-tgt-conductive-test-environment-metrics-proposal- draft-text.doc

doc.: IEEE /0661r0 Submission July 2005 Alexander Tolpin, IntelSlide 3 Summary Purpose Test Equipment Typical setup Specific metrics –Throughput vs. Attenuation –TX Rate Adaptation –Antenna Diversity

doc.: IEEE /0661r0 Submission July 2005 Alexander Tolpin, IntelSlide 4 Purpose of Conductive Test Environment Provide good control of test parameters Provide good visibility of test results Minimize an impact of extraneous signals Guarantee high repeatability of test results (+/-3%) – over-time and location. Model real-life experience –for example, TPT vs. Attenuation correlates with TPT vs. range in LOS.

doc.: IEEE /0661r0 Submission July 2005 Alexander Tolpin, IntelSlide 5 Main Test Equipment DUT – any wireless device (AP or Client) that includes relevant SW running on the specific platform WLCP (WireLess CounterPart) - reference AP or a reference Client or just a RF signal generator depending on test objectives –WLCP may be also a source of internal interference signals Shielded enclosure for DUTs and WLCPs in order to isolate from extraneous signals Cables –RF-cables – connected to antenna connectors. –Wired LAN cables –Control cables Attenuators – to control attenuation (which means path loss) and RF signal input power

doc.: IEEE /0661r0 Submission July 2005 Alexander Tolpin, IntelSlide 6 Main Test Equipment (cont.) Calibrated combiners, splitters and couplers – to handle different RF path, including antennas entries. Power Meter Devices – to measure RF signal power per packet Traffic Analyzer – to gather and analyze traffic through RF cables Wired Traffic Generator to generate data traffic from DUT to WLCP and from WLCP to DUT on top of layer 2. Wired Traffic Analyzer to gather delivered data payload over time through wired interface on top of layer 2.

doc.: IEEE /0661r0 Submission July 2005 Alexander Tolpin, IntelSlide 7 Main Test Equipment (cont.) Test controller includes the following capabilities, likely automated and controlled by dedicated SW: The ability to control TX rates and TX power of WLCP and DUT The ability to control power meters. The ability to control attenuators The ability to control Wired Traffic Analyzer The ability to control Wired Traffic Generator. The ability to control Traffic Analyzer

doc.: IEEE /0661r0 Submission July 2005 Alexander Tolpin, IntelSlide 8 Calibration and isolation of the setup The setup shall be properly isolated from external interference and other unwanted signals. Prior to beginning the specific test, the test equipment shall be calibrated, taking into account losses due to cables, couplers, splitters etc. All test software shall be verified. The test setup may be monitored during the test to ensure that the test conditions do not change unintentionally.

doc.: IEEE /0661r0 Submission July 2005 Alexander Tolpin, IntelSlide 9 Typical conductive setup - superset

doc.: IEEE /0661r0 Submission July 2005 Alexander Tolpin, IntelSlide 10 Examples of setups Same test equipment may be used for many test setups depending on test objectives, for example: –Few DUTs may work together (bandwidth sharing test, fairness tests) –Few WLCPs may be used together, some of them, for example, to generate internal interference signals (ACI, Roaming) –Different RF cables with different length controlled by attenuators may simulate multi-path effects IEEE /0419r1 [7] briefly describes the following conductive setups –TPT vs Attenuation –Roaming –ACI – two configurations –Bandwidth sharing and fairness –Antenna Diversity

doc.: IEEE /0661r0 Submission July 2005 Alexander Tolpin, IntelSlide 11 TPT vs Attenuation – purpose The primary metric is the average data payload successfully transferred during 1 second from/to MAC SAP of DUT to/from MAC SAP of WLCP for specific attenuation of DUT. Attenuation corresponds to RX Power of the RF signal measured at the antenna connector of DUT Applicable for wireless clients, both IBSS and BSS Provides the basic measure of the ability of DUT to transmit and receive frames without loss across the wireless interface in conductive environment. Correlates with end-user experiences of TPT vs range in a real life environment The additional secondary metrics may be measured concurrently: the retry rate and the non-acked rate.

doc.: IEEE /0661r0 Submission July 2005 Alexander Tolpin, IntelSlide 12 Metrics The throughput is computed and reported as the average payload per second of all data frames that were injected to DUT or WLCP through wired interface by Traffic Generator and successfully transmitted, delivered via conductive wireless media and received by WLCP or DUT respectively, and then captured by Wired Traffic Analyzer via wired interface. –The only one instance of the specific injected frame is counted on the other side. –If the frame was duplicated and few instances were delivered and captured by Wired Traffic analyzer, the first instance only must be counted and rest must be ignored. The retry rate is measured as the number of packets were retransmitted via wireless interface when each retransmission is counted separately, divided to the number of packets were transmitted at least once. The non-acked rate is measured as the number of packets were transmitted but not acked if ack is required, divided to the number of packets were transmitted. –For this metric each retransmission is counted separately.

doc.: IEEE /0661r0 Submission July 2005 Alexander Tolpin, IntelSlide 13 TPT vs. Attenuation conductive setup

doc.: IEEE /0661r0 Submission July 2005 Alexander Tolpin, IntelSlide 14 Procedure Setup DUT to the initial configuration and associate with WLCP. Step 0: Set attenuation current_value=min_value. Step 1: –Test controller generates data traffic higher or equal to maximum theoretical throughput for specific frame sizes during required duration. –The WLCP TX power is measured and recorded. –The traffic captured by the Wired Traffic Analyzer –TPT is measured and recorded. –The retry rate and non-ack rate are extracted by Traffic Analyzer and recorded. Step 2: Set attenuation current_value+=step_value Step 3: Repeat steps 1-2 until current_value>max_value Each attenuation value should be translated to DUT RX Power based on measured TX Power of WLCP, cables lost and attenuation values

doc.: IEEE /0661r0 Submission July 2005 Alexander Tolpin, IntelSlide 15 Attenuation (dB) Setup_Path_Loss (dB) WLCP_TX_Power (dBm) RX_Input_Power = WLCP_TX_Power- Attenuation- Setup_Path_Loss (dBm) Average TPT (MBps) Retry Rate - optional Non- acked rate - optional Reporting The results can be summarized in the following table: The results should be reported as a table or a graph of throughput vs. attenuation

doc.: IEEE /0661r0 Submission July 2005 Alexander Tolpin, IntelSlide 16 Example of test results – 3 measurements The tests were repeated 3 times using off the shelf NIC using same setup, the precision is within +/- 3% from average

doc.: IEEE /0661r0 Submission July 2005 Alexander Tolpin, IntelSlide 17 Variation of measured data Data collected on a off-the-shelf a/b/g NIC Data collected for 3 runs each time % variation was calculated for each attenuation value as follows % variation = (Max Throughput – Min Throughput) / Avg. Throughput ModeMin variationMax variationAverage variation g – Rx0.18%6.41%1.58% g – Tx0.18%4.6%1.43% a – Rx0.53%6.08%2.08% a - Tx0.09%5.46%1.47%

doc.: IEEE /0661r0 Submission July 2005 Alexander Tolpin, IntelSlide 18 Example of reporting table Attenuation (dB) Setup_Path_Loss (dB) WLCP_TX_Power (dBm) RX_Input_Power = WLCP_TX_Power-Attenuation- Setup_Path_Loss (dBm) TX Average TPT (mBps) RX Average TPT (mBps)

doc.: IEEE /0661r0 Submission July 2005 Alexander Tolpin, IntelSlide 19 TX Rate Adaptation – purpose This is the secondary metric already discussed in [6]. Based on the previously defined TPT vs Attenuation test. The purpose of this test is to determine the ability of DUT to select the TX rate that is the most efficient for current link condition. Applicable for clients, both IBSS and BSS. Allows discovering the root cause of TPT degradation.

doc.: IEEE /0661r0 Submission July 2005 Alexander Tolpin, IntelSlide 20 Metrics The TPT measured using automatic TX rate selection for specific attenuation (RX Power Level) is ActualTPT (RxPowerLevel). The maximum of TPT results achieved for the specific attenuation (RX Power) using fixed TX rates is BestTPT(RxPowerLevel). The Efficiency(RxPowerLevel) is the metric for a specific attenuation. The AverageEfficiency is the metric for the certain range of RX Power Level with n points of measurements.

doc.: IEEE /0661r0 Submission July 2005 Alexander Tolpin, IntelSlide 21 Rate Adaptation conductive setup

doc.: IEEE /0661r0 Submission July 2005 Alexander Tolpin, IntelSlide 22 Procedure Setup DUT according to the initial configuration and associate with WLCP. Measure TPT vs. Attenuation using automatic rate selection, and record the ActualTPT Enforce DUT to transmit data frames on the specific rate, measure TPT vs Attenuation and record TPT per fixed rate. –Repeat for all relevant fixed rates. –Calculate the BestTPT (the maximum TPT achieved using different fixed rates) for each attenuation point Calculate Efficiency (RxPowerLevel) = ActualTPT/BestTPT for each attenuation point.

doc.: IEEE /0661r0 Submission July 2005 Alexander Tolpin, IntelSlide 23 Reporting The results can be summarized in the following table: Attenuation (dB) RX Power Level (dBm) ActualTPT (MBps) Fixed rate TPT (MBps) BestTPT (MBps) Efficiency = ActualTPT/ BestTPT TxRate#1…TxRate#n The results should be reported as a table or a graph of Efficiency(RxPowerLevel) The Average Efficiency should be calculated and reported for short range (above --35dBm), mid range (between -35dBm and - 65dBm inclusive), and far range (below -65dBm till -85dBm)

doc.: IEEE /0661r0 Submission July 2005 Alexander Tolpin, IntelSlide 24 Example of TX Rate Adaptation test results Short Range Average Efficiency =0.95 Mid Range Average Efficiency=0.99 Far Range Average Efficiency=0.97

doc.: IEEE /0661r0 Submission July 2005 Alexander Tolpin, IntelSlide 25 Antenna Diversity – purpose This is the secondary metric already discussed in [5] This test is based on the previously defined TPT vs Attenuation test. Applicable for clients, both BSS and IBSS The purpose of this test is to determine how the changing of RF signal power for single MAIN and AUX antennas impacts TPT of DUT. The test provides the basic measure of an ability of DUT to select the optimal antenna when input RX power of each antenna is changing over time.

doc.: IEEE /0661r0 Submission July 2005 Alexander Tolpin, IntelSlide 26 Metric The required metric is the difference between TPT achieved when both antennas have same RX Power and when one antenna has less RX Power The expectation is that DUT selects the antenna with higher RX Power and TPT is not be impacted significantly.

doc.: IEEE /0661r0 Submission July 2005 Alexander Tolpin, IntelSlide 27 Antenna Diversity conductive setup

doc.: IEEE /0661r0 Submission July 2005 Alexander Tolpin, IntelSlide 28 Antenna Diversity measurement procedure A = B Dwell Time Transition Time Time Power A B A = B Dwell Time Transition Time Time Power B A The TPT is measured and recorded separately for each antenna combinations during dwell time and transition time Test parameters –Dwell time –Transition time –Minimum attenuation (maximum RF input power) for each antenna –Maximum attenuation (minimum RF input power) for each antenna

doc.: IEEE /0661r0 Submission July 2005 Alexander Tolpin, IntelSlide 29 Reporting The results can be summarized in the following table: StepCombinations of RX PowerTimeTPT measuredComments 0A=B=maximum0N/AInitial setting 1A=B=maximumDwell Expect maximum TPT 2A=maximum, B-> minimumTransition Expect a little TPT impact 3A=maximum, B=minimumDwell Expect antenna A selected 4A=maximum, B->maximumTransition Expect TPT -> maximum 5B=A=maximumDwell Expect maximum TPT 6A->minimum, B=maximumTransition Expect a little TPT impact 7A=minimum, B=maximumDwell Expect antenna B selected 8A->maximum, B=maximumTransition Expect TPT -> maximum

doc.: IEEE /0661r0 Submission July 2005 Alexander Tolpin, IntelSlide 30 Conclusions The conductive environment has been proposed The proposal contains: –Requirements for equipment –Requirements for test conditions –General test setup –Examples of specific setups for specific tests The three specific metrics have been proposed - Throughput vs. attenuation, TX rate adaptation and Antenna Diversity The proposal for each metric uses the template [2] and contains: –Purpose of the metric –Description of the metric –Description of the setup –Baseline, modifiers and test parameters –Procedure –Reporting results requirements Recommendation: –Recommend TGT to adopt the content of document 11-05/0660r0 into the P draft.

doc.: IEEE /0661r0 Submission July 2005 Alexander Tolpin, IntelSlide 31 References [1] IEEE [2] IEEE /1540r1.Tom Alexander. Task Group T (WPP) Metrics Template. [3] IEEE /1641r1. Tom Alexander. Metrics Template Example [4] P D0.1. Draft Recommended Practice for the Evaluation of Wireless Performance [5] IEEE /0194r0. Craig Warren. Performance Testing of Diversity for [6] IEEE /1466r1. Mike Wilhoyte. Some Concepts of Rate Management Testing [7] IEEE /419r1. Alexander Tolpin. Conductive Test Environment