Using Stop Motion Turntables in Open OTA Throughput Tests IEEE 802.11 - TO be obtained November 2005 Using Stop Motion Turntables in Open OTA Throughput Tests August/2006 Date: 2006-11-9 Authors: 5355 Ave Encinas, Carlsbad, CA 92008 Pertti Visuri Airgain, Inc (760) 597 0200 pvisuri@airgain.com Notice: This document has been prepared to assist IEEE 802.11. It is offered as a basis for discussion and is not binding on the contributing individual(s) or organization(s). The material in this document is subject to change in form and content after further study. The contributor(s) reserve(s) the right to add, amend or withdraw material contained herein. Release: The contributor grants a free, irrevocable license to the IEEE to incorporate material contained in this contribution, and any modifications thereof, in the creation of an IEEE Standards publication; to copyright in the IEEE’s name any IEEE Standards publication even though it may include portions of this contribution; and at the IEEE’s sole discretion to permit others to reproduce in whole or in part the resulting IEEE Standards publication. The contributor also acknowledges and accepts that this contribution may be made public by IEEE 802.11. Patent Policy and Procedures: The contributor is familiar with the IEEE 802 Patent Policy and Procedures including the statement "IEEE standards may include the known use of patent(s), including patent applications, provided the IEEE receives assurance from the patent holder or applicant with respect to patents essential for compliance with both mandatory and optional portions of the standard." Early disclosure to the Working Group of patent information that might be relevant to the standard is essential to reduce the possibility for delays in the development process and increase the likelihood that the draft publication will be approved for publication. Please notify the Chair <stuart.kerry@philips.com> as early as possible, in written or electronic form, if patented technology (or technology under patent application) might be incorporated into a draft standard being developed within the IEEE 802.11 Working Group. If you have questions, contact the IEEE Patent Committee Administrator at <patcom@ieee.org>. Pertti Visuri, Airgain, Inc. Pertti Visuri, AIrgain, Inc.

Eliminating Inconsistencies in Open OTA tests August/2006 Eliminating Inconsistencies in Open OTA tests The purpose of this presentation is to provide sample results on how using Stop Motion Turntables can help to eliminate the inconsistencies that plaque Open Over The Air measurements when comparing devices that have different antenna systems In one test three 802.11g access points that have different radio systems and also different antenna systems were used in throughput comparison tests in a residential environment In another test two client station wireless cards that were identical except for different antennas were tested Finally in a test of three access points with different antenna systems the effect of the number of stops of the turntable was evaluated Pertti Visuri, Airgain, Inc.

August/2006 Three AP Comparison Three AP devices were compared in an Open Over the Air test The throughput of each device was measured over six different links Tests at each link included 15 measurements for each Device Under Test (DUT) 5 6 4 1 2 3 Access Point Client Second Floor Pertti Visuri, Airgain, Inc.

Dual Turntables for Each Link August/2006 Dual Turntables for Each Link The 15 tests in each location were performed by placing both the DUT and the wireless counter part (WLCP) on a stop motion turntable. Both tables were turned 24º between each measurement and stopped for the duration of the throughput test. The 15 stops covered the full circle on both turntables. The tables were turning in opposite directions Client Station (WLCP) Access Point (DUT) Stop-motion turntable with controller Stop-motion turntable with controller Test control system Traffic generation device Traffic analyzer Automatic control of stop-motion turn tables The connection to one or the turntables was wireless but it was not operated during the throughput test Pertti Visuri, Airgain, Inc.

Measured Throughputs in All Tests August/2006 Measured Throughputs in All Tests Here are the results of all measurements. The only difference between the curves on the graph of each DUT is the precise location and orientation of the DUT and WLCP As can be seen, the throughput of each device varies dramatically based on the precise location and orientation of the DUT and the WLCP Throughput of DUT1 Throughput of DUT2 Throughput of DUT3 Throughput (Mbits/s) Link number Link number Link number Pertti Visuri, Airgain, Inc.

Comparison Results in Different Tests August/2006 Comparison Results in Different Tests Here is a part of the same data of measuring the same six links with exactly the same AP and Client Stations. The only differences between the graphs are the precise location and the orientation of the AP and the Client station. The results appear inconsistent DUT1 DUT2 DUT3 Throughputs in orientation 3 Throughputs in orientation 4 Throughputs in orientation 1 Throughput (Mbits/s) Link number Link number Link number Throughputs in orientation 5 Throughputs in orientation 9 Throughputs in orientation 10 Throughput (Mbits/s) Link number Link number Link number Pertti Visuri, Airgain, Inc.

Averaging the Results of Slightly Different Locations August/2006 Averaging the Results of Slightly Different Locations Fortunately it is possible to eliminate the inconsistencies and achieve repeatable results by simply performing several measurements at slightly different locations and orientations (of both the DUT and WLCP) for each of the links (distances) and averaging the results for each link Averaging all 15 measurements for each of the six links shown on the previous slide provides consistent results. Average Throughput of all 15 orientations Link number Pertti Visuri, Airgain, Inc.

Two Client Cards with Different Antennas August/2006 Two Client Cards with Different Antennas In another similar test the devices under test (DUTs) were both cardbus cards that were identical except that they had different antenna systems The same AP (wireless counter part or WLCP) was used in all tests The throughput of seven different links (general AP and client station locations) were measured for both Cards Again the seven tests were repeated 15 times using slightly different locations and varying orientations of both the AP and the client stations Pertti Visuri, Airgain, Inc.

Measured Throughputs in All Tests August/2006 Measured Throughputs in All Tests Here are the results of all measurements. The only difference between the curves on the graph of each DUT is the precise location and orientation of the DUT and WLCP Throughput of DUT1 Throughput of DUT2 Step of the turn tables Throughput (Mbits/s) Link number Link number Pertti Visuri, Airgain, Inc.

Comparison Results in Different Test Runs August/2006 Comparison Results in Different Test Runs Here are a number of graphs comparing the two devices in exactly the same locations. The only differences between the graphs are the precise location and the orientation of the AP and the Client station DUT1 DUT2 Throughputs in orientation 1 Throughputs in orientation 2 Throughputs in orientation 5 Throughput (Mbits/s) Link number Link number Link number Throughputs in orientation 9 Throughputs in orientation 11 Throughputs in orientation 13 Throughput (Mbits/s) Link number Link number Link number Pertti Visuri, Airgain, Inc.

Averaging the Results of Slightly Different Locations August/2006 Averaging the Results of Slightly Different Locations Again, calculating the average of all 15 measurements for each of the links provides consistent results. One of the DUTs performs better at all distances. Average Throughput of all 15 orientations DUT1 DUT2 Link number Pertti Visuri, Airgain, Inc.

The Physics Causing the Large Variation August/2006 The Physics Causing the Large Variation The reason causing the wide variation and apparent inconsistency in Open Over the Air (OOTA) testing is multipath fading Wireless signals reflect to varying degree from all surfaces that they reach. Reflected signals arrive at the receiving antenna in different phases depending on the distance they travel The RF field vectors add or subtract from one another depending on their phase and polarization Moving either of the antennas or any of the reflecting surfaces will result in a change in signal strength receiving antenna Transmitting antenna Pertti Visuri, Airgain, Inc.

Effect of Antenna Gain Patterns on Multipath Fading August/2006 Effect of Antenna Gain Patterns on Multipath Fading If the gain pattern of either the receiving or transmitting antenna is different than the gain patterns in a reference system the resulting multipath fading will be different Reflected signals from all directions are included in the net signal strength and their contributions are affected by the antenna gain in each direction This results in a different signal strength (in each location and for each orientation of the antennas) if the gain patterns of both the antennas in the compared systems are not identical Transmitting antenna receiving antenna + + + = Pertti Visuri, Airgain, Inc.

Signal Variation with Antenna Location August/2006 Signal Variation with Antenna Location To measure the effect of multipath fading an access point with two different antenna systems was moved over a grid of 100 locations and the signal strength was measured in each location The client station connected to the access point was about 40m (120 feet) away in a non-line of sight location The client station was not moved at all during the test The signal strength was measured using the RSSI reporting feature of an 802.11 radio card and averaged over hundreds of samples during a few minutes to even out the effect of small changes in environment during the test The local signal strength variation is about 12 to 15dB and the locations of high and low signals depend strongly on the antenna system Antenna system 1 Signal Strength Antenna system 2 Signal Strength Pertti Visuri, Airgain, Inc.

Multipath Variation for Different Antenna Systems August/2006 Single radio two antenna diversity unit Multipath Variation for Different Antenna Systems The multipath fading affects all systems, including MIMO systems In this test throughput of four different systems were compared using the physical test arrangement on slide 11 Throughput was measured using a standard Chariot test in 100 locations for each of the tested systems Each unit experienced very high throughput variations as a result of small movements and the local patterns were different Pertti Visuri, Airgain, Inc.

How many Measurements in Each Location August/2006 How many Measurements in Each Location Clearly taking several measurements in slightly different orientations and locations helps achieve more consistent results and eliminates much of the random variations between locations (or links) To determine how many measurements in each location would be needed to achieve consistent results a test of three DUTs was performed in three locations using 36 stops on a stop-motion turntable for both downlink and uplink throughput. Then different samples were taken out of the 36 stops (turntable turning 10 degrees between stops) By taking every sixth measurement six graphs each representing averaging six measurements at 60degree intervals were obtained By taking every third point three graphs each averaging 12 measurements at 30 degrees apart were generated By taking every other point two graphs averaging 18 measurements 15 degrees apart, and finally One graph averaging all 36 measurements taken 10 degrees apart on the turntable Pertti Visuri, Airgain, Inc.

All measurements August/2006 DUT3 Pertti Visuri, Airgain, Inc. DUT1 DUT2 DUT3 Pertti Visuri, Airgain, Inc.

Sets of Averages of Six Measurements August/2006 Sets of Averages of Six Measurements DUT1 DUT2 Downlink DUT3 Results are quite inconsistent. The choice of which six measurements are chosen for the average affects the relative performance results Different DUTs appear to be best in different locations Uplink Pertti Visuri, Airgain, Inc.

Sets of Averages of Twelve Measurements August/2006 Sets of Averages of Twelve Measurements Downlink DUT1 DUT2 DUT3 Results are a little less inconsistent. However, The choice of which twelve measurements are chosen for the average still affects the relative performance results Pertti Visuri, Airgain, Inc.

Sets of Averages of Eighteen Measurements August/2006 Sets of Averages of Eighteen Measurements Downlink DUT1 DUT2 DUT3 Results are much more consistent. However, The choice of which eighteen measurements are chosen for the average still affects the relative performance results in some locations Pertti Visuri, Airgain, Inc.

Averaging all 36 Measurements for Each Location August/2006 Averaging all 36 Measurements for Each Location DUT1 DUT2 DUT3 Finally, averaging 36 measurements taken 10 degrees apart on the turntable appear to provide quite consistent results for both downlink and uplink Even though this evidence is somewhat anecdotal, taken together with similar results from many other tests, it supports a recommendation that in stop motion turntable testing the standard number of measurement stops should be 36, 10 degrees apart Pertti Visuri, Airgain, Inc.

August/2006 Conclusions It is not possible to evaluate the relative performance of wireless devices with different antenna systems by performing measurements only in a few locations at various distances Performing several tests at each distance at slightly different locations and orientations and averaging the results at each distance is a very effective way to obtain consistent results and achieve repeatability of test results in open over the air (OTA) testing. (It is important to note that it is not helpful to repeat the same test in exactly the same locations of the AP and the client station and average the results together. The variations are caused by the location and orientation of the devices.) It appears that performing 36 measurements using a stop motion turntable at both ends of the link will provide reasonably consistent comparison test results even in cases when the antenna systems are different in the DUTs The proposed text of the Draft Recommended Practice for the Evaluation of 802.11 Wireless Performance under preparation in the IEEE 802.11 Task Group T incorporates this methodology for over the air testing. Pertti Visuri, Airgain, Inc.