1. Using Stop Motion Turntables in Open OTA Throughput Tests
IEEE TO be obtained
November 2005
Using Stop Motion Turntables in Open OTA Throughput Tests
August/2006
Date:
Authors:
5355 Ave Encinas, Carlsbad, CA 92008
Pertti Visuri
Airgain, Inc
(760)
Notice: This document has been prepared to assist IEEE It is offered as a basis for discussion and is not binding on the contributing individual(s) or organization(s). The material in this document is subject to change in form and content after further study. The contributor(s) reserve(s) the right to add, amend or withdraw material contained herein.
Release: The contributor grants a free, irrevocable license to the IEEE to incorporate material contained in this contribution, and any modifications thereof, in the creation of an IEEE Standards publication; to copyright in the IEEE’s name any IEEE Standards publication even though it may include portions of this contribution; and at the IEEE’s sole discretion to permit others to reproduce in whole or in part the resulting IEEE Standards publication. The contributor also acknowledges and accepts that this contribution may be made public by IEEE
Patent Policy and Procedures: The contributor is familiar with the IEEE 802 Patent Policy and Procedures including the statement "IEEE standards may include the known use of patent(s), including patent applications, provided the IEEE receives assurance from the patent holder or applicant with respect to patents essential for compliance with both mandatory and optional portions of the standard." Early disclosure to the Working Group of patent information that might be relevant to the standard is essential to reduce the possibility for delays in the development process and increase the likelihood that the draft publication will be approved for publication. Please notify the Chair as early as possible, in written or electronic form, if patented technology (or technology under patent application) might be incorporated into a draft standard being developed within the IEEE Working Group. If you have questions, contact the IEEE Patent Committee Administrator at
Pertti Visuri, Airgain, Inc.
Pertti Visuri, AIrgain, Inc.

2. Eliminating Inconsistencies in Open OTA tests
August/2006
Eliminating Inconsistencies in Open OTA tests
The purpose of this presentation is to provide sample results on how using Stop Motion Turntables can help to eliminate the inconsistencies that plaque Open Over The Air measurements when comparing devices that have different antenna systems
In one test three g access points that have different radio systems and also different antenna systems were used in throughput comparison tests in a residential environment
In another test two client station wireless cards that were identical except for different antennas were tested
Finally in a test of three access points with different antenna systems the effect of the number of stops of the turntable was evaluated
Pertti Visuri, Airgain, Inc.

3. August/2006
Three AP Comparison
Three AP devices were compared in an Open Over the Air test
The throughput of each device was measured over six different links
Tests at each link included 15 measurements for each Device Under Test (DUT) 5 6 4 1 2 3
Access Point
Client
Second Floor
Pertti Visuri, Airgain, Inc.

4. Dual Turntables for Each Link
August/2006
Dual Turntables for Each Link
The 15 tests in each location were performed by placing both the DUT and the wireless counter part (WLCP) on a stop motion turntable. Both tables were turned 24º between each measurement and stopped for the duration of the throughput test.
The 15 stops covered the full circle on both turntables. The tables were turning in opposite directions
Client Station (WLCP)
Access Point (DUT)
Stop-motion turntable with controller
Stop-motion turntable with controller
Test control system
Traffic generation device
Traffic analyzer
Automatic control of stop-motion turn tables
The connection to one or the turntables was wireless but it was not operated during the throughput test
Pertti Visuri, Airgain, Inc.

5. Measured Throughputs in All Tests
August/2006
Measured Throughputs in All Tests
Here are the results of all measurements. The only difference between the curves on the graph of each DUT is the precise location and orientation of the DUT and WLCP
As can be seen, the throughput of each device varies dramatically based on the precise location and orientation of the DUT and the WLCP
Throughput of DUT1
Throughput of DUT2
Throughput of DUT3
Throughput (Mbits/s)
Link number
Link number
Link number
Pertti Visuri, Airgain, Inc.

6. Comparison Results in Different Tests
August/2006
Comparison Results in Different Tests
Here is a part of the same data of measuring the same six links with exactly the same AP and Client Stations. The only differences between the graphs are the precise location and the orientation of the AP and the Client station. The results appear inconsistent
DUT1 DUT2
DUT3
Throughputs in orientation 3
Throughputs in orientation 4
Throughputs in orientation 1
Throughput (Mbits/s)
Link number
Link number
Link number
Throughputs in orientation 5
Throughputs in orientation 9
Throughputs in orientation 10
Throughput (Mbits/s)
Link number
Link number
Link number
Pertti Visuri, Airgain, Inc.

7. Averaging the Results of Slightly Different Locations
August/2006
Averaging the Results of Slightly Different Locations
Fortunately it is possible to eliminate the inconsistencies and achieve repeatable results by simply performing several measurements at slightly different locations and orientations (of both the DUT and WLCP) for each of the links (distances) and averaging the results for each link
Averaging all 15 measurements for each of the six links shown on the previous slide provides consistent results.
Average Throughput of all 15 orientations
Link number
Pertti Visuri, Airgain, Inc.

8. Two Client Cards with Different Antennas
August/2006
Two Client Cards with Different Antennas
In another similar test the devices under test (DUTs) were both cardbus cards that were identical except that they had different antenna systems
The same AP (wireless counter part or WLCP) was used in all tests
The throughput of seven different links (general AP and client station locations) were measured for both Cards
Again the seven tests were repeated 15 times using slightly different locations and varying orientations of both the AP and the client stations
Pertti Visuri, Airgain, Inc.

9. Measured Throughputs in All Tests
August/2006
Measured Throughputs in All Tests
Here are the results of all measurements. The only difference between the curves on the graph of each DUT is the precise location and orientation of the DUT and WLCP
Throughput of DUT1
Throughput of DUT2
Step of the turn tables
Throughput (Mbits/s)
Link number
Link number
Pertti Visuri, Airgain, Inc.

10. Comparison Results in Different Test Runs
August/2006
Comparison Results in Different Test Runs
Here are a number of graphs comparing the two devices in exactly the same locations. The only differences between the graphs are the precise location and the orientation of the AP and the Client station
DUT1 DUT2
Throughputs in orientation 1
Throughputs in orientation 2
Throughputs in orientation 5
Throughput (Mbits/s)
Link number
Link number
Link number
Throughputs in orientation 9
Throughputs in orientation 11
Throughputs in orientation 13
Throughput (Mbits/s)
Link number
Link number
Link number
Pertti Visuri, Airgain, Inc.

11. Averaging the Results of Slightly Different Locations
August/2006
Averaging the Results of Slightly Different Locations
Again, calculating the average of all 15 measurements for each of the links provides consistent results. One of the DUTs performs better at all distances.
Average Throughput of all 15 orientations
DUT1 DUT2
Link number
Pertti Visuri, Airgain, Inc.

12. The Physics Causing the Large Variation
August/2006
The Physics Causing the Large Variation
The reason causing the wide variation and apparent inconsistency in Open Over the Air (OOTA) testing is multipath fading
Wireless signals reflect to varying degree from all surfaces that they reach.
Reflected signals arrive at the receiving antenna in different phases depending on the distance they travel
The RF field vectors add or subtract from one another depending on their phase and polarization
Moving either of the antennas or any of the reflecting surfaces will result in a change in signal strength
receiving antenna
Transmitting antenna
Pertti Visuri, Airgain, Inc.

13. Effect of Antenna Gain Patterns on Multipath Fading
August/2006
Effect of Antenna Gain Patterns on Multipath Fading
If the gain pattern of either the receiving or transmitting antenna is different than the gain patterns in a reference system the resulting multipath fading will be different
Reflected signals from all directions are included in the net signal strength and their contributions are affected by the antenna gain in each direction
This results in a different signal strength (in each location and for each orientation of the antennas) if the gain patterns of both the antennas in the compared systems are not identical
Transmitting antenna
receiving antenna + + + =
Pertti Visuri, Airgain, Inc.

14. Signal Variation with Antenna Location
August/2006
Signal Variation with Antenna Location
To measure the effect of multipath fading an access point with two different antenna systems was moved over a grid of 100 locations and the signal strength was measured in each location
The client station connected to the access point was about 40m (120 feet) away in a non-line of sight location
The client station was not moved at all during the test
The signal strength was measured using the RSSI reporting feature of an radio card and averaged over hundreds of samples during a few minutes to even out the effect of small changes in environment during the test
The local signal strength variation is about 12 to 15dB and the locations of high and low signals depend strongly on the antenna system
Antenna system 1 Signal Strength
Antenna system 2 Signal Strength
Pertti Visuri, Airgain, Inc.

15. Multipath Variation for Different Antenna Systems
August/2006
Single radio two antenna diversity unit
Multipath Variation for Different Antenna Systems
The multipath fading affects all systems, including MIMO systems
In this test throughput of four different systems were compared using the physical test arrangement on slide 11
Throughput was measured using a standard Chariot test in 100 locations for each of the tested systems
Each unit experienced very high throughput variations as a result of small movements and the local patterns were different
Pertti Visuri, Airgain, Inc.

16. How many Measurements in Each Location
August/2006
How many Measurements in Each Location
Clearly taking several measurements in slightly different orientations and locations helps achieve more consistent results and eliminates much of the random variations between locations (or links)
To determine how many measurements in each location would be needed to achieve consistent results a test of three DUTs was performed in three locations using 36 stops on a stop-motion turntable for both downlink and uplink throughput.
Then different samples were taken out of the 36 stops (turntable turning 10 degrees between stops)
By taking every sixth measurement six graphs each representing averaging six measurements at 60degree intervals were obtained
By taking every third point three graphs each averaging 12 measurements at 30 degrees apart were generated
By taking every other point two graphs averaging 18 measurements 15 degrees apart, and finally
One graph averaging all 36 measurements taken 10 degrees apart on the turntable
Pertti Visuri, Airgain, Inc.

17. All measurements August/2006 DUT3 Pertti Visuri, Airgain, Inc.
DUT1 DUT2
DUT3
Pertti Visuri, Airgain, Inc.

18. Sets of Averages of Six Measurements
August/2006
Sets of Averages of Six Measurements
DUT1 DUT2
Downlink
DUT3
Results are quite inconsistent.
The choice of which six measurements are chosen for the average affects the relative performance results
Different DUTs appear to be best in different locations
Uplink
Pertti Visuri, Airgain, Inc.

19. Sets of Averages of Twelve Measurements
August/2006
Sets of Averages of Twelve Measurements
Downlink
DUT1 DUT2
DUT3
Results are a little less inconsistent.
However, The choice of which twelve measurements are chosen for the average still affects the relative performance results
Pertti Visuri, Airgain, Inc.

20. Sets of Averages of Eighteen Measurements
August/2006
Sets of Averages of Eighteen Measurements
Downlink
DUT1 DUT2
DUT3
Results are much more consistent.
However, The choice of which eighteen measurements are chosen for the average still affects the relative performance results in some locations
Pertti Visuri, Airgain, Inc.

21. Averaging all 36 Measurements for Each Location
August/2006
Averaging all 36 Measurements for Each Location
DUT1 DUT2
DUT3
Finally, averaging 36 measurements taken 10 degrees apart on the turntable appear to provide quite consistent results for both downlink and uplink
Even though this evidence is somewhat anecdotal, taken together with similar results from many other tests, it supports a recommendation that in stop motion turntable testing the standard number of measurement stops should be 36, 10 degrees apart
Pertti Visuri, Airgain, Inc.

22. August/2006
Conclusions
It is not possible to evaluate the relative performance of wireless devices with different antenna systems by performing measurements only in a few locations at various distances
Performing several tests at each distance at slightly different locations and orientations and averaging the results at each distance is a very effective way to obtain consistent results and achieve repeatability of test results in open over the air (OTA) testing. (It is important to note that it is not helpful to repeat the same test in exactly the same locations of the AP and the client station and average the results together. The variations are caused by the location and orientation of the devices.)
It appears that performing 36 measurements using a stop motion turntable at both ends of the link will provide reasonably consistent comparison test results even in cases when the antenna systems are different in the DUTs
The proposed text of the Draft Recommended Practice for the Evaluation of Wireless Performance under preparation in the IEEE Task Group T incorporates this methodology for over the air testing.
Pertti Visuri, Airgain, Inc.