1. OTA Comparison test results and test design
IEEE TO be obtained
November 2005
OTA Comparison test results and test design
January/2006
Date:
Authors:
5355 Ave Encinas, Carlsbad, CA 92008
Pertti Visuri
Airgain, Inc
(760)
Notice: This document has been prepared to assist IEEE It is offered as a basis for discussion and is not binding on the contributing individual(s) or organization(s). The material in this document is subject to change in form and content after further study. The contributor(s) reserve(s) the right to add, amend or withdraw material contained herein.
Release: The contributor grants a free, irrevocable license to the IEEE to incorporate material contained in this contribution, and any modifications thereof, in the creation of an IEEE Standards publication; to copyright in the IEEE’s name any IEEE Standards publication even though it may include portions of this contribution; and at the IEEE’s sole discretion to permit others to reproduce in whole or in part the resulting IEEE Standards publication. The contributor also acknowledges and accepts that this contribution may be made public by IEEE
Patent Policy and Procedures: The contributor is familiar with the IEEE 802 Patent Policy and Procedures including the statement "IEEE standards may include the known use of patent(s), including patent applications, provided the IEEE receives assurance from the patent holder or applicant with respect to patents essential for compliance with both mandatory and optional portions of the standard." Early disclosure to the Working Group of patent information that might be relevant to the standard is essential to reduce the possibility for delays in the development process and increase the likelihood that the draft publication will be approved for publication. Please notify the Chair as early as possible, in written or electronic form, if patented technology (or technology under patent application) might be incorporated into a draft standard being developed within the IEEE Working Group. If you have questions, contact the IEEE Patent Committee Administrator at
Pertti Visuri, Airgain, Inc.
Pertti Visuri, AIrgain, Inc.

2. Presentation Objective
January/2006
Presentation Objective
This presentation discusses effective Test design for OTA throughput comparison tests
When two Devices Under Test (DUTs) are compared in Over the Air (OTA) environment it is necessary to perform a number of measurements in slightly different locations to manage the effect of multipath fading
To test the effectiveness of this approach in revealing differences that exist in wireless devices a set of tests was performed and then two comparisons were performed based on the test data
One comparison was between DUTs that actually are different
The other was between identical devises measured in slightly different multipath conditions
15dB
50 cm (20 inches)
An examples of signal strength variations caused by multipath
Pertti Visuri, Airgain, Inc.

3. Throughput Comparison Test
January/2006
Throughput Comparison Test
The tests were done by measuring the throughput of two AP devices (DUT1 and DUT2 ) that were otherwise identical, except one had a higher maximum throughput and the other had a smart antenna that improves the signal by 4dB
A total of about 380 one minute throughput tests were performed. These consisted of
19 different AP locations that were the same for both DUT1 and DUT2
10 different orientations of the client laptop (WLCP) for each of the AP locations
Gateway
Examples of Measured Links (between floors in dashed line)
Pertti Visuri, Airgain, Inc.

4. Throughput Comparison Test
January/2006
Throughput Comparison Test
Changing the client laptop (WLCP) orientation/location between measurements was facilitated by using an automated stop-motion turntable at the client end in all of the testing
It turned 36 degrees and stopped for a 60 second throughput test before recording the result and turning to the next location.
The client was a cardbus card so every turn of the laptop changed both the WLCP location and orientation
WLCP
10 repetitions
Stop-motion turntable with controller
Pertti Visuri, Airgain, Inc.

5. Demonstrating the Test Design
January/2006
Demonstrating the Test Design
To demonstrate the effectiveness of the “throughput comparison” test design the collected data of 380 throughput tests was compared in two different ways:
First the data measured with DUT1 was compared to the data measured with DUT2. A clear difference is expected in the results
Then the data for DUT2 was divided in two groups based on the orientation of the WLCP during the measurement. Results with even numbered rotations were compared to measurements with odd numbered rotations. No difference should be evident in the results
Comparison 1
Comparison 2
Two different DUTs
The same WLCP orientation
The same DUT
WLCP rotated 36º
DUT1
WLCP angle 1
DUT2
WLCP angle 2
DUT2
Pertti Visuri, Airgain, Inc.

6. January/2006
Raw Data Points
All data points (each a set of two compared measurements) in the order they were first recorded
Comparison 1
Comparison 2
Two different DUTs
The same WLCP orientation
The same DUT
WLCP rotated 36º
DUT1
WLCP angle 1
DUT2
WLCP angle 2
DUT2
Throughput (Mbits/s)
WLCP orientation
DUT location
WLCP orientation
DUT location
Pertti Visuri, Airgain, Inc.

7. Average of Compared Entities within Each Data Point
January/2006
Average of Compared Entities within Each Data Point
The average of the compared entities is calculated for each data point to be used as a help variable for forming sub-samples
Comparison 1
Comparison 2
Two different DUTs
The same WLCP orientation
The same DUT
WLCP rotated 36º
DUT1
WLCP angle 1
DUT2
WLCP angle 2
DUT2
Average of DUT2 and DUT1
Average of angle1and angle2
Throughput (Mbits/s)
WLCP orientation
DUT location
WLCP orientation
DUT location
Pertti Visuri, Airgain, Inc.

8. Data Points Arranged into Sub-samples
January/2006
Data Points Arranged into Sub-samples
The data points are organized into sub-samples based on their averages
Comparison 1
Comparison 2
Two different DUTs
The same WLCP orientation
The same DUT
WLCP rotated 36º
DUT1
WLCP angle 1
DUT2
WLCP angle 2
Throughput (Mbits/s)
Range of average throughput of DUT2 and DUT1
0 to 5
5 to 10
10 to 15
15 to 20
over 20
Average of DUT2 and DUT1
Average of angle1and angle2
Throughput (Mbits/s)
over 20
15 to 20
10 to 15
5 to 10
0 to 5
Range of average throughput of DUT2 and DUT1
Pertti Visuri, Airgain, Inc.

9. Averages of Sub-samples Calculated
January/2006
Averages of Sub-samples Calculated
The average of each sub sample is calculated for both compared entities
Comparison 1
Comparison 2
Two different DUTs
The same WLCP orientation
The same DUT
WLCP rotated 36º
DUT1
WLCP angle 1
DUT2
WLCP angle 2
Average of DUT2 and DUT1
Average of angle1and angle2
Throughput (Mbits/s)
Throughput (Mbits/s)
over 20
15 to 20
10 to 15
5 to 10
0 to 5
over 20
15 to 20
10 to 15
5 to 10
0 to 5
Range of average throughput of DUT2 and DUT1
Range of average throughput of DUT2 and DUT1
Pertti Visuri, Airgain, Inc.

10. January/2006
Final Test Results
The averages of the sub-samples are the final results of the test
Comparison 1
Comparison 2
Two different DUTs
The same WLCP orientation
WLCP rotated 36º
The same DUT
DUT1
WLCP angle 1
DUT2
WLCP angle 2
Throughput (Mbits/s)
Throughput (Mbits/s)
The results show the expected difference both at high and at low throughout levels
As expected, the results show no difference between the compared data point sets
Range of average throughput of DUT2 and DUT1
0 to 5
5 to 10
10 to 15
15 to 20
over 20
Range of average throughput of DUT2 and DUT1
0 to 5
5 to 10
10 to 15
15 to 20
over 20
Pertti Visuri, Airgain, Inc.

11. Final Test Results Percent difference
January/2006
Final Test Results Percent difference
Another way to present the result is the percentage difference of the compared entities as a function of general throughput level
Comparison 1
Comparison 2
Two different DUTs
The same WLCP orientation
WLCP rotated 36º
The same DUT
DUT1
WLCP angle 1
DUT2
WLCP angle 2 % % %
Percentage difference in Throughput
Percentage difference in Throughput
Range of average throughput of DUT2 and DUT1
0 to 5
5 to 10
10 to 15
15 to 20
over 20
Range of average throughput of DUT2 and DUT1
0 to 5
5 to 10
10 to 15
15 to 20
over 20
Range of average throughput of DUT2 and DUT1
0 to 5
5 to 10
10 to 15
15 to 20
over 20
Pertti Visuri, Airgain, Inc.

12. Using the Throughput vs. place test design
January/2006
Using the Throughput vs. place test design
The same exact measurement data can also be processed using a “throughput vs. place” test design
In this case there are 19 different ranges, but for each of them only one AP location
The following slides show this way of processing the data for the two comparisons presented in the preceding slides
The “throughput vs. place” test design is essentially the one used in the throughput vs. range tests
Gateway
Examples of Measured Links (between floors in dashed line)
Pertti Visuri, Airgain, Inc.

13. Comparison Results when averaging in each Place
January/2006
Comparison Results when averaging in each Place
Here are the results for the throughput vs. place comparison of DUT1 and DUT2 in the residential environment shown as a function of the “places” of the AP
All 10 WLCP (client) location/orientation combinations averaged for each place and the results are organized according to the average throughput of the two DUTs in each place
Throughput (Mbits/s)
AP location
Pertti Visuri, Airgain, Inc.

14. Comparison Results for DUT1 and DUT2
January/2006
Comparison Results for DUT1 and DUT2
Here are a number of test results obtained using only one WLCP location/orientation
The throughput vs. place graphs display a lot of variation reflecting the effect of multipath fading making results random
This demonstrates that it is important to include both several AP and DUT locations in all samples
DUT1 DUT2
WLCP orientation 1
WLCP orientation 2
WLCP orientation 6
Average of all WLCP orientations
WLCP orientation 7
WLCP orientation 9
Pertti Visuri, Airgain, Inc.

15. Comparison Results for Identical units
January/2006
Comparison Results for Identical units
Here are the same results for the throughput vs. range comparison of two identical devices where the only difference in between the two graphs is that the WLCP (laptop) was rotated 36 degrees between measurements
DUT1 DUT2
WLCP orientation 1
WLCP orientation 3
WLCP orientation 5
Average of all WLCP orientations
WLCP orientation 7
WLCP orientation 9
Pertti Visuri, Airgain, Inc.

16. Comparison Results for RANGE
January/2006
Comparison Results for RANGE
Here are the results for a throughput vs. range test comparison of DUT1 and DUT2 in a large office environment shown as a function of the distance between DUT and WLCP (Range)
The WLCP (client laptop with a wireless card) was on a stop motion turntable. Each point on the graph is the average of 18 throughput measurements.
Throughput (Mbits/s) 25 35 55 60 75 95 90
120
120
Range (feet)
Pertti Visuri, Airgain, Inc.

17. Comparison Results for RANGE
January/2006
Comparison Results for RANGE
Here are the same results for the throughput vs. range comparison of two identical devices where the only difference in between the two graphs is that the WLCP (laptop) was rotated 36 degrees between measurements
DUT1 DUT2
WLCP orientation 1
WLCP orientation 7
WLCP orientation 11
WLCP orientation 6
WLCP orientation 4
Average of all WLCP orientations
Pertti Visuri, Airgain, Inc.

18. Throughput Comparison Test Design
January/2006
Throughput Comparison Test Design
Here are all of the data points measured in the various WLCP orientations
Throughput (Mbits/s) 25 35 55 60 75 95 90
120
120
Range (feet)
Pertti Visuri, Airgain, Inc.

19. Throughput Comparison Test Design
January/2006
Throughput Comparison Test Design
Calculating the averages of the two DUTs in each location
Throughput (Mbits/s) 25 35 55 60 75 95 90
120
120
Range (feet)
Pertti Visuri, Airgain, Inc.

20. Comparison Results for RANGE
January/2006
Comparison Results for RANGE
Organizing the results according to the average throughout of the two DUTs
Throughput (Mbits/s)
10 to 15
Over 25
20 to 25
15 to 20
5 to 10
0 to 5
Range of average throughput of DUT2 and DUT1
Pertti Visuri, Airgain, Inc.

21. Comparison Results for RANGE
January/2006
Comparison Results for RANGE
Final results using the “Throughput Comparison”
Throughput (Mbits/s)
Range of average throughput of DUT2 and DUT1
0 to 5
5 to 10
10 to 15
15 to 20
20 to 25
Over 25
Pertti Visuri, Airgain, Inc.

22. Two methods of processing the data
January/2006
Two methods of processing the data
Here are the results of the two ways of processing the same data
“Throughput vs. Range” (using a turntable)
“Throughput Comparison”
Range (feet) 25 35 55 60 75 95 90
120
Throughput (Mbits/s)
Throughput (Mbits/s)
Over 25
20 to 25
15 to 20
10 to 15
5 to 10
0 to 5
Range of average throughput of DUT2 and DUT1
Pertti Visuri, Airgain, Inc.

23. January/2006
Straw Poll
Should the Multipath fading in OTA tests be addressed in the TGT draft in the general way discussed in the proposed addendum?
Pertti Visuri, Airgain, Inc.