doc.: IEEE t Submission November 2004 Tom AlexanderSlide 1 A Link Layer Metrics Proposal for TGT Tom Alexander VeriWave, Inc. November 2004

doc.: IEEE t Submission November 2004 Tom AlexanderSlide 2 Outline Background and Scope General Approach Functional Model Configuration Parameters and Test Conditions Specific Metrics and Tests

doc.: IEEE t Submission November 2004 Tom AlexanderSlide 3 Background & Scope Proposal covers only link layer metrics –Examples: throughput, latency at MAC level Uses general methodology principles followed by the IETF BMWG Based on an Internet Draft by Tom Alexander and Scott Bradner –Available as draft-alexander-wlan-meth-00.txt Available on any IETF mirror, e.g.: –This presentation gives a summary Interested people referred to the full document (54 pages) for more information Device Level Metrics (e.g., battery life) Link Layer Metrics (e.g., forwarding rate) Layer 1.5 Metrics (e.g., roaming) PHY Layer Metrics (e.g., total radiated power) System-level Metrics (e.g., R factor) This is Tom’s view of the world! :-)

doc.: IEEE t Submission November 2004 Tom AlexanderSlide 4 General Approach Establish and test a baseline first, then test variations –Standard process followed by the BMWG, also contemporary practice –Vary ONE element at a time –Example: perform a baseline throughput test with all parameters at their defaults, then repeat with RTS/CTS enabled, then disable RTS/CTS and repeat with fragmentation enabled, etc. Both client and AP testing covered Testers must stress the DUT but cannot violate the protocol Test data consists of specific frame sizes and frame formats, following principles laid down by many preceding BMWG test specifications Specify test conditions, not test equipment –For example: it is irrelevant as to whether shielded or open-air environments are used, provided that specified signal level, SNR and interference limits are met during a trial 802 policy: Do not unduly restrict implementations –Wireless is complex => there are a lot of test conditions to be met

doc.: IEEE t Submission November 2004 Tom AlexanderSlide 5 Functional Model Simple functional model, consisting of DUT and tester –Implementation details left up to test implementer AP DUT Tester Client DUT Tester Client DUT (NIC and stack only) Tester Wireless Media Interface Wired Media Interface Wireless Media Interface (Under Test) Secondary Wired or Wireless Interface Test Software on DUT Wireless Media Interface (Under Test) AP/switch DUT Testing Client DUT Testing (preferred) Client DUT Testing (alternative)

doc.: IEEE t Submission November 2004 Tom AlexanderSlide 6 Configuration Parameters Configuration parameters deal with setup parameters internal to the DUT –Different parameters for client DUTs vs. AP/switch DUTs These parameters must be reported with the results –Multiple trials may be carried out using different configuration parameters (where applicable) Parameters covered –Hardware/software version, interface state (active, inactive), etc. –AP operating mode (WDS, ESS, etc) and client operating mode (IBSS, infrastructure BSS, etc.) –Channel assignment, DUT power level –RTS and fragmentation thresholds –Power management modes, service priorities (QoS levels), security modes

doc.: IEEE t Submission November 2004 Tom AlexanderSlide 7 Test Conditions Test conditions are generally external to the DUT –Related to tester state, test environment, equipment setup, test traffic, etc. Must also be reported with the test results Test conditions covered: –Test environment, signal level, signal-to-noise and signal-to- interference ratios –Test traffic frame sizes and frame formats, connection setup –Offered load, PHY data rates, management traffic parameters –Number of clients, trial duration –Combination of various conditions/configurations to create sets of test parameters Test conditions for wireless are much more complex than for wired LAN testing

doc.: IEEE t Submission November 2004 Tom AlexanderSlide 8 Categories of Link Layer Metrics Metrics are divided into three categories –Throughput-related –Latency/timing-related –Capacity-related Throughput-related tests deal with rates –Examples: throughput, forwarding rate, loss ratio, etc. –Should probably be called “rate-related”, but that’s a tongue-twister! Latency/timing-related tests deal with times –Examples: latency, rate adaptation time, roaming time, etc. Capacity-related tests deal with amounts –Examples: number of clients supported, power save buffer capacity, etc.

doc.: IEEE t Submission November 2004 Tom AlexanderSlide 9 Intended Load vs. Offered Load Quantifying and achieving a pre-set offered load is a difficult problem in WLANs –Medium access is half-duplex and requires carrier-sense/deference DUT contends with tester for access to medium –Random backoff required after all transmission attempts –Backoff timers of contending clients are suspended when medium is busy –Unicast frames must be acknowledged, and retried if necessary –High error rates imply high levels of retry and backoff The actual offered load achievable by the tester may thus vary significantly from the intended load Therefore, the tester should present a specified intended load (Iload), and measure and report the offered load (Oload) that is actually achieved (along with the performance of the DUT, of course) –Appendix A in document describes how to calculate Iload –Any bidirectional Iload > 50% of theoretical capacity of the wireless medium oversubscribes the DUT

doc.: IEEE t Submission November 2004 Tom AlexanderSlide 10 Throughput Related Metrics Can use either frame-based or time-based load (RFC2889) –Examples: throughput, latency at MAC level Tests covered: –Unicast intra-BSS throughput, forwarding rate, frame loss Standard throughput test on APs and clients –Unicast “ESS-mode” throughput, forwarding rate, frame loss APs only –Multicast forwarding rate APs only –Forward pressure (see RFC2285, RFC2889) Detect cheaters ;-) –Authentication and association rates –Power management mode throughput, forwarding rate, frame loss Test parameters that are varied per trial include frame sizes, number of STAs, signal level, fragmentation, RTS/CTS, security mode, etc.

doc.: IEEE t Submission November 2004 Tom AlexanderSlide 11 What Does Throughput Mean in ? The standard definition of maximum throughput is: Maximum Forwarding Rate at which zero loss occurs –Forwarding Rate => loss expected (and measured) –Throughput => no loss allowed is a high-loss medium –PHY signal level, SNR, ACI rejection parameters are specified at a 10% Frame Error Ratio!! –If one out of 10 frames can be trashed by a conforming PHY, is there much point defining a throughput metric (in the traditional sense)? is a reliable, in-order-delivery link-layer protocol, so … –Link layer retries and retries and retries until frames are delivered –A DUT that is incapable of high forwarding rate could conceivably throttle the tester by refusing to issue ACKs Can the tester tell the difference? Probably not, thanks to the high loss –Any form of frame loss is probably more due to an overloaded medium rather than an overloaded DUT anyway Does a “throughput” metric even make sense?

doc.: IEEE t Submission November 2004 Tom AlexanderSlide 12 Latency/Timing Related Metrics Can use either last-bit-in to first-bit-out (“store-and-forward”) or first- bit-in to first-bit-out (“cut-through”) approach –See RFC1242 Tests covered –Intra-BSS latency and latency variation (peak-to-peak jitter) Clients and APs –ESS latency and latency variation APs only –Roaming and reassociation time Note: this test covers what can be referred to as “failover roaming”; signal- strength based roaming is considered to be a Layer 1.5 metric. –Rate adaptation time Note: this test measures the time taken for the DUT to react to step changes in signal strength; rate adaptation in the presence of linear variations in signal strength is considered to be a Layer 1.5 metric –Beacon interval and timing –Reset recovery time

doc.: IEEE t Submission November 2004 Tom AlexanderSlide 13 Capacity Related Metrics These tests are only applicable to APs and switches Tests covered: –Burst capacity Ability to accept and forward bursts of back-to-back data frames Kept separate from the standard forwarding rate tests because of the issues with half-duplex and achieving uninterrupted bursts of frames –Authentication and association database capacity –Power-save buffer capacity These really quantify the internal packet buffer and database capacity of the AP or switch Some of these tests should be done prior to performing the corresponding throughput-related tests –Test results constrain test parameters during dependent tests

doc.: IEEE t Submission November 2004 Tom AlexanderSlide 14 Conclusions A set of link layer metrics have been proposed The proposal contains: –List of metrics –General test setup, DUT configuration, test conditions –Results reporting requirements –Baseline test parameters –Detailed description of each metric and associated test, including: Objective of test and metric, and expected correlation to user experience Relevant test parameters Test procedure, including normative requirements Analysis Results reporting The proposal could form a starting point for the link layer metrics work in TGT –More work required on things such as throughput and fully specifying test conditions