doc.: wng Submission - Study Project Proposal WPP – Tools & Parameters November 2003 Bob Mandeville, Iometrics Bob Mandeville Wireless Performance Prediction (WPP) - Approaches to the Issue

doc.: wng Submission - Study Project Proposal WPP – Tools & Parameters November 2003 Bob Mandeville, Iometrics Two Approaches to Creating Testing Standards IETF (BMWG) –Based on two-step approach to definitions: Terminology document (all relevant functional characteristcs are defined) Methodology document This method is most appropriate for performance testing ATM Forum, Metro Ethernet Forum –Based on ratified standards documents –Each test definition is referenced to standards source text This method is most appropriate for conformance testing

doc.: wng Submission - Study Project Proposal WPP – Tools & Parameters November 2003 Bob Mandeville, Iometrics IETF BMWG Test Standards Templates Terminology Definition Template: –Term to be defined. (e.g., Latency) –Definition: The specific definition for the term. Discussion: A brief discussion about the term, it's application and any restrictions on measurement procedures. –Measurement units: The units used to report measurements of this term, if applicable. –Issues: List of issues or conditions that effect this term. –See Also: List of other terms that are relevant to the discussion of this term. Methodology Definition Template: –Objectives –Setup parameters –Procedures –Measurements –Reporting formats

doc.: wng Submission - Study Project Proposal WPP – Tools & Parameters November 2003 Bob Mandeville, Iometrics Conformance Oriented Test Methods Template 1/2 Test Name Name derived from reference document Test Definition ID A punctuated alphanumeric string assigned to each defined requirement and test procedure couple using the following convention: ‘one to three letter abbreviated source document name’. ‘section number’ - ‘paragraph number in the section from which requirement is derived’. This number always figures as the last number of an ID. Ethernet Services Model=M; Ethernet Services Definitions=S; Traffic and Performance Parameters for SLSs=T. Example: M Reference Document Source Reference document and section (and paragraph when useful for clarity) Test Type Functional, Conformance, Interoperability or Performance Test Status Normative, optional, additional Requirement Description Brief description of the service requirement that the device must or should satisfy Description of DUT/SUT Type of Ethernet frame forwarding Device Under Test (DUT). Common designations used in this document are: CE Device (Customer Equipment Device); UNI Device (User Network Interface Device); MEN Device (Metro Ethernet Network Device). A UNI device may be considered as a kind of MEN ‘border’ device.

doc.: wng Submission - Study Project Proposal WPP – Tools & Parameters November 2003 Bob Mandeville, Iometrics Conformance Oriented Test Methods Template 2/2 Test Object Succinct description of test purpose Test Bed Configuration Succinct description of test bed configuration Test Procedure Succinct description of the test procedure with mention of the test stimulus and expected output Units Units can be time units, rates and counts in integers such as milliseconds, frames per second and numbers of frames transmitted or received. For the most part units used are defined in RFCs 2285, 2544, 2889 Variables Variables such as frame length, test duration, numbers of interfaces under test must be described Results Description of the textual, numerical and/or graphical format in which to display test results Remarks Description of any particular observations that might effect the test result

doc.: wng Submission - Study Project Proposal WPP – Tools & Parameters November 2003 Bob Mandeville, Iometrics Sample List of Terms to be Defined by Category:

doc.: wng Submission - Study Project Proposal WPP – Tools & Parameters November 2003 Bob Mandeville, Iometrics Standardized Approach A sample application - Roaming

doc.: wng Submission - Study Project Proposal WPP – Tools & Parameters November 2003 Bob Mandeville, Iometrics Sample Application to a Key Variable Roaming Performance Roaming Test –A practical example which shows poor performance related to a lack of test standards definitions To roam a a/b/g device will: 1.disassociate from one AP 2.search for a stronger RF signal from another AP 3.then associate and authenticate with that A 4.resume normal data transmission Roaming can fail due to: –transient RF conditions –the time that APs and devices take to complete the four step roaming process

doc.: wng Submission - Study Project Proposal WPP – Tools & Parameters November 2003 Bob Mandeville, Iometrics Test Configuration

doc.: wng Submission - Study Project Proposal WPP – Tools & Parameters November 2003 Bob Mandeville, Iometrics Test Procedure Roaming Test recorded: Total Roaming Time = Decision Time + Switch Over Time –The Decision Time is the time it took the NIC to stop attempting to transmit packets to AP 1 after the attenuation of the RF signal –The Switch Over Time is the time it took the NIC to complete its association with AP2 after it stopped attempting to transmit packets to AP 1 –During the Decision Time cards recognized the signal attenuation and invoked proprietary algorithms to adapt their rates to slower speeds. –Switch Over Time ends when the NIC receives the AP’s acknowledgement to its association request. –This time should only be recorded as valid if data traffic from the NIC to the AP successfully resumes transmission after association.

doc.: wng Submission - Study Project Proposal WPP – Tools & Parameters November 2003 Bob Mandeville, Iometrics Test Results

doc.: wng Submission - Study Project Proposal WPP – Tools & Parameters November 2003 Bob Mandeville, Iometrics

doc.: wng Submission - Study Project Proposal WPP – Tools & Parameters November 2003 Bob Mandeville, Iometrics Test Conclusions Need to break out Decision Time and Switch over Time Switch Over Times are as low as 60 milliseconds and averages a little over 700 milliseconds over all the combinations excepting two outliers which took over 8 seconds. In the majority Decision Time is the largest contributor to the overall Roaming Times. Packet traces show that most implementations of the rate adaptation algorithms maintain transmission at the lowest 1 Mbps rate for several seconds after loss of RF signal has been detected. These algorithms will need to be revisited to deliver quality roaming. Test standards for measuring roaming times can make a significant contribution by aligning all vendors and users on a common set of definitions This applies to roaming but also to a large number of other undefined terms

doc.: wng Submission - Study Project Proposal WPP – Tools & Parameters November 2003 Bob Mandeville, Iometrics Thank you