Analyze Assure Accelerate TM Spirent Communications Vodafone UK UE Performance Test Readout 15 April 2005

Analyze Assure Accelerate TM The following information is all subject to NDA and is for Spirent internal information ONLY

Analyze Assure Accelerate TM Introduction Vodafone is the world’s largest network operator:  Either owns or has a substantial interest in 26 networks around the world (including a 45% stake in Verizon Wireless)  400 million subscribers on its wholly-owned and associated networks  Current market capitalization: $170 billion  Vodafone UK launched 3G service in November ‘04 - only 200,000 3G subscribers as of end March ‘05 A key ‘Must Win’ account since Q104:  Must win team: Nigel Wright, Paul Collins, Paul Farren -> Tash Tseng  2 ULTS system orders secured at Vodafone KK (Japan)

Analyze Assure Accelerate TM Introduction Initial contact: February ‘04, Düsseldorf, Germany  Confirmed that globally Vodafone tests handsets only with Vodafone live! services - all other handset testing responsibility of manufacturer  Made us aware of ‘one-off’ UE demodulation performance testing initiative at Vodafone UK Initially pursued sale of U-ATS FLEX3 - budget denied Switched to strategy of testing on U-ATS at BABT Between August 04 and February ’05, Vodafone and Spirent negotiated test plan contents Testing carried out March 14 – 25 ‘05 at BABT with Rob Short support

Analyze Assure Accelerate TM Vodafone Test Plan Summary Exerts from VF Test Plan “Executive Summary”: inspired from the RF conformance tests focus on characterising the RF performance  Although the tests are largely inspired from the RF conformance tests specified by the 3GPP Terminal group in TS , the intention of the tests is not to enquire about conformance of UE but to focus on characterising the RF performance of a number of terminals in specified conditions. magnitude of the performance difference  One objective is notably to enquire about the magnitude of the performance difference between various UE implementations. Spirent executed 95% of VF Test Plan by using “standard” U-ATS test cases with customized test parameters

Analyze Assure Accelerate TM Final Vodafone Test Plan Test Cases: # Test Case 33 Demodulation with a single radio link 2 Demodulation with soft handover 1 TPC Command Combining with soft handover 9 Downlink Outer Loop Power Control 3 Adjacent Channel Interference 1 Maximum Output Power 2 Receiver Sensitivity 4 Measurement Accuracy

Analyze Assure Accelerate TM How did VF Test Plan diverge from ? Asymmetric data rate testing (384 DL, 64 UL) to match real network Typical Urban multi-path propagation channels  Defined as 20-path fading model  We simplified to 12-path model Downlink Power control enabled for many tests based on Section 7 (not present in 3GPP spec)  Average downlink DPCH Ec/Ior was a required test result  VF wanted to see which UEs required the lowest average DL DPCH levels to maintain a specified quality level Varied Ior/Ioc (signal/noise ratio) Defined their own pass/fail limits for tests

Analyze Assure Accelerate TM Additional VF-Defined Performance Tests Reference Sensitivity Sweep  When UE is far from base station and Ior is low, which UE will perform best?  Algorithm: Sweep Ior level and record BER Loss of Synchronization  Custom Vodafone test that WE DID NOT SUPPORT  Sweep Ior/Ioc in 1dB steps every 2 seconds until UE loses sync DL Power Control Convergence in DTX  Custom test that WE DID NOT SUPPORT

Analyze Assure Accelerate TM Compromises Made with VF Radio Bearer Configurations  VF specified “standard” configurations (voice, video, packet data)  U-ATS only supports loopback (test mode) configurations Downlink Power control not supported in all tests  U-ATS only supports in 7.8.x test cases  This was good enough for majority of testing, but we could not meet every VF test requirement

Analyze Assure Accelerate TM Review of Results – Summary Purchased 3 commercial VF UEs to allow us to mirror testing in NJ (already had Nokia) Final results file is available on the network: \\Njeng\Common\Vodafone U-ATS Testing March 2005\Vodafone All UE's - Final.xls \\Njeng\Common\Vodafone U-ATS Testing March 2005\Vodafone All UE's - Final.xls  Outcome largely a success  More “undetermined” test results than we had hoped to see  Results did serve to highlight some performance differences between UE’s

Analyze Assure Accelerate TM Results – Performance Differentiators Demodulation of DCH (7.3.1) at 384 DL/64 UL  Nokia 6630 clear winner – other UEs could not keep call up Downlink Power Control tests  Nokia 6630 and Motorola V980 outperformed the rest – other couldn’t converge on the target  Motorola V980 averaged 3dB lower DL DPCH Demodulation of DCH in Soft Handover (7.7.1)  Nokia 6630 and Sony Ericsson V800 passed while all others failed

Analyze Assure Accelerate TM Results – Performance Differentiators (cont.) Measurement Accuracy Tests (8.7.x.x.x)  Motorola V980 and Samsung Z107 failed (drop call)  Nokia 6630 and Sharp 902SH passed Reference Sensitivity Sweep UEIor (dBm) level at which BER > 1% Samsung Z Sharp 902SH Nokia Motorola V Sony Ericsson V

Analyze Assure Accelerate TM Results – Areas of concern Downlink Power Control tests  Many “undetermined” results with data rate > 12.2 kbps  There is a known issue in 3410 L1 that should improve this in next release Combining of TPC Commands (7.7.2) test  In the testing at BABT, no UEs were able to go into Soft Handover successfully, so all were “undetermined” – not seen in NJ Measurement Accuracy tests  Some UE’s dropping call in middle of test

Analyze Assure Accelerate TM Results – Areas of concern (cont.) Call Setup reliability  Several tests had to be run multiple times before a call could be successfully setup. 384 kbps DL, 64 kbps UL Data Rate  All UE’s – except Nokia – were unable to set up call at this data rate Other comments: \\Njeng\Common\Vodafone U-ATS Testing March 2005\Vodafone All UE's - Final.xls\\Njeng\Common\Vodafone U-ATS Testing March 2005\Vodafone All UE's - Final.xls

Analyze Assure Accelerate TM Usability Lessons of U-ATS Changing of test parameter levels is not as robust as it should have been  Lots of test time was lost because of “out-of- range” levels  U-ATS needs to do a better job range checking prior to suite execution Frequent need to reset UE’s between tests  Can be automated with X-API - not an easy thing to set up as all UEs are quite different Lack of Registration and Call Setup reliability drastically increased test time

Analyze Assure Accelerate TM Usability Lessons of U-ATS (cont.) Results Analysis is extremely cumbersome  In order to compile “proper” results comparison spreadsheet for Vodafone, we spent approximately 30 hours (including analysis)  U-ATS currently generates reports individually for each suite execution Huge amount of results data from multiple UE testing There is no way to compare results (other than manually) We have recently developed a tool to extract and compare results from database This tool could be improved and commercialized  Inadequate reporting of reasons for test failures and ‘undetermined’s in Event Log