University of Kansas Dr. Joseph B. Evans Dr. Daniel Deavours Leon S. Searl Information & Telecommunication Technology Center Bluetooth Interoperability Testing at ITTC

University of Kansas Overview What is the Bluetooth SIG Why Test Interoperability Test Philosophy Types of Bluetooth Applications Tested Types of Devices Tested Bluetooth Interoperability Tests Test Result Types General Results So Far Future Work

University of Kansas Bluetooth Special Interest Group The Bluetooth SIG is a trade association comprised of telecommunication, computing, and network industry companies that is driving the development of a low-cost short-range wireless specification for connection mobile products Protocols – RF, Link Layer, etc. Profiles (application APIs) – Serial Port, Object Push, Print, Audio, etc. Promoters - 3COM, Agere, Ericsson, IBM, Intel, Microsoft, Motorola, Nokia, Toshiba, etc. HQ in Overland Park, KS Established in February, 1998 First Specification release in July, 1999

University of Kansas Why Do Interoperability Testing? Bluetooth SIG would like to obtain snapshot of current state of interoperability of commercially shipped Bluetooth products Assess the feasibility of having an Interoperability Laboratory Determine if additional Bluetooth specifications are required

University of Kansas Interoperability Test Philosophy Test interoperability from user point of view Did it work or fail? How easy was it to use User level debugging of failures See if parameters are correctly set Were user manual instructions followed? User cant tell which which device is responsible for Bluetooth related failure Save all test data to SQL database for use by SIG

University of Kansas Bluetooth Device Categories Handheld (HH) Mobile Phone (MP) Personal Computer/USB/PCMCIA (PC) Printer (PR) Headset (HS) Hands-free (HF) Human Interface Device (HID)

University of Kansas Interoperability Testing Determine what Bluetooth operations are successful between pairs of Bluetooth capable devices Business Card Transfer Meeting Request Transfer Memo Transfer File Transfer Send Fax Dialup Networking Data Synchronization Print Object Human Interface Device Input Headset Audio

University of Kansas Bluetooth Application Test Types HH-HHBizcard, Meeting Request, Memo HH-PRBizcard, Meeting Request MP-MPBizcard, Meeting Request MP-PRBizcard, Meeting Request MP-HSAudio, Voice Dial MP-HFAudio, Voice Dial PC-PCBizcard, Meeting Request, File Exchange PC-PRBizcard, Meeting Request, File PC-HHBizcard, Meeting Request, File Exchange, Sync PC-MPBizcard, Meeting Request, Dial-up Networking, Fax HH-MPBizcard, Meeting Request, Dial-up Networking

University of Kansas Bluetooth Application Test Types All applications are tested using Public connections An application from each device pair type test suite is chosen to test the following Bluetooth features for each pair of devices Bonding Secure Data

University of Kansas Bluetooth Test Organization Device Pair Suite There is a Test Suite for each Device Type Pairing (Example HH-PR) A Test Suite is made up of Step Groups Step Groups A Step Group exercises a Bluetooth Application (Bizcard, etc.) between two Bluetooth devices or initializes the devices A Step Group is composed of individual Steps Step Instruction to a Test Operator to perform an application operation on one or both devices –Non-Bluetooth Setup: 1 device, no Step Result Data –Bluetooth Setup: 1 device, may have Step Result Data –Bluetooth Interoperation: 2 device, Step Result Data for Both Devices

University of Kansas Example Test Organization HH-MP Test Suite Setup for other Step Groups … Business Card Transfer Step Group –Create Bizcard on HH –Send Bizcard from HH to MP –Alter name on Bizcard on MP –Send Bizcard from MP to HH –Remove all Bizcards from MP –Remove second Bizcard from HH Meeting Request Step Group … Bonded, Non-secure Step Group … Public, Secure Step Group … Dial-up Networking Step Group …

University of Kansas Bluetooth Test Step Group Data Device 1 ID Device 2 ID Test Operator Name Start Date and Time Unique Execution ID Support of Device 1 for application tested by the Step Group Support of Device 2 for application tested by the Step Group

University of Kansas Bluetooth Test Step Measurements Qualitative Subjective ease of use (0-5 scale) Ease of Use Comments for subjective scale < 3 Step comment Quantitative Start date/time, Stop date/time Step duration (seconds) Action count (integer) Connection failures/retries (integer) Success/Failure (boolean) Group execution ID Unique step ID

University of Kansas Measurement Summaries Summary data includes the following Number of device pairs tested Number of measurement steps Number of measurement step failures Percent of step failures Average subjective scale Average connection failure loss Summaries are produced for the following groupings Total database Suites Step groups Individual devices

University of Kansas General Results 20 devices tested HF = 1 HS = 4 PR = 2 MP = 2 HH = 2 PC = 7 of device pairings tested (out of 75 possible) 371 steps with measurements most of the steps succeeded 4.2 subjective usability (out of 5) of the steps that succeeded

University of Kansas General Results Terms used by various manufactures are inconsistent with each other Examples: Business Card, Contact, Address Inter-operability failure rates correspond to complexity of the user interface HS & HF = 14.3% failure rate –1 or 2 button interface PC = 23.9 failure rate –Graphical user interface Software, PC user interfaces are still in infancy

University of Kansas Future Work Refined interoperability measurements Create well designed test application for test operator input (instead of current rapid prototype) More specific interoperability data captured –Breakup subjective scale –Intuitive interface –User manual –Audio quality –Add new measures –Robustness (PC crashes) Web-based test operator application Multiple test operators to reduce measurement bias Add standard deviation to test summaries in addition to mean