Measuring Test Execution Complexity Eduardo Aranha and Paulo Borba Informatics Center Federal University of Pernambuco Recife, PE, Brazil {ehsa, Mobile Devices R&D Motorola Industrial Ltda Jaguariuna, SP – Brazil PROMISE 2006

Mobile Devices Software Group Page 1 Motorola Proprietary, MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office. All other product or service names are the property of their respective owners. © Motorola, Inc Software Testing Companies that release products with poor quality may quickly lose its clients A usual activity performed to ensure quality is software testing Test team Developers teams Developer team

Mobile Devices Software Group Page 2 Motorola Proprietary, MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office. All other product or service names are the property of their respective owners. © Motorola, Inc Research Context Software Requirements Test Cases Requirements and Code Coverage Test Execution Effort Estimates Test Generation Effort Estimation Coverage Analysis Test Case Selection Test Case Selection Test Team and Resource Planning Planning Quality Model This Research Others Motorola Brazil Test Center Researches Some Others Motorola Researches Motorola Brazil Test Center Research Project

Mobile Devices Software Group Page 3 Motorola Proprietary, MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office. All other product or service names are the property of their respective owners. © Motorola, Inc Test Specification Language Controlled natural language (CNL) –Subset of English with restricted grammar and vocabulary –Domain specific language (e.g., mobile phone application) –Sentences written in a more concise and standard way –Make easer to understand and to have text processing Example of test procedure: StepDescriptionExpected Results 1Start the message center.The phone is in message center. 2Select the new message option.The phone is in message composer. 3 Insert a recipient address into the recipients field. The recipients field is filled. 4 Insert a SMS content into the message body. The message body is populated. 5Send the message. The send message transient is displayed. The message is sent.

Mobile Devices Software Group Page 4 Motorola Proprietary, MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office. All other product or service names are the property of their respective owners. © Motorola, Inc Measuring Test Execution Complexity of a Test Case For each single test action (step) –Evaluate execution complexity levels –Rate execution complexity levels –Add points to the whole test execution complexity Example: StepDescriptionExpected Results 1Start the message center.The phone is in message center. 2Select the new message option.The phone is in message composer. 3 Insert a recipient address into the recipients field. The recipients field is filled. 4 Insert a SMS content into the message body. The message body is populated. 5Send the message. The send message transient is displayed. The message is sent. 100 Points Complexity LevelCharactistic 78Total LowNFC2 0 N/ANFC HighFC2 30 LowFC1 2 Total: 630 execution points 31 Points Complexity LevelCharactistic Total... LowNFC2 N/ANFC1... HighFC2 LowFC1 Points Complexity LevelCharactistic Total... NFC2 NFC1... FC2 FC1

Mobile Devices Software Group Page 5 Motorola Proprietary, MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office. All other product or service names are the property of their respective owners. © Motorola, Inc Step 1 – Evaluating execution complexity levels Knowledge base for the execution complexity levels of the test action –Verb and optionally its arguments mapped into complexity levels If it is a new action: –Evaluate it according to the exercised system functional and non functional characteristics –Record evaluation in the knowledge base FC2Average size of date inputs. Less than 3030 to 100More than 100 LowMediumHigh NFC2Use of Bluetooth. File transfer, print messages, use headset Application data synchronization 3050

Mobile Devices Software Group Page 6 Motorola Proprietary, MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office. All other product or service names are the property of their respective owners. © Motorola, Inc Step 2 and 3 Step 2: Rating execution complexity levels –Search the knowledge base for the values assigned for each execution complexity levels of each characteristic Step 3: Assigning the value for the test case –The execution complexity of a test case is calculated by summing the execution points of each of its test steps FC2Average size of date inputs. LowMediumHigh

Mobile Devices Software Group Page 7 Motorola Proprietary, MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office. All other product or service names are the property of their respective owners. © Motorola, Inc Using Test Execution Complexity for Effort Estimation Test productivity –The proportion between execution points and time spent for executing each test –For example, a test with 600 execution points executed in 5 minutes gives a productivity of 120 execution points / minute –A new test case with 160 execution points can be estimated to be executed in approximately 1 minute and 20 seconds Risk factors –Environment conditions, team experience, the use of tools, the reuse of test setups and other factors must be considered –Effort multipliers

Mobile Devices Software Group Page 8 Motorola Proprietary, MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office. All other product or service names are the property of their respective owners. © Motorola, Inc Method Validation Initial experiment –We observed the relation between execution times and the measured complexity of test cases –Qualitative analyses based on historical data with different environment conditions After submitting final paper version, we executed and analyzed 33 tests in a controlled environment –We extended the characteristics using a Delphi panel –Improvements: TestMMREPRED(20)PRED(30) %100.00%37.50% %33.33%25.00% %50.00%0.00%

Mobile Devices Software Group Page 9 Motorola Proprietary, MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office. All other product or service names are the property of their respective owners. © Motorola, Inc Conclusions New method for measuring test execution complexity Applicable for new or existing tests CNL improves specifications and supports automation –Knowledge base reduces estimation effort Some drawbacks –Effort to create the knowledge base –CNL, functional and non functional characteristics are dependent of application domain More experiments to run –ANOVA, covariance –Estimation accuracy, regression analysis, Bayesian calibration, etc.

Mobile Devices Software Group Page 10 Motorola Proprietary, MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office. All other product or service names are the property of their respective owners. © Motorola, Inc Questions?

Mobile Devices Software Group Page 11 Motorola Proprietary, MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office. All other product or service names are the property of their respective owners. © Motorola, Inc Functional Characteristics IDDescription Complexity Level / Guidelines / Rates LowAverageHigh 1 Average number of navigations between screens. Up to 5More than Average size of date inputs. Less than 3030 to 100More than Software configuration. or IM account configuration 40 4File manipulation. save files Move files from different memories types or transfer files though network List manipulation. search list entries, delete entriesadd new entries Multimedia manipulation. Standard sound/picture/animation manipulation camera, emoticon specific sound/picture/animation, edit picture Server access type. StandardAuthenticated Type of screen items to be verified. Valid charactersInvalid charactersNumber of pixels

Mobile Devices Software Group Page 12 Motorola Proprietary, MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office. All other product or service names are the property of their respective owners. © Motorola, Inc Non Functional Characteristics IDDescription Complexity Level / Guidelines / Rates LowAverageHigh 1Application delay. Wait a transient message 10 2Use of Bluetooth. File transfer, print messages, use headset Application data synchronization Use of network. Sending IM messages or short s or short messages Sending large messages or large e- mails