ConfidentialPA Testing Mobile Applications A Model for Mobile Testing

ConfidentialPA Introduction ▪This presentation outlines a model for mobile applications testing ▪Focus on what could be included in a mobile application test scope ▪The model is context independent, and should therefore be adapted and applied to specific contexts ▪Four different dimensions

ConfidentialPA Mobile Application Test Scope ▪The model outlines what could be part of a mobile application test scope ▪Which tests are actually selected/created/executed are dependant on different priorities and risks ▪Business priorities/risks? ▪Customer priorities/risks? ▪User priorities/risks? ▪Technical priorities/risks?

ConfidentialPA Model Overview Application Coverage Items Configurations Software Quality Attributes Environments

ConfidentialPA Application Coverage Items ▪Divide the application into different functional areas ▪Each functional area is called a Coverage Item ▪The size of the coverage item is dependant on the granularity needed for planning and reporting

ConfidentialPA Example: Alarm & Clock Application World Time Alarm Clock Stop Watch Timer Alarm & Clock Application

ConfidentialPA Software Quality Attributes ▪ISO/IEC ▪Functional Suitability ▪Performance Efficiency ▪Compatibility ▪Usability ▪Reliability ▪Security ▪Maintainability ▪Portability

ConfidentialPA Example: Functional Suitability ▪Functional Completeness ▪Are the application features there? ▪Functional Correctness ▪Are the application features working? ▪Installability ▪Is the install/uninstall process for the application working? ▪Back-end functionality

ConfidentialPA Example: Performance Efficiency ▪Application start up time ▪UI transitions ▪Memory usage ▪Loading times ▪Save time ▪Back-end capabilities ▪Performance and capacity testing ▪Stress tests

ConfidentialPA Example: Reliability ▪Availability ▪Service up-time ▪Recoverability ▪What happens when the application crashes? ▪How does the application handle data during crash? ▪Service back-end recovery

ConfidentialPA Example: Reliability ▪Random stability tests ▪Monkey testing for Android applications ▪Random walkthrough of UI until application crash or time limit is reached ▪Aging & Duration ▪What happens when the application is used over a long period of time? ▪What happens when the application is used for a long time? ▪What happens to the back-end over time?

ConfidentialPA Example: Usability ▪Different user personas ▪Test the application through different user personas to cover the most important use cases ▪Learnability / Understandability ▪It is easy to learn how to use the application? ▪Aesthetics ▪Usable for people with disabilities? ▪User Error Protection ▪Is the application mistake-proof?

ConfidentialPA Example: Compatibility ▪Interoperability testing ▪Support for different accessories ▪Headsets, watches, wristbands, external storage, external displays ▪Support for communication with third-party devices ▪TV, Playstation, Computer ▪Co-existence ▪Sharing resources with other applications ▪Interruptions from other applications or systems ▪Sharing resources with other users ▪Back-end access ▪Multiplayer usage with different other mobile devices

ConfidentialPA Example: Maintainability ▪Application upgrade ▪What happens when you upgrade to a new version of the application, especially with user data ▪Testability ▪Has the application been created to enable efficient testing? ▪Replacing the back-end ▪Backwards Compatibility

ConfidentialPA Example: Security ▪Fuzz testing ▪Fuzz testing or fuzzing is a software testing technique, often automated or semi-automated, that involves providing invalid, unexpected, or random data to the inputs of a computer program. The program is then monitored for exceptions such as crashes, or failing built-in code assertions or for finding potential memory leaks. Fuzzing is commonly used to test for security problems in software or computer systems. [2] ▪There are of course many ways of improving application security [3]

ConfidentialPA Example: Portability ▪Portability testing refers to the process of testing the ease with which a computer software component can be moved from one environment to another, e.g. moving between two different Android cookies, or between Android and Firefox OS. This is typically measured in terms of the maximum amount of effort permitted. [4] ▪Activity performed by Software Developer (in Test)?

ConfidentialPA Configurations ▪Different Operating Systems ▪Different Chipsets & HW ▪Tablets vs. Smart Phones ▪Free vs. Premium ▪Targeted customers

ConfidentialPA Different Operating Systems ▪Is the application working for all intended operating systems? ▪Android ▪Firefox OS ▪Windows ▪IOS ▪Different versions of operating systems ▪New Android cookies ▪Old versions? ▪Different software releases ▪Different OS configurations and customizations

ConfidentialPA Different Chipsets & HW ▪Mobile Devices have different chipsets with different capabilities ▪Qualcom ▪Nvidia ▪Samsung ▪Texas Instruments ▪Devices have different HW support for different features ▪Devices could have different capabilities, even though they have the same chipset, such as RAM, internal memory, etc.

ConfidentialPA Tablet vs. Smartphone ▪Default screen orientation differs between smartphones and tablets ▪There are many different screen sizes for the applications to support ▪Some tablets only support WiFi ▪Speakers / sound may be different

ConfidentialPA Free vs. Premium ▪If your application has both a free and a premium verison, you need to consider what testing can be re-used, and what must be re-tested for both versions ▪Depends on the implementation of the different versions ▪With a good implementation, the tests on the free version should be minimal

ConfidentialPA Targeted Customers ▪If an application has several customized versions which are going to different customers, it is necessary to consider what testing can be re-used between version, and what is customer specific

ConfidentialPA Environments ▪Mobile Networks ▪WiFi and Bluetooth ▪Content ▪Servers & Services

ConfidentialPA Mobile Networks ▪Different mobile networks and operators have different configurations and different capabilities ▪Does the application have access to the correct servers? ▪Is the back-end working in a live environment ▪Can the application handle loss of connection or low bandwidth?

ConfidentialPA WiFi / BT ▪What happens if there is a loss of WiFi network, or BT connection during application usage? ▪What happens if reception is bad or fluctuating?

ConfidentialPA Content ▪If the application handles different content it is relevant to try out live customer content to secure compatibility ▪Different file types ▪Some content could have DRM protection, and also different DRM solutions ▪NFC tags can have different data types and different amounts of data

ConfidentialPA Servers & Services ▪How does the application work with different content and streaming servers? ▪Different solutions ▪Other services

ConfidentialPA Testing the Complete Ecosystem

ConfidentialPA Creating Tests based on the Model ▪Each test case / test session / test mission / test should be marked as covering: ▪One or multiple Coverage Items ▪One or multiple Software Quality Attributes ▪Low level tests usually only cover one coverage item and one quality attribute, while high level system tests usually cover multiple ▪Which tests are created should be based on risk and priorities

ConfidentialPA Test Runs / Executions / Activity ▪Each test run / execution / activity should be marked as covering: ▪One configuration ▪One environment ▪You can run the same tests on multiple configurations and environments ▪Which tests are executed should be based on risks and priorities

ConfidentialPA Test Reporting ▪One (sub-)report for each configuration and environment ▪Coverage items and software quality attributes can be used in the reports to facilitate analysis and understanding ▪Group results based on different coverage items and/or quality attributes ▪Coverage Item / Software Quality Attribute Matrix

ConfidentialPA Test Reporting Example Configuration XEnvironment Y FunctionalPerformanceCompatibilityUsabilityReliabilitySecurityMaintainabilityPortability Alarm Clock World Time Stop Watch Timer Test OKTest Not OKNot Sufficient CoverageNo Test Needed

ConfidentialPA Conclusion ▪The model described in this presentation can be applied to most contexts ▪When using the scope created with this model, there are many testing practices that must be applied to make it effective ▪Risk-based testing ▪10-minute test plans ▪Session-based testing ▪Etc. ▪The model alone will not solve all your mobile application testing needs, but it can be a good starting point for setting a relevant test scope

ConfidentialPA References [1] ISO/IEC [2] Fuzz Testing [3] Application Security [4] Portability testing