Designing Tests for Smart TVs Ceren Şahin Gebizli, Ph.D. Test Architect, ISTQB-CTAL-TA Vestel Electronics, R&D 13-15 November 2017, Budapest, Hungary

Challenges Short time-to-market Limited resources Large code base 5M LOC in total Large models Thousands of states and transitions Importance of User Perception

Traditional Testing Checklists based on experiences Lack of coverage information Lack of inputs for test automation Long time for preparing and executing tests. Some critical faults are missing

Model-based Testing Effective test case generation; Focus on features that are mostly used Focus on scenarios that are mostly error-prone Focus on scenarios that reveal different failures

System Models used for MBT start state transition probabilities finish state Here you can see one of our test models, we are using MaTeLo model-based testing tools to design our test models. Our test models actually based on Finite state automatas and Markov chains. What is that mean? We are modelling our tests with states and transitions and also assigning probabilities to each transition to be able to generate test cases based on those probability values. Normally, if there is one transition between 2 states, the probability of that transition is 1. When there are 2 transitions coming out from one state to two different states, the probability is disturbed as 0.5, 0.5. We designed hundreds of models which include thousands of states and transitions. After starting model-based testing, out test automation rates increased ROIs increased. The quality of our test cases and the bugs we found got better and the coverage became measurable. After preparing all test models for all features and working on test automation and identifying the automation rates, we started to search for different tests to be able to found specific bugs which are reported from the field , the real end users. When you buy a product, you are informed that this product’s life time is 10 years. This information is given by analyzing the materials, components, the hw of the product. What if we analyze the sw of the TV and give a lifetime until a major problem occurs? states that can comprise sub-models Hierarchical Markov chains defined with the MaTeLo tool (http://www.all4tec.net)

Longevity Tests Detecting the mostly used modules of TVs by getting data from real users Designing a usage model Execution of each generated test case on test automation system Executing test cases simulates 1 year usage of product in 23 days. 1 test case ~1.5 h 1.5 saatte 1 gün simule ediliyor. 24 saatte 16 gün simule edilir. Yani 1 günde 16 gün simule edilir. 23 günde 365 gün simule edilmiş olur. By this thought, we designed a test called Longevity test which simulates 1 year usage of the product in 23 days. For this, we detected the mostly used modules by the users, we designed a usage module.

Usage Profile How we collected the real data to design a usage model. We gave 30 TVs to 30 field test users for 30 days and they were informed that we were going to collect their usage information. After 30 days, we collected the log files full of execution traces and analyzed the data to identify the mostly used modules. After detecting the usage of modules, we calculated those usage as probability values for our Longevity test model.

Selected Modules with Probabilities Software Module # of Visits Calculated Proability Portal 1900 0.146 Youtube 2250 0.173 HBBTV 500 0.038 Media Browser (Video) 1750 0.134 Media Browser (Audio) 400 0.03 Media Browser (Picture) 100 0.007 PVR Ready 1000 0.076 Channel List EPG 2000 0.153 Teletext 1250 0.096 HDMI-SCART

Test Model

Test Automation Embedding test scripts to the corresponding transitions. Hundreds of test cases and test scripts are generated automatically by Model Based Test tool. Test scripts are executed automatically on our VesTA automation system.

Test Automation

Test Automation

Test Automation

Results and Future Work Traditional methods: Test cases preparation : 2d Test scripting of these cases : 3d Number of test cases: 300 Test execution time : 15h (%100 automatic) Faults: 4 No idea about simulating the yearly usage Model-based Testing: Test design and scripting in parallel: 1d Number of test cases: 79 Test execution time : 1.5h for each test case (%100 automatic) Faults: 10 (Additional critical faults) Detection of new critical faults by simulating the 1 year usage of end-users in 23 days.

Conclusions Consumer electronics domain Context of an industrial case study for MBT of a Smart TV system User profiles give an idea about most important modules, features and test scenarios. Generated automatic tests can be run for different software versions as regression test.

Thanks! Questions welcome..