1. Designing Tests for Smart TVs
Ceren Şahin Gebizli, Ph.D.
Test Architect, ISTQB-CTAL-TA
Vestel Electronics, R&D
13-15 November 2017, Budapest, Hungary

2. Challenges Short time-to-market Limited resources Large code base
5M LOC in total
Large models
Thousands of states and transitions
Importance of User Perception

3. Traditional Testing Checklists based on experiences
Lack of coverage information
Lack of inputs for test automation
Long time for preparing and executing tests.
Some critical faults are missing

4. Model-based Testing Effective test case generation;
Focus on features that are mostly used
Focus on scenarios that are mostly error-prone
Focus on scenarios that reveal different failures

5. System Models used for MBT
start state
transition probabilities
finish state
Here you can see one of our test models, we are using MaTeLo model-based testing tools to design our test models. Our test models actually based on Finite state automatas and Markov chains. What is that mean? We are modelling our tests with states and transitions and also assigning probabilities to each transition to be able to generate test cases based on those probability values. Normally, if there is one transition between 2 states, the probability of that transition is 1. When there are 2 transitions coming out from one state to two different states, the probability is disturbed as 0.5, 0.5.
We designed hundreds of models which include thousands of states and transitions. After starting model-based testing, out test automation rates increased ROIs increased. The quality of our test cases and the bugs we found got better and the coverage became measurable.
After preparing all test models for all features and working on test automation and identifying the automation rates, we started to search for different tests to be able to found specific bugs which are reported from the field , the real end users.
When you buy a product, you are informed that this product’s life time is 10 years. This information is given by analyzing the materials, components, the hw of the product. What if we analyze the sw of the TV and give a lifetime until a major problem occurs?
states that can comprise sub-models
Hierarchical Markov chains defined with the MaTeLo tool (

6. Longevity Tests
Detecting the mostly used modules of TVs by getting data from real users
Designing a usage model
Execution of each generated test case on test automation system
Executing test cases simulates 1 year usage of product in 23 days.
1 test case ~1.5 h
1.5 saatte 1 gün simule ediliyor.
24 saatte 16 gün simule edilir.
Yani 1 günde 16 gün simule edilir.
23 günde 365 gün simule edilmiş olur.
By this thought, we designed a test called Longevity test which simulates 1 year usage of the product in 23 days.
For this, we detected the mostly used modules by the users, we designed a usage module.

7. Usage Profile How we collected the real data to design a usage model.
We gave 30 TVs to 30 field test users for 30 days and they were informed that we were going to collect their usage information.
After 30 days, we collected the log files full of execution traces and analyzed the data to identify the mostly used modules. After detecting the usage of modules, we calculated those usage as probability values for our Longevity test model.

8. Selected Modules with Probabilities
Software Module
# of Visits
Calculated Proability
Portal
1900
0.146
Youtube
2250
0.173
HBBTV
500
0.038
Media Browser (Video)
1750
0.134
Media Browser (Audio)
400
0.03
Media Browser (Picture)
100
0.007
PVR Ready
1000
0.076
Channel List
EPG
2000
0.153
Teletext
1250
0.096
HDMI-SCART

9. Test Model

10. Test Automation
Embedding test scripts to the corresponding transitions.
Hundreds of test cases and test scripts are generated automatically by Model Based Test tool.
Test scripts are executed automatically on our VesTA automation system.

11. Test Automation

12. Test Automation

13. Test Automation

14. Results and Future Work
Traditional methods:
Test cases preparation : 2d
Test scripting of these cases : 3d
Number of test cases: 300
Test execution time : 15h (%100 automatic)
Faults: 4
No idea about simulating the yearly usage
Model-based Testing:
Test design and scripting in parallel: 1d
Number of test cases: 79
Test execution time : 1.5h for each test case (%100 automatic)
Faults: 10 (Additional critical faults)
Detection of new critical faults by simulating the 1 year usage of end-users in 23 days.

15. Conclusions Consumer electronics domain
Context of an industrial case study for MBT of a Smart TV system
User profiles give an idea about most important modules, features and test scenarios.
Generated automatic tests can be run for different software versions as regression test.

16. Thanks! Questions welcome..