System Testing Challenge Meeting the IoT System Testing Challenge

Agenda Why do we test? What is system testing? What are requirements and test specifications? What is Software System Testing? What does IoT mean? What is SeqZap Customer stories SKOV Grundfos Conclusions

Rasmus Toftdahl Olesen Product Manager for SeqZap Has helped customers adopt SeqZap for the last 10 years Master of Computer Science from University of Aalborg Loves testing! Knows enough about electronics to avoid burning his fingers on the soldering iron

SeqZap is distruted in Scandinavia by Nohau. Distributor SeqZap is distruted in Scandinavia by Nohau.

Why do we test?

Test Saves Money Cost to Fix Requirements Design Verification Production Release

New Product Average Selling Price Complexity and Cost Cost of Test Time

Why do we test? Testing is knowing Known quality

What is System Testing?

Software System Testing Requirements System testing? Design Unit Test System Test <-> Kravspecifikation Funktionel test, men også ikke-funktionelle test som performance test, sikkerhedstest og andre. Jeg vil lige præcisere hvad vi mener når vi siger at vi arbejder med system test. Når vi siger "system test" mener vi det som i den traditionelle V-model for software-udvikling bliver kaldt "system test". System-testen er her beskrevet som den test der viser om kravspecifikationen bliver overholdt. Det er hovedsagelig en funktionel test, som viser om softwaren funktionelt opfører sig som forventet. Alt efter typen af produktet vil der i systemtesten også kunne være forskellige ikke-funktionelle test, som: Performance test Sikkerhedstest Hardwaretest Mekaniktest

System Testing ”Classic” Requirements Test Specification Perform

What are requirements?

Requirements 300 page document with references to 100 standard reference documents Bullet points from marketing material ”Just like the last version, but with this feature added” Requirements database ”Go ask that guy”

Requirement example Requirement B-67 Target should respond on protocol XYZ within 30 seconds of being powered on.

Test Specification example Test Case Name: Start responding within 30 seconds Validates: Requirement B-67 Definition: Turn on target Wait 30 seconds Expect that the target responds on protocol XYZ using the STATUS? request message

SeqZap Implementation example

SeqZap Implementation example

Collapsed implementation

Specification and Implementation

Print (read: PDF)

System Testing ”SeqZap style” Requirements Executable Test Specification

System Testing ”Agile” Implement Feature A Write test for Feature A Test Test All Features Fix failing tests

System Testing ”Agile” Implement Feature A Write test for Feature A Test Test All Features Fix failing tests

What is Software System Testing?

Software System Testing Functionality testing Abstraction! Does not require coding skills Requires domain knowledge

Domain language (ventilation) Airflow (m3/h) Pressure (atmos) Automatic windows (0-50% opened) Temperature (celcius)

Engineering language Airflow (0-5V) Pressure (0-3.3V) Actuator (60W) Temperature (PT100)

In SeqZap

We want as many people as possible to be able to read and write tests. Developers Testers Service Technicians Project managers? Secretaries? Sales people? Write the test in domain ”terms” Domain terms are the only thing that all people in an organisation share.

What is IoT?

What is IoT? Cloud Internet of Things Big Data Industry 4.0 Putting data from embedded controllers into ”databases”

What is IoT? Primary feature Secondary features Putting data from embedded controllers into ”databases” Secondary features Online service Firmware update Usage statistics

How? Complexity Internet protocols Larger operating systems IP TCP/UDP HTTP Larger operating systems Linux, WindowsCE, Android User Interfaces Touch screen Web Complexity

R&D Challenges Increasing complexity Internet of Things (IoT), Big Data, Industry 4.0 Shorter release-cycle Manual testing is hard/impossible

What is SeqZap?

System or Device Under Test Test system System or Device Under Test Test Interface Instrument Test Interface Tool Tool Script Report Script Script Test Sequencer Script Script Log

Key features Write the test in domain language Testers, Technicians, Programmers, Engineers, Sales, Management, etc. Support more than one interface Analog IO, Digital IO, Serial, HTTP, ModBus, etc. Adding support for a new interface is simple Integrated Test Environment Live connection to the DUT

Customer Stories SKOV

SKOV SKOV is providing climate solutions and farm management solutions for animal production worldwide, which ensure optimum conditions for animals, humans and environment.

SKOV Mobile app’s (Android, iPhone, WinPhone Cloud solutions for Enterprise Management PC based Management systems Embedded controllers Embedded controllers

SKOV Embedded Controller ARM/Linux target QT Touch interface Simple IO control

SKOV Embedded Controller Flere forskellige typer controllere – klima, produktion eller kombinerede klima og produktionsstyringer til fjerkræ eller grise. Alarmanlæg.

SKOV Embedded Controller SKOV TCP Completely internal protocol

SKOV Embedded Controller SKOV TCP Analog Digital IO Measurement Computing boxes

SKOV Embedded Controller SKOV TCP BFN Analog Digital IO BigFarmNetwork Integration protocol between SKOV products.

SKOV Embedded Controller SKOV TCP BFN SSH Analog Digital IO Terminal Access (it’s a linux target)

SKOV Embedded Controller SKOV TCP BFN SSH UDP Analog Digital IO Device discovery

SKOV Embedded Controller SKOV TCP BFN SSH UDP Remote UI Analog Digital IO Inspect widget tree Capture framebuffer Simulate touch

FarmOnline Explorer (FOX) Management Interface Windows Application Collects data from embedded controllers

FarmOnline Explorer (FOX) Management system FarmOnline. App til både Android og Apple.

FarmOnline Explorer (FOX) Windows Accessibility Standard accessibility layer in windows ”Hjælp til handicappede”

FarmOnline Explorer (FOX) Windows Accessibility BFN BigFarmNetwork Integration protocol between SKOV products.

Integration Testing BFN BFN Windows Accessibility Remote UI Analog Digital IO

Using only external interfaces Integration Testing Windows Accessibility BFN Remote UI BFN Analog Digital IO Using only external interfaces

System Integration Testing (SKOV) JSON Windows UI Remote UI Analog/Digital IO

Interfaces (SKOV) Controller (Chicken and Pig farms) Analog/Digital IO Remote UI Proprietary Protocols SSH (Remote Command Line) UDP Device Discovery GSM (SMS/Call) PC based Management System Windows UI Proprietary Protocol Web Service interface (JSON) JSON Web/Cloud access Web Browser Mobile access Mobile UI

SKOV Organisation & Methods 25 Software developers at SKOV in Glyngøre 7 Software developers in Malaysia Organized in 6 Scrum Teams (3 – 8 persons) Prioritized backlogs Daily Scrum meetings / 2 weeks sprints Continous integration Automatic test is developed inside each Scrum Team (no dedicated test department)

Test Machines

Test Targets

Test Process Automatic test execution between 18:00 – 06:00 A fresh result every morning at SCRUM Did we break anything yesterday? Continous Integration (Cruise Control) Build Unit Test System Test Nightly test on real hardware Developer debugging using PC hardware

Continous Integration Each commit triggers a rebuild Nightly release builds (6 virtual build servers) Nightly test of all release builds in SKOV’s Software Test Center (~2.000 individual test cases) Functionality UI testing Stability Automatic test reports available

Continous Integration TestCenter Automatic test execution between 18:00 – 06:00 Total test time: 12 hours Total test cycles per year: 365 - - - - - - - - - - - - - - - - If we only had one target : Total test time: ~ 1.300 hours Equal to: (24 hours/day) ~ 2 months Total test cycles per year: ~ 6

Test Steps and procedure re-use Her ses et eksempel på en Test Case, hvor kan kalder en del procedurer fra Shared script filer. Senere i testen kaldes procedurer fra ”Procedures.szs” filen i test projektets mappe, som altid har namespacet ”LocalProcedures” Bemærk at der benyttes Test Steps og at de stemmer overens med steps i Test Specificationen.

Test data is configured in a DataTable GSM Tool benytter også parametre fra datatabel afhængigt af hvilket test site, der er valgt. Bemærk at Test Site og HostName er sat som local values

Notes / Conclusions Introduced errors in existing features are detected immediately during regression tests Automatic test / continuous Integration enables SKOV to release software products with known and reproducible quality level! At SKOV we aim to have automatic test on all software product – but we are not there yet. (Mobile solutions are still manually tested) Automatic test is time consuming, but worth the effort ! - Ebbe Kronborg, R&D Department Manager, Software

Customer Stories Grundfos

Grundfos Grundfos is a global leader in advanced pump solutions and a trendsetter in water technology. We contribute to global sustainability by pioneering technologies that improve quality of life for people and care for the planet. Worldwide organisation (17-18.000 employees) SeqZap used globally as the go-to tool for embedded software test

System Test GENI

System Test GENI PLC IO via OPC

System Test GENI PLC IO via OPC PSU Via SCPI

System Integration Testing PLC GENI Pump GSM Comm. Module Web Cloud

System Integration Testing PLC GENI Pump GSM Comm. Module Web Cloud

How does this relate to us? Conclusions How does this relate to us?

Conclusions (Development) Automate as much as possible Build Unit test System test Integration test Use the tools from the software industry Continous Integration (Jenkins, etc.) Unit testing frameworks Test results should be available as fast as possible

Conclusions (System Testing) GUI testing is actually possible on an embedded target Easy to understand Easy to communicate to the rest of the organisation Wow effect Move real-time stuff to a non-PC system Use PC to drive the real-time system Most system tests are not real-time

Conclusions (Adoption) Start testing! One test is always better than zero test Start testing one system Then expand! No big-bang solutions How do you eat an elephant? Bite by bite

Questions?

Thank you for your time!