1. System Testing Challenge
Meeting the IoT
System Testing Challenge

2. Agenda Why do we test? What is system testing?
What are requirements and test specifications?
What is Software System Testing?
What does IoT mean?
What is SeqZap
Customer stories
SKOV
Grundfos
Conclusions

3. Rasmus Toftdahl Olesen
Product Manager for SeqZap
Has helped customers adopt SeqZap for the last 10 years
Master of Computer Science from University of Aalborg
Loves testing!
Knows enough about electronics to avoid burning his fingers on the soldering iron

4. SeqZap is distruted in Scandinavia by Nohau.
Distributor
SeqZap is distruted in Scandinavia by Nohau.

5. Why do we test?

6. Test Saves Money Cost to Fix Requirements Design Verification
Production
Release

7. New Product
Average Selling Price
Complexity and Cost
Cost of Test
Time

8. Why do we test?
Testing is knowing
Known quality

9. What is System Testing?

10. Software System Testing
Requirements
System testing?
Design
Unit Test
System Test <-> Kravspecifikation
Funktionel test, men også ikke-funktionelle test som performance test, sikkerhedstest og andre.
Jeg vil lige præcisere hvad vi mener når vi siger at vi arbejder med system test.
Når vi siger "system test" mener vi det som i den traditionelle V-model for software-udvikling bliver kaldt "system test".
System-testen er her beskrevet som den test der viser om kravspecifikationen bliver overholdt.
Det er hovedsagelig en funktionel test, som viser om softwaren funktionelt opfører sig som forventet.
Alt efter typen af produktet vil der i systemtesten også kunne være forskellige ikke-funktionelle test, som:
Performance test
Sikkerhedstest
Hardwaretest
Mekaniktest

11. System Testing ”Classic”
Requirements
Test
Specification
Perform

12. What are requirements?

13. Requirements
300 page document with references to 100 standard reference documents
Bullet points from marketing material
”Just like the last version, but with this feature added”
Requirements database
”Go ask that guy”

14. Requirement example
Requirement B-67 Target should respond on protocol XYZ within 30 seconds of being powered on.

15. Test Specification example
Test Case Name: Start responding within 30 seconds
Validates: Requirement B-67
Definition:
Turn on target
Wait 30 seconds
Expect that the target responds on protocol XYZ using the STATUS? request message

16. SeqZap Implementation example

17. SeqZap Implementation example

18. Collapsed implementation

19. Specification and Implementation

20. Print (read: PDF)

21. System Testing ”SeqZap style”
Requirements
Executable
Test Specification

22. System Testing ”Agile”
Implement Feature A
Write test for
Feature A
Test
Test All Features
Fix failing tests

23. System Testing ”Agile”
Implement Feature A
Write test for
Feature A
Test
Test All Features
Fix failing tests

24. What is Software System Testing?

25. Software System Testing
Functionality testing
Abstraction!
Does not require coding skills
Requires domain knowledge

26. Domain language (ventilation)
Airflow (m3/h)
Pressure (atmos)
Automatic windows (0-50% opened)
Temperature (celcius)

27. Engineering language Airflow (0-5V) Pressure (0-3.3V) Actuator (60W)
Temperature (PT100)

28. In SeqZap

29. We want as many people as possible to be able to read and write tests.
Developers
Testers
Service Technicians
Project managers? Secretaries? Sales people?
Write the test in domain ”terms”
Domain terms are the only thing that all people in an organisation share.

30. What is IoT?

31. What is IoT? Cloud Internet of Things Big Data Industry 4.0
Putting data from embedded controllers into ”databases”

32. What is IoT? Primary feature Secondary features
Putting data from embedded controllers into ”databases”
Secondary features
Online service
Firmware update
Usage statistics

33. How? Complexity Internet protocols Larger operating systemsIP
TCP/UDP
HTTP
Larger operating systems
Linux, WindowsCE, Android
User Interfaces
Touch screen
Web
Complexity

34. R&D Challenges Increasing complexity
Internet of Things (IoT), Big Data, Industry 4.0
Shorter release-cycle
Manual testing is hard/impossible

35. What is SeqZap?

36. System or Device Under Test
Test system
System or Device Under Test
Test Interface
Instrument
Test Interface Tool
Tool
Script
Report
Script
Script
Test Sequencer
Script
Script
Log

38. Key features Write the test in domain language
Testers, Technicians, Programmers, Engineers, Sales, Management, etc.
Support more than one interface
Analog IO, Digital IO, Serial, HTTP, ModBus, etc.
Adding support for a new interface is simple
Integrated Test Environment
Live connection to the DUT

39. Customer Stories
SKOV

40. SKOV SKOV is providing climate solutions and farm management solutions
for animal production worldwide,
which ensure optimum conditions
for animals, humans
and environment.

41. SKOV Mobile app’s (Android, iPhone, WinPhone
Cloud solutions for Enterprise Management
PC based Management systems
Embedded controllers
Embedded controllers

42. SKOV Embedded Controller
ARM/Linux target
QT Touch interface
Simple IO control

43. SKOV Embedded Controller
Flere forskellige typer controllere – klima, produktion eller kombinerede klima og produktionsstyringer til fjerkræ eller grise.
Alarmanlæg.

44. SKOV Embedded Controller
SKOV TCP
Completely internal protocol

45. SKOV Embedded Controller
SKOV TCP
Analog Digital IO
Measurement Computing boxes

46. SKOV Embedded Controller
SKOV TCP
BFN
Analog Digital IO
BigFarmNetwork
Integration protocol between SKOV products.

47. SKOV Embedded Controller
SKOV TCP
BFN
SSH
Analog Digital IO
Terminal Access (it’s a linux target)

48. SKOV Embedded Controller
SKOV TCP
BFN
SSH
UDP
Analog Digital IO
Device discovery

49. SKOV Embedded Controller
SKOV TCP
BFN
SSH
UDP
Remote UI
Analog Digital IO
Inspect widget tree
Capture framebuffer
Simulate touch

50. FarmOnline Explorer (FOX)
Management Interface
Windows Application
Collects data from embedded controllers

51. FarmOnline Explorer (FOX)
Management system FarmOnline. App til både Android og Apple.

52. FarmOnline Explorer (FOX)
Windows
Accessibility
Standard accessibility layer in windows
”Hjælp til handicappede”

53. FarmOnline Explorer (FOX)
Windows
Accessibility
BFN
BigFarmNetwork
Integration protocol between SKOV products.

54. Integration Testing BFN BFN Windows Accessibility Remote UI
Analog Digital IO

55. Using only external interfaces
Integration Testing
Windows
Accessibility
BFN
Remote UI
BFN
Analog Digital IO
Using only external interfaces

56. System Integration Testing (SKOV)
JSON
Windows UI
Remote UI
Analog/Digital IO

57. Interfaces (SKOV) Controller (Chicken and Pig farms)
Analog/Digital IO
Remote UI
Proprietary Protocols
SSH (Remote Command Line)
UDP Device Discovery
GSM (SMS/Call)
PC based Management System
Windows UI
Proprietary Protocol
Web Service interface (JSON)
JSON
Web/Cloud access
Web Browser
Mobile access
Mobile UI

58. SKOV Organisation & Methods
25 Software developers at SKOV in Glyngøre
7 Software developers in Malaysia
Organized in 6 Scrum Teams (3 – 8 persons)
Prioritized backlogs
Daily Scrum meetings / 2 weeks sprints
Continous integration
Automatic test is developed inside each Scrum Team (no dedicated test department)

59. Test Machines

60. Test Targets

61. Test Process Automatic test execution between 18:00 – 06:00
A fresh result every morning at SCRUM
Did we break anything yesterday?
Continous Integration (Cruise Control)
Build
Unit Test
System Test
Nightly test on real hardware
Developer debugging using PC hardware

62. Continous Integration
Each commit triggers a rebuild
Nightly release builds (6 virtual build servers)
Nightly test of all release builds in SKOV’s Software Test Center (~2.000 individual test cases)
Functionality
UI testing
Stability
Automatic test reports available

63. Continous Integration
TestCenter
Automatic test execution between 18:00 – 06:00
Total test time: hours Total test cycles per year:
If we only had one target :
Total test time: ~ hours
Equal to:
(24 hours/day) ~ 2 months
Total test cycles per year: ~ 6

64. Test Steps and procedure re-use
Her ses et eksempel på en Test Case, hvor kan kalder en del procedurer fra Shared script filer.
Senere i testen kaldes procedurer fra ”Procedures.szs” filen i test projektets mappe, som altid har namespacet ”LocalProcedures”
Bemærk at der benyttes Test Steps og at de stemmer overens med steps i Test Specificationen.

65. Test data is configured in a DataTable
GSM Tool benytter også parametre fra datatabel afhængigt af hvilket test site, der er valgt.
Bemærk at Test Site og HostName er sat som local values

66. Notes / Conclusions
Introduced errors in existing features are detected immediately during regression tests
Automatic test / continuous Integration enables SKOV to release software products with known and reproducible quality level!
At SKOV we aim to have automatic test on all software product – but we are not there yet. (Mobile solutions are still manually tested)
Automatic test is time consuming, but worth the effort !
- Ebbe Kronborg, R&D Department Manager, Software

67. Customer Stories
Grundfos

68. Grundfos
Grundfos is a global leader in advanced pump solutions and a trendsetter in water technology. We contribute to global sustainability by pioneering technologies that improve quality of life for people and care for the planet.
Worldwide organisation ( employees)
SeqZap used globally as the go-to tool for embedded software test

69. System Test
GENI

70. System Test
GENI
PLC IO via OPC

71. System Test
GENI
PLC IO via OPC
PSU
Via SCPI

72. System Integration Testing
PLC
GENI
Pump
GSM
Comm.
Module
Web
Cloud

73. System Integration Testing
PLC
GENI
Pump
GSM
Comm.
Module
Web
Cloud

74. How does this relate to us?
Conclusions
How does this relate to us?

75. Conclusions (Development)
Automate as much as possible
Build
Unit test
System test
Integration test
Use the tools from the software industry
Continous Integration (Jenkins, etc.)
Unit testing frameworks
Test results should be available as fast as possible

76. Conclusions (System Testing)
GUI testing is actually possible on an embedded target
Easy to understand
Easy to communicate to the rest of the organisation
Wow effect
Move real-time stuff to a non-PC system
Use PC to drive the real-time system
Most system tests are not real-time

77. Conclusions (Adoption)
Start testing!
One test is always better than zero test
Start testing one system
Then expand!
No big-bang solutions
How do you eat an elephant?
Bite by bite

78. Questions?

79. Thank you for your time!