1. How to Test the Internet of Everything
Intelligent Assurance and Testing
28-Apr-19
How to Test the Internet of Everything
Paul Gerrard
@paul_gerrard
gerrardconsulting.com

2. Intelligent Definition and Assurance
Agenda
The Testing Challenge
The Scope of IoE Testing
Test Strategy for the IoE
The Future for Testing and Testers
Close
Intelligent Definition and Assurance

3. Intelligent Definition and Assurance
Background
The IoE will dominate our industry for 10 to 20 years
Right now, it is very confusing
Standards emerging, but current applications are bleeding edge, speculative or exploratory
Security and privacy risks dominate today
But functional and integration complexity challenges must also be overcome
There are also broader societal risks
Government, companies and individuals need to pay attention.
Intelligent Definition and Assurance

4. Assurance, Re-Training, Mentoring
28-Apr-19
Introduction
IoE will be 'everything on the internet' soon
A strategy for 'Testing Everything'?
Dimensions, influences, and opportunities
Can only paint a picture
Suggest what testers need to think about
The challenge we face is quite different from what we have tested before.
Intelligent Definition and Assurance
© Aqastra Limited 2008

5. Scope of the Internet of Everything
The scope of IoE is growing and includes:
Vast number of static devices, mostly sensors
Mobile devices: cars, buses, trains, planes, ships, satellites...
It will include:
Objects in homes, cities, work, hospitals, social venues
Wearable, human-embedded devices, phones and tablets
IoE integrates with 100s of services, payment and legacy systems
Eventually, IoE will be called the Internet
The test strategy for the IoE is the test strategy for everything.
Intelligent Definition and Assurance

6. The Biggest Thing in 30 Years?
Continuous delivery and DevOps bring factory automation processes into software
After Waterfall and Agile, CD and DevOps become 'the third way'
The IoE will affect everyone on the planet.
From bn devices over the next 5 to 20 years; no one knows
We are on a journey that will increase the scale of the internet by one hundred times
How will we test it?
Intelligent Definition and Assurance

7. The Testing Journey (from the 1980s to the present)
Screen based applications running on dumb terminals – 1980s
GUI applications and Client/Server – 1990s
The emergence of the Internet
'Mobile Revolution' - today
Internet of Everything - emerging
Smart phones, buildings, cars, ships, planes and cities
BigData, pervasive, wearable computing
Massive scale
IoE brings new levels of complexity and scale
Non-functional risks are well-understood – what's new?
Functional testing and simulation at very large scale.
Intelligent Definition and Assurance

8. If CMOs hold the budget…
There is a shift in power in the mobile space
If marketers hold the money for building IoE:
IT will have to meet short and long term business goals
Development must deliver, experiment and learn
Market testing and experimentation to drive rapid evolution
DevOps, Continuous Delivery, high levels of automation and simulation
Test analytics blended with production analytics.
Intelligent Definition and Assurance

9. Connecting the Unconnected
'Internet.org by Facebook' wants to provide cheaper access to data by (to perhaps 1/100th of the current price)
Developing economies will probably lead the way in implementing the IoE with mobile internet
But what networking technology/protocols will be used for local connectivity?
"No technology dominates yet. We are waiting for technologies to improve in range and performance. The standard technology for connecting the IoT hasn't been invented yet."
We are still at the start of the IoE journey
Intelligent Definition and Assurance

10. The Scope of IoE Testing

11. Intelligent Definition and Assurance
Functional testing
Hardware-Level functionality
Most of this testing will be performed by manufacturers
Object and Server level functionality
We'll have to test simple web-apps to systems with many complex sub-systems
Integration and Interaction at all levels
Companies, testers will have to use judgement on deciding what to test and how much
Collaboration confusion
Use expectations and user testing of collaborating systems will be difficult.
Intelligent Definition and Assurance

12. Intelligent Definition and Assurance
Scale
Scalability
We'll have to do some large-scale load, performance and stress testing
Wearables and embedded
We will need to simulate and test thousands of devices carried by or embedded in people
Everything connected
We'll need to do location-based tests at very large scale.
Intelligent Definition and Assurance

13. Intelligent Definition and Assurance
Networks
Moving networks
We'll have to test security and resilience of our systems against alien or rogue networks
Network security risks at multiple levels
Networks will need to be hardened and tested at all levels, not just 'perimeter'
Device registration, provisioning, failure and security
Power-down, power-up and automated authentication, configuration and registration processes will need to be tested
Mobile objects
Power, interference, network strength, roaming and jamming issues will all need testing.
Intelligent Definition and Assurance

14. Intelligent Definition and Assurance
Big Data
Logistics
A substantial data storage service will have to be tested (performance, backup/recovery etc.)
Analysis and visualisation
These outputs and processes will need to be tested.
Personal and corporate privacy
Testing will need to embrace the ethical use of data and it's protection from misuse.
Intelligent Definition and Assurance

15. The Scope of IoE Testing
Not all IoE systems will be huge or complex
But we face a wider range of concerns than ever before:
Most bring a new technology and risk profile
The range of testing issues we need to consider
The scale of the testing required has increased
The approaches we will need to tackle the risks of failure will have to change.
Intelligent Definition and Assurance

16. Test Strategy for the IoE

17. Modelling, Testing and Test Data
Tools to model the IoE are only just emerging
We'll need to use Big Data test techniques
'Pattern-based test design'
Generate tests from patterns to simulate scenarios
Oracles will derive from conventions or common-sense
Personas as patterns of user or system behaviour
Goal: to feed automated tools large numbers of scenarios
Developers will need to make systems testable through High Volume Automated Testing.
Intelligent Definition and Assurance

18. Test Environments; Testing in the Field
Test environments will vary widely
A test lab for a home could be set up in any office
An urban environmental management system will be harder to create
Scale of field testing will vary widely, but will become a common (and possibly mandatory) testing stage
Barcelona has set up an urban-scale testing facility
Barcelona Urban Lab is positioned as a field test environment to prove the viability of innovative systems.
Expect other cities around the world to offer similar services.
Intelligent Definition and Assurance

19. Intelligent Definition and Assurance
Tool Support
Automated support will be essential and probably dominate testing
Most components tested through APIs, web or other services
New protocols emerging; custom drivers may be required
The challenge will be, "how do we design the thousands or millions of tests to feed the tools?"
Intelligent Definition and Assurance

20. Intelligent Definition and Assurance
Test Analytics
DevOps processes and production analytics monitor system vital signs like sensors
These analytics are essentially the same
Production systems are platforms for experimentation
The code that serves marketers can also serve developers and testers
Analytics code is your sensor network
DevOps processes are 'things', too.
Intelligent Definition and Assurance

21. Performance Testing and Test Data
Local network protocols
IP based (XMPP, MQTT); few tools, but cloud services emerging
Low Power/Lossy Network (ZigBee, 6LowPAN, DASH7/RFID, Bluetooth and NFC); only need simple drivers to 'chirp'
Coherent data may be a problem, e.g. location data
Cars must follow the road network
People can only walk through certain places in a shopping mall
So you will have to capture the GPS data using a tool perhaps
To create unique paths, you could edit together segments of intersecting paths.
Intelligent Definition and Assurance

22. The Future for Testing and Testers
We need to change the way we think about testing

23. Intelligent Definition and Assurance
New Model for Testing
"We explore sources of knowledge to build test models that inform our testing"
Forget logistics; focus on 10 thinking skills
Not a process with sequence, inputs/outputs, entry/exit criteria
What is the model useful for?
Separates logistics from the core testing skills
Clarifies two modes: exploration and testing
Supports all testing, regardless of methodology, manual v automated, structured or agile.
Intelligent Definition and Assurance

24. New Model Testing
29 page paper:

25. Manual v Automated Testing?
Automation will not make testing easy; it will make testing possible
More and more, DevOps processes are a source of data for analysis just like production
You might not need to be a programmer but you may need to:
Write your own utilities, download, configure and adapt open source tools or create automated tests
Have informed conversations with developers
Should testers learn how to write code? Yes!
Intelligent Definition and Assurance

26. Intelligent Definition and Assurance
Shift-Left
Every company seems to be 'shifting left'
Developers responsible for testing – no testers
Testers embedded in development teams
Testers morphing into business or systems analysts
Testing thinking is being moved left and earlier in the process
The New Model identifies these thinking activities
Testers need to learn how to 'let go' of testing, so view this as an opportunity, rather than a threat.
Intelligent Definition and Assurance

27. Intelligent Definition and Assurance
Close
Testing the IoE is the next level of testing in terms of complexity and scale
The tools we need don't yet exist
Test models, test data generators and automatic oracles
Simulation/High Volume Test Automation tools
Testing in the lab and experimentation in production will become one discipline – test analytics
Some new test approaches will be required
The New Model is an attempt to trigger new thinking in this area.
Intelligent Definition and Assurance

28. How to Test the Internet of Everything
Intelligent Assurance and Testing
28-Apr-19
How to Test the Internet of Everything
Paul Gerrard
@paul_gerrard
gerrardconsulting.com