1. How to Test the Internet of Everyting @paul_gerrard Paul Gerrard paul@gerrardconsulting.com gerrardconsulting.com

2. Background The IoE will dominate our industry for 10 to 20 years Right now, it is very confusing Standards emerging, but current applications are bleeding edge, speculative, exploratory (often simple) Security and privacy risks dominate today But functional and integration complexity challenges must also be overcome There are also broader societal risks Government, companies and individuals need to pay attention. Intelligent Definition and AssuranceSlide 2

3. Testing the Internet of Everything An article and webinar series 1."Internet of Everything – What is it and how will it affect you?" 2."Internet of Everything – Architecture and Risks" 3."Internet of Everything – Test Strategy" All available at: http://gerrardconsulting.com/?q=node/652 Intelligent Definition and AssuranceSlide 3

4. What is the IoE? Devices, makers, corporates, smart cities, drones, 3-D printed cars

5. Some cheap devices Slide 5

6. My Robot (and a better one) Slide 6 https://www.youtube.com/watch?v=f0CluQiwLRg

7. Slide 7

8. Smart City World Congress Slide 8

9. Slide 9

10. Test this? Slide 10

11. Architecture(s)

12. A Layered Model of the IoE No standard for networking cheap devices No standard/conventional architecture for IoE But... – some patterns are emerging and the architecture can be split into logical layers – Standards are emerging (slowly) – The 7 layer model describes the features of elements in an IoT architecture. Intelligent Definition and AssuranceSlide 12

13. Seven layer architecture Not every system has seven separate layers. Layers are often combined into one object/system Internet connected Data-Centre Based Not Internet connected Local Network Local, metro network internet connected Slide 13

14. Drivers

15. Introduction IoE test strategy – for testing everything? – Can only paint a picture – Dimensions, influences, and opportunities – Suggest what testers need to think about The challenge we face is somewhat different to what we have tested before. Intelligent Definition and AssuranceSlide 15

16. What's in Scope? The scope of IoE is growing and includes: – Vast numbers of static devices, mostly sensors – Mobile devices: cars, buses, trains, planes, ships, satellites... It will include: – Objects in homes, cities, work, hospitals, social venues – Wearable, human-embedded devices, phones and tablets IoE integrates with 100s of services, payment and legacy systems Eventually, IoE will be called the Internet The test strategy for the IoE is the test strategy for everything Once more with feeling... Intelligent Definition and AssuranceSlide 16

17. The test strategy for the IoE is the test strategy for everything

18. The Biggest Thing in 30 Years? Continuous delivery and DevOps bring factory automation processes into software After Waterfall and Agile, CD and DevOps become 'the third way' The IoE will affect everyone on the planet. From 50-700bn devices over the next 5 to 20 years; no one knows We are on a journey that could increase the scale of the internet by one hundred times How will we test it? Intelligent Definition and AssuranceSlide 18

19. The Testing Journey (from the 1980s to the present) Screen based applications running on dumb terminals – 1980s GUI applications and Client/Server – 1990s The emergence of the Internet - 1995 'Mobile Revolution' - today Internet of Everything - emerging – Smart phones, buildings, cars, ships, planes and cities – BigData, pervasive, wearable computing – Massive scale IoE brings new levels of complexity and scale Non-functional risks are well-understood – what's new? Functional testing and simulation at very large scale. Intelligent Definition and AssuranceSlide 19

20. If CMOs hold the budget… There is a shift in power in the mobile space If marketers hold the money for building IoE: – IT must meet short and long term business goals – Software teams must deliver, experiment and learn Market testing and experimentation to drive rapid evolution DevOps, Continuous Delivery, high levels of automation and simulation Test analytics* blended with production analytics. Intelligent Definition and AssuranceSlide 20 * Thinking Big: Introducing Test Analytics, http://gerrardconsulting.com/?q=node/630

21. Connecting the Unconnected 'Internet.org by Facebook' wants to provide cheaper access to data (perhaps 1/100th of the current price) Developing economies will probably lead the way in implementing the IoE with mobile internet But what networking technology/protocols will be used for local connectivity? "No technology dominates yet. We are waiting for technologies to improve in range and performance. The standard technology for connecting the IoT hasn't been invented yet." We are still at the start of the IoE journey Intelligent Definition and AssuranceSlide 21

22. The Scope of IoE Testing

23. Functional testing Hardware-Level functionality – Most of this testing will be performed by manufacturers Object and Server level functionality – We'll have to test simple web-apps to systems with many complex sub-systems Integration and Interaction at all levels – Companies, testers will have to use judgement on deciding what to test and how much Collaboration confusion – Use expectations and user testing of collaborating systems will be difficult. Intelligent Definition and AssuranceSlide 23

24. Testing @ Scale Scalability – We'll have to do some large-scale load, performance and stress testing Wearables and embedded – We will need to simulate and test thousands of devices carried by or embedded in people Everything connected – We'll need to do location-based tests at very large scale. Intelligent Definition and AssuranceSlide 24

25. Networks Moving networks – We'll have to test security and resilience of our systems against alien or rogue networks Network security risks at multiple levels – Networks will need to be hardened and tested at all levels, not just 'perimeter' Device registration, provisioning, failure and security – Power-down, power-up and automated authentication, configuration and registration processes will need to be tested Mobile objects – Power, interference, network strength, roaming and jamming issues will all need testing. Intelligent Definition and AssuranceSlide 25

26. Big Data Logistics – A substantial data storage service will have to be tested (performance, backup/recovery etc.) Analysis and visualisation – These outputs and processes will need to be tested Personal and corporate privacy – Testing will need to embrace the ethical use of data and it's protection from misuse. Intelligent Definition and AssuranceSlide 26

27. The Scope of IoE Testing Not all IoE systems will be huge or complex But we face a wider range of concerns than ever before: – Most bring a new technology and risk profile – The range of testing issues we need to consider – The scale of the testing required has increased We'll need new approaches to tackle the risks of failure. Intelligent Definition and AssuranceSlide 27

28. Test Strategy for the IoE

29. Modelling, Testing and Test Data Tools to model the IoE are only just emerging We'll need to use Big Data test techniques 'Pattern-based test design' – Generate tests from patterns to simulate scenarios – 'Heuristic oracles' will derive from conventions or common-sense – Personas as patterns of user or system behaviour Developers will need to make systems testable through High Volume Automated Testing. Intelligent Definition and AssuranceSlide 29

30. Test Environments; Testing in the Field Test environments will vary widely – A test lab for a home could be set up in any office – An urban environmental management system will be harder to create Scale of field testing will vary widely, but will become a common (and possibly mandatory) testing stage Barcelona has set up an urban-scale testing facility – Barcelona Urban Lab is positioned as a field test environment to prove the viability of innovative systems Cities around the world will offer similar services. Intelligent Definition and AssuranceSlide 30

31. Slide 31

32. Tool Support Automated support will be essential and probably dominate testing Most components tested through APIs, web or other services New protocols emerging; custom drivers may be required The challenge: "How do we design the thousands or millions of tests to feed the tools?" Intelligent Definition and AssuranceSlide 32

33. Test Analytics DevOps processes and production analytics monitor system vital signs (like sensors) – These analytics are essentially the same Production systems are platforms for experimentation – The code that serves marketers can also serve developers and testers Analytics code is your sensor network DevOps processes are 'things', too. Intelligent Definition and AssuranceSlide 33

34. Performance Testing and Test Data Local network protocols – IP based (XMPP, MQTT); few tools, but cloud services emerging – Low Power/Lossy Network (ZigBee, 6LowPAN, DASH7/RFID, Bluetooth and NFC); only need simple drivers to 'chirp' Coherent data may be a problem, e.g. location data – Cars must follow the road network – People can only walk through certain places in a shopping mall – So you will have to capture the GPS data using a tool perhaps To create unique paths, you could edit together segments of intersecting paths. Intelligent Definition and AssuranceSlide 34

35. The Future for Testing and Testers We need to think a little differently A New Model for Testing

36. Manual v Automated Testing? Automation will not make testing easy; it will make testing possible More and more, DevOps processes are a source of data for analysis just like production You might not need to be a programmer but you may need to: – Write your own utilities, download, configure and adapt open source tools or create automated tests – Have informed conversations with developers Should testers learn to code? Probably Intelligent Definition and AssuranceSlide 36

37. New Model Testing 29 page paper: http://dev.sp.qa/download/newModelhttp://dev.sp.qa/download/newModel Slide 37

38. Shift-Left Every company seems to be 'shifting left' – Developers responsible for testing – no testers – Testers embedded in development teams – Testers morphing into business or systems analysts Testing thinking is being moved left and earlier in the process The New Model identifies these thinking activities Testers need to learn how to 'let go' of testing, so view this as an opportunity, rather than a threat. Intelligent Definition and AssuranceSlide 38

39. Close The strategy for testing the IoE is the strategy for testing everything Some of the tools we need don't yet exist – Test modelling, test pattern/data generators and automated oracles – Simulation/high volume automation/visualisation tools Testing in the lab and experimentation in the field become one discipline informed by Test Analytics New test approaches will be required The New Model aims to trigger some new thinking in this area. Intelligent Definition and AssuranceSlide 39

40. Slide 40

41. Slide 41

42. Slide 42

43. How to Test the Internet of Everything @paul_gerrard Paul Gerrard paul@gerrardconsulting.com gerrardconsulting.com