IoT Performance Testing: Challenges & Considerations

Current leading IoT protocols – HTTP is a known quantity. Hard to find a load test tool that doesn’t speak it – MQTT is IBM’s now open standard publisher\subscriber messaging protocol – CoAP is a document transfer protocol designed for M2M communication over constrained networks and nodes

How do current tool capabilities align with IoT? If the system being tested relies on HTTP IoT traffic then you have a wide range of vendors offering load test tools at competitive prices. End-to- end testing (barring device level) is also simplified If you are emulating MQTT or CoAP IoT traffic then your options narrow and costs & effort go up as proxies and test harnesses come into the picture If there is a requirement to emulate multiple protocols during testing then things become further complicated It remains to be seen how tool vendors will respond to the challenges presented by IoT

IoT Performance Test Tool Considerations Physical level: Devices, sensors, actuators Session protocols level: e.g. MQTT, CoAP, HTTP System level: Database, processing, analytics End user level: Business IoT (e.g. Healthcare, retail) Consumer IoT (e.g. Home, mobility) Cellular & short range data link communications Middleware processes incoming data Cellular & short range data link communications; Conventional networking Performance testing of the physical devices will likely be out of scope of full scale PT. Cellular and short range data links likely to fall out of scope and should be covered by provider SLAs. If your injection tool of choice supports the protocol(s) required then this will simplify things. If not then some form of driver or proxy may need to be implemented at this level in order to allow load application. If database performance is out of scope or not the key focus of a particular test phase then some form of stub or data recipient will need to be provisioned in order to receive data processed by the middleware. Having a single test tool which supports protocols to both inject and receive data would be very helpful here. If your IoT solution incorporates Big data (which it almost certainly will) then skipping performance testing is not an option. Finding a tool which is capable of injecting sufficient load to stress the database becomes critical. Data verification becomes further complicated with Big Data. Added to this it is critical to ensure that data being processed through the system results in the correct outputs being processed at acceptable rates. Again, having a single test tool which supports protocols to both inject and receive data would be very helpful here. Performance testing of the downstream systems and devices will likely be outside the scope of full scale PT.

Creating IoT load test environments Cost of creation of IoT-specific test environments may be preclusive This problem could be magnified when the requirement for Big Data environments is introduced Organisations may have to compromise on scale and extent of load test coverage Understanding areas of key risk will be critical when making priority calls

Creation & management of IoT test data IoT and Big Data go hand-in-hand Big Data favours non-conventional database solutions such as Apache Hadoop or any of the so-called “NoSQL” databases Performance testers frequently orchestrate the creation and management of their data-sets, something which will be complicated by a lack of familiarity with the technologies mentioned above There will be a lag between introduction of these technologies and rate at which performance testers become competent in them

Verifying the success of IoT tests IoT performance tests entail not only large amounts of input data, but also large amounts of processing and output data Verification can be a complicated, time-consuming process at the best of times, but the amounts of data anticipated with the advent of IoT and Big Data will present significant challenges To overcome these challenges performance testers will need to be become more innovative and efficient in their approach to test planning, execution and analysis Organisations will also need to seriously consider these factors when assessing timescales for IoT and Big Data projects

Hopefully these slides will have given you a high level insight into some of the challenges IoT will present from a performance testing perspective If you would like to know more about performance testing IoT solutions and what we can do to help then get in touch Please contact us at or give us a call on