JMeter and why it fails at IoT

IoT testing has been a space which has been quite overlooked in the past with no real testing platforms being built to tackle the problems specific to IoT testing. In such a case, a lot of teams have to settle down with a legacy tool like JMeter which was built to test web applications. While JMeter is an easier path than creating your own set of scripts to automate these testing scenarios, it leaves much to be desired, especially concerning IoT. This is where IoTIFY comes into the picture, it is a testing platform built from scratch to utilize the newest technologies, built with IoT testing in mind. Now you don’t have to choose between ease of use and a robust testing platform which can tackle complex problems specific to IoT testing,

Here are some of the features that set IoTIFY apart from other legacy web application testing tools.

• Create a truly realistic digital twin of your device

• Stateful device models to accommodate complex behaviors

• Bi-directional communication so that you can test server-initiated functions like OTA updates

• Device-to-device communication, so you can model a full IoT system

• Extremely scalable to millions of concurrent devices at once

• Extensible JavaScript-based scripting with full npm support

• Easy provisioning of devices with our cloud connectors

• In-depth tracking of metrics and payloads so you can track any issues

• Automate complex workflows with ease using a graphical interface

• Easy to collaborate with your team with workspaces

Let’s take a more in-depth look at some of these issues that IoTIFY solves.

Test a device’s complete lifecycle

An IoT device will have multiple states, where it will behave differently, there are cases where the server may want to initiate an action on the device. These are all complex device interactions where tools like JMeter fail. With IoTIFY, you can create stateful device twins of your physical device and model behavior that can be as granular as required. Tools like JMeter can simulate MQTT payloads, but the communication isn’t bidirectional. This severely limits the scope of testing that can be done on these tools.

Another issue is the fact that JMeter virtual devices lack support for things like device states, memory and so on, which are basic requirements for IoT devices.

With IoTIFY, you get a true digital representation of your device, which allows you to test each and every complex workflow at scale.

Full IoT system orchestration with LAN communication between devices

IoT systems often not only communicate with a central server but also between devices on the local networks. This multichannel communication is a problem for legacy testing tools to simulate, however with IoTIFY you can have edge devices communicating with gateways which in turn communicate with a central server. This is a scenario unique to IoT systems and is often not viable to simulate owing to the complexity of having multiple different devices all communicating differently.

Scaling up the IoT systems to a million scale

In an IoT system, the scale can go to millions of unique devices all working at the same time and having their own memory and states. This can be extremely difficult to test with tools such as JMeter. However IoTIFY was built with scale in mind, so scaling your tests to a million nodes is as easy as clicking a button. Another advantage is the fact that, unlike simple load generators which will at best simulate a fixed scenario, IoTIFY device twins are true-life representations of the devices, which means that they can create a meaningful simulation of how the system will work in production. With full bidirectional communication and each node being a unique device with its individual memory and states, you can simulate a real-world usage of your system.

Provisioning and maintaining devices with cloud providers

A lot of IoT systems nowadays use cloud providers such as AWS or Azure to maintain the identity and data for their IoT devices. When creating a testing workflow, it can be a task to provision these devices and orchestrate the individual device identity and memory. Take into account the fact that the scale can move to millions and it can become a truly time and resource-consuming endeavor. With IoTIFY, using our cloud connectors, it becomes a matter of a few clicks. Our cloud connectors make it a painless task to provision and maintain millions of devices on cloud providers saving time and resources.

Quality metrics and their necessity

When dealing with any IoT system, it can be rather important to know and understand exactly what is happening with each individual device at any given time. This becomes increasingly necessary in case of any problems or bugs while testing an IoT system. Such granular data is not available with legacy testing tools such as JMeter, which will mostly provide high-level data like success and failure rates. With IoTIFY you get an insight into the workings, payloads and states of each individual device at all time intervals making it a very powerful tool for understanding what went wrong in any given test run. Tracking of test assertions, latencies and such are also taken care of by default. A powerful metrics engine also allows you to define any custom metric and easily visualize the data in graphs and charts.

Automate complex workflows easily

It can become quite a challenge to create very complex scenarios where multiple different devices are working together and each one depends on another simulation. It can become a nightmare to create such complex scenarios with legacy testing tools. However, with IoTIFY, you can easily automate such workflows using our graphical tool. The powerful scenario composer allows you to create such scenarios using a drag-and-drop tool. So it becomes a task which can be completed in minutes.

Collaborate with ease

While working in teams it is important that everyone has access to resources like the code base, the historical results and such. This can be a hassle with a tool like JMeter. However, IoTIFY was bult with collaboration at its core. Working with team members is as easy as sending them an invite by email. Once the accept the invite, they are added to a shared workspace. This makes it much more convenient to work and share resources.