Analyzing Results

How to understand the simulation results

Once the job is running, it could be monitored in real time with Job status page

Status page allows you to pause/resume or stop the currently ongoing simulation. If the simulation is finished, you could delete the results or rerun the simulation with the same settings by icons available in action column. The results could also be analyzed in detail by Result explorer.

Network Timestamps

This chart provides you a high precision timestamped overview about every iteration. Think of this as a test timeline, indicating which iteration started when and lasted how long. You will see an overlap of iteration, which is natural considering the delays in processing the messages.

Let's look at this chart in more detail by zooming through the chart into a particular time interval. In this test run there were 1200 clients sending messages at 1 second interval. We see the blue bars in the start depict the connection establishment completion. At timestamp 21:40:10, 210 out of 1200 clients were marked connected with the MQTT broker. At 21:40:11, remaining 990 clients were also marked connected. So the entire connection process of 1200 clients took roughly around 2 seconds.

After the connection has been established, there was a 1 second delay, and then the client started sending their messages (Iteration 0). We this in the chart with black marked lines.

Iteration 0 started at 21:40:13 and continued to execute until 21:40:17, that is a total of 4 seconds. There was an overlap between iteration 1, which also started at 21:40:16 and continued until 21:40:20

Overall, throughout the test cycle, the server wasn't accepting more than 500 messages/second on an average. Now we know that server was throttled to not receive more than 500 messages/second from a single IP source, so that would explain that the iterations were spread over time.

If you are sending more messages or connecting more clients than what your server could handle, every client is eventually going to wait for its turn to connect and send, thereby stretching the overall test duration. It is important to carefully tune timeout values in your template to handle this worst case, otherwise the iteration will be marked as failed.