Measuring Publish-Subscribe (loopback) latency of MQTT

Search…

MQTT pub/sub latency measurement is important classification criterion for selecting a broker. Let's understand the impact of latency vs. the number of clients connected with your Broker.

Introduction
IoTIFY’s network simulator is a powerful cloud based tool to create intelligent IoT simulation. Unlike other load testers which generate dumb traffic to cloud, IoTIFY Network simulator is smart and completely customizable. With its Javascript syntax, the tool could be used to perform variety of use cases.
In this post we will use network simulator to measure the round trip performance of a public MQTT broker. We will measure average delay between a publication and reception of the message (loop back). Then we will plot this latency value change with respect to number of connected clients, and see how does it changes as number of connected clients change. 
Step 1. Create a loopback client template
For this demo we will use a simple template named latency. Go to the top level Template and click on the import button. Provide the following URL in import dialog
​https://raw.githubusercontent.com/iotify/nsim-examples/master/performance-testing/latency.json​
For this test, we will use public MQTT broker broker.hivemq.com:1883 but you are free to change the broker if you want. 
In this template, each client will publish a timestamped message to a unique topic /latency/{{client()}} and will also subscribe to it. The expression in the topic will evaluate to /latency/0 for client 0 and so on at the run time. Once the client receives back the message it has published, it will subtract the received timestamp from the current timestamp and will calculate the loopback delay in millisecond. The delay value will be recorded via metric() API. The key to metric.add() API will be based on the total number of clients used in this demo along with QoS, so we can plot separate metrics for a group of 100 clients, 1000 clients and so on. 
Step 2. Run the template
Lets run this template with following settings 
Number of clients: 100
Messages sent: 100
Gap between each message: 1 second
Depending on your account type, you could see a different limits on the parameters above. 
Once the simulation has been finished, go to the top level Metrics Menu and plot the recently created latency_qos0_100 metric for last the 5 minutes. You will see a chart like this
Average publish/subscribe latency of 100 MQTT clients with QoS0
You could change the aggregation function on the right hand side to see Min/Max values as well. So we observe that on average MQTT loopback latency remains approx 60 ms for a group of 100 clients. 
Now lets change the number of clients in the simulation run to 1000 and run the job again. This time you will see a new metric value is being created with key named latency_qos0_1000. Go to Metrics menu and plot both latency values for last 30 minutes. You will notice that latency value becomes significantly higher when 1000 clients were connected instead of 100. 
MQTT loopback Latency comparison of 100 vs. 1000 clients on QoS0
With these charts we observe the following: 
A) The average latency tends to decrease slowly as the test progresses. The values are significantly higher in the start of the test rather than end. This is expected because of caching and reduced overhead of an already established connection. 
B) Average Latency increases with number of connected clients. 
Effect of QoS Settings on MQTT Latency
What happens when we change the QoS Values of MQTT protocol and repeat the same experiment? Here is a chart depicting the different QoS latencies with the same number of clients. It appears that QoS 2 has the highest latency values of them, which is also expected. 
Optional Exercises
Change the QoS value to 1 and 2 and repeat the same experiment. 
Change the MQTT retain flag and repeat the same experiment.   

IoT Testing - Previous

Performance Testing

Next - IoT Testing

Security Testing

Last modified 1yr ago

Copy link

Contents

Introduction

Step 1. Create a loopback client template

Step 2. Run the template

Effect of QoS Settings on MQTT Latency

Optional Exercises