rabbitmq,

Resolve RabbitMQ Node statistics not available in Cluster Node

Nov 08, 2021 · 2 mins read · Post a comment
Resolve RabbitMQ Node statistics not available in Cluster Node

Setting up RabbitMQ cluster with master and slave instances can invoke some node destabilizations like the error Node statistics not available. Here I’m going to show you few steps how can you resolve it.

Prerequisites

  • RabbitMQ

Solution

Step 1. The most common symptom is the 100% disk usage. So ssh into the instance and check the status of the rabbitmq service:

systemctl status rabbitmq-server

If the status is Exited like below follow the step 2

● rabbitmq-server.service
   Loaded: loaded (/usr/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code)
  Process: 15630 ExecStart=/usr/sbin/rabbitmq-server (code=exited, status=1/FAILURE)
 Main PID: 15630 (code=exited, status=1/FAILURE)
   Status: "Exited."

Step 2. Check if /opt/rabbitmq directory has reached 100% usage:

df -h

If the Output is similar like:

Filesystem                       Size  Used Avail Use% Mounted on
udev                             7,7G     0  7,7G   0% /dev
tmpfs                            1,6G  2,2M  1,6G   1% /run
/dev/mapper/datavg-lv_rabbitmq   3.0G  3.0G   34M 100% /opt/rabbitmq

Then you should find out which directory from /opt/rabbitmq is using majority of the space:

du -sch /opt/rabbitmq/*

So if you figure out that the directory is old or contains messages that are not useful you can clean them up. Afterwards restart the rabbitmq service, so it can be in a running status and fix the error:

systemctl status rabbitmq-server
...
● rabbitmq-server.service - RabbitMQ broker
   Loaded: loaded (/usr/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: disabled)
   Active: active (running)

Step 3. Even if the service status is running, it may still show the same message Node statistics not available. There is one more possibility. Check the /var/log/rabbitmq/*.log log files and see if the net tick timeout issue is popping up:

=INFO REPORT==== 1-Nov-2021::17:50:33 ===
node rabbit@server down: net_tick_timeout

To resolve this open the rabbitmq default conf file and set the net_ticktime if it’s not defined:

net_ticktime = 60

Restart the rabbitmq service, so the changes can take effect and that’s it. The error should be gone:

systemctl restart rabbitmq-server

Conclusion

If there is something that you wanna raise please do it in the comment section below. Feel free to leave a comment below and if you find this tutorial useful, follow our official channel on telegram.