How to Set Up a High Availability RabbitMQ Cluster on CentOS 7

RabbitMQ is a widely adopted open-source message broker that supports AMQP, STOMP, and various messaging technologies. It plays a key role in enterprise environments and microservice-based systems by functioning as an asynchronous communication layer between distinct services. This tutorial outlines how to deploy a RabbitMQ cluster across multiple CentOS 7 machines to ensure high availability. In this setup, one server serves as the primary node, while additional servers act as mirrored nodes for redundancy.

System Requirements

• Two or more clean CentOS 7 servers updated and within the same private network subnet
• RabbitMQ installed with its web-based management interface activated (refer to the RabbitMQ installation guide for CentOS 7)
• A non-root user with sudo permissions (see how to configure sudo on CentOS, Debian, and FreeBSD)

Firewall Configuration

By default, the CentOS firewall tool, firewalld, blocks all incoming connections. To allow access to RabbitMQ and its management interface from other systems, including external ones, specific ports must be opened.

The RabbitMQ management UI listens on port 15672. To make it accessible remotely, configure the firewall to open this port in the public zone:

sudo firewall-cmd --zone=public --add-port=15672/tcp --permanent

To facilitate internal communication among the RabbitMQ nodes while keeping access restricted to internal IPs, additional ports must be opened exclusively within the subnet 192.168.0.100/24.

Enable epmd Discovery (Port 4369)

sudo firewall-cmd --permanent --zone=public --add-rich-rule='
  rule family="ipv4"
  source address="192.168.0.100/24"
  port protocol="tcp" port="4369" accept'

Enable RabbitMQ Inter-Node Communication (Port 25672)

sudo firewall-cmd --permanent --zone=public --add-rich-rule='
  rule family="ipv4"
  source address="192.168.0.100/24"
  port protocol="tcp" port="25672" accept'

Enable CLI Tools Communication (Ports 35672-35682)

sudo firewall-cmd --permanent --zone=public --add-rich-rule='
  rule family="ipv4"
  source address="192.168.0.100/24"
  port protocol="tcp" port="35672-35682" accept'

Open AMQP Ports (5671 and 5672)

If your services use AMQP, you need to open the following ports:

sudo firewall-cmd --permanent --zone=public --add-rich-rule='
  rule family="ipv4"
  source address="192.168.0.100/24"
  port protocol="tcp" port="5672" accept'

sudo firewall-cmd --permanent --zone=public --add-rich-rule='
  rule family="ipv4"
  source address="192.168.0.100/24"
  port protocol="tcp" port="5671" accept'

Check the official RabbitMQ documentation for a full list of networking requirements if your setup uses additional protocols.

Reload Firewall Rules

Once all rules have been applied, reload firewalld to activate the changes:

sudo firewall-cmd --reload

Repeat these firewall setup steps on each server that will be part of the RabbitMQ cluster.

Installing rabbitmqadmin CLI Tool

The management extension of RabbitMQ provides a Python-based utility called rabbitmqadmin, which becomes available once the management interface is active. You can easily install this tool using the commands below:

sudo wget http://localhost:15672/cli/rabbitmqadmin
sudo mv rabbitmqadmin /usr/local/bin/
sudo chmod +x /usr/local/bin/rabbitmqadmin

Setting Up DNS for Hostname Resolution

Hostname recognition is crucial for forming RabbitMQ clusters. As servers typically lack DNS records by default, you must edit the /etc/hosts file on each server to map hostnames to local IP addresses. This ensures nodes can locate each other by name.

Below is a sample /etc/hosts configuration for a master server. Modify the IP addresses and hostnames to match your infrastructure.

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

127.0.0.1 guest
::1       guest

127.0.0.1 YOUR_MASTER_SERVER_HOST_NAME
::1       YOUR_MASTER_SERVER_HOST_NAME

192.168.0.101 YOUR_MASTER_SERVER_HOST_NAME
192.168.0.102 YOUR_MIRROR_SERVER_HOST_NAME

Clustering RabbitMQ Servers

For RabbitMQ servers to join the same cluster, their Erlang cookies must be identical. Each node generates a unique cookie by default, so it must be manually set to the same value on all machines.

You can set the Erlang cookie with the following command (feel free to use a custom value):

sudo sh -c "echo 'WE<3COOKIES' > /var/lib/rabbitmq/.erlang.cookie"

After updating the Erlang cookie, restart the RabbitMQ service to apply the changes:

sudo systemctl restart rabbitmq-server.service

On each server except the master, execute the following commands to add them to the cluster:

sudo rabbitmqctl stop_app
sudo rabbitmqctl join_cluster "rabbit@<YOUR_MASTER_SERVER_HOST_NAME>"
sudo rabbitmqctl start_app

Confirm the clustering setup with the following command:

sudo rabbitmqctl cluster_status

All participating nodes will appear under the nodes and running_nodes sections. From this point onward, configurations will automatically replicate across all nodes.

Enabling High-Availability for Queues and Exchanges

To ensure durability and fault tolerance, define a high-availability policy using either the management UI or the command line. The following example creates a policy for two-node data mirroring:

sudo rabbitmqctl set_policy -p "/" --priority 1 --apply-to "all" ha ".*" '{ "ha-mode": "exactly", "ha-params": 2, "ha-sync-mode": "automatic"}'

Explanation of the parameters:

• -p "/": Applies to the default virtual host
• --priority 1: Policy precedence
• --apply-to "all": Targets both queues and exchanges
• ha: Policy identifier
• ".*": Regex to match all resources
• JSON object: Defines mirroring to exactly 2 nodes with automatic sync

You can increase ha-params for more replicas. For best fault tolerance, aim for a quorum (N/2 + 1). Be aware that increasing replicas may degrade performance due to added load on disk, network, and CPU.

Alternative Mirroring Strategies

Mirror to all nodes:

'{ "ha-mode": "all", "ha-sync-mode": "automatic"}'

Mirror to specific nodes (e.g., node-1 and node-2):

'{ "ha-mode": "nodes", "ha-params" :["rabbit@node-1", "rabbit@node-2"], "ha-sync-mode": "automatic"}'

You can also assign rules to match specific queue names. Suppose you have the following nodes:

• rabbit@master
• rabbit@client-ha
• rabbit@product-ha

To match queues with names starting with “client” and “product” respectively:

sudo rabbitmqctl set_policy -p "/" --priority 1 --apply-to "queues" ha-client "client.*" '{ "ha-mode": "nodes", "ha-params": ["rabbit@master", "rabbit@client-ha"], "ha-sync-mode": "automatic"}'
sudo rabbitmqctl set_policy -p "/" --priority 1 --apply-to "queues" ha-product "product.*" '{ "ha-mode": "nodes", "ha-params": ["rabbit@master", "rabbit@product-ha"], "ha-sync-mode": "automatic"}'

Note: Exclusive queues, which are deleted once the client disconnects, cannot be mirrored. Even if the policy matches such queues, they will not be replicated as their temporary nature makes mirroring redundant.

Verifying the RabbitMQ Cluster Setup

To validate the clustering configuration, start by creating a new queue using the command-line interface via the management plugin:

sudo rabbitmqadmin declare queue --vhost "/" name=my-ha-queue durable=true

This command adds a durable queue named my-ha-queue to the default virtual host /.

Next, confirm that this queue is managed under the high-availability policy and replicated across nodes using:

sudo rabbitmqctl list_queues name policy state pid slave_pids

Now, publish a test message to the queue from the master server:

sudo rabbitmqadmin -u user_name -p password publish routing_key=my-ha-queue payload="hello world"

Temporarily shut down RabbitMQ on the master node to simulate a failover scenario:

sudo systemctl rabbitmqctl shutdown

Retrieve the message by connecting to the mirror node using its IP or DNS name:

sudo rabbitmqadmin -H MIRROR_NODE_IP_OR_DNS -u user_name -p password get queue=my-ha-queue

Finally, bring the master node back online:

sudo systemctl start rabbitmq-server.service

Removing the Default Guest Account

RabbitMQ ships with a pre-configured guest user. Retaining this user on a publicly accessible server is considered insecure and should be avoided.

Delete the default guest account using this command:

sudo rabbitmqctl delete_user guest

Conclusion

By following this complete setup guide, you’ve successfully configured a resilient and highly available RabbitMQ cluster on CentOS 7. With multiple nodes working together, message queues and exchanges are mirrored to ensure data durability and system reliability in case of failures. From configuring firewalls and host resolution to establishing clustering and applying HA policies, every step contributes to a robust asynchronous messaging infrastructure suitable for scalable microservice architectures.

Make sure to regularly monitor your cluster, review performance metrics, and update policies as needed to accommodate growing application demands. With proper configuration and maintenance, your RabbitMQ cluster will provide stable and secure messaging for your entire service ecosystem.