Why Not Just Cluster Across Data Centers?
RabbitMQ clustering assumes low-latency, reliable network links between nodes because Erlang distribution and consensus protocols like Raft are sensitive to latency and partitions; stretching a single cluster across geographically distant data centers over a WAN is explicitly discouraged and tends to produce constant partition churn. Federation and Shovel exist to solve the cross-data-center or cross-organization messaging problem differently: instead of nodes joining one tightly-coupled cluster, independent brokers stay fully autonomous and simply forward messages to each other over normal AMQP connections, tolerating WAN latency and even temporary disconnects gracefully.
Cricket analogy: It's like trying to run one live match umpiring system across two stadiums on different continents versus simply having each stadium run its own independent match and forwarding highlights footage to the other afterward — the latter tolerates delay far better.
Federation: Linking Exchanges and Queues
The federation plugin lets an exchange or queue on one broker (the downstream) receive messages published to a corresponding exchange or queue on another broker (the upstream), via a federation link that behaves like a persistent AMQP client connection. Federated exchanges are commonly used to fan messages out to multiple regions, for example publishing once to a topic exchange in a US data center and having a federation link automatically replicate matching messages to a federated exchange in an EU data center, all while each broker keeps its own independent queues, bindings, and policies. Federation is configured declaratively: you define an upstream (connection details to the remote broker) and a policy that matches exchange or queue names to that upstream.
Cricket analogy: It's like a franchise's US broadcast feed being picked up and rebroadcast by an EU sports network via a dedicated feed agreement, while each network still runs its own independent ad breaks and commentary teams.
# Define an upstream broker connection
rabbitmqctl set_parameter federation-upstream eu-upstream \
'{"uri":"amqp://user:[email protected]","expires":3600000}'
# Apply a policy federating exchanges matching 'orders.*' to that upstream
rabbitmqctl set_policy --apply-to exchanges federate-orders "^orders\." \
'{"federation-upstream-set":"all"}'Shovel: Point-to-Point Message Forwarding
The shovel plugin is simpler and more explicit than federation: a shovel is a configured worker that consumes messages from a specific source queue (or exchange, via a bound queue) on one broker and republishes them to a specific destination exchange on another broker, running either as a static shovel defined in configuration or as a dynamic shovel set via a runtime parameter. Shovels are ideal for one-off or well-defined data migration and bridging tasks, such as moving all orders from a legacy on-premise broker into a cloud broker during a migration window, because unlike federation they don't require matching topology on both sides — you explicitly wire a source to a destination.
Cricket analogy: It's like a courier physically carrying match footage from a stadium with no broadcast feed to a TV studio that will air it, rather than relying on an automatic broadcast relay — you're explicitly wiring one specific source to one specific destination.
Use federation when you need ongoing, topology-aware replication between brokers that should stay loosely coupled but continuously synchronized (e.g., multi-region fanout). Use shovel when you need an explicit, one-directional pipe from a known source to a known destination, especially for migrations or bridging dissimilar topologies.
Neither federation nor shovel provides exactly-once delivery across the link — both are typically configured with at-least-once semantics, so consumers on the destination side must be idempotent to safely handle occasional duplicate deliveries after a reconnect.
- RabbitMQ clustering is not designed for high-latency WAN links between data centers.
- Federation links an exchange or queue on one broker to a corresponding one on another via a persistent AMQP connection.
- Federation is configured with an upstream definition plus a policy matching exchange or queue names.
- Shovel explicitly wires a specific source queue/exchange to a specific destination exchange on another broker.
- Shovels can be static (config file) or dynamic (runtime parameter) and suit migrations and bridging.
- Federation suits ongoing multi-broker replication; shovel suits explicit point-to-point forwarding.
- Both federation and shovel typically provide at-least-once delivery, requiring idempotent consumers.
Practice what you learned
1. Why is standard RabbitMQ clustering discouraged across geographically distant data centers?
2. How is a federation link established between two brokers?
3. What is a key difference between shovel and federation?
4. What delivery guarantee do federation and shovel typically provide across the link?
Was this page helpful?
You May Also Like
RabbitMQ Clustering Basics
Learn how RabbitMQ nodes join together to form a cluster, how metadata and queue state are shared, and what happens when nodes fail.
Monitoring RabbitMQ
Learn the key metrics, tools, and alerting strategies for keeping a RabbitMQ cluster observable and catching problems before they cause outages.
RabbitMQ Performance Tuning
Practical techniques for maximizing RabbitMQ throughput and minimizing latency, covering prefetch, publisher confirms, queue type choice, and memory/disk alarms.