Draining a Redis Cluster Node Before Removal

You need to retire a primary from a live Redis Cluster — the instance is oversized, the host is being recycled, or a scale-in event is reclaiming capacity — and the one outcome you cannot accept is losing keys. A primary that still owns hash slots holds the only authoritative copy of every key mapped to those slots, so tearing it down or calling CLUSTER FORGET while it is still serving data drops those keys and returns CLUSTERDOWN or MOVED errors to clients mid-request. Draining is the ordered procedure that makes removal safe: migrate every slot the node owns onto the remaining primaries, detach or promote its replicas, confirm the node holds zero slots, and only then forget and deprovision it. This page walks that sequence against Redis 7.x with redis-cli and shows how to automate the fragile CLUSTER FORGET broadcast in Python so no surviving node is left with a stale view of the departed one. It is the removal half of automated node provisioning; the additive path is covered in rebalancing cluster slots after adding a node.

# Prerequisites

• A Redis 7.x cluster with at least three primaries after the target is removed — a Redis cluster cannot drop below three healthy primaries and stay operational.
• redis-cli built from the same major version as the server, reachable from a host that can talk to every node's cluster bus port (client port + 10000).
• redis-py 5.x on Python 3.10+ (pip install "redis>=5,<6") for the automated forget-broadcast.
• Enough free memory on the remaining primaries to absorb the drained node's slots and their keys without tripping maxmemory.
• The node's id and address, plus credentials with permission to run CLUSTER subcommands and --cluster reshard / rebalance.
• A maintenance understanding that draining moves data over the wire; run it during a low-write window when practical, coordinating with any in-flight zero-downtime slot migration.

# Step-by-Step Drain and Removal

# 1. Identify the target node's id

Every cluster operation addresses a node by its 40-character id, not its IP, because addresses can change while the id is stable for the life of the node. Read the topology and record the id, slot count, and replica links of the node you intend to remove.

# List all nodes; the target line shows its id, role, and owned slot ranges.
redis-cli -h 10.0.1.11 -p 6379 CLUSTER NODES | grep -E 'myself|master'
# Example row for the drain target (slots 0-5460 owned, one replica attached):
# 3f7a...c19  10.0.1.11:6379@16379 master - 0 1710160000000 4 connected 0-5460

# 2. Handle the target's replicas first

A primary being drained usually has one or more replicas pinned to it; if you forget the primary while a replica still points at it, that replica is orphaned and will not be promoted. Either promote a replica to take over the slots (then you are draining the old primary as a now-empty node) or, more commonly, reattach the replica to a primary you are keeping so replica coverage is preserved. Reassign it explicitly with CLUSTER REPLICATE against the replica, naming a surviving primary's id.

# Run ON the replica node; point it at a primary you are keeping (id ec2b...).
redis-cli -h 10.0.1.21 -p 6379 CLUSTER REPLICATE ec2b9d4f7a1c5e8b0a2d6f3c9e11ab77c4d20e56
# Verify the replica now follows the new primary, not the drain target.
redis-cli -h 10.0.1.21 -p 6379 CLUSTER NODES | grep slave

# 3. Migrate every slot off the node

With replicas handled, move the slots. For a full drain the cleanest command is rebalance with a weight of zero, which tells Redis to redistribute the target's slots across the other primaries proportionally in one pass. Use reshard when you want to steer all slots to one specific destination instead.

# Weight 0 drains the target across all remaining primaries automatically.
redis-cli --cluster rebalance 10.0.1.11:6379 \
  --cluster-weight 3f7a...c19=0 --cluster-yes

# Alternative: send all N slots to one named destination primary.
redis-cli --cluster reshard 10.0.1.11:6379 \
  --cluster-from 3f7a...c19 \
  --cluster-to   ec2b...e56 \
  --cluster-slots 5461 --cluster-yes

# 4. Confirm zero slots remain

Draining is not complete until the node owns no slots. Re-read CLUSTER NODES and assert the target's line ends with no slot ranges; a lingering range means the reshard was interrupted and must be re-run before you proceed. The same slot-accounting discipline underpins rebalancing cluster slots after adding a node.

# The target row must now show 'connected' with NO trailing slot range.
redis-cli -h 10.0.1.11 -p 6379 CLUSTER NODES | grep 3f7a
# Cross-check the cluster still owns all 16384 slots in aggregate.
redis-cli --cluster check 10.0.1.11:6379 | grep -E 'slots|covered'

# 5. Broadcast CLUSTER FORGET from every remaining node

CLUSTER FORGET is node-local: it only removes the target from the node you send it to, and it installs a 60-second ban that stops that node from re-adding the target through gossip. To evict the node cluster-wide you must send FORGET to every surviving node inside that window — miss one and gossip re-propagates the target back to the nodes that already forgot it. Automate the fan-out so it completes well under 60 seconds. Never send FORGET to the target itself.

import asyncio
import redis.asyncio as redis

TARGET_ID = "3f7ac19...c19"  # 40-char id of the node being removed
SURVIVORS = [                # every node that will REMAIN in the cluster
    ("10.0.1.12", 6379), ("10.0.1.13", 6379),
    ("10.0.1.21", 6379), ("10.0.1.22", 6379),
]

async def forget_on(host: str, port: int) -> str:
    client = redis.Redis(host=host, port=port, decode_responses=True)
    try:
        # FORGET is idempotent per node; a node that never knew the target
        # raises "Unknown node", which we treat as already-forgotten.
        await client.execute_command("CLUSTER", "FORGET", TARGET_ID)
        return f"{host}: forgotten"
    except redis.ResponseError as exc:
        return f"{host}: {exc}"
    finally:
        await client.aclose()

async def broadcast_forget() -> None:
    # Fan out concurrently so the whole cluster forgets inside the 60s ban.
    results = await asyncio.gather(*(forget_on(h, p) for h, p in SURVIVORS))
    for line in results:
        print(line)

asyncio.run(broadcast_forget())

# 6. Shut down and deprovision the node

Once every survivor has forgotten the target and the 60-second ban has held, the node is isolated — it owns no slots, has no replicas, and no peer will gossip it back. Shut the process down cleanly so it flushes any pending state, then release the underlying host. Wiring this final step into your infrastructure-as-code closes the loop with automating node scaling with Terraform and Ansible.

# Graceful shutdown on the drained node, then destroy the machine.
redis-cli -h 10.0.1.11 -p 6379 SHUTDOWN NOSAVE
# e.g. terraform destroy -target=aws_instance.redis_node_4  (or ansible teardown)

# Failure Modes

Forgetting or shutting down before slots are drained. If you call CLUSTER FORGET or SHUTDOWN while the target still owns slots, those slots become uncovered and every key mapped to them is lost; clients hitting them get CLUSTERDOWN Hash slot not served. Diagnose by checking slot coverage the instant symptoms appear:

redis-cli --cluster check 10.0.1.12:6379 | grep -i 'not covered\|slots'
# "[ERR] Not all 16384 slots are covered" confirms a premature removal.

Fix by restoring the node if it is still alive (restart it and let gossip re-attach, then re-run the drain from Step 3), or by re-sharding the orphaned range onto a healthy primary and reloading the lost keys from your source of truth.

Incomplete FORGET broadcast (node reappears). You forgot the target on most nodes but missed one; that node keeps gossiping the target, and within seconds every node that forgot it re-learns it, undoing the removal. Diagnose by looking for the id reappearing after you thought it was gone:

# Run against several nodes; the target id must appear on NONE of them.
for h in 10.0.1.12 10.0.1.13 10.0.1.21 10.0.1.22; do
  redis-cli -h $h -p 6379 CLUSTER NODES | grep -c 3f7a
done   # every line must print 0

Fix by re-running the Python broadcast so the fan-out reaches all survivors together, then shutting the target down immediately so it cannot gossip itself back during the ban window.

Orphaned replica after removal. The target's replica was never reassigned, so when the primary leaves, the replica is left following a node the Redis cluster has forgotten and provides no coverage. Diagnose with:

redis-cli -h 10.0.1.21 -p 6379 CLUSTER NODES | grep slave
# A slave line whose master id is the forgotten target is orphaned.

Fix by pointing it at a surviving primary with CLUSTER REPLICATE <surviving-primary-id> (Step 2), which you should have done before draining.

# Verification

Confirm the Redis cluster owns all 16,384 slots and reports a healthy state with the target gone:

redis-cli --cluster check 10.0.1.12:6379 | grep -E 'covered|All 16384'
redis-cli -h 10.0.1.12 -p 6379 CLUSTER INFO | grep -E 'cluster_state|cluster_known_nodes'
# Expect: cluster_state:ok and cluster_known_nodes reduced by exactly one.

Assert no surviving node still references the removed id, and that the drained node itself is unreachable:

redis-cli -h 10.0.1.12 -p 6379 CLUSTER NODES | grep -c 3f7a   # expect 0
redis-cli -h 10.0.1.11 -p 6379 PING                            # expect a connection error

Finally, read a sampling of keys that lived on the drained slots to prove no data was lost in the move — every key should now resolve through a MOVED redirect to its new owner and return its value.

# FAQ

# What happens if I remove the node before draining its slots?

The keys mapped to the slots that node still owned are lost, and the Redis cluster enters a degraded state. Because a primary holds the only authoritative copy of its slots' keys (replicas aside), calling CLUSTER FORGET or SHUTDOWN on a slot-owning node leaves those slots uncovered, and any client request that hashes to them returns CLUSTERDOWN Hash slot not served until you re-shard the range onto a live primary and repopulate it from your database. Always drive the target's owned-slot count to zero, verified with redis-cluster check, before you forget it.

# Why must CLUSTER FORGET run on every node within 60 seconds?

CLUSTER FORGET acts only on the single node you send it to and installs a 60-second ban that prevents that node from re-adding the target via the gossip protocol. That ban is what stops a half-completed removal from healing itself: if you forget the target on four of five nodes, the fifth keeps advertising it, and once any node's ban expires it re-learns the target from that fifth node. Broadcasting FORGET to all remaining nodes inside the 60-second window ensures every node is simultaneously banning the target, so gossip has no surviving source to re-propagate it before the node is shut down.

# Do I really need to deal with replicas before draining the primary?

Yes. A replica pinned to the primary you are removing must be reassigned to a surviving primary (or promoted) first, because forgetting a primary while a replica still follows it orphans that replica — it keeps pointing at a node the rest of the Redis cluster no longer knows and contributes no failover coverage. Reassigning with CLUSTER REPLICATE before you drain preserves your replication factor throughout the operation, so the Redis cluster never briefly runs a primary with no standby.

# How do I automate draining in Terraform or Ansible?

Treat the ordered steps here as the provider or playbook's teardown hook rather than letting the tool destroy the instance directly. In practice a terraform destroy on a Redis node should trigger a pre-destroy provisioner (or an Ansible pre-task) that runs the drain: reassign replicas, rebalance --cluster-weight <id>=0, verify zero slots, then execute the Python FORGET broadcast against the surviving nodes — only after that returns success does the machine get torn down. The full pattern for wiring these lifecycle hooks into declarative infrastructure is covered in automating node scaling with Terraform and Ansible.

---

Up: Automated Node Provisioning & Removal

# Related

• Automating Node Scaling with Terraform and Ansible
• Rebalancing Cluster Slots After Adding a Node
• Zero-Downtime Slot Migration
• Redis Cluster Slot Allocation Basics