Zero-Downtime Slot Migration: Production Playbook for Horizontal Redis Scaling

This page covers how to move Redis hash slots between nodes while live traffic keeps flowing — comparing manual state-machine orchestration against the automated redis-cli --cluster tooling, and the production signals that decide which one you reach for.

Horizontal scaling of a Redis deployment is fundamentally about redistributing the 16,384 hash slots across a new topology without violating latency SLAs or dropping in-flight requests. When backend and DevOps teams push past a node's memory or throughput ceiling, ownership of a slot range must transfer incrementally while every routing guarantee stays intact. Because Redis maps each key to a slot with a deterministic CRC16(key) % 16384, the correctness of a migration hinges entirely on the slot-to-node allocation model: get the handoff sequence wrong and clients see stale routing, ASK storms, or — worst case — a slot owned by two primaries at once. Treating redistribution as a continuous, observable workflow is the operational core of Redis Cluster Scaling, Sharding & Automation, where automation pipelines enforce idempotency and telemetry drives every go/no-go decision inside a live traffic window.

# Protocol semantics: the state machine you cannot skip

Migration is governed by a strict distributed state machine. The source node marks the target range MIGRATING; the destination marks it IMPORTING. Keys move one at a time via the MIGRATE command, which is atomic per key — the key exists on exactly one node at every instant. During the transition, a client asking for a key that has already moved receives an ASK redirect, telling it to query the destination for that one operation only. This is categorically different from a MOVED redirect: MOVED signals a permanent ownership change and updates the client's cached slot map, whereas ASK is transient and requires the client to send an ASKING command before retrying the original command against the destination.

A client library that ignores ASK semantics will surface elevated latency, connection resets, or outright data-access failures during the window. The handoff order is non-negotiable — the destination must be readied before the source begins redirecting, or a client can be pointed at a node that is not yet importing:

# Architectural trade-offs: three ways to move a slot

There are three viable ways to drive the state machine above in production. Manual CLUSTER SETSLOT + MIGRATE gives you per-key control and is the only option when you need to move individual hot keys or script custom batching. The redis-cli --cluster reshard utility wraps that sequence for a fixed slot count between two named nodes. redis-cli --cluster rebalance computes target weights across the whole cluster and issues the moves for you. They differ sharply in blast radius and operator burden.

The rows share the same underlying protocol; what changes is how much of the sequencing, retry, and finalization logic you hand to the tool versus keep in your own orchestrator.

# Approach A — manual state-machine orchestration

Reach for manual orchestration when you need surgical control: moving a single hot key or hash-tagged key group off an overloaded node, or scripting a migration whose batching must respect application-level backpressure. You drive each transition explicitly. The critical ordering rule: set IMPORTING on the destination first, MIGRATING on the source second, stream the keys, then assign the new owner on the destination and the source before gossip converges.

import redis  # redis-py 5.x

def migrate_slot(slot: int, src: redis.Redis, dst: redis.Redis,
                 dst_id: str, src_id: str, dst_host: str, dst_port: int) -> None:
    """Move one slot from src to dst with the correct handoff ordering."""
    # 1. Ready the destination BEFORE the source starts redirecting.
    dst.execute_command("CLUSTER", "SETSLOT", slot, "IMPORTING", src_id)
    src.execute_command("CLUSTER", "SETSLOT", slot, "MIGRATING", dst_id)

    # 2. Drain the slot in bounded batches so a single MIGRATE never blocks
    #    the source event loop for long. COUNT caps keys per call.
    while True:
        keys = src.execute_command("CLUSTER", "GETKEYSINSLOT", slot, 100)
        if not keys:
            break
        # Atomic, per-key move. TIMEOUT is ms; REPLACE guards a partial retry.
        src.execute_command(
            "MIGRATE", dst_host, dst_port, "", 0, 5000,
            "REPLACE", "KEYS", *keys,
        )

    # 3. Assign the new owner on BOTH nodes, then let gossip propagate.
    #    Destination first so it accepts writes before the source stops.
    dst.execute_command("CLUSTER", "SETSLOT", slot, "NODE", dst_id)
    src.execute_command("CLUSTER", "SETSLOT", slot, "NODE", dst_id)

Two subtleties bite teams here. First, CLUSTER SETSLOT <slot> STABLE only clears the transient MIGRATING/IMPORTING flags — it does not transfer ownership, so use it to abort a stalled move, never to finish one. Second, the empty-key/REPLACE form of MIGRATE is what lets a retried batch succeed idempotently after a network blip, rather than failing on a key that already landed.

# Approach B — automated reshard and rebalance

For routine capacity changes — adding a node during automated node provisioning, or evening out slots after a scale event — the packaged tooling is safer and faster to operate because it handles finalization and per-slot ordering internally. Run migrations in controlled batches (typically 100–500 slots per iteration) sized to your key distribution and network bandwidth. The --cluster-pipeline flag batches MIGRATE commands to saturate throughput while respecting TCP backpressure.

#!/usr/bin/env bash
set -euo pipefail

SOURCE_NODE="10.0.1.10:6379"
DEST_NODE="10.0.1.20:6379"
SLOT_COUNT=256

echo "Validating cluster state before migration..."
redis-cli --cluster check "${SOURCE_NODE}"

echo "Migrating ${SLOT_COUNT} slots to ${DEST_NODE}..."
redis-cli --cluster reshard "${SOURCE_NODE}" \
  --cluster-from "$(redis-cli -h 10.0.1.10 -p 6379 CLUSTER MYID)" \
  --cluster-to   "$(redis-cli -h 10.0.1.20 -p 6379 CLUSTER MYID)" \
  --cluster-slots "${SLOT_COUNT}" \
  --cluster-yes \
  --cluster-pipeline 10000

echo "Migration batch complete. Verifying slot ownership..."
redis-cli --cluster check "${SOURCE_NODE}"

To spread load across every primary in one pass rather than a single node pair, redis-cli --cluster rebalance <any-node> --cluster-use-empty-masters computes target weights and issues all the moves. It is the lowest-effort option but the bluntest: on a skewed keyspace it can over-move slots and briefly amplify write copy volume, so gate it behind the same telemetry you use for a manual run. Wrap either command in a retry-aware orchestrator so a transient stall triggers a clean re-run instead of leaving partial state.

# When to choose which

Tie the decision to concrete production signals rather than preference:

1. Moving specific hot keys or a hash-tag group → manual SETSLOT + MIGRATE. It is the only approach that operates below slot granularity and lets you throttle per batch against your own metrics.
2. Adding or draining one node with a known slot count → redis-cli --cluster reshard. One node pair, bounded blast radius, automatic finalization. This is the default for scripted scale-up.
3. Cluster-wide skew after several topology changes → redis-cli --cluster rebalance, run in a low-traffic window. Accept the coarser control in exchange for a single command.
4. Dataset per slot is large (>10 GB node, or any slot holding >15% of a node's memory) → shrink batches to ≤200 slots and prefer manual orchestration so a single MIGRATE of a multi-megabyte value never stalls the source event loop past cluster-node-timeout.
5. Consistency SLA forbids any redirect-induced error → whichever approach you pick, pair it with a modern cluster-aware client (below) and canary a 50-slot batch before committing the full move.

# Pre-migration configuration and failure boundaries

Before any transfer, tune the parameters that define your failure boundaries, then inject them through your provisioning templates so a new node joins the gossip ring already configured. The cluster-node-timeout parameter dictates how long a node may be unreachable before the Redis cluster triggers failover; bulk key transfers inflate round-trip times, so temporarily raising it to 15000–20000 ms buys a safety buffer without blinding partition detection. Calibrate repl-backlog-size to absorb the replication surge as the destination synchronizes, and raise client-output-buffer-limit for cluster nodes so a redirect storm does not trip an OOM kill. Set cluster-require-full-coverage no so the deployment keeps serving unaffected slots even if one slot is briefly blocked mid-migration.

# Failure modes and diagnostics

Three failure modes account for most stalled or degraded migrations.

ASK redirect storm. A client that does not send ASKING before retrying against the destination will loop on redirects, exhausting its connection pool and driving p99 latency up. Diagnose by watching redirect and rejection rates:

redis-cli -h 10.0.1.20 -p 6379 INFO stats | grep -E "total_reads_processed|rejected_connections"
redis-cli --cluster check 10.0.1.10:6379   # look for slots stuck in [migrating]/[importing]

Half-migrated / orphaned slot. If an orchestrator dies between the final SETSLOT NODE on the destination and the source, one node still advertises MIGRATING while the other claims ownership — clients ping-pong between them. Detect the mismatch by comparing ownership across nodes; resolve by re-issuing SETSLOT NODE <dst_id> on the lagging node (never STABLE, which would abandon the transfer):

redis-cli -h 10.0.1.10 -p 6379 CLUSTER NODES | grep -E "migrating|importing"

MIGRATE timeout on a large value. A single multi-megabyte key can exceed the MIGRATE TIMEOUT and block the source event loop, cascading into a false failover. Find the offenders before you start and split the slot into smaller batches:

redis-cli --bigkeys
redis-cli -h 10.0.1.10 -p 6379 MEMORY USAGE session:whale-tenant

# Client-side resilience

Backend services must ride out topology shifts without restarts. The official redis-py cluster client implements both MOVED and ASK handling natively — it issues ASKING automatically on redirect — but production deployments still need explicit retry and pool tuning so a redirect burst degrades gracefully.

from redis.cluster import RedisCluster, ClusterNode
from redis.retry import Retry
from redis.backoff import ExponentialBackoff
from redis.exceptions import ConnectionError, TimeoutError

nodes = [ClusterNode("10.0.1.10", 6379), ClusterNode("10.0.1.20", 6379)]

client = RedisCluster(
    startup_nodes=nodes,
    decode_responses=True,
    retry=Retry(ExponentialBackoff(), retries=5),
    retry_on_error=[ConnectionError, TimeoutError],
    read_from_replicas=True,
    cluster_error_retry_attempts=3,   # bounds MOVED/ASK follow-ups per command
    socket_timeout=2.0,
    socket_connect_timeout=1.0,
)

def get_user_session(user_id: str) -> dict:
    try:
        return client.hgetall(f"session:{user_id}")
    except Exception as e:  # routing exhausted — surface, do not silently drop
        raise RuntimeError(f"Cluster routing exhausted for session:{user_id}") from e

Monitor routing exceptions to catch a slot map that has drifted out of sync with the live topology, and keep socket_timeout low enough that a redirect follow-up cannot pin a worker thread through the whole gossip-convergence delay.

# Observability and telemetry

Blind migration is an operational liability. A redis_exporter scrape into Prometheus gives real-time visibility into migration velocity, ownership drift, and redirect rates.

# prometheus.yml snippet
scrape_configs:
  - job_name: 'redis-cluster'
    static_configs:
      - targets: ['10.0.1.10:9121', '10.0.1.20:9121']
    metrics_path: /scrape
    params:
      redis.addr: ['redis://10.0.1.10:6379', 'redis://10.0.1.20:6379']

Key PromQL for tracking a migration:

# A gap between assigned and healthy slots means migrating or failed slots
redis_cluster_slots_assigned - redis_cluster_slots_ok

# Total coverage must hold at 16384 for the whole cluster
redis_cluster_slots_assigned != 16384

# Connection rejections during a redirect storm
rate(redis_client_rejected_connections_total[5m]) > 10

Alert when redis_cluster_slots_assigned != redis_cluster_slots_ok persists for more than ten minutes — that is the signature of a stalled migration degrading cluster health.

# Verification

After each batch, confirm the topology converged before moving on or decommissioning the old node:

# Every slot covered, no errors, one owner per slot
redis-cli --cluster check 10.0.1.10:6379

# Ownership map, sorted — no slot should appear under two masters
redis-cli CLUSTER SLOTS

# Gossip has converged when message counters stabilize within ~5-10s
redis-cli -h 10.0.1.10 -p 6379 CLUSTER INFO | grep -E "cluster_state|cluster_slots_ok|messages"

Coordinate cache coherence alongside the topology check. Keep a ~10% TTL overlap during the window so brief routing uncertainty resolves to a refresh rather than a stale read; use SCAN with COUNT 100 to verify key presence on the destination; and avoid bulk DEL/UNLINK sweeps mid-migration, since they contend for I/O with the MIGRATE pipeline. Hold a 24-hour observation window on connection-pool utilization and replica sync lag before retiring legacy nodes.

For a full command-by-command runbook with CI/CD gating, follow the step-by-step Redis Cluster slot migration guide.

---

Up: Redis Cluster Scaling, Sharding & Automation

# Related

• Redis Cluster Slot Allocation Basics — how CRC16 maps keys to the 16,384 slots you are moving.
• Automated Node Provisioning & Removal — inject migration-safe config as nodes join and leave.
• Step-by-Step Redis Cluster Slot Migration Guide — the full production runbook and pipeline gates.
• TTL vs Explicit Invalidation — coordinate cache coherence across the migration window.