Automated Node Provisioning & Removal in Redis Cluster

Modern distributed caching architectures treat Redis nodes as ephemeral compute units rather than permanent infrastructure fixtures. Manual redis-cli --cluster invocations during traffic spikes or capacity contraction introduce unacceptable operational risk. Production-grade environments require deterministic orchestration, atomic slot reallocation, and strict failure boundary enforcement. The architectural paradigm outlined in Redis Cluster Scaling, Sharding & Automation decouples compute provisioning from storage topology, enabling infrastructure pipelines to scale horizontally while preserving read/write continuity.

This guide details the operational playbook for automated node lifecycle management, covering IaC bootstrapping, gossip validation, deterministic slot migration, telemetry-driven triggers, and controlled decommissioning.

flowchart LR
    PROV[Provision node] --> JOIN[Cluster handshake / gossip]
    JOIN --> VAL{cluster_state ok?}
    VAL -->|no| JOIN
    VAL -->|yes| RESH[Reshard slots in]
    RESH --> SERVE[Serve traffic]
    SERVE --> DRAIN[Drain: reshard slots out]
    DRAIN --> DEL[del-node]

# Phase 1: Infrastructure-as-Code Bootstrapping

Provisioning begins with declarative compute allocation and strict configuration enforcement. Terraform defines the instance footprint, while Ansible handles Redis bootstrap. The automation controller must inject non-negotiable cluster parameters before the service starts:

# ansible/roles/redis-cluster/templates/redis.conf.j2
cluster-enabled yes
cluster-config-file nodes-${ansible_fqdn}.conf
cluster-node-timeout 5000
cluster-migration-barrier 1
cluster-announce-ip {{ hostvars[inventory_hostname]['ansible_default_ipv4']['address'] }}
cluster-announce-port 6379
cluster-announce-bus-port 16379
save ""
appendonly yes

The cluster-announce-ip and cluster-announce-bus-port directives are critical in cloud environments with NAT or VPC routing. Misconfiguration here causes gossip fragmentation and split-brain conditions. Once Ansible converges, the automation layer validates the bootstrap sequence before proceeding to topology integration. For a complete IaC implementation pattern, refer to Automating Node Scaling with Terraform and Ansible.

# Phase 2: Gossip Integration & Topology Validation

After service startup, the node enters the handshake state and begins participating in the gossip protocol. The orchestrator must verify cluster health before assigning hash slots:

# Validate cluster state and gossip convergence
redis-cli -h <new-node-ip> -p 6379 CLUSTER INFO | grep -E "cluster_state|cluster_known_nodes"
redis-cli -h <new-node-ip> -p 6379 CLUSTER NODES | grep -E "myself|master|slave"

The orchestrator should poll cluster_state:ok and ensure cluster_known_nodes matches the expected topology. Only after gossip convergence can the controller safely query slot distribution. Understanding Redis Cluster Slot Allocation Basics is mandatory here: the automation must calculate exact slot deltas to maintain hash ring equilibrium without exceeding the 16384-slot ceiling.

# Phase 3: Deterministic Slot Reallocation

Slot migration is the operational bottleneck. The orchestrator must execute a strict state machine to move slots from overloaded primaries to the new node:

1. Mark Target as Importing: CLUSTER SETSLOT <slot> IMPORTING <source_node_id> (the destination must be ready first)
2. Mark Source as Migrating: CLUSTER SETSLOT <slot> MIGRATING <target_node_id>
3. Stream Keys: MIGRATE <target_ip> <target_port> "" 0 5000 KEYS <key1> <key2> ...
4. Atomic Handoff: CLUSTER SETSLOT <slot> NODE <target_node_id> (on both nodes)

The MIGRATE command must use COPY and REPLACE flags to prevent data loss during transient failures. The orchestrator tracks migrating and importing states via CLUSTER SLOTS output. If a migration stalls due to large key serialization or network jitter, the controller implements exponential backoff and adjusts MIGRATE timeout thresholds before aborting. A comprehensive breakdown of this atomic flow is documented in Zero-Downtime Slot Migration.

# Phase 4: Telemetry-Driven Scaling Triggers

Automated scaling must react to real-time metrics, not static schedules. The controller subscribes to Prometheus-exported Redis metrics and OpenTelemetry traces:

During extreme load events, the orchestrator must prioritize slot migration over replica synchronization to prevent cascading latency. The playbook for high-velocity capacity expansion is detailed in Scaling Redis Nodes During Black Friday Traffic.

# Phase 5: Controlled Node Decommissioning

Node removal is inherently riskier than provisioning. The orchestrator must drain slots, respect cluster-migration-barrier, and ensure replica failover completes before terminating the instance:

# 1. Drain the node: migrate all of its slots to remaining primaries while it is
#    still a primary (resharding moves slots; it does not promote/demote nodes).
redis-cli --cluster reshard <any-cluster-node>:6379 --cluster-from <node-id-to-remove> --cluster-to <target-node-id> --cluster-slots <count> --cluster-yes

# 2. Verify the node now owns zero slots
redis-cli -h <node-to-remove> -p 6379 CLUSTER NODES | grep myself

# 3. Remove the now-empty node from the cluster topology
redis-cli --cluster del-node <any-cluster-node>:6379 <node-id-to-remove>

In containerized environments, the orchestrator must coordinate with the Kubernetes control plane to prevent premature pod termination. The exact sequence for graceful eviction is covered in Zero-Downtime Node Removal in Kubernetes.

# Production Python Orchestrator

The following Python module implements a deterministic scaling controller using redis-py, structured logging, and exponential backoff. It handles slot migration state transitions, observability hooks, and failure boundary enforcement.

import time
import logging
from typing import List, Dict, Optional
from redis.cluster import RedisCluster, ClusterNode
from redis.exceptions import RedisError, ConnectionError

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger("redis_cluster_orchestrator")

class RedisClusterScaler:
    def __init__(self, startup_nodes: List[Dict[str, str]], max_retries: int = 5, base_delay: float = 1.0):
        # startup_nodes expects ClusterNode objects, not {"host": ..., "port": ...} dicts.
        nodes = [ClusterNode(n["host"], int(n["port"])) for n in startup_nodes]
        self.client = RedisCluster(startup_nodes=nodes, decode_responses=True, socket_timeout=5)
        self.max_retries = max_retries
        self.base_delay = base_delay

    def _execute_with_backoff(self, func, *args, **kwargs):
        for attempt in range(self.max_retries):
            try:
                return func(*args, **kwargs)
            except (ConnectionError, RedisError) as e:
                delay = self.base_delay * (2 ** attempt)
                logger.warning(f"Attempt {attempt+1} failed: {e}. Retrying in {delay}s...")
                time.sleep(delay)
        raise RuntimeError("Max retries exceeded for cluster operation")

    def migrate_slots(self, source_id: str, target_id: str,
                      target_host: str, target_port: int,
                      slots: List[int], timeout_ms: int = 5000):
        # Pass the target host/port explicitly. get_node() resolves by host/port,
        # not by the 40-char cluster node id, so it cannot look up `target_id`.
        for slot in slots:
            logger.info(f"Migrating slot {slot} from {source_id} to {target_id}")

            # Step 1: Mark states. IMPORTING on the target MUST precede MIGRATING
            # on the source, or the source's -ASK redirects reach a target that is
            # not yet importing and bounce the client back with -MOVED.
            self._execute_with_backoff(self.client.execute_command, "CLUSTER", "SETSLOT", slot, "IMPORTING", source_id)
            self._execute_with_backoff(self.client.execute_command, "CLUSTER", "SETSLOT", slot, "MIGRATING", target_id)

            # Step 2: Stream keys. Pass arguments explicitly — splitting a formatted
            # string mangles the empty auth token ("") and any key with whitespace.
            keys = self._get_keys_in_slot(slot)
            if keys:
                self._execute_with_backoff(
                    self.client.execute_command,
                    "MIGRATE", target_host, target_port, "", 0, timeout_ms, "KEYS", *keys,
                )

            # Step 3: Atomic ownership handoff
            self._execute_with_backoff(self.client.execute_command, "CLUSTER", "SETSLOT", slot, "NODE", target_id)
            logger.info(f"Slot {slot} successfully migrated.")

    def _get_keys_in_slot(self, slot: int, count: int = 100) -> List[str]:
        # Production implementations should use CLUSTER GETKEYSINSLOT or SCAN with slot filtering
        try:
            return self.client.execute_command("CLUSTER", "GETKEYSINSLOT", slot, count)
        except Exception as e:
            logger.error(f"Failed to fetch keys for slot {slot}: {e}")
            return []

    def validate_cluster_health(self) -> bool:
        info = self.client.cluster_info()
        state = info.get("cluster_state", "fail")
        known_nodes = int(info.get("cluster_known_nodes", 0))
        logger.info(f"Cluster state: {state} | Known nodes: {known_nodes}")
        return state == "ok"

# Usage Example:
# scaler = RedisClusterScaler(startup_nodes=[{"host": "10.0.1.10", "port": 6379}])
# if scaler.validate_cluster_health():
#     scaler.migrate_slots("source_node_id", "target_node_id", "10.0.1.20", 6379, list(range(1000, 1050)))

# Operational Guardrails

1. Never exceed 16384 total slots. The orchestrator must validate slot sums before committing migrations.
2. Respect cluster-migration-barrier. Ensure at least one replica remains online for every primary during rebalancing.
3. Enforce idempotency. All provisioning and removal scripts must be safe to re-run. Use CLUSTER NODES snapshots as state anchors.
4. Integrate with service mesh. Route traffic away from draining nodes using Istio/Envoy weight adjustments before executing CLUSTER SETSLOT.

Automated Redis Cluster scaling is not a feature toggle; it is a reliability engineering discipline. By combining deterministic IaC, atomic slot migration, and telemetry-driven triggers, backend teams can scale caching infrastructure predictably under any load profile.