Redis High Availability & Failover Automation

A cache that vanishes when a single node dies is not a cache — it is a latent outage waiting for a hardware fault, a rolling deploy, or a network partition to expose it. High availability is the discipline of keeping Redis answering reads and accepting writes through the loss of any one node, without leaking split-brain writes, serving stale reads off a lagging replica, or letting a failover collapse into a reconnect storm that buries the survivor.

This reference ties the availability decisions together — which failover model detects and heals a dead primary, how replication both protects data and scales reads, and how the client behaves in the seconds when the topology is in flux — so backend engineers, caching specialists, Python developers, and DevOps teams converge on one coherent posture instead of three that fight each other under load. Each section states the concept, shows canonical redis-py 5.x (redis.asyncio) or redis-cli code, and links to a deeper page where the trade-offs are actually resolved.

The three concerns below are not independent switches. The failover model decides who becomes primary and how fast; replication decides what data the new primary starts with and how much read traffic the replicas can absorb; and the client decides how gracefully the application rides through the promotion window. A weak choice in one surfaces as a symptom in another — an aggressive failover with no client-side backoff turns a two-second promotion into a self-inflicted thundering herd, and a fast client with a lagging replica cheerfully serves data that is minutes out of date.

# Failover Models — Sentinel, Cluster, and Managed

The failover model is the mechanism that notices a primary is gone and appoints a successor. Redis Sentinel vs Cluster Failover is where this decision is resolved, but the shape of the choice matters here because it colors everything downstream. Sentinel runs a separate fleet of monitoring processes that gossip about the health of one or more primaries; when a configured quorum of them independently mark a primary as subjectively down, they elect a leader Sentinel to run the promotion and rewrite each replica's REPLICAOF target. Redis Cluster folds detection and failover into the data nodes themselves: every primary owns a slice of the 16,384 hash slots, each has its own replicas, and the primaries vote among themselves to promote a replica when one of their peers stops answering cluster-bus pings. Managed offerings hide both behind a stable endpoint but still expose the same failure surface — a promotion window during which some writes may be lost.

The critical variable is not which system detects the fault but how many independent voters must agree before a promotion happens, because that number is exactly what prevents two primaries from existing at once. With Sentinel, that agreement is expressed as the quorum value and confirmed by querying the fleet directly:

# Ask each Sentinel what it currently believes about the monitored primary.
redis-cli -h sentinel-a -p 26379 SENTINEL master cache-primary | \
  grep -E "name|ip|port|flags|num-slaves|quorum|num-other-sentinels"

# A healthy quorum: num-other-sentinels is (fleet size - 1) and flags shows
# only "master" — never "s_down" or "o_down" during normal operation.
redis-cli -h sentinel-a -p 26379 SENTINEL ckquorum cache-primary

Set the quorum below a strict majority and a network partition can let a minority of Sentinels promote a replica while the original primary keeps taking writes on the other side of the split — the classic split-brain. Set it too high and a genuine single-node failure never reaches the threshold, leaving you with an outage no automation will heal. The safe rule is a quorum of floor(N/2) + 1 over an odd-sized fleet of at least three, deployed across independent failure domains so no single rack or availability zone loss can eliminate the majority. Which failover model to run, and how to size that fleet, is walked end to end in Redis Sentinel vs Cluster Failover.

# Replication and Read Scaling

Replication is doing two jobs at once, and conflating them causes most availability surprises. Its first job is durability-through-redundancy: a replica holds a near-live copy of the primary's dataset, so a promotion has something recent to promote. Its second job is horizontal read capacity: because replicas answer read commands, they let a read-heavy workload scale beyond a single node's CPU and network budget. Replication and Read Scaling with Redis Replicas develops both jobs in full. The tension is that Redis replication is asynchronous by default — the primary acknowledges a write to the client before the replica has applied it — so a replica always trails the primary by some offset, and reading from a replica means accepting a small, variable staleness window that a read from the primary would not have.

For durable state you cannot afford to read stale, route reads to the primary or bound the acceptable lag explicitly; for tolerant data — rendered fragments, catalog snapshots, counters — replica reads are free scale. The pattern below measures the offset gap before trusting a replica, and falls back to the primary when the replica has drifted too far, using redis.asyncio:

import redis.asyncio as redis

primary = redis.Redis(host="cache-primary", port=6379, decode_responses=True)
replica = redis.Redis(host="cache-replica-1", port=6379, decode_responses=True)


async def read_with_bounded_lag(key: str, max_lag_bytes: int = 1_048_576) -> str | None:
    # master_repl_offset (primary) minus slave_repl_offset (replica) is the
    # replication backlog still unapplied on this replica, measured in bytes.
    p_info = await primary.info("replication")
    r_info = await replica.info("replication")
    lag = int(p_info["master_repl_offset"]) - int(r_info["slave_repl_offset"])

    if lag > max_lag_bytes or r_info.get("master_link_status") != "up":
        # Replica is behind or disconnected: read the source of truth instead
        # of serving a value the primary has already moved past.
        return await primary.get(key)
    return await replica.get(key)

The staleness introduced by replica reads compounds with any staleness already allowed by your invalidation strategy: a value protected only by a safety-net TTL can be stale by the TTL window plus the replication lag when read off a trailing replica. When a write must invalidate copies scattered across replicas and services, drive those busts through pub/sub routing rather than assuming replication alone will carry the delete promptly. The mechanics of splitting a read fleet, weighting replicas, and keeping lag inside an SLA are the subject of Replication and Read Scaling with Redis Replicas.

# Connection Pooling and Client Resilience

The availability of the server means nothing if the client cannot follow a failover. In the seconds between a primary dying and a replica being promoted, every in-flight command fails, every pooled connection to the dead node is now useless, and every application instance tries to reconnect at once. Without discipline, that synchronized reconnect becomes a thundering herd that lands on the freshly promoted primary before it has warmed its caches, and the recovery itself causes a second outage. Connection Pooling and Client Resilience is where the client-side contract is specified. Three settings carry most of the weight: a bounded pool so a stampede cannot open unlimited sockets, aggressive but finite timeouts so a hung connection fails fast instead of pinning a worker, and capped exponential backoff with jitter so retries spread across a window instead of arriving as one wall.

The configuration below builds a resilient async client whose retries are bounded and whose connections are health-checked, so a failover degrades into a brief blip rather than a cascading reconnect storm:

import redis.asyncio as redis
from redis.asyncio.retry import Retry
from redis.backoff import ExponentialBackoff
from redis.exceptions import (
    ConnectionError as RedisConnError,
    TimeoutError as RedisTimeout,
)

# A capped pool plus capped, jittered retry keeps a failover from turning into
# a reconnect storm that overruns the newly promoted primary.
pool = redis.ConnectionPool(
    host="cache-primary",
    port=6379,
    max_connections=64,            # hard ceiling on concurrent sockets
    socket_timeout=0.5,            # fail a hung command fast, do not pin a worker
    socket_connect_timeout=0.5,    # bound the reconnect attempt during promotion
    health_check_interval=15,      # PING idle connections, retire stale ones
    retry=Retry(ExponentialBackoff(base=0.05, cap=1.0), retries=3),
    retry_on_error=[RedisConnError, RedisTimeout],
    decode_responses=True,
)
client = redis.Redis(connection_pool=pool)


async def resilient_get(key: str) -> str | None:
    try:
        return await client.get(key)
    except (RedisConnError, RedisTimeout):
        # Retries are already exhausted here; degrade to the source of truth
        # rather than blocking the request on a topology still settling.
        return None

Pool sizing is a balance, not a maximum-is-better dial. An over-provisioned pool exhausts file descriptors and floods a recovering node; an under-provisioned pool bottlenecks under normal peak concurrency and creates artificial contention that looks like a Redis problem but lives entirely in the client. Pair the ceiling with the health check so connections severed by a silent partition are retired before a request tries to reuse them. In a Redis Cluster deployment, the same discipline extends to per-node pools and the client's handling of MOVED and ASK redirection during a slot migration, where a topology change and a failover can land in the same window. The full tuning model — pool math against peak concurrency, timeout budgets, and the async-versus-sync trade-off — lives in Connection Pooling and Client Resilience.

# Comparing the Failover Models

The two native models and the managed option resolve the same problem — detect a dead primary, appoint a successor, re-point clients — but they place the cost in different places. This table summarizes where each lands so you can match the model to your data shape and operational appetite.

The rightmost column is where teams underinvest and the middle column is where they get surprised. Sentinel is the lower-complexity choice when a single dataset fits one node's memory and you only need availability, not sharding; Cluster earns its operational weight when the dataset outgrows one node and you need both partitioning and per-shard failover in one system. A managed endpoint trades control for convenience, which is the right trade until an incident review needs to know exactly how many writes the last promotion dropped and the answer is not in your hands.

# Production HA Readiness Checklist

Before a Redis deployment is trusted to survive node loss unattended, confirm the following across the control plane, the data plane, and the client:

• The Sentinel fleet (or Cluster) is an odd number of at least three, spread across independent failure domains so no single zone loss removes the voting majority.
• quorum is set to floor(N/2) + 1 — a strict majority — never lower, so a partition cannot let a minority promote a second primary.
• Replica reads are either bounded by an explicit lag check or reserved for data that tolerates a staleness window; strongly-consistent reads go to the primary.
• Every client uses a bounded pool with finite socket_timeout, socket_connect_timeout, and capped, jittered retry so a failover cannot become a reconnect storm.
• health_check_interval is set so connections severed by a silent partition are retired before a request reuses them.
• The application degrades to the source of truth (or a bounded error) when Redis is unreachable, rather than blocking a worker on a topology that is still settling.
• min-replicas-to-write (and min-replicas-max-lag) is configured if lost writes during a partition are unacceptable, accepting reduced write availability in exchange.
• Failover has been exercised in a game day — an actual SENTINEL failover or a killed primary — not merely assumed to work from configuration.
• role, master_link_status, master_repl_offset, and connected_slaves are alerting metrics, not just dashboard lines.

# Failure Modes at a Glance

Most availability incidents reduce to a handful of named failure modes. Each has a one-line diagnosis and a page that treats the fix in depth.

• Split-brain (two primaries accepting writes) — a partition lets a minority quorum promote a replica while the original primary still takes writes; diagnose via two nodes simultaneously reporting role:master for the same dataset and divergent master_repl_offset. See Redis Sentinel vs Cluster Failover.
• Failover storm / thundering reconnect — a promotion invalidates every pooled connection at once and all instances reconnect simultaneously, burying the new primary; diagnose via a connected_clients spike synchronized with the promotion and elevated command latency on the survivor. See Connection Pooling and Client Resilience.
• Replication-lag stale reads — a trailing replica serves values the primary has already superseded; diagnose via a growing master_repl_offset minus slave_repl_offset gap while reads still route to that replica. See Replication and Read Scaling with Redis Replicas.
• Quorum misconfiguration — the quorum is set below a strict majority (enabling split-brain) or above it (blocking legitimate failover); diagnose via SENTINEL ckquorum failing or a known-dead primary that never gets promoted. See Redis Sentinel vs Cluster Failover.

# Monitoring & Observability

Instrument the topology from INFO replication and the Sentinel API before you need them in an incident. The fields below map directly to the failure modes above, and each answers a specific question about who is primary, how far behind the replicas are, and whether a promotion is imminent.

The two commands below cover the day-to-day check — the data plane's replication health and the control plane's view of it:

# Data plane: role, link health, offsets, and attached replicas on one node.
redis-cli -h cache-replica-1 INFO replication | \
  grep -E "^(role|master_link_status|master_repl_offset|slave_repl_offset|connected_slaves)"

# Control plane: the quorum's authoritative view of every monitored primary,
# including how many Sentinels currently agree it is reachable.
redis-cli -h sentinel-a -p 26379 SENTINEL masters

Alert on the derivative of the replication-offset gap, not its absolute value — a slow, steady climb is the early signal that a replica is falling permanently behind and will make a poor promotion candidate long before it disconnects. Watch role transitions as events, so a promotion pages you with context instead of arriving as an unexplained latency spike, and correlate connected_clients against those transitions to catch a reconnect storm while it is still building rather than after it has toppled the survivor.

---

Up one level: Home

# Related

• Redis Sentinel vs Cluster Failover — which model detects and heals a dead primary, and how to size the quorum.
• Replication and Read Scaling with Redis Replicas — using replicas for both durability and read capacity without serving dangerously stale data.
• Connection Pooling and Client Resilience — riding through a failover without a reconnect storm.