CacheInvalidation.org

Designing Graceful Fallback Routing for Cache Misses

A single Redis cache miss is cheap; a synchronized wave of them is an incident. Under memory pressure, a network partition, or a mid-flight resharding event, thousands of keys stop resolving in the same window and every request stampedes the origin database at once — saturating connection pools, inflating P99 latency, and cascading into dependent services. This page shows how to make each miss route deterministically through a tiered fallback hierarchy in async Python: classify why the key was absent (natural TTL expiry, LRU eviction under maxmemory-policy, or a MOVED redirection during slot migration), then route accordingly within a sub-millisecond budget instead of collapsing every failure into one blind origin read.

Prerequisites

  • Redis 7.x deployed as a sharded cluster (or a primary/replica pair) with maxmemory-policy set explicitly — volatile-ttl, allkeys-lru, or allkeys-lfu.
  • redis-py 5.x on Python 3.10+ using redis.asyncio for non-blocking reads on the request hot path.
  • pybreaker for per-tier circuit breaking and tenacity for bounded, jittered retries.
  • A local process-memory tier (an in-process LRU dict or aiocache) as the last fast tier before the origin database.
  • Client-side keyspace-miss metrics wired to your observability stack so eviction and redirect deltas are queryable.

Step-by-Step Implementation

1. Instrument the miss so you can classify it. Before any routing logic runs, capture the miss/evict/expire deltas and topology state so a lifecycle expiry is never confused with an infrastructure fault.

# Real-time miss / eviction / expiry deltas
redis-cli --stat 1 | grep -E "miss|evict|expir"

# Slot ownership and topology validation
redis-cli CLUSTER SHARDS

# Blocking operations stalling the event loop
redis-cli SLOWLOG GET 10

# Memory-policy pressure indicators
redis-cli INFO memory | grep -E "used_memory|maxmemory|mem_fragmentation"

2. Declare the fallback hierarchy and a strict timeout budget. Model the tiers as an explicit ordered list — Primary Cluster → Read Replica → Local Memory → Origin DB — where each tier owns an independent timeout so a slow tier can never consume the whole request budget.

# Per-tier deadlines in seconds — the sum is your worst-case read budget (~11ms)
TIER_TIMEOUTS = {
    "primary":  0.005,   # cluster primary
    "replica":  0.004,   # regional read replica
    "local":    0.001,   # in-process memory
    "origin":   0.250,   # relational source of truth (last resort)
}

3. Build the async cluster client with replica reads enabled. Construct a single long-lived RedisCluster so slot-aware routing and the connection pool are reused across requests rather than rebuilt per call.

from redis.asyncio import RedisCluster

async def build_client() -> RedisCluster:
    return RedisCluster(
        host="redis-cluster",
        port=6379,
        socket_timeout=0.005,
        decode_responses=True,
        read_from_replicas=True,   # serve reads from replicas automatically
        retry_on_timeout=False,    # we own retries — never retry silently
    )

4. Classify the error and route to the correct tier. Catch each exception type distinctly so a ClusterDownError (no reachable slot owner) bypasses to the origin, while an unclassified ResponseError is re-raised instead of being masked as a cache miss.

from redis.exceptions import (
    TimeoutError as RedisTimeout,
    ConnectionError as RedisConnError,
    ClusterDownError,
    ResponseError,
)

async def classify_and_route(client, key, fallback):
    try:
        return await client.get(key)          # None == legitimate miss
    except RedisTimeout:
        return await fallback(key)            # network degradation → next tier
    except ClusterDownError:
        return await fallback(key)            # slot has no owner → origin
    except RedisConnError:
        raise                                 # pool exhaustion → caller decides
    except ResponseError:
        raise                                 # never mask an unclassified error

5. Wrap the read with a circuit breaker and bounded, jittered retry. Guard the origin tier with a breaker that opens after repeated failures, and cap retries at two attempts with exponential jitter so a degraded primary never triggers a retry storm.

import pybreaker
from tenacity import (
    retry, stop_after_attempt, wait_exponential_jitter, retry_if_exception_type,
)

db_breaker = pybreaker.CircuitBreaker(fail_max=5, reset_timeout=30)

@retry(
    stop=stop_after_attempt(2),
    wait=wait_exponential_jitter(initial=0.001, max=0.01),
    retry=retry_if_exception_type(RedisTimeout),
)
async def fetch_with_fallback(client, key, local_cache, query_origin):
    async def to_origin(k):
        return await db_breaker.call_async(query_origin, k)
    value = await classify_and_route(client, key, to_origin)
    if value is None:
        # miss confirmed — serve local/stale and warm out of band (step 6)
        return await local_cache.get_or_warm(key)
    return value

6. Warm asynchronously under eviction pressure instead of hammering the origin. When diagnostics show evicted_keys climbing, route the miss to the local tier and enqueue a background refresh — the same decoupling used for write-behind caching and driven here by an async invalidation workflow — so read latency is never coupled to a write-heavy invalidation storm.

import asyncio

async def get_or_warm(local_cache, key, query_origin, redis_client, ttl=60):
    cached = local_cache.get(key)
    if cached is not None:
        # serve stale immediately, refresh Redis + local out of band
        asyncio.create_task(_warm(key, query_origin, redis_client, local_cache, ttl))
        return cached
    value = await query_origin(key)              # cold: one synchronous origin read
    await redis_client.set(key, value, ex=ttl)
    local_cache.set(key, value)
    return value

async def _warm(key, query_origin, redis_client, local_cache, ttl):
    value = await query_origin(key)
    await redis_client.set(key, value, ex=ttl)
    local_cache.set(key, value)

7. Gate the routing SLA in CI/CD with chaos injection. Fail the build if a simulated topology shift breaks the routing contract, so fallback regressions are caught before deploy rather than in production.

- name: Validate fallback routing SLA
  run: pytest tests/test_cache_routing.py --junitxml=results.xml

- name: Inject cluster topology shift and re-test
  run: |
    docker exec redis-node-1 redis-cli DEBUG SLEEP 0.5
    docker pause redis-node-1
    sleep 2
    pytest tests/test_cluster_failover_routing.py
    docker unpause redis-node-1

Critical Path of a Fallback Decision

The routing hierarchy below shows how one read walks the tiers — each hop enforces its own deadline and a miss at the bottom schedules an out-of-band warm rather than blocking the caller.

Tiered fallback routing ladder with per-tier timeout budgets A read request enters at the top and descends an ordered ladder of tiers, each owning an independent timeout: Primary cluster at 5ms, Read replica at 4ms, Local in-process memory at 1ms, and the Origin database at 250ms as the last resort. A hit at any tier short-circuits right into the shared "Return value" success rail. A miss or timeout drops the request down to the next tier. When the request reaches the origin, a synchronous read is issued and an out-of-band cache-warm task is enqueued before the value is returned, so read latency is never coupled to the warm. Return value served to caller Read request Primary cluster slot-aware read 5ms Read replica regional, may lag 4ms Local memory in-process, stale ok 1ms Origin database last resort · circuit-broken 250ms Schedule async cache warm out-of-band refresh — caller not blocked miss / timeout degraded miss read + enqueue hit hit hit then return

Handling MOVED and ASK correctly. The async RedisCluster follows MOVED <slot> <ip:port> and ASK redirects internally, refreshing its slot map on MOVED; you only surface ClusterDownError when a slot has no reachable owner. If you ever route at a lower level, an ASK must be answered with a single ASKING command before the operation — issuing the read without it just earns another redirect. The mechanics of why these redirects appear are covered in hash slot allocation.

Failure Modes

Retry storm on a degraded primary. If retry_on_timeout=True (or tenacity is unbounded), a slow primary multiplies every read into a burst that finishes exhausting the connection pool. Diagnose with redis-cli INFO clients | grep connected_clients climbing while redis-cli --stat throughput flatlines; fix by pinning retry_on_timeout=False and capping attempts at two with jitter (step 5).

Synchronous origin fetch amplifying an eviction storm. When allkeys-lru eviction spikes and every miss falls straight through to the database, origin load explodes exactly when Redis is already thrashing. Diagnose with redis-cli INFO stats | grep -E "evicted_keys|keyspace_misses" showing a rising eviction-to-miss ratio; fix by routing to the local tier and warming asynchronously (step 6) rather than reading through synchronously.

Masking a real error as a cache miss. Collapsing every exception into return fallback(key) turns an auth failure or a malformed command into a silent, uncacheable origin read that never recovers. Diagnose with redis-cli SLOWLOG GET 10 and application error logs showing repeated origin reads for a key that should be cached; fix by re-raising unclassified ResponseError and ConnectionError (step 4) instead of swallowing them.

Verification

Confirm the fallback contract holds under load and failover before trusting it in production.

# 1. Miss classification is queryable — expiry vs eviction must be distinguishable
redis-cli INFO stats | grep -E "keyspace_hits|keyspace_misses|expired_keys|evicted_keys"

# 2. No retry storm — connected_clients stays bounded while a node is paused
docker pause redis-node-1 && redis-cli -h redis-node-2 INFO clients | grep connected_clients

# 3. The routing tests assert per-tier timeouts and breaker isolation
pytest tests/test_cache_routing.py -q

A correct implementation keeps connected_clients flat during the paused-node window, isolates the degraded tier via the circuit breaker within ~50ms, and shows evicted_keys growth triggering background warms rather than a keyspace_misses spike against the origin.

FAQ

Should fallback routing ever read through to the database synchronously? Only on a genuine cold miss where no local or stale copy exists. Under eviction pressure a synchronous read-through amplifies origin load; route to the local tier and warm asynchronously instead (step 6).

How is this different from a cache stampede lock? A stampede lock deduplicates concurrent cold reads for one key; fallback routing decides which tier answers when a key is absent for any reason. They compose — apply a per-key lock inside the origin tier so only one warm task runs per key.

Do I need ASKING if I use the high-level RedisCluster client? No. The async RedisCluster handles MOVED/ASK internally and refreshes its slot map. You only issue ASKING manually if you route at the raw-connection level.

What timeout should the primary tier use? Start at 5ms socket_timeout for an in-region cluster and derive it from your P99 read latency plus headroom — low enough that a hung node fails over fast, high enough that normal reads never time out. The sum of all tier timeouts is your worst-case read budget.

Can I reuse this pattern for cross-service invalidation? The routing tiers are read-path only. Propagating invalidations across services belongs to Redis Pub/Sub; fallback routing simply reacts to the misses those invalidations produce.

Keep exploring

Up one level: Fallback Routing Strategies in Redis Cluster Environments