Invalidating Related Caches with Secondary Index Sets

One product row changes and suddenly a dozen unrelated cache entries are wrong: the new-arrivals listing that featured it, three faceted-search results that matched its price band, a rendered product card, and a personalized recommendation slice. None of those keys carry the product id in their name, so you cannot reconstruct them from the mutation alone, and walking the keyspace to find them is both slow and unsafe. This page builds a secondary index set — a Redis SET, keyed on the entity, that records the exact name of every derived cache key the moment it is written — so a single change reads that set and purges everything it touched without guessing and without scanning. It is the fan-out primitive behind key tagging strategies for bulk updates, applied to the specific case where dependencies are discovered at write time rather than declared up front.

# Prerequisites

• Redis 7.x, standalone or in Cluster mode, reachable from redis-cli and your application host.
• redis-py 5.x on Python 3.10+ (pip install "redis>=5,<6" tenacity); every example uses the redis.asyncio client.
• A clear notion of the owning entity for each cache key — the record whose change makes that key stale (a product, a user, an order).
• Permission to read INFO memory, SCARD, and MEMORY USAGE for capacity checks.
• If you deploy on Cluster, an understanding of hash tags and how they pin keys to one of the 16,384 slots via slot allocation.

# Step-by-Step Implementation

The mechanism is deliberately small: on every cache write, record the key in its entity's index set with SADD; on invalidation, read the set with SMEMBERS, then UNLINK every member plus the index itself in one atomic unit. Each step below runs independently against a local Redis.

# 1. Register every derived key in the entity's index on write

Whenever you populate a cache entry that depends on an entity, add its name to that entity's index in the same round trip, so the index can never fall behind the data it tracks.

import redis.asyncio as redis

client = redis.Redis(host="localhost", port=6379, decode_responses=True)

async def cache_derived(entity: str, key: str, value: str, ttl: int = 3600) -> None:
    # The value and its index membership are written together in one pipeline,
    # so a derived key is never live without an index entry pointing at it.
    idx = f"secidx:{{{entity}}}"          # {entity} hash tag co-locates in Cluster
    async with client.pipeline(transaction=True) as pipe:
        pipe.set(key, value, ex=ttl)      # the derived cache entry itself
        pipe.sadd(idx, key)               # record it under the owning entity
        await pipe.execute()

# a listing that features product 42, registered under that product
await cache_derived("product:42", "{product:42}list:new-arrivals", rendered_html)

# 2. Refresh a guard TTL on the index so orphans self-expire

An index set that outlives all its members leaks memory forever. Push its expiry out on every write to a little beyond your longest member TTL, so an idle entity's index eventually evaporates on its own.

async def register(entity: str, key: str, member_ttl: int) -> None:
    idx = f"secidx:{{{entity}}}"
    async with client.pipeline(transaction=True) as pipe:
        pipe.sadd(idx, key)
        # Guard TTL sits above the longest member so the index never
        # expires out from under a live key, but does not live forever.
        pipe.expire(idx, member_ttl + 300)
        await pipe.execute()

# 3. Bound index growth for hot entities

A perpetually re-cached entity can accumulate an unbounded set. Cap it: once the index crosses a threshold, treat it as "purge everything for this prefix" rather than tracking each member, so a runaway set degrades gracefully instead of exhausting memory.

CAP = 5000

async def register_bounded(entity: str, key: str) -> None:
    idx = f"secidx:{{{entity}}}"
    added = await client.sadd(idx, key)
    if added and await client.scard(idx) > CAP:
        # Mark the index saturated; the purge path will fall back to a
        # bounded SCAN over {entity} instead of trusting an oversized set.
        await client.set(f"{idx}:saturated", 1, ex=3600)

# 4. Purge the whole set atomically

On invalidation, read the members and delete them together with the index in a single indivisible operation. A pipeline transaction (MULTI/EXEC) suffices when the read and the deletes can be split; when you need the read and delete to be strictly atomic against a concurrent re-registration, do it in a Lua script instead. Prefer UNLINK over DEL so reclaiming a large rendered fragment never blocks the event loop.

_PURGE = """
local members = redis.call('SMEMBERS', KEYS[1])
for _, k in ipairs(members) do
    redis.call('UNLINK', k)          -- non-blocking free of each derived key
end
redis.call('UNLINK', KEYS[1])        -- and the index set itself
return #members
"""

async def purge_entity(entity: str) -> int:
    idx = f"secidx:{{{entity}}}"
    # One server-side call: read, unlink every member, unlink the index.
    return await client.eval(_PURGE, 1, idx)

# 5. Co-locate the index and its members in one slot

In Cluster mode a multi-key UNLINK and any Lua script only run when every key hashes to the same slot. Wrap the entity id in a hash tag on both the index and each member so {product:42} is the only substring hashed, keeping the purge on one node.

# Both the index and its members share the {product:42} hash tag.
idx = "secidx:{product:42}"
members = ["{product:42}list:new-arrivals", "{product:42}query:price-lt-50"]
# All hash to the same slot, so the atomic purge stays on one shard.

# 6. Wrap the purge in bounded, idempotent retries

A master promotion or transient partition can interrupt the purge. Because deleting an already-deleted key is a no-op, the whole operation is idempotent and safe to retry with backoff and jitter.

from tenacity import (
    retry, wait_exponential_jitter, stop_after_attempt, retry_if_exception_type,
)
from redis.exceptions import ConnectionError as RedisConnError, TimeoutError as RedisTimeout

@retry(
    retry=retry_if_exception_type((RedisConnError, RedisTimeout)),
    wait=wait_exponential_jitter(initial=0.05, max=2.0),
    stop=stop_after_attempt(4),
    reraise=True,
)
async def purge_entity_safe(entity: str) -> int:
    # Re-running a partial purge only deletes what remains — idempotent.
    return await purge_entity(entity)

# Failure Modes

CROSSSLOT on the purge. In Cluster mode, if the index and a member hash to different slots, the script aborts with CROSSSLOT Keys in request don't hash to the same slot and nothing is deleted. Diagnose by comparing the two slots directly:

redis-cli CLUSTER KEYSLOT "secidx:{product:42}"
redis-cli CLUSTER KEYSLOT "{product:42}list:new-arrivals"

Matching numbers confirm the hash tag co-locates them; divergent numbers mean a member was written without the {product:42} tag. If some derived keys genuinely must live on their own slots, you cannot purge them in one atomic call — group members by slot and issue one purge per slot, accepting that the fan-out is no longer indivisible.

Stale index entries (dangling members). A member key expires by TTL or is evicted while its name lingers in the set, so the index over-reports and every purge wastes work on names that no longer exist. Detect drift by comparing set cardinality against how many members still exist:

redis-cli SCARD "secidx:{product:42}"
redis-cli SMISMEMBER "secidx:{product:42}" "{product:42}list:new-arrivals"

The dangling names are harmless to UNLINK (deleting a missing key is a no-op), but they inflate memory and SMEMBERS cost. Prune them by having the purge script SREM any member UNLINK reports as already gone, or run a periodic reconciliation sweep that rebuilds the index from source-of-truth metadata.

Oversized index blocking the server. SMEMBERS on a set with hundreds of thousands of members is an O(N) call that stalls the single-threaded server and spikes p99 for every client. Catch it in the slow log and check the set size:

redis-cli SLOWLOG GET 10
redis-cli SCARD "secidx:{product:42}"

Above roughly 10,000 members, stop enumerating the whole set in one call. Iterate with SSCAN in batches and UNLINK in chunks, or fall back to a bounded prefix purge — the same enumeration-safety principle covered in SCAN vs KEYS for safe key enumeration.

# Verification

Confirm the full round-trip against a live instance — register two derived keys, purge the entity, and assert everything is gone:

# Seed two derived keys and register both under the entity index.
redis-cli SET "{product:42}list:new-arrivals" 1
redis-cli SET "{product:42}query:price-lt-50" 1
redis-cli SADD "secidx:{product:42}" "{product:42}list:new-arrivals" "{product:42}query:price-lt-50"

# Read, unlink members, unlink the index — expect a count of 2.
redis-cli EVAL "local m=redis.call('SMEMBERS',KEYS[1]) for _,k in ipairs(m) do redis.call('UNLINK',k) end redis.call('UNLINK',KEYS[1]) return #m" 1 "secidx:{product:42}"
# => (integer) 2

# Both members and the index must now be absent.
redis-cli EXISTS "{product:42}list:new-arrivals" "{product:42}query:price-lt-50" "secidx:{product:42}"
# => (integer) 0

For ongoing health, watch MEMORY USAGE on the largest index keys to catch bloat before it matters, track SCARD growth per hot entity against your cap, and export the purge EVAL duration from redis-py so a p99 above ~50 ms alerts before it degrades write endpoints. Gate deployments on a CI job that runs the purge against an ephemeral Redis:

- name: Validate secondary-index purge
  run: |
    docker run -d --name ci-redis -p 6379:6379 redis:7.2-alpine
    pytest tests/test_secondary_index.py --redis-url=redis://localhost:6379

# FAQ

# Why maintain an index set instead of a SCAN over a key pattern?

SCAN MATCH finds keys by name shape, but derived caches rarely encode the owning entity in their name — a new-arrivals listing that happens to feature product 42 has no 42 in its key. A secondary index records the true dependency at write time, so purging is an O(members) read of an explicit list rather than an O(total-keys) walk that cannot even express the relationship. SCAN remains the right tool only when the dependency genuinely lives in the key name.

# Is reading the set and deleting its members really atomic?

Only if you do both inside one server-side unit. A Lua script (or a MULTI/EXEC transaction that carries the member list) runs without interleaving other commands, so no client can read a half-purged state. If you split the SMEMBERS and the UNLINK across two separate round trips, a concurrent write can register a new member in the gap and that key will survive the purge — which is why Step 4 keeps the read and the deletes together.

# How does this behave in Redis Cluster?

The index and every member must hash to one slot, which you force with a shared hash tag such as {product:42}. That pins all of an entity's caches to a single shard so the atomic purge runs on one node. The trade-off is load: a very hot entity concentrates its keys on one slot, so extremely skewed entities may need their fan-out sharded by member group instead of a single index.

# What happens to stale entries when a member key expires on its own?

The member's name stays in the set until something removes it, so the index temporarily over-reports. This is safe — UNLINK on a missing key is a no-op — but wasteful. Bound the damage with the guard TTL from Step 2 so idle indexes expire wholesale, and have the purge SREM members that came back as already-deleted, or run a periodic reconciliation sweep.

# How much memory does the index add?

Each index set stores one copy of every member key name, so the overhead is roughly the summed length of the derived key names per entity, plus Redis set bookkeeping. For entities with a handful of derived caches this is negligible; for entities that fan out to thousands of keys it is real, which is why Step 3 caps set size and falls back to a bounded prefix purge once an index would grow without limit.

---

Up: Key Tagging Strategies for Bulk Updates

# Related

• Using Key Tags to Invalidate Related Data Sets
• SCAN vs KEYS for Safe Key Enumeration
• Redis Cluster Slot Allocation Basics
• Building Async Invalidation Queues with Celery