Implementing Redlock with redis-py

You are coordinating a job that must run on exactly one worker — a cache rebuild, a batch invalidation, a leader election for a scheduler — and a single-instance SET NX lock is not safe enough. That lock lives on one primary, and if that primary fails over to a replica that had not yet replicated the lock key, two workers can each believe they hold it. Redlock addresses this by spreading the lock over N fully independent Redis primaries and requiring a majority to agree, so the loss of any single node cannot silently hand the lock to a second holder. This page implements Redlock end to end with the redis.asyncio client, acquiring across all nodes concurrently, computing the true remaining validity of the lock, releasing it safely, and pairing it with a fencing token for the resource it protects. It is the concrete implementation behind the trade-offs in distributed locks for cache coordination.

# Prerequisites

• Redis 7.x running on N = 5 independent primaries — separate hosts or availability zones, not a primary and its replicas, and not one Redis Cluster. Independence is the entire point.
• redis-py 5.x on Python 3.10+ (pip install "redis>=5,<6"). All examples use the redis.asyncio client.
• Loosely synchronized clocks. Redlock tolerates bounded drift but not arbitrary clock jumps; keep ntpd/chrony running on every node.
• A monotonic clock in the application (time.monotonic()) for measuring elapsed acquisition time — never the wall clock, which can step backward.
• An understanding that this lock is for coordination, not for guarding a correctness-critical resource without a fencing token as well (see the FAQ).

# Step-by-Step Implementation

# 1. Connect to five independent primaries

Open one redis.asyncio client per primary. Each address must be a distinct failure domain; two of these being replicas of one another would collapse the quorum's independence.

import redis.asyncio as redis

NODES = [
    "redis://10.0.1.10:6379", "redis://10.0.2.10:6379",
    "redis://10.0.3.10:6379", "redis://10.0.4.10:6379",
    "redis://10.0.5.10:6379",
]
# One dedicated client per primary; short timeouts so a dead node fails fast.
clients = [
    redis.from_url(url, socket_connect_timeout=0.1, socket_timeout=0.1)
    for url in NODES
]
QUORUM = len(clients) // 2 + 1   # 3 of 5

# 2. Generate a unique lock token

Every acquisition attempt must carry a value no other client could guess or reuse, so that release only ever deletes a lock this client still owns. A random 20-byte token is standard.

import secrets

def new_token() -> str:
    # Unique per acquisition; used later for a compare-and-delete release.
    return secrets.token_hex(20)

# 3. Acquire on one node with SET NX PX

The per-node primitive is a single atomic SET with NX (only if absent) and PX (millisecond TTL). The TTL bounds how long a crashed holder can block everyone else. A node that is down or slow must be treated as a plain failure, not an error, so the quorum loop can continue.

async def acquire_one(client: redis.Redis, key: str, token: str, ttl_ms: int) -> bool:
    try:
        # SET key token NX PX ttl_ms — atomic acquire-if-absent with expiry.
        return bool(await client.set(key, token, nx=True, px=ttl_ms))
    except (redis.RedisError, OSError):
        return False   # unreachable/slow node counts as "not acquired"

# 4. Fan out across all nodes concurrently

Issue every SET at once with asyncio.gather rather than sequentially. Concurrency keeps total elapsed time close to the slowest single node instead of the sum of all five, which directly preserves more of the lock's validity (Step 5).

import asyncio, time

async def try_acquire_all(key: str, token: str, ttl_ms: int) -> tuple[int, float]:
    start = time.monotonic()
    results = await asyncio.gather(
        *(acquire_one(c, key, token, ttl_ms) for c in clients)
    )
    elapsed_ms = (time.monotonic() - start) * 1000
    return sum(results), elapsed_ms   # (nodes acquired, wall time spent)

# 5. Require a majority AND positive remaining validity

Holding a majority is necessary but not sufficient. Acquisition itself consumes part of the TTL, and clocks drift, so you must compute the lock's remaining validity and reject the lock if that value is not strictly positive. The drift allowance is a small factor of the TTL plus a few milliseconds:

validity = ttl_ms − elapsed_ms − drift, where drift = ttl_ms * 0.01 + 2.

DRIFT_FACTOR, DRIFT_CONST_MS = 0.01, 2

async def acquire(key: str, ttl_ms: int = 10_000):
    token = new_token()
    acquired, elapsed = await try_acquire_all(key, token, ttl_ms)
    drift = ttl_ms * DRIFT_FACTOR + DRIFT_CONST_MS
    validity = ttl_ms - elapsed - drift
    if acquired >= QUORUM and validity > 0:
        return token, validity          # lock is held and safe to use
    await release(key, token)           # rolled back on every node
    return None                         # failed — caller must not proceed

# 6. Use the lock only within its validity window

The returned validity — not the nominal TTL — is your real budget. If the protected work might exceed it, do not run it: acquire with a longer TTL or, better, attach a fencing token (Step 7). This is the same discipline that keeps a single-flight cache rebuild from being executed twice under a stampede.

lock = await acquire("lock:rebuild:catalog", ttl_ms=10_000)
if lock:
    token, validity_ms = lock
    if validity_ms > 3_000:             # leave headroom for the actual work
        await rebuild_catalog_cache()   # runs on exactly one worker
    await release("lock:rebuild:catalog", token)

# 7. Release on every node with a compare-and-delete Lua script

Release must delete the key on all N nodes, including any that were acquired before a partition healed. It must also be conditional: delete only if the stored value still equals your token, so a lock that already expired and was re-acquired by another worker is never deleted out from under them. GET-then-DEL is racy; a Lua script makes the check-and-delete atomic.

RELEASE_LUA = """
if redis.call('GET', KEYS[1]) == ARGV[1] then
  return redis.call('DEL', KEYS[1])
else
  return 0
end
"""

async def release(key: str, token: str) -> None:
    # Fire the compare-and-delete at every node; ignore individual failures.
    async def _rel(client: redis.Redis) -> None:
        try:
            await client.eval(RELEASE_LUA, 1, key, token)
        except (redis.RedisError, OSError):
            pass
    await asyncio.gather(*(_rel(c) for c in clients))

# 8. Fence the protected resource with a monotonic token

Redlock cannot stop a holder that was paused mid-critical-section (a long GC pause) from waking up after its lock expired and writing anyway. The defence is a fencing token: a strictly increasing number the protected resource records and uses to reject any stale writer. Redis can mint it with INCR on a majority; the external store enforces it.

async def fencing_token(key: str = "fence:catalog") -> int:
    # Monotonic counter; the downstream store must reject any token <= last seen.
    return await clients[0].incr(key)   # replicate/quorum in production

# Failure Modes

Split acquisition (no node holds a majority). Under a partition or heavy contention, your SETs may land on 2 of 5 while a competitor takes the other 2 and one node is down — nobody reaches quorum, and both callers correctly fail. The danger is leaving orphaned keys behind on the minority nodes, which block the next attempt until the TTL expires. Diagnose which nodes still hold your token:

for h in 10.0.1.10 10.0.2.10 10.0.3.10 10.0.4.10 10.0.5.10; do
  redis-cli -h "$h" GET lock:rebuild:catalog
done

The fix is already in Step 5: on any sub-quorum result, call release unconditionally so partial acquisitions are rolled back immediately instead of waiting out the TTL.

Lock lost on primary failover. If you weaken the setup to a primary with replicas instead of independent primaries, a SET NX can be acknowledged by the primary and lost when an un-replicated replica is promoted — the exact hazard covered under Sentinel versus Cluster failover. Detect it by comparing the primary's replication offset against what the replica actually received:

redis-cli -h 10.0.1.10 INFO replication | grep -E "master_repl_offset|slave0"

The fix is structural: use five genuinely independent primaries with no replication between them, so no single promotion can erase a majority.

Clock jump shrinks the real validity. A backward ntp step or a stalled VM makes the nominal TTL diverge from wall time, so a lock you believe is valid has already expired on the nodes. Diagnose by watching for large offsets:

redis-cli -h 10.0.1.10 TIME     # compare against local `date +%s` across nodes

The fix is to keep the acquisition fast (Step 4's concurrency), keep the drift allowance honest (Step 5), and never let critical work run past the returned validity.

# Verification

Confirm a clean acquisition holds the key on a majority with a positive TTL, and that release removes it everywhere:

# After acquire(): a majority of nodes should return the same token.
for h in 10.0.1.10 10.0.2.10 10.0.3.10 10.0.4.10 10.0.5.10; do
  redis-cli -h "$h" PTTL lock:rebuild:catalog   # expect a positive ms value on >= 3
done

Assert in code that the algorithm refuses the lock when quorum is not met — a fast, deterministic test that a broken quorum never returns a token:

async def test_rejects_without_quorum():
    # Simulate 3 of 5 nodes down; acquire must return None, not a token.
    for c in clients[:3]:
        await c.close()
    assert await acquire("lock:test", ttl_ms=2_000) is None

Verify the compare-and-delete refuses to release a lock that a different worker now owns:

redis-cli -h 10.0.1.10 SET lock:t othertoken PX 5000
redis-cli -h 10.0.1.10 EVAL "if redis.call('GET',KEYS[1])==ARGV[1] then return redis.call('DEL',KEYS[1]) else return 0 end" 1 lock:t mytoken
# expect (integer) 0 — the key is untouched because the token does not match

# FAQ

# Does Redlock guarantee mutual exclusion?

Not on its own for correctness-critical resources. Martin Kleppmann's well-known critique shows that a process pause (a long GC pause or an OS scheduling stall) can exceed the lock TTL, so the lock expires and is granted to a second worker while the first still believes it holds it. Redlock's timing assumptions do not survive an unbounded pause. The robust remedy is a fencing token (Step 8): a monotonically increasing number the protected resource records and uses to reject any write carrying a stale token. Use Redlock for efficiency — stopping duplicate work most of the time — and fencing for safety when a double execution would be a correctness bug.

# Why five nodes, and can I use three or seven?

N must be odd so a majority is unambiguous, and the nodes must be independent primaries. Five is the common choice because it tolerates the loss of two nodes while still forming a quorum of three; three nodes tolerate only one loss, and seven add coordination cost for marginal extra resilience. Never build the quorum from a primary and its replicas — replicas are not independent, so a single failover can wipe out the majority at once.

# How does clock drift affect the lock, and what is the drift factor?

Redlock's validity is ttl − elapsed − drift, where drift accounts for the maximum clock divergence you expect across nodes during the lock's lifetime, typically 1% of the TTL plus a small constant. Subtracting it means you always treat the lock as expiring slightly earlier than its nominal TTL, so a modestly fast clock on one node cannot make you use a lock past its real expiry. It assumes bounded drift; a large clock step voids the guarantee, which is why every node must run time synchronization.

# How is this different from a single-instance SET NX lock?

A single-instance lock is one SET key token NX PX ttl against one Redis. It is simpler and lower-latency and is perfectly adequate when the lock is an optimization and occasional double execution is tolerable. Its weakness is that the lock lives on exactly one node: if that node fails over to a replica that had not replicated the key, the lock evaporates. Redlock trades that latency and simplicity for surviving the loss of a minority of independent nodes, which matters when a failover during the critical section is a realistic risk.

# What happens if my job runs longer than the lock TTL?

The lock expires and another worker can acquire it, so two workers may run concurrently. Never size the TTL below your worst-case job time and hope; instead, gate on the returned validity (Step 6) and refuse to start work that will not finish inside it. For genuinely long jobs, either extend the TTL periodically from the holder (re-SET with the same token while work continues) or make the downstream write idempotent and fenced so a late writer is rejected rather than trusted.

---

Up: Distributed Locks for Cache Coordination

# Related

• Distributed Locks for Cache Coordination
• Cache Stampede Prevention with Probabilistic Early Expiration
• Implementing Probabilistic Early Recomputation in Python
• Redis Sentinel vs Cluster Failover