The Bug That Cost Millions: How Distributed Locks Actually Work

A reconciliation job runs every night, processing millions of dollars in payouts. One worker picks up the batch, acquires a lock, and starts processing. Then a garbage collection pause freezes it for ten seconds. Its lock had a five-second TTL. By the time it resumes, a second worker has already acquired the same lock and started processing the exact same batch — and neither worker has any idea the other exists.

I’ve seen this exact shape of bug take down systems that everyone assumed were protected. The lock did precisely what it was designed to do. That’s what makes this failure mode so dangerous: it isn’t a bug in the implementation. It’s a wrong assumption about what a distributed lock actually guarantees.

The Problem, Properly Framed

A lock is supposed to guarantee mutual exclusion: only one process does the protected work at a time. The naive implementation — a Redis key set with SETNX, expiring after a TTL — looks correct in every test you’ll run locally. No garbage collection pauses, no network delays, no CPU steal on shared infrastructure. The TTL exists as a safety net so a crashed worker doesn’t hold the lock forever, but it never actually fires in dev.

The problem is that a TTL is a timer, not a guarantee tied to a worker’s actual progress. It has no idea whether the worker holding the lock is still running, paused, or dead — it just counts down. Anything that pauses a worker longer than the TTL — a GC pause, CPU steal, a slow downstream call — causes the lock to expire while the worker still believes it holds it. When that worker resumes, it has no signal that anything changed.

The Core Mechanics

Why the Naive Lock Breaks Under Real Conditions

sequenceDiagram
    participant A as Worker A
    participant L as Lock Service (Redis)
    participant DB as Payouts DB

    A->>L: SETNX lock:batch-42 (TTL 5s)
    L-->>A: lock acquired
    A->>DB: process payouts for batch-42
    DB-->>A: success
    A->>L: release lock:batch-42
    Note over A,DB: Single worker, no overlap — looks correct

This is the happy path, and it’s the only path most teams ever test. Here’s what actually happens under real production conditions:

sequenceDiagram
    participant A as Worker A
    participant L as Lock Service (Redis)
    participant B as Worker B
    participant DB as Payouts DB

    A->>L: SETNX lock:batch-42 (TTL 5s)
    L-->>A: lock acquired
    A->>DB: start processing batch-42
    Note over A: GC pause — Worker A frozen for 10s
    Note over L: TTL expires at 5s, lock auto-releases
    B->>L: SETNX lock:batch-42
    L-->>B: lock acquired (A's lock already expired)
    B->>DB: start processing batch-42
    Note over A: Worker A resumes, unaware its lock is gone
    A->>DB: finish processing batch-42
    B->>DB: finish processing batch-42
    Note over DB: Every payout in batch-42 written twice

Worker A’s pause doesn’t have to be a bug — GC pauses, CPU steal on a noisy neighbor’s VM, and slow network calls are all normal, expected occurrences at scale. The lock doesn’t fail because something went wrong. It fails because the design assumed a TTL could stand in for “is this worker still alive and working,” and that assumption doesn’t hold.

The Fencing Token Fix

The instinct is to make the lock “more correct” — retry acquisition more carefully, extend the TTL, add heartbeats. None of that actually closes the gap, because the lock service can never know for certain that the old holder has stopped running. The fix has to move the responsibility somewhere else entirely: to the resource being protected.

Every time the lock is granted, the lock service issues a monotonically increasing fencing token along with it. The resource — not the lock service — is responsible for tracking the highest token it has ever accepted, and rejecting any write that arrives with a lower one.

sequenceDiagram
    participant A as Worker A
    participant L as Lock Service
    participant B as Worker B
    participant DB as Payouts DB (fencing-aware)

    A->>L: acquire lock:batch-42
    L-->>A: granted, token=33
    Note over A: GC pause — Worker A frozen
    Note over L: TTL expires, lock released
    B->>L: acquire lock:batch-42
    L-->>B: granted, token=34
    B->>DB: write payouts (token=34)
    DB-->>B: accepted, last_token=34
    Note over A: Worker A resumes, still holds token=33
    A->>DB: write payouts (token=33)
    DB-->>A: rejected — 33 < last_token 34
    Note over DB: Only Worker B's writes are accepted

The database becomes the actual enforcement point, not a passive victim of whatever the lock decided. Its acceptance logic is a simple state machine:

stateDiagram-v2
    [*] --> NoWrites: last_token = 0
    NoWrites --> Accepted: write arrives, token > last_token
    Accepted --> Accepted: write arrives, token > last_token — last_token updated
    Accepted --> Rejected: write arrives, token <= last_token — stale holder, dropped
    Rejected --> Accepted: a later write with a higher token still succeeds

    note right of Rejected
        This is what actually stops
        the double-write — not the lock,
        the resource refusing stale tokens
    end note

There’s no ceremony here. The resource just refuses to move backward.

Consensus & the Redlock Debate

A single Redis instance holding the lock is itself a single point of failure — if it goes down, every worker is blocked. Redlock proposes acquiring the lock across multiple independent Redis instances and requiring a majority to agree, on the theory that it survives any one instance failing.

It’s worth being honest about how contested this is. Redlock’s safety depends on bounded clock drift and bounded pause times — assumptions that distributed systems, by their nature, can’t actually guarantee. A majority quorum across several Redis instances doesn’t turn Redis into a real consensus system; it just makes the failure mode described above harder to trigger, not impossible.

The production-grade alternative at real scale is a system built for this from the ground up: ZooKeeper or etcd, which provide session-based locks with fencing tokens (zxid or revision numbers) built in natively, backed by actual distributed consensus protocols (ZAB, Raft). If you’re protecting something where a duplicate write costs real money, this is where the guarantee should live — not in a client library’s retry logic against a Redis quorum.

There’s a subtler reason this matters beyond raw availability: a fencing token only works if the counter issuing it survives a failover intact. If the lock service is a single Redis instance — or async-replicated — a failover to a stale replica can reissue a token value that was already used, silently breaking the entire fencing guarantee. This is exactly why ZooKeeper’s zxid and etcd’s revision numbers are consensus-replicated rather than a plain counter: they can’t move backward across a failover the way a hand-rolled Redis counter can.

The Tradeoffs

The lock service is a single point of failure in the critical path. If it’s unreachable, the correct behavior is to refuse to process — fail closed — not to fall back to running unprotected. This has to be a design-time decision, not something improvised mid-incident.

Migrating to fencing tokens creates its own risk window. If old, non-fencing-aware workers and new fencing-aware workers are both live while the resource hasn’t fully enforced fencing yet, you’ve recreated the exact bug this pattern fixes. The rollout order matters: the resource’s enforcement has to be live before you can trust any worker’s token.

You can’t verify exclusivity by “the lock said so.” The only real signal of correctness is that the resource’s fencing check accepted exactly one write and rejected every other one. Trusting the lock’s state instead of the resource’s enforcement is the same mistake that caused the original bug.

Running a consensus system is real operational overhead. ZooKeeper or etcd means a cluster to run, monitor, and maintain — which is exactly why teams reach for a single Redis instance first, and only justify the switch once the cost of a duplicate write is high enough to warrant it.

Not every problem needs a lock at all. If the operation is naturally idempotent, an idempotency key on the write itself often solves the double-processing problem without needing exclusivity in the first place — no clock-dependent failure mode, no lock service to keep available. Locks prevent concurrent execution; idempotency keys prevent duplicate side effects even when concurrent execution happens anyway. When idempotency alone is enough, it’s the simpler, safer choice.

Takeaways

• A TTL is a timer, not a liveness guarantee. It has no idea whether the lock holder is still alive — it just counts down.
• The fix lives at the resource, not the lock. A fencing token only works if the thing being protected enforces it.
• Redlock’s safety is clock-dependent, and that’s a real, debated limitation — not a settled solved problem.
• ZooKeeper/etcd-style consensus locks with native fencing are the production-grade answer when the cost of a duplicate write is high — partly because their token counters are consensus-replicated and can’t be reissued after a failover the way a plain Redis counter can.
• Migration order matters. Fencing has to be enforced at the resource before any worker’s token can be trusted.
• Ask whether you need a lock at all. Idempotency often solves the same problem with fewer moving parts and no clock dependency.

A distributed lock only ever proves one thing: who asked for exclusivity first. It can’t prove who’s still running. The moment you build a system that trusts the lock instead of the resource enforcing it, you’ve built the exact bug that took down that reconciliation job — you just haven’t hit the GC pause yet.

If you found this useful or want to discuss it further, connect with me on GitHub or LinkedIn.