A deadlock hangs forever; a race condition gives the wrong answer when the timing is unlucky. Both depend on scheduling you don’t control.
The worst bugs are the ones that don’t reproduce. It passes every test, runs fine in staging, then corrupts a balance or hangs a worker exactly once a week in production, always at the worst time, never when you’re watching. That signature — intermittent, timing-dependent, load-correlated — is almost always a concurrency bug. This is a refresher on the two big ones, race conditions and deadlocks, because understanding them is the difference between “we added retries and hoped” and actually fixing it.
Race conditions: when the answer depends on timing
A race condition happens when two threads touch shared state without coordination, and the result depends on who happens to run first. The right side of the diagram is the canonical example: two threads each do a deposit. Both read the balance (100), both add 50, both write back (150). One deposit vanished. The balance should be 200; it’s 150. Nobody wrote buggy arithmetic — the bug is that “read, modify, write” isn’t a single indivisible step, and the two threads interleaved in the middle of it.
The key concept is atomicity. An operation is atomic if it happens all-at-once from every other thread’s point of view — no one can observe it half-done. balance += 50 looks atomic in your source code and is absolutely not atomic at the machine level; it’s a read, an add, and a write, and any of those can be interrupted. Race conditions are what live in that gap.
The fixes all enforce atomicity:
- Locks (mutexes) — a thread grabs the lock, does its read-modify-write, releases it. Others wait. The critical section becomes effectively atomic.
- Atomic instructions / compare-and-swap (CAS) — hardware primitives that do “read-modify-write, but atomically” without a full lock, the basis of lock-free data structures.
- Don’t share mutable state — the most robust fix. Message passing, immutability, and single-owner designs sidestep the whole problem. This is the philosophy behind Go’s “share memory by communicating” and the actor model.
Deadlocks: when locks fix one problem and create another
So you add locks. Now you can create a deadlock — the left side of the diagram. Thread 1 holds Lock A and wants Lock B. Thread 2 holds Lock B and wants Lock A. Each is waiting for a resource the other holds, so neither ever proceeds. The program doesn’t crash; it just… stops. Forever.
Deadlock requires four conditions to hold simultaneously (the Coffman conditions): mutual exclusion (resources can’t be shared), hold-and-wait (you keep what you have while waiting for more), no preemption (locks can’t be forcibly taken), and circular wait (a cycle in who’s waiting for whom). Break any one and deadlock is impossible.
The most practical one to break is circular wait, and the fix is beautifully simple: always acquire locks in the same global order. If every thread that needs both A and B always grabs A before B, the cycle can’t form — nobody’s ever holding B while waiting for A. Consistent lock ordering prevents a huge fraction of real deadlocks and costs nothing but discipline.
The general principle
Locks are the obvious tool and a double-edged one:
- Too little locking → race conditions (corrupted state).
- Too much locking → deadlocks (hangs) and contention (everything queues on one lock, killing the parallelism you were trying to use).
The craft is holding the smallest lock for the shortest time in a consistent order — protecting exactly the shared state that needs it and no more.
Why this still matters in the AI/cloud era
You might think managed runtimes and higher-level frameworks retired these problems. They relocated them.
Distributed systems have distributed versions of both. A “lock” across machines is a distributed lock (etcd, ZooKeeper, Redis locks), and it has all the same failure modes plus new ones — what happens when the lock-holder dies, or the network partitions and two nodes both think they hold it? The race condition becomes a distributed race, and “two workers both processed the same job” is the read-modify-write bug wearing a cloud costume. Idempotency and optimistic concurrency (version checks on write) are the distributed-scale answers.
Agents deadlock too. An agentic system where agents wait on each other’s outputs can form the exact circular-wait pattern in the diagram — agent A blocked on a result from agent B, which is blocked on A. It presents as “the workflow just hung,” and it’s a deadlock by another name. Timeouts are the crude preemption that breaks the cycle; better orchestration avoids the cycle entirely.
Shared state under concurrent load is everywhere in inference serving. A serving system juggling many concurrent requests over shared caches, batch queues, and GPU memory is a giant concurrency problem. The “it only fails under load” bug — where everything’s fine until traffic is high enough to make the unlucky interleaving likely — is the race condition from the diagram, and load is just what makes the dice roll often enough to hit it.
The takeaway
When a bug is intermittent, timing-dependent, and worse under load, stop looking for a logic error and start looking for shared state touched without coordination. Race conditions come from non-atomic access to shared data; deadlocks come from the locks you add to fix them, waiting on each other in a cycle. The tools are old — atomicity, consistent lock ordering, not sharing mutable state — and they map directly onto the newest distributed and agentic systems, where the same two bugs just operate at a larger scale and hide a little better.