Concurrency interleaves tasks on one core. Parallelism runs them at the same instant on many. They’re different problems.
Rob Pike gave the cleanest definition anyone has: concurrency is dealing with many things at once; parallelism is doing many things at once. They sound like synonyms and they are not, and mixing them up is behind a startling share of “why didn’t adding threads make it faster?” confusion. This is a refresher on the distinction, because getting it straight is what lets you actually scale — including the very modern problem of fanning out model calls and saturating a GPU.
The core difference
Concurrency is a structuring concept. It’s about a program juggling multiple tasks whose lifetimes overlap — starting task B while task A is waiting on something, then coming back to A. On a single core, as in the left of the diagram, only one thing truly runs at any instant; the system just switches between tasks fast enough that they all make progress. Concurrency is about composition and coordination.
Parallelism is an execution concept. It’s about literally doing multiple things at the same physical instant, which requires multiple execution units — multiple cores, multiple machines, or the thousands of lanes on a GPU. The right side of the diagram: Task A on Core 1 and Task B on Core 2, genuinely simultaneous.
You can have either without the other. A single-core async web server is highly concurrent with zero parallelism. A tight numerical loop split across cores is parallel without needing to be concurrent in any interesting way. Most real systems want both, for different parts of the work.
The question that tells you which you need
Ask: is this task waiting, or computing?
- I/O-bound work — waiting on the network, disk, a database, an external API. The CPU is idle during the wait. Here you want concurrency: while one request waits, do useful work on another. Adding more cores barely helps, because the bottleneck is waiting, not computing.
- CPU-bound work — heavy math, encoding, compression, tensor operations. The CPU is pegged. Here you want parallelism: split the work across cores so more of it happens at once. Async gives you nothing here, because there’s no waiting to overlap.
Getting this backwards is the classic mistake: throwing threads at an I/O-bound problem (where async would’ve been simpler and lighter) or reaching for async on a CPU-bound problem (where it does nothing but add complexity).
The tools, and Python’s famous asterisk
- async/await, event loops, goroutines → concurrency. Lightweight, great for holding tens of thousands of waiting connections without a thread each. This is the answer to the old C10K problem (how do you serve 10,000 simultaneous connections?) and its modern C10M sequel.
- Threads and multiprocessing → the path to parallelism, letting the OS schedule work across cores.
- SIMD and GPUs → data parallelism at a massive scale — the same operation over thousands of data elements at once.
And the asterisk every Python engineer hits: the GIL (Global Interpreter Lock) means standard CPython runs only one thread of Python bytecode at a time. So Python threads give you concurrency but not CPU parallelism — they’re great for I/O-bound work (the GIL is released during I/O waits) and useless for speeding up pure-Python computation. For CPU-bound parallelism in Python you reach for multiprocessing, or native extensions (NumPy, PyTorch) that release the GIL and parallelize in C/CUDA. Understanding this one fact resolves most “my Python threads didn’t speed anything up” bugs. (Recent CPython has an experimental no-GIL build, but the mental model still holds for the code most people run today.)
Why this is central to AI infrastructure
This distinction is not academic when you’re building AI systems — it is the performance story.
Model calls are I/O-bound; fan them out with concurrency. An agent or a batch job making many LLM calls spends almost all its time waiting on the provider. The right tool is async concurrency: fire off many requests and await them together. Do this and a job that would take 100 seconds sequentially finishes in the time of the slowest single call. Reach for threads or processes here and you’ve added overhead to solve a waiting problem. This one pattern — concurrent fan-out of model calls — is among the biggest latency wins available in agentic systems.
Inference itself is a parallelism problem. The GPU is a massive data-parallel machine; a matrix multiply is thousands of lanes doing the same op at once. Serving throughput comes from batching — running many requests’ tokens through the hardware in parallel. That’s why inference servers work so hard to fill batches: an under-full batch wastes the parallel hardware you’re paying for.
The two compose. A well-built inference service is concurrent at the request-handling layer (async accept and queue many requests while they wait) and parallel at the compute layer (batch them onto the GPU). Confuse the layers — try to make request handling “parallel” with a thread per request, or try to make GPU work “concurrent” without batching — and you leave enormous throughput on the table.
The takeaway
Keep the one question handy: is this work waiting or computing? Waiting wants concurrency; computing wants parallelism. That single test tells you whether to reach for async or for more cores, saves you from the GIL trap, and — in the AI era — tells you to fan out your model calls concurrently while batching your tensor work in parallel. The words sound interchangeable. The systems you build with them are not.