The queue is a buffer between a fast producer and a slower consumer — absorbing bursts instead of dropping them.
If you connect a fast component directly to a slow one, the fast one either overwhelms the slow one or has to wait for it. Put a queue in between and everything changes: the producer drops work in and moves on, the consumer takes work out at its own pace, and a burst that would’ve crushed the consumer just makes the queue temporarily deeper. That single idea — a buffer that decouples producers from consumers — is one of the most important patterns in distributed systems, and it’s become the standard way to put a shock absorber in front of expensive, rate-limited LLM calls. This is the refresher.
What the queue buys you
Look at the diagram. A producer publishes messages to a broker (Kafka, RabbitMQ, SQS, Redis Streams); the messages wait there, in order and persisted, until a consumer takes one, processes it, and acknowledges it. From that arrangement you get four superpowers:
- Decoupling. The producer doesn’t know or care who consumes, how many consumers there are, or whether they’re up right now. You can deploy, restart, or scale the consumer independently. The producer just needs the broker.
- Burst absorption. When the producer spikes (a traffic surge, a batch job kicking off), the queue grows instead of the consumer falling over. The buffer trades latency for survival — messages wait a bit longer, but nothing gets dropped.
- Resilience. If the consumer crashes, messages sit safely in the queue until it comes back. Work is persisted, not lost. Contrast with a direct call, where a dead consumer means a failed request.
- Scalability via competing consumers. Need to drain the queue faster? Add more consumers (the three in the diagram) and they share the load. Horizontal scaling becomes “add workers.”
Backpressure: the queue tells you the truth
The queue’s depth is one of the most honest health signals a system produces. It’s literally demand minus capacity, measured live. If the queue stays shallow, consumers are keeping up. If it’s steadily growing, your consumers are underwater — the producer is outpacing them, and you need more consumers, faster ones, or (if it’s a genuine overload) a way to shed load. This is backpressure: the system pushing back when downstream can’t keep up.
The crucial discipline is to monitor queue lag and alert on it. A growing queue is an early warning that something downstream is struggling, often before users feel it — you have the buffer’s worth of time to react. Ignore it and the queue grows until it exhausts storage or blows past your latency budget, at which point the shock absorber has bottomed out. “Consumer lag” is to a queue-based system what a check-engine light is to a car.
The semantics that trip everyone up
Distributed queues force you to confront delivery guarantees, and the reality is humbler than people expect:
- At-most-once — might lose messages, never duplicates. Rarely what you want.
- At-least-once — never loses messages, but may deliver duplicates (e.g., the consumer processed a message but crashed before acknowledging, so it’s redelivered). This is the normal, practical default.
- Exactly-once — the holy grail, and largely a fiction across system boundaries. Where it exists it’s expensive and narrowly scoped. The ACID/consistency post explains why: true exactly-once across a network is the same hard coordination problem as strong distributed consistency.
The practical answer is to embrace at-least-once and make your consumers idempotent — processing the same message twice produces the same result as processing it once (dedupe on a message ID, use upserts, check-then-act atomically). Idempotency is how you live comfortably with a guarantee weaker than “exactly once,” and it’s one of the most valuable habits in distributed systems.
And when a message simply can’t be processed — malformed, or triggering a bug every time — you don’t want it redelivered forever, blocking everything behind it (a “poison message”). That’s what a dead-letter queue is for: after N failed attempts, the broker sets the message aside so the rest of the queue keeps flowing, and you inspect the failures later.
The AI-era relevance
This pattern didn’t just survive into the AI era — it became one of the primary tools for building sane LLM systems.
Queues tame expensive, rate-limited model calls. LLM inference is slow, costly, and rate-limited by providers. Calling a model synchronously inside a user request ties up a connection for the whole generation and falls apart under a burst. Put a queue in front: accept the request fast, enqueue the work, let a pool of workers pull jobs and call the model at a controlled rate that respects provider limits. The queue is your rate limiter and your burst buffer. This is the standard architecture for any non-trivial AI workload — batch embedding jobs, document processing pipelines, async generation.
Agentic and event-driven systems run on message passing. Recall from the concurrency post that “share memory by communicating” avoids whole classes of bugs. Multi-agent systems and event-driven AI pipelines lean on exactly this: agents and steps communicate through queues and event streams rather than shared mutable state, which decouples them, makes the flow observable, and lets you scale each stage independently. The dead-letter queue becomes where an agent step that keeps failing gets parked instead of looping forever.
Consumer lag is a first-class AI-ops metric. When your embedding backlog or inference queue is growing, that’s your system telling you demand exceeds your model-serving capacity — the single clearest signal for when to scale workers, raise rate limits, or shed load. It’s the same backpressure lesson, now guarding your GPU budget.
The takeaway
A queue is a buffer that decouples the thing producing work from the thing doing it, and from that one idea you get burst absorption, resilience, independent scaling, and an honest, real-time health signal in the queue depth. The subtleties — at-least-once delivery, idempotent consumers, dead-letter queues, backpressure — are the difference between a queue that smooths your system and one that silently drops or duplicates work. None of it changed when LLMs arrived; it just found its most important modern job, standing between your users and the slow, expensive, rate-limited models they’re waiting on. When a fast thing needs to talk to a slow thing, put a queue between them.