Timeouts and Deadline Propagation: The Budget Nobody Sets

The client stopped waiting at two seconds. Without a propagated deadline, nothing below it knows.

Go and grep your codebase for timeout. Note the values. Then ask, for each one, where the number came from.

The honest answer is almost always: someone picked it. It was 30 because 30 is a number. Or it was copied from the example in the client library’s README. Or it is missing entirely, which means it is whatever the transport layer defaults to — and a surprising number of HTTP clients default to no timeout at all, which means forever.

This is one of the highest-leverage pieces of neglected configuration in most systems, and it causes a specific, recognisable class of outage.

The problem with a timeout

A timeout is a local, independent guess about how long something should take. Every hop makes its own guess, with no knowledge of any other hop’s guess and no knowledge of how much time the request has already spent.

client   → 2s
gateway  → 5s
service  → 10s
database → 30s

The client gives up at two seconds. It is now the only participant that knows this. The gateway, service, and database continue for up to another twenty-eight seconds, doing work whose result has nowhere to go — holding a connection, a thread, possibly a row lock, definitely CPU.

Then the client retries. Now there are two copies of the work in flight, and the first one is still holding the lock the second one needs. Retries make this worse in a hurry, because the pattern compounds: each abandoned attempt leaves a resource pinned, and each retry adds another.

The second problem is subtler and shows up in latency budgets. A duration-based timeout restarts at every hop. A “2 second timeout” applied at five hops in sequence permits up to ten seconds of total latency, because each hop starts its own two-second clock when it receives the request. Nobody intended ten seconds; nobody configured ten seconds; ten seconds is nonetheless what the system permits.

The fix: propagate an absolute deadline

Send when to stop, not how long to wait.

The entry point computes one deadline — now + 2s — and every downstream call carries it. Each service computes its remaining budget by subtracting the current time, and the whole chain stops at the same instant because they are all quoting the same absolute moment.

gRPC has this built in. The grpc-timeout header is set from the deadline on every hop, and the value shrinks as time is consumed:

func Handler(ctx context.Context, req *Request) (*Response, error) {
    // ctx already carries the caller's deadline — gRPC decoded it from the header.
    // Reserve a little time to write our own response.
    ctx, cancel := context.WithDeadline(ctx, deadline(ctx).Add(-50*time.Millisecond))
    defer cancel()

    // Shed doomed work before doing any of it.
    if remaining := time.Until(deadline(ctx)); remaining < 100*time.Millisecond {
        return nil, status.Error(codes.DeadlineExceeded, "insufficient budget remaining")
    }

    // The deadline travels automatically on outbound calls made with this ctx.
    return s.downstream.Fetch(ctx, req)
}

Two details in that snippet carry most of the value.

WithDeadline, not WithTimeout. context.WithTimeout(ctx, 2*time.Second) inside a handler sets a fresh two seconds and reintroduces exactly the restarting-clock problem you were trying to remove. Go’s context honours the earlier of the two deadlines, so it is not catastrophic — but writing it as a duration means the code no longer expresses the budget it is participating in.

Shed work you cannot finish. If 40ms remain and the call typically takes 300ms, starting it is strictly worse than failing immediately: you burn a connection and a thread to produce a result that will be discarded. Checking the remaining budget before doing expensive work is the single highest-value line in the function, and it is the one almost nobody writes.

Over plain HTTP there is no universal standard. Pick a header — many teams use a Deadline or X-Request-Deadline carrying a Unix timestamp in milliseconds — and enforce it in middleware so individual handlers cannot forget. What matters is that it is absolute and that it is set once, at the edge.

Deadlines have a real dependency on clock sync. An absolute timestamp is only meaningful if the two machines agree on what time it is. In practice NTP-disciplined hosts are close enough for millisecond-scale budgets, but it is worth knowing the assumption exists — and worth treating a large negative remaining budget as “my clock is wrong” rather than as “expired”, so a clock skew event does not manifest as a total outage.

Push it all the way down

A deadline that stops at your application boundary is doing half the job. The expensive resource is usually a level lower.

Databases. PostgreSQL has statement_timeout, which you can set per transaction from the remaining budget:

remaining_ms = int(deadline_remaining().total_seconds() * 1000)
cur.execute("SET LOCAL statement_timeout = %s", (remaining_ms,))

Without this, cancelling the client-side call does not stop the query. The database keeps executing, keeps holding its locks, and keeps consuming a connection from a pool whose size is your actual concurrency limit. A pile of abandoned-but-still-running queries is one of the fastest ways to take down a database that is otherwise perfectly healthy.

HTTP clients. Set all four kinds, because one of them is not a timeout on the thing you think it is: connect, TLS handshake, read/write, and total request. Many client libraries default the total to none — requests in Python has no default timeout at all, and http.Client in Go has Timeout: 0 meaning forever unless you set it.

Queues and background work. A visibility timeout that is shorter than the processing time means the message is redelivered while still being processed, so two workers do the same job. Longer than necessary means a crashed worker’s message sits invisible for that long. Both are deadline problems wearing different clothes.

How to choose the number

Derive it, do not pick it.

Start from measured latency. A defensible default is a small multiple of the observed p99 for that specific call — roughly 2–3× is a common starting point. That sits well clear of normal variation while still catching genuine hangs. The important property is not the multiplier; it is that the number is traceable to a measurement and gets revisited when the measurement changes.

Then check it against the budget. If the user-facing target is 2 seconds and the request touches five services, they cannot each have a 2-second timeout. Work top-down: allocate the 2 seconds across the chain, leave headroom, and let each service’s timeout be its allocation rather than an independent opinion.

Then check the invariant. For every caller/callee pair:

timeout(caller) > timeout(callee) + (retries × (timeout(callee) + backoff))

Violate it and timeouts fire inside-out: the caller abandons a call the callee is about to answer successfully. This is the arithmetic behind a lot of “intermittent” failures that vanish when someone bumps a number without understanding why it helped.

Differentiate by operation, not by service. A health check and a report generation on the same service need wildly different deadlines. One timeout per service is a number chosen for the slowest thing, which means it is far too generous for everything else — and “far too generous” is the condition under which a slow dependency becomes an outage.

Why the missing timeout is an outage

This is the failure mode worth being able to recite, because it is extremely common and it looks like something else while it is happening.

A dependency slows from 50ms to 5 seconds. Not down — slow. Your service has no timeout on that call.

Every request that touches it now holds its thread for 5 seconds instead of 50ms, a 100× increase in occupancy. Your thread pool is 200. At any meaningful request rate, all 200 threads are parked on that dependency within seconds. Requests that have nothing to do with the slow dependency now queue behind them and then fail. Health checks queue too, so the load balancer marks the instance unhealthy and removes it — sending its traffic to the remaining instances, which fill up faster.

The whole service is down because one dependency got slow, and by the time anyone is looking at dashboards, the symptom is “everything is broken” rather than “one dependency is slow”. The cascading failure that follows is well understood; the trigger was a missing timeout.

The timeout is what converts “a dependency is slow” into “calls to that dependency fail fast”, which is a survivable condition. It is not a nicety; it is the mechanism that keeps a local problem local.

Agents make budgets harder, not optional

Two properties of LLM-backed systems interact badly with everything above.

Latency is high and variable. A model call taking 400ms or 25 seconds depending on output length is normal. A timeout derived from p99 is not obviously meaningful when the distribution has that shape — and a generous timeout chosen to accommodate the tail is a generous timeout applied to every call.

Agent loops nest. An agent calls a tool, which calls a service, which calls a model. The budget has to span that whole tree, and the model call in the middle is the least predictable part of it.

The approach that works is to treat the deadline as a budget the loop spends, checked before each step rather than only at the boundary:

def run_agent(task: Task, deadline: float) -> Result:
    while time.monotonic() < deadline:
        remaining = deadline - time.monotonic()

        # Do not start a step there is no room to finish.
        if remaining < MIN_STEP_BUDGET:
            return Result.partial(reason="budget exhausted")

        step = model.next_step(task, timeout=min(remaining, MODEL_MAX))
        if step.is_final:
            return step.result

        # The tool call inherits the same absolute deadline, not a fresh timer.
        execute_tool(step, deadline=deadline)

    return Result.partial(reason="deadline exceeded")

Three things this buys you, and the third is the one that matters operationally:

• An unbounded loop becomes impossible. The deadline is the termination condition, independent of whether the model ever decides it is finished.
• Partial results beat nothing. Returning what has been established plus “budget exhausted” is far more useful than a bare timeout error — for the user and for whatever retries.
• Cost gets a bound. Wall-clock budget is a proxy for token spend, which is why it belongs in the same conversation as token FinOps. An agent with no deadline has no spend ceiling, and the first time that matters is a bill.

The rule worth remembering

Set a deadline at the edge, propagate it as an absolute time, check the remaining budget before starting expensive work, and make sure every caller’s timeout exceeds its callee’s plus retries.

Missing timeouts turn slow dependencies into outages. Duration-based timeouts multiply the latency budget by the depth of the call graph. A propagated deadline fixes both, and it is a few lines of middleware.