· 6 min read AISREOpen Source

Your runbooks are lying to your agents: RAG for on-call knowledge

On-call agents are only as good as what they retrieve. Making runbook RAG fresh, exact-match aware, and poison-resistant — with a runnable starter repo.

Diagram: runbook RAG pipeline from a naive starter to production hardening — hybrid retrieval, freshness signals, permissions, provenance, and a verification loop

Every SRE team has a runbook that lies. It describes the architecture from before the migration, names a dashboard that was deleted, and ends with “escalate to Dave,” who left in 2024. Documentation rot is so universal that “the runbook is already lying to you” has become shorthand for the whole genre.

For years the rot was survivable, because the reader was a human who’d squint at the stale instruction and route around it. That defense is now leaving the building: we’re wiring runbooks into retrieval pipelines and handing the answers to on-call agents — mine included. Retrieval-augmented generation turns your runbook corpus from documentation into behavior. Which means retrieval quality just became a production concern, and the runbook that lies is no longer an annoyance. It’s an incident with a delay timer.

This post is about making runbook retrieval trustworthy enough to sit under an agent. It comes with code: runbook-rag-starter (Apache-2.0), the smallest useful version, plus the hardening path in the order the failures actually bite.

Start insultingly simple — on purpose

The starter is about 40 lines of Python around three decisions: a folder of Markdown runbooks, embeddings in a single SQLite file via sqlite-vec, and a CLI — runbook-rag index ./runbooks, then runbook-rag ask "what do we do when checkout-svc latency spikes?". Top-k chunks get stuffed into the prompt; the answer cites the files it used. No Postgres, no vector service, no Docker.

It is deliberately mediocre, and the README says so. Each file is one chunk, so a 50-page runbook is one vector. There’s no keyword fallback, no reranker, no freshness signal, and if retrieval comes back empty the model will improvise. These aren’t oversights — they’re the curriculum. Run your real on-call questions through it, watch exactly where it fails, and you’ll know which of the following sections to build first, with evidence instead of a vendor’s roadmap. (Vector embeddings doing the heavy lifting here, if you want the primer.)

The failures cluster into five patterns, and they’re predictable enough to be a checklist.

Where chatbot RAG intuitions fail on-call

1. On-call queries are full of exact strings. “What’s the runbook for payments-gw-prod-7?” Pure cosine similarity is spectacular at paraphrase and mediocre at identifiers — hostnames, error codes, ticket IDs, flag names get embedded into mush. Chatbot corpora mostly dodge this; on-call queries are made of it. The fix is hybrid retrieval: BM25 keyword search alongside the vector search, merged before ranking. This is the single highest-value upgrade to the naive version, and it’s usually the first failure you’ll observe in replay.

2. Freshness is a correctness property. A news chatbot serving last month’s article is mildly annoying. A retrieval layer serving the pre-migration failover procedure is actively harmful — the words are fluent, the steps are confident, and the topology they describe no longer exists. Runbook RAG needs freshness as a first-class signal: a last_verified date on every document, a staleness penalty at ranking time, an age flag in the answer (“this runbook was last verified 14 months ago”), and expiry for anything past your tolerance. If the corpus has no verification dates, that metadata backfill is the project.

3. Retrieval must respect permissions. Runbooks contain topology, credentials-adjacent detail, and customer names. The moment retrieval sits behind an agent that anyone can query — a Slack bot, an on-call assistant — it will happily surface documents the asker couldn’t open. Permission-aware retrieval (filter by the requester’s ACLs at query time, not index time — people change teams) is table stakes for unified knowledge platforms precisely because retrofitting it is miserable.

4. The corpus is an attack surface. Once agents act on retrieved text, whoever can write to the runbook wiki can influence production actions. That’s memory-and-context poisoning — ASI06 in the OWASP agentic Top 10 — and the defense is treating runbook writes like code: reviewed changes, provenance on every document, and retrieval that surfaces where each answer came from so a poisoned source is findable and evictable rather than ambient.

5. Empty retrieval must not become fiction. When nothing relevant exists, a naive pipeline stuffs the prompt anyway and the model fills the silence — plausible, wrong, mid-incident. Require citations; make “no runbook found” an explicit, honorable answer; and log those misses, because the query log of unanswered on-call questions is the best runbook-writing backlog your team has ever had.

From retrieval to execution — slowly

Reading runbooks is the entry level. The 2026 tooling wave is pushing further — incident.io calls it “intelligent runbook execution”: context-aware workflows where the agent doesn’t just quote the procedure but runs it. That’s also where the blast radius lives, so the promotion path matters. The three-tier model that’s emerging as consensus — advisory → approval-gated → conditional autonomy — maps exactly onto the bounded-autonomy argument I’ve made before: each tier is earned with a measured track record, never granted by roadmap date.

The dependency is directional: execution trust is capped by retrieval trust. An agent with flawless judgment acting on a stale runbook is a flawless executor of the wrong procedure. Harden the retrieval layer first; it’s the foundation the autonomy tiers stand on. (And when the incident’s over, close the loop the other way — the postmortem that updates the runbook is what keeps the corpus worth retrieving from.)

Honest caveats

RAG does not fix wrong documentation — it distributes it more efficiently, with better production values. If your runbooks are rotten, retrieval is an amplifier, and the real fix is the boring loop where every incident review updates or retires the runbook it exercised. Measure retrieval quality separately from answer quality (replay real queries; check the right document lands in the top results) or you’ll tune the model to compensate for a search problem. And a starter repo is a starting point: sqlite-vec on one box is honest about what it is, and what it is not is a multi-tenant, permission-aware knowledge platform.

What to do Monday

  • Clone the starter and index your real runbooks. runbook-rag-starter, ten minutes, your actual corpus. The failures you see are your roadmap.
  • Replay 20 real on-call questions from your incident channel and score whether the right runbook lands in the top 3. That number is your retrieval baseline, and it’s probably lower than you think.
  • Add last_verified to your runbook template today. The metadata is free at write time and unaffordable to reconstruct later.
  • Route exact-identifier queries through keyword search before investing in anything fancier — hybrid retrieval is the biggest single jump in on-call retrieval quality.
  • Log every empty retrieval. Each one is either a missing runbook or a missing alias for an existing one. Both are cheap to fix once you can see them.

The uncomfortable truth about on-call AI is that most of its quality ceiling is set before the model ever runs — by whether the right document exists, is current, and gets retrieved. Teams chase agent quality because it’s exciting, and ignore corpus quality because it’s chores. The chores are the product. Your agent is only ever as honest as the shelf you point it at.

Frequently asked questions

Why is RAG for runbooks different from ordinary chatbot RAG?

Because the queries and the stakes are different. On-call questions are full of exact identifiers — hostnames, error codes, ticket IDs — that pure vector similarity handles badly, so runbook retrieval needs a hybrid of keyword and semantic search. Freshness is a correctness property, not a nice-to-have: a runbook describing the pre-migration architecture isn't slightly stale, it's wrong, and an agent acting on it does damage. And the consumer is often an agent mid-incident rather than a human who'll sanity-check the answer, which raises the bar on citations, permission-aware retrieval, and refusing to answer when retrieval comes back empty.

What is the biggest failure mode of feeding runbooks to an AI agent?

Confident answers built on stale or missing documents. If retrieval returns nothing relevant, a naive pipeline still stuffs the prompt and the model improvises — plausible, wrong, and delivered mid-incident. The fixes are mechanical: require the answer to cite its source documents, penalize or flag documents past a verification age, and make the empty-retrieval case an explicit 'I found nothing' rather than a generation. Retrieval quality is measurable on its own — if the right document isn't in the top results for replayed real queries, no amount of model quality downstream can save the answer.

How should I start building runbook RAG — what's the minimal version?

A folder of Markdown runbooks, an embedding model, a SQLite file with a vector extension like sqlite-vec, and a retrieval loop that stuffs top-k chunks into the prompt with citations — about 40 lines of Python, no separate vector database or Docker required. That version is deliberately naive: one vector per file, no keyword fallback, no reranking, no freshness signal. Start there, replay real on-call questions against it, and fix the failure you actually observe first — usually chunking or exact-match misses. The runbook-rag-starter repo on GitHub is exactly this starting point, with the missing pieces documented as the roadmap.

Can an on-call agent execute runbooks it retrieves, not just read them?

Eventually, but gate it by tier. The sane progression is advisory (the agent quotes the runbook and a human acts), then approval-gated (the agent proposes the runbook's commands and a human approves each execution), then conditional autonomy (the agent executes pre-vetted, low-blast-radius steps within limits and escalates the rest). Each promotion should be earned with a track record, and the runbook corpus itself becomes part of your attack surface once agents act on it — a poisoned or stale document is now an instruction to production, which is why write-path controls and provenance matter as much as retrieval quality.