The Evidence an AI Operator Owes You Before It Acts

In August I wrote about the postmortem nobody is writing for agent-caused incidents and proposed five sections the template needs, starting with agent context at decision time. A reader asked the obvious follow-up: where does that section’s content come from?

The honest answer is that in most deployments it doesn’t come from anywhere. The incident happens, someone opens the template, and the only artifact is a tool-call log that says restart_deployment(namespace=checkout, name=api) at 14:02. The prompt is gone, the retrieved runbook was edited since, the model alias now points at a newer build, and the policy file has had two commits. You are writing fiction with timestamps.

Digital forensics has a name for the discipline that fixes this: forensic readiness. Rowlingson’s ten-step process for forensic readiness (International Journal of Digital Evidence, 2004) starts from the scenarios that will need evidence and works back to what must be collected before they happen. Agents need the same move, and they make it easier than usual, because there is a single choke point every consequential action passes through: the tool call.

Start from the questions, not the fields

Most logging designs start with “what can we capture?” and end with a schema nobody can map to an investigation. Turn it around. Every agent incident review I’ve seen, or read about, asks some version of these eight questions:

#	Investigation question	Fields that answer it
1	Which agent, and which build of it, acted?	agent id + version, model id (resolved, not the alias)
2	What did it see when it decided?	prompt hash, every context source with URI + content hash
3	What exactly did it try to do?	tool name, arguments, hash of the tool’s schema at call time
4	Who or what allowed it?	policy decision, policy version, matching rule id
5	How much authority did it hold at that moment?	authority rung, and what granted it
6	Was a human involved, and how?	approval mode: none, notified, approved, by whom
7	How old was what it saw?	decision time, oldest input timestamp
8	Has this record been altered since?	previous record hash, this record’s hash

Three of those rows are worth defending, because teams skip them.

The tool schema hash (row 3). The tool’s name tells you nothing about what it could do on that day. MCP servers ship releases. A restart_deployment tool that took one deployment last month may take a label selector this month. If you only logged the name, you cannot tell whether the agent did something dumb with a narrow tool or something reasonable with a tool that had quietly widened. Schema drift is one of the ordinary failures worth injecting on purpose for exactly this reason.

The resolved model id (row 1). model: latest in a log is an admission that you don’t know which model acted. Log what the provider returned, not what you asked for.

Input age (row 7). Agents act on telemetry, and telemetry lags or goes partial without announcing it. I argued in telemetry integrity that an agent can’t tell a quiet system from a broken collector. The evidence record should at least let the reviewer tell. If the newest metric the agent used was four minutes old during a fast-moving incident, that is a finding.

The record, as code

Here is the whole thing: a capture function that runs before the tool executes, a check that maps each question to its fields, and a hash chain. Standard library only. Save it as evidence_record.py and run it with python3.

"""Decision-time evidence record for an AI operator. Stdlib only.

Each investigation question names the fields that answer it. A record that
cannot answer a question is rejected *before* the action runs.
"""
import hashlib, json
from datetime import datetime, timezone

QUESTIONS = {
    "Which agent, and which build of it, acted?":       ["agent.id", "agent.version", "model.id"],
    "What did it see when it decided?":                  ["context.prompt_sha256", "context.sources"],
    "What exactly did it try to do?":                    ["action.tool", "action.args", "action.tool_schema_sha256"],
    "Who or what allowed it?":                           ["policy.decision", "policy.version", "policy.rule_id"],
    "How much authority did it hold at that moment?":    ["authority.rung", "authority.granted_by"],
    "Was a human involved, and how?":                    ["human.mode"],
    "How old was what it saw?":                         ["timing.decided_at", "timing.context_as_of"],
    "Has this record been altered since?":               ["integrity.prev_sha256", "integrity.sha256"],
}

def get(rec, path):
    for key in path.split("."):
        if not isinstance(rec, dict) or key not in rec:
            return None
        rec = rec[key]
    return rec

def unanswerable(rec):
    """Questions this record cannot answer. Empty list = admissible."""
    return [q for q, fields in QUESTIONS.items()
            if any(get(rec, f) in (None, "", []) for f in fields)]

def sha(obj):
    return hashlib.sha256(json.dumps(obj, sort_keys=True).encode()).hexdigest()

def seal(rec, prev_sha):
    """Chain the record to its predecessor so a deletion or edit is detectable."""
    body = {k: v for k, v in rec.items() if k != "integrity"}
    rec["integrity"] = {"prev_sha256": prev_sha, "sha256": sha({**body, "prev": prev_sha})}
    return rec

def verify_chain(records):
    prev = "0" * 64
    for i, r in enumerate(records):
        body = {k: v for k, v in r.items() if k != "integrity"}
        if r["integrity"]["prev_sha256"] != prev or r["integrity"]["sha256"] != sha({**body, "prev": prev}):
            return f"chain broken at record {i}"
        prev = r["integrity"]["sha256"]
    return "chain intact"

def capture(prompt, sources, tool, args, schema, policy, authority, human_mode):
    now = datetime.now(timezone.utc).isoformat()
    return {
        "agent": {"id": "k8s-remediator", "version": "1.4.2"},
        "model": {"id": "provider/model-2026-09-01"},
        "context": {"prompt_sha256": sha(prompt),
                    "sources": [{"uri": u, "sha256": sha(c), "as_of": t} for u, c, t in sources]},
        "action": {"tool": tool, "args": args, "tool_schema_sha256": sha(schema)},
        "policy": policy,
        "authority": authority,
        "human": {"mode": human_mode},
        "timing": {"decided_at": now, "context_as_of": min(t for _, _, t in sources)},
    }

if __name__ == "__main__":
    schema = {"name": "restart_deployment", "params": {"namespace": "str", "name": "str"}}
    src = [("prom://checkout/p99", "p99=4.2s", "2026-11-13T14:02:10Z"),
           ("k8s://checkout/events", "OOMKilled x3", "2026-11-13T14:01:55Z")]
    good = capture("diagnose checkout latency", src, "restart_deployment",
                   {"namespace": "checkout", "name": "api"}, schema,
                   {"decision": "allow", "version": "pol-218", "rule_id": "reversible-restart"},
                   {"rung": "supervised", "granted_by": "budget:checkout/reversible"},
                   "approved:oncall")
    # A typical tool-log line: it knows the call, not the decision.
    thin = {"action": {"tool": "restart_deployment", "args": {"namespace": "checkout", "name": "api"}},
            "timing": {"decided_at": good["timing"]["decided_at"]}}

    chain, prev = [], "0" * 64
    for rec in (good, json.loads(json.dumps(good))):
        chain.append(seal(rec, prev)); prev = rec["integrity"]["sha256"]

    print("full record, unanswerable:", unanswerable(chain[0]))
    print("tool-log line, unanswerable:", len(unanswerable(thin)), "of", len(QUESTIONS))
    for q in unanswerable(thin):
        print("   -", q)
    print(verify_chain(chain))
    chain[0]["action"]["args"]["name"] = "payments"   # someone edits history
    print("after edit:", verify_chain(chain))
    assert unanswerable(chain[1]) == [] and len(unanswerable(thin)) == len(QUESTIONS)

Output:

full record, unanswerable: []
tool-log line, unanswerable: 8 of 8
   - Which agent, and which build of it, acted?
   - What did it see when it decided?
   - What exactly did it try to do?
   - Who or what allowed it?
   - How much authority did it hold at that moment?
   - Was a human involved, and how?
   - How old was what it saw?
   - Has this record been altered since?
chain intact
after edit: chain broken at record 0

The line that matters is the second one. The “typical tool-log line” in the script holds the tool name, the arguments and a timestamp, which is what a basic tool-call log looks like. It fully answers none of the eight questions. Even “what exactly did it try to do?” fails, because without the schema hash you don’t know what the tool meant that day.

Make it a precondition, not a log

The design decision that turns this from documentation into a control: unanswerable(record) must return an empty list before the tool call is dispatched. If the runtime can’t say which policy rule allowed the action or what authority the agent held, the action doesn’t run.

That sounds strict. It’s the same rule we already apply to humans with production access in regulated environments: no change without a ticket that says who approved it and under what procedure. We just never wrote it down for the thing that doesn’t get tired at 3 a.m.

It also puts the check in the right place. The MCP gateway or tool proxy already sees every call, already knows the tool schema, and usually makes the policy decision. It is the natural emitter. Asking each agent to log itself is asking the system under investigation to write its own alibi.

There’s a regulatory reason to start now as well. The EU AI Act’s Article 12 requires high-risk AI systems to “technically allow for the automatic recording of events (logs) over the lifetime of the system.” Whether a given agent is high-risk is a legal question I won’t answer. The engineering point is that “we logged the tool calls” will not read as traceability to anyone who has to reconstruct a decision.

What it costs

Less than you’d think. The record is a few hundred bytes plus hashes. The expensive part, the prompt and the retrieved context, is stored by hash, with the content kept in object storage under whatever retention and privacy policy you already use for traces. That separation matters: the content-capture decision is a privacy decision, and the evidence record shouldn’t force it. A hash proves which content was used without holding the content in the audit log.

Limits

• A hash chain is tamper-evident, not tamper-proof. Anyone who can rewrite every record can rebuild the chain. Anchor the latest hash periodically somewhere the agent’s credentials can’t write: write-once object storage in a separate account, or your SIEM.
• Hashes prove identity, not meaning. You’ll know exactly which runbook version the agent retrieved. You still need a person to read it.
• Eight questions is my list, not a standard. It came from working backwards through agent incident reviews. Yours may need a ninth. The useful property is that each field is there because a question needs it, so adding one is a deliberate act.
• It records the decision, not the reasoning. Chain-of-thought text, where you have it, is not a faithful account of why a model acted. Treat it as a supporting exhibit, never as the explanation.

What I’d do this week

1. Take your last agent-involved incident and try to answer the eight questions from what you actually have. Count the gaps.
2. Put the capture in the gateway or tool proxy, not in the agent.
3. Start with rows 3 and 1, the schema hash and the resolved model id. They’re the cheapest to add and the most often missing.
4. Turn unanswerable() == [] into a precondition for one action class, the most consequential one, and watch what breaks.

The postmortem template says what to write. This record decides whether you can write it.