Secrets Management: Scope, Lifetime, and the Blast Radius of a Leak

Storing it safely is the easy half. The hard half is what it can do, for how long, and whether you dare revoke it.

Most secrets-management effort goes into the vault: which one, how to encrypt it, how to sync it. That work matters, and it is the easy half.

The half that decides what a breach actually costs is answered long before the leak, by three properties of the credential itself:

• Scope — what can it do?
• Lifetime — for how long?
• Revocability — can you kill it without causing an outage?

A perfectly stored credential that grants full database access, never expires, and is embedded in forty services is a bad credential. A credential in a slightly worse store that grants read on one table for fifteen minutes is a good one. The store is not the variable that dominates.

The right frame is not “how do we prevent leaks”. Leaks happen — a logged header, a debug endpoint, a compromised dependency, a laptop, a screenshot. The frame is: assume it leaks. What does that cost?

Where secrets actually leak from

Ranked roughly by how often they cause real incidents, and none of these involve an attacker doing anything clever:

Source control. Still number one, and the history is what gets people — deleting the file does not remove it from the repo, so a secret committed once is present forever unless the history is rewritten. Every organisation should have pre-commit and server-side scanning, and every commit containing a live credential should trigger revocation automatically. The uncomfortable version: a secret that has ever been in a repo, however briefly, must be treated as compromised.

Logs. Your framework logs the request. The request has an Authorization header. It is now in your log aggregator, replicated, indexed, backed up, and readable by everyone with log access — a much larger group than the people who can read the vault. Redact at the logging layer, by allowlist, not by pattern-matching for things that look like secrets.

Environment variables. The whole environment appears in crash dumps, in error-reporting payloads, in /proc/<pid>/environ, in debug endpoints that print config, and in the Pod spec that anyone with get pod can read. They are convenient precisely because everything can read them, which is the problem.

CI/CD. Build logs, cached artifacts, and — the sharp edge — pull requests from forks. A CI configuration that exposes production secrets to a workflow triggered by an outside contributor’s PR is a widely-exploited pattern, not a hypothetical one.

Kubernetes Secrets, understood accurately. A Kubernetes Secret is base64-encoded, not encrypted. Base64 is an encoding. Without encryption at rest for etcd they sit in etcd in the clear, and anyone with get secrets in the namespace can read every one. They are a distribution mechanism, which is fine; they are not a protection mechanism, which is what people assume.

Scope: the lever with the most leverage

Least privilege is repeated so often it has stopped landing. The concrete version is a question you can ask about any credential in your system right now: if an attacker had this, what could they do?

If the answer is “read every table”, the scope is wrong, whatever the storage looks like.

Scope has four dimensions and most credentials are over-broad on all four:

Action. Read, write, delete, administer. A reporting service needs SELECT. It very often has db_owner, because that is what was easiest to grant on day one and nobody revisited it.

Resource. One table, one bucket, one prefix. s3:GetObject on arn:aws:s3:::* is common and nearly always wrong.

Time. Fifteen minutes versus forever. This is the cheapest dimension to tighten and the one most often left at “forever”.

Network. A credential usable only from inside your VPC is worth much less to an attacker outside it. Condition keys and resource policies make this enforceable rather than aspirational.

A useful audit that takes an afternoon: list every credential in production, and for each write down the worst thing someone could do with it. The list will contain at least one entry that changes what you work on next.

Lifetime: make rotation unnecessary

Rotation is the standard advice and it is the second-best answer. Rotating a long-lived credential every ninety days means that in the worst case a leaked credential is valid for ninety days, and it means someone maintains a rotation process that runs four times a year — which is to say, an untested process.

The better answer is to shorten the lifetime until rotation stops being an event. A credential valid for fifteen minutes does not need a rotation policy; expiry is built into it. The operational burden goes away because the problem goes away.

This is what workload identity provides, and it is the single highest-value change available in this area:

1. The platform attests to what the workload is — a signed JWT bound to its service account, issued by the cluster.
2. A credential broker verifies that attestation against policy.
3. It mints a short-lived, scoped credential.
4. The workload holds it in memory and refreshes before expiry.

# The token is projected by the platform, signed, short-lived, audience-bound.
with open("/var/run/secrets/tokens/vault-token") as f:
    attestation = f.read()

# Exchange identity for a credential. No long-lived secret exists anywhere here.
creds = broker.exchange(
    attestation=attestation,
    role="reporting-read-only",
    ttl="15m",
)

The property that matters is in the comment: identity is proven, not presented. There is no long-lived string to steal, log, or commit. The bootstrap problem — “how do I securely give the service the secret it needs to fetch its secrets” — dissolves, because the platform already knows what the workload is and can say so cryptographically.

Every major platform now supports this: IRSA and Pod Identity on EKS, Workload Identity on GKE and AKS, Vault’s Kubernetes auth method, SPIFFE/SPIRE for the vendor-neutral version. If you are still distributing static cloud credentials to Pods, this is the migration with the best ratio of risk reduced to effort spent.

Revocability: the property nobody tests

Here is the question that separates a secrets strategy from a secrets store: if you learned right now that a production credential was compromised, could you revoke it in the next five minutes?

For most organisations the honest answer is no, and the reason is always the same: the credential is shared. Revoking it breaks eleven services, three of which nobody is sure about, so the decision escalates, and the credential stays live while people talk.

That is the failure. The credential was designed to be un-revokable, years before it leaked.

What makes revocation possible:

One credential per consumer. Shared credentials are un-revokable by construction. Per-service credentials mean revocation has a blast radius of exactly one service, and you know which one. This also gives you attribution — an audit log showing which service did something, rather than “the app user”.

Support two valid credentials at once. Rotation without downtime requires an overlap window: issue the new one, let consumers pick it up, then revoke the old one. A system that permits only one valid credential forces a simultaneous switch, which is why rotations get scheduled for maintenance windows and then postponed.

Practise it. Revoke a real production credential on purpose, on an ordinary afternoon. You will discover the service that reads it once at startup and never again, the cached connection that survives revocation for hours, and the runbook step referencing a tool that no longer exists. Finding those on a Tuesday is free.

Revoke first, investigate second. The instinct during a suspected leak is to understand the scope before acting, because revocation might cause an outage. That ordering is backwards: every minute of investigation is a minute the credential still works. If revoking immediately is too frightening, that fear is itself the finding.

The practices that hold up

Never commit secrets; scan for them anyway. Pre-commit hooks, server-side scanning, and automatic revocation on detection. Treat any credential that has ever touched a repository as compromised.

Prefer files over environment variables. A mounted file can have restrictive permissions, does not appear in crash dumps or process listings, and can be re-read without restarting — which is what makes rotation possible without a deploy.

Fail closed on startup. Validate that every required secret is present and well-formed before serving traffic. A service that starts without a credential and fails on the first request that needs it has converted a deploy failure into a user-facing incident.

Encrypt etcd at rest and lock down RBAC. If you use Kubernetes Secrets — which is reasonable as a distribution mechanism — know that get secrets in a namespace means every secret in it. That permission should be rare and audited.

Never send a secret through the model. Obvious in principle, routinely violated in practice: a credential placed in a prompt, a system message, or a tool result is in the context window, in the provider’s logs, in your traces, and potentially in a cache. Tools that need credentials should obtain them themselves, from the broker, using their own identity — never receive them as arguments.

Agent credentials are the hard version of all of this

Everything above gets harder when the thing holding the credential is a non-deterministic agent, and the difficulty is structural rather than incidental.

The agent’s permissions are the union of everything it might need. A conventional service does one job, so its credential can be scoped to that job. An agent’s whole value proposition is handling tasks you did not enumerate, which pushes toward broad grants. That tension is real and does not have a clean resolution — but the wrong resolution is a single broad credential held for the agent’s lifetime.

The tool-call boundary is where the leak happens. Tool arguments are constructed by a model from context that may include attacker-controlled text — a page it fetched, a ticket it read, an email. A model that has a credential in its context can be induced to put that credential in a tool argument. Prompt injection is the delivery mechanism; the credential being reachable at all is the vulnerability.

The shape that works:

• Per-task, per-tool credentials. Mint at the gateway when a tool call is authorised, scoped to that call, expiring in minutes. The agent never holds a credential; the gateway attaches one.
• The model never sees credential material. Not in the context, not in tool results, not in error messages. If a credential is not in the context window, no amount of injection can exfiltrate it.
• Carry the human’s identity, not just the agent’s. An agent acting on behalf of a user should produce an audit trail naming both, or the audit trail cannot answer who authorised what — which is the question every post-incident review and every auditor starts with.
• Scope down for autonomy. An agent acting without a human in the loop should have strictly narrower permissions than one whose actions are approved. Autonomy and privilege should move in opposite directions.

The rule worth remembering

Assume every credential leaks. Design so that when it does, it grants little, expires soon, and can be revoked without a meeting.

Scope it to one consumer and one job. Make its lifetime short enough that rotation is automatic rather than scheduled. And find out today whether you could actually revoke it — because the answer is a property you can change now and cannot change during an incident.