The Deployment Pipeline: Build Once, Promote Everywhere

One artifact, many environments. The moment you rebuild per environment, every test result upstream stops meaning anything.

Here is a pipeline that most organisations have shipped at some point:

deploy-staging:
  script:
    - docker build -t myapp:staging .
    - docker push myapp:staging
    - kubectl set image deploy/myapp myapp=myapp:staging

deploy-prod:
  script:
    - docker build -t myapp:prod .        # ← this line
    - docker push myapp:prod
    - kubectl set image deploy/myapp myapp=myapp:prod

It looks symmetrical and reasonable. It is neither. docker build ran twice, at two different times, and there is no mechanism anywhere that guarantees the two results are identical. A transitive dependency published a patch. A base image tag moved. The build cache was warm one time and cold the other. A test-only file was present in one context and not the other.

Which means the staging tests passed against an artifact that does not exist in production. You tested something. It was not the thing you shipped.

The rule

Build once. Promote the same bytes. Change only the configuration.

The pipeline produces exactly one artifact, identified by a content digest — sha256:9f2c…, not a tag like latest or prod, because tags are mutable pointers and digests are the content-addressed identity of a specific set of layers. That digest moves through every stage untouched. Each gate it passes is evidence attached to that digest, and the evidence stays valid because the subject cannot change.

build:
  script:
    - docker build -t $REG/myapp:$GIT_SHA .
    - docker push $REG/myapp:$GIT_SHA
    - DIGEST=$(crane digest $REG/myapp:$GIT_SHA)
    - echo "IMAGE=$REG/myapp@$DIGEST" >> build.env   # carried through every later stage

deploy-staging:
  script: [ "./deploy.sh staging $IMAGE" ]

deploy-prod:
  needs: [ deploy-staging ]
  script: [ "./deploy.sh production $IMAGE" ]        # same $IMAGE. No build step exists here.

Everything else in continuous delivery rests on this. Progressive rollout assumes the canary and the stable version are comparable builds. Rollback assumes a previous artifact still exists to point at. Supply-chain attestation assumes a provenance record binds to something immutable. Remove build-once and each of those quietly becomes approximate.

Configuration is the thing that varies

If the artifact is identical everywhere, everything environment-specific has to arrive from outside it. This is 12-factor’s config principle and the test it gives you is unusually sharp: could you open-source this repository right now without leaking anything? If not, configuration is baked in somewhere it should not be.

Three rules make it work:

Inject at deploy time. Environment variables, mounted ConfigMaps, or a config service — read at startup, never compiled in. A build that consumes NODE_ENV to decide what to include has produced an environment-specific artifact and has quietly opted out of build-once.

Fail fast on missing config. Validate the complete configuration at startup and exit non-zero if something required is absent. The alternative is a service that starts happily and fails on the first request that touches the missing value — at which point it is already taking traffic and the failure is a user-visible 500 rather than a failed deploy.

Secrets are not configuration. They travel a separate path with its own access control, rotation, and audit trail — a secrets manager, or short-lived credentials minted per workload identity. A secret in an environment variable is readable by anything that can describe the Pod, is in every crash dump, and does not rotate.

The stages, and what each one is for

A pipeline is a sequence of increasingly expensive filters. Each stage exists to reject a class of defect as cheaply as possible, and the ordering should be by cost, not by category.

Stage	Rejects	Budget
Lint, type check, unit tests	Logic errors, contract breaks	< 5 min
Build + sign + SBOM	Non-compiling code, known CVEs	< 5 min
Integration tests	Wiring, schema, dependency mismatches	< 15 min
Deploy to staging + smoke	Deployment mechanics, config errors	minutes
Canary in production	Everything the earlier stages cannot see	hours

The last row is the one people resist and it is the honest one. Staging does not have production’s traffic shape, data volume, cache state, or neighbours. There is a class of defect that only production can reject, and pretending otherwise just means finding it during a full rollout instead of during a 5% one.

Speed is a correctness property, not a convenience. Once the main pipeline exceeds roughly ten minutes, people stop waiting for it. They batch changes to amortise the wait, which makes each deploy bigger, which makes each failure harder to attribute — the opposite of what the pipeline is for. The DORA research is consistent on this point: the elite pattern is small changes shipped often, and long pipelines are structurally incompatible with small changes.

If the pipeline is slow, the fixes in order of value: parallelise independent stages, cache dependencies properly by lockfile hash, run only the tests affected by the change on pre-merge and the full suite post-merge, and move anything that does not gate the merge off the critical path.

Trunk-based development, and why branches fight the pipeline

A pipeline can only validate what has been integrated. A two-week feature branch is two weeks during which the pipeline is testing a world nobody will ship.

Long-lived branches produce a specific failure: the merge is a large, untested change that arrives all at once, and the integration problems it contains were created gradually over two weeks but are all discovered in one afternoon. Merge conflicts scale with branch age and so does the risk that the merge itself is the defect.

Trunk-based development — short-lived branches merged to main within a day or two, main always releasable — is not a cultural preference. It is what makes the pipeline’s verdict mean something, because the thing being tested is the thing everyone is building on.

The obvious objection is incomplete features, and the answer is to separate the two things that “merging” conflates:

Deployment is an engineering operation. Release is a product decision. Feature flags decouple them. Incomplete code merges to main, ships to production dark, and is enabled for users when it is ready — and disabled in seconds if it is not, with no redeploy and no rollback.

if flags.enabled("new-pricing-engine", user=user):
    return new_pricing(cart)
return legacy_pricing(cart)

Two warnings from everyone who has done this. Flags are code with a lifecycle, and an un-removed flag is permanent branching complexity — give each one an owner and an expiry, and treat removal as part of the work rather than cleanup. And flags in the request path are a runtime dependency: if the flag service is down, every call site needs a defined default, evaluated locally, that fails safe.

For database changes the same decoupling has a name — expand/contract. Add the new column, backfill it, write to both, switch reads, then remove the old column in a later deploy. Each step is independently deployable and independently revertible, which is the only way a schema change and a rollback can coexist.

Rollback is a first-class path, not an exception

Two questions worth answering honestly about your current system: how long does a rollback take, and when did you last do one on purpose?

With build-once, rollback is re-pointing the deployment at the previous digest. It is fast because nothing is built, and safe because that artifact already passed every gate. Without build-once, rollback means rebuilding an old commit — which takes as long as a build, and may not produce what shipped last time.

Three things make rollback real rather than theoretical:

• Automate it on SLO violation. A human deciding to roll back during an incident costs minutes of triage at the worst possible moment. A rollout that watches error rate and latency and aborts itself costs seconds.
• Keep previous artifacts and their config together. Rolling the image back while leaving the new config in place is a state nobody tested.
• Rehearse it. A rollback path exercised only during incidents is an untested code path exercised only under stress. Roll back a real deploy in production on an ordinary afternoon, on purpose, and find out what breaks.

And know which changes are not rollback-able. A destructive migration, a consumed message, a sent email, a third-party write — for these, forward-fix is the only path, and that is a reason to structure them as expand/contract in the first place.

Measure the pipeline the way you measure a service

The four DORA metrics are useful mainly because they come in pairs that resist gaming:

• Deployment frequency and lead time for changes — throughput.
• Change failure rate and time to restore service — stability.

Optimise throughput alone and you ship breakage faster. Optimise stability alone and you ship nothing, which reads as perfect stability and is its own kind of failure. The research’s persistent finding is that these move together in high-performing organisations, because the practices that make deployment safe are the same ones that make it fast: small changes, fast feedback, automated verification, quick reversal.

Treat them as a diagnostic. A high change failure rate points at the gates. A long time to restore points at rollback and observability. A long lead time with a fast pipeline points at review and approval queues, not at engineering.

The supply chain runs through here too

The pipeline is the only place where you can attach provenance to what ships, because it is the only place that knows how the artifact was made.

The practical baseline, in increasing order of effort:

• Generate an SBOM at build and store it against the digest. You cannot answer “are we exposed to this CVE?” in an hour without one.
• Sign the artifact — Sigstore / cosign makes keyless signing tied to workload identity straightforward — and verify the signature at admission, so an unsigned or foreign image cannot run.
• Pin everything by digest: base images, actions, plugins. A mutable tag in a build file is a place where your build can change without your code changing.
• Emit provenance — the SLSA framework defines build provenance attestations recording which source, which builder, which inputs produced this digest.

All four are only possible because there is exactly one artifact to talk about. Build-once is the precondition for the entire supply-chain story, which is a good argument for it even if you found the testing argument unpersuasive.

Where AI-generated code changes the shape

Volume. That is the whole change, and it is enough to matter.

The pipeline was always the mechanism that decided what reaches production. When most changes were hand-written, review was a meaningful second filter and the pipeline could afford gaps that human attention covered. With a large share of changes machine-generated, review attention per change drops, and the gaps become the primary path to production.

That argues for making the pipeline’s gates mechanical and non-negotiable rather than adding more review:

• Coverage on changed lines, enforced, not reported.
• Contract and schema compatibility checks that fail the build — a generated change to a serialiser is the classic silent break.
• Dependency-addition review as a distinct, human-required gate. Models add plausible-sounding packages, and dependency confusion is a real attack surface, not a hypothetical one.
• Every change through the same pipeline, with no bypass. “It’s a small fix” is exactly when the mechanism is doing its job.

The uncomfortable version: if your pipeline currently depends on a careful human reading every diff, it does not have the gates it needs, and it has not had them for a while. Generated code did not create that gap. It removed the thing that was covering it.

The rule worth remembering

One artifact, identified by digest, promoted unchanged through every environment, with configuration injected at deploy and release decoupled from deployment by flags.

Everything else — canaries, rollback, attestation, DORA — is downstream of that one property. A pipeline that rebuilds per environment has an interesting collection of test results about software that nobody is running.