Infrastructure as Code: State, Drift, and Why the Plan Is the Product

Config, state, reality. Plan is the diff. Almost every IaC surprise is two of these three disagreeing in a way nobody read closely enough.

Most people learn Terraform as a syntax. You learn resource, you learn variable, you learn what an output is, and you get quite far writing HCL that produces infrastructure. Then one day terraform plan proposes to destroy the production database, and it becomes clear that the syntax was never the part worth understanding.

The part worth understanding is that Infrastructure as Code is a three-way comparison, and that only one of the three is authoritative.

The three things

Config is what you wrote — the .tf files in Git. Reviewed, versioned, and the only one a human directly controls.

State is what the tool believes exists. It is a cache: a mapping from your resource addresses to real provider IDs, plus the attribute values last observed. It is not the truth, and it can be wrong in both directions.

Reality is what the cloud provider actually has. It is the only authority, and the tool can only see it through API calls.

plan is the operation that reconciles them:

1. Refresh — call the provider API for every resource in state and update state to match reality.
2. Diff — compare the refreshed state against the config.
3. Report — print the actions that would close the gap.

apply executes that diff. Nothing else happens. The mental model is exactly the reconciliation loop with a human standing in the middle of it, and the human’s job is to read the diff.

Why state exists, and why it is dangerous

State is often described as an optimisation. It is not; it is load-bearing for two things you cannot get otherwise.

Identity. Your config says resource "aws_instance" "web". AWS says i-0abc123def456. Something has to hold that mapping, because nothing in AWS knows or cares about the name web. Without it, every run creates a new instance.

Deletion detection. You delete a resource block from your config. Reality still has the resource. Config no longer mentions it. Only state remembers it used to be managed, which is how the tool knows to destroy it rather than ignore it. This is also why state is the mechanism behind the scariest plans — a resource that falls out of state stops being managed silently, and a resource that falls into an unexpected state address gets destroyed.

Two rules follow, and neither is negotiable:

Never commit state to Git. It contains every attribute of every resource — RDS passwords, generated keys, connection strings — in plain text. Terraform has never encrypted state at rest on its own; that is the backend’s job. And Git has no locking, so two engineers applying at the same time will both write and one will silently win.

Use a remote backend with locking. S3 plus a lock (DynamoDB historically, S3-native conditional writes in current versions), GCS, or Terraform Cloud. Locking is the part people skip and the part that matters: two concurrent applies against one state file is how you get a state that describes neither the old world nor the new one.

terraform {
  backend "s3" {
    bucket       = "acme-tfstate"
    key          = "prod/network/terraform.tfstate"
    region       = "us-east-1"
    encrypt      = true
    use_lockfile = true          # S3-native state locking
  }
}

Note the key. One state file per environment per blast-radius boundary. A single monolithic state for the whole estate means every plan refreshes every resource (slow), every apply locks everybody (contention), and one corrupted file loses the whole estate (unrecoverable). Split state by what fails together.

Drift: the disagreement that matters

Drift is reality diverging from state. It happens because the console button exists, and at 3am the console button works.

Here is the uncomfortable part. When refresh finds drift, the default behaviour is to propose undoing it, because config is treated as authoritative. That is correct when someone poked at a resource carelessly. It is badly wrong when the console change was the emergency fix that stopped the outage, and the next routine apply — for an unrelated tag change — quietly reverts it.

This is the central operational risk of IaC and it is a process problem, not a tool problem:

• Detect drift on a schedule, not at apply time. A nightly terraform plan -detailed-exitcode that alerts on exit code 2 turns drift into a ticket you handle deliberately instead of a surprise inside an unrelated change.
• Make console access to production read-only by default. If break-glass write access exists, make using it generate a ticket automatically — the point is not to prevent the 3am fix, it is to guarantee someone reconciles it afterwards.
• After every out-of-band change, port it back to config. The window between the fix and the port-back is when the revert happens.

The deeper framing: config changes cause a large share of serious outages, and IaC does not reduce the number of config changes. It makes them reviewable, diffable, and revertible — but only if someone actually reads the diff. A rubber-stamped plan is a config change with extra ceremony.

Reading a plan properly

The plan is the product. Everything else is a way of producing it. There are four verbs and they are not equally dangerous:

  + create              new resource
  ~ update in-place     changed, no interruption
-/+ destroy and then create replacement       ← read this one
  - destroy             gone

-/+ is the line that ends careers. It means an attribute changed that the provider cannot modify in place, so the resource must be torn down and rebuilt. On an EC2 instance that is a few minutes of downtime. On an RDS instance it is your data. Terraform tells you exactly why, in a comment on the offending attribute:

-/+ resource "aws_db_instance" "main" {
      ~ availability_zone = "us-east-1a" -> "us-east-1b"
        # forces replacement

The habits that make this safe are unglamorous and completely effective:

• terraform plan -out=tfplan, then terraform apply tfplan. Applying a saved plan guarantees you execute the diff you read. A bare apply re-plans, and the world may have changed since you looked.
• Post the plan on the pull request. Review infrastructure changes as diffs of effects, not diffs of HCL. Two-line HCL changes routinely produce replacement plans, and you cannot see that in the HCL.
• Add prevent_destroy to the things that must never be rebuilt. It converts a catastrophe into an error message.

lifecycle {
  prevent_destroy = true
}

• Grep CI plans for must be replaced and require an explicit approval to proceed. Machines are better at not skimming than humans are.

Modules, and the abstraction trap

Modules are functions: inputs, a body, outputs. The same rules apply as for any function — one responsibility, a small interface, no hidden global state.

The failure mode specific to IaC is the kitchen-sink module: a module "service" that creates a cluster, a database, a load balancer, DNS, IAM, and dashboards, with forty variables. It is genuinely convenient right up until one consumer needs a slightly different load balancer, and there is no way to express that except adding variable forty-one. Do that five times and the module is a worse API than the provider it wraps, because it has all the surface area and none of the documentation.

The heuristic that holds up: a module should have a smaller interface than the resources it composes. If it does not, it is not an abstraction, it is a rename. And the things inside a module should share a lifecycle — you should never want to destroy half of one.

Pin module and provider versions the same way you pin dependencies, for the same reason: a build that can change without you changing anything is not reproducible.

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "5.13.0"      # not ">= 5.0" — exact
}

What IaC does not give you

Three things get assumed and are not true.

It is not a rollback mechanism. Reverting the commit and applying does not restore the previous world. It computes a new diff from current reality toward the old config, and for a replaced database that means creating an empty one. Version control gives you the old description, not the old state. Backups give you the old state.

It does not prevent a bad change from applying everywhere at once. IaC is a force multiplier in both directions: a typo in a shared module, applied across thirty accounts by automation, is a very efficient outage. The counterweight is the same as for application code — stage the rollout, apply to one environment first, and give the pipeline the same progressive delivery discipline you would give a deploy.

It does not make the state file safe by existing. State is a plaintext inventory of your entire estate including secrets. Its read access list should look like a production database’s, not like a build artifact’s.

Generated infrastructure raises the stakes on the plan

This is worth saying plainly because the tooling is now good enough that people are doing it.

An AI agent writing HCL is a genuinely good fit: the language is declarative, well documented, and heavily represented in training data, and the feedback loop is fast. The output is frequently correct. But “frequently correct” interacts badly with a workflow whose only safety mechanism is a human reading a diff — because the volume of plausible-looking changes goes up, and human plan-reading attention does not.

The failure is not the model writing something obviously wrong. It is the model writing something reasonable whose plan contains one -/+ on a stateful resource, arriving in a batch of changes that all look fine. Review attention is the scarce resource, and generation makes it scarcer per change.

What actually helps is mechanical, not procedural:

• Machine-check the plan, do not just show it. Parse terraform show -json tfplan and hard-fail CI on any replacement or destroy of a resource type on a protected list. A policy engine — OPA, Sentinel, Conftest — does this properly, but forty lines of Python over the JSON plan catches the important cases on day one.
• Let the agent open a PR; never let it hold apply credentials. The plan-in-CI, human-approves, pipeline-applies path is the same gate you already wanted, and it is the one that does not depend on anyone’s attention holding up.
• prevent_destroy on everything stateful becomes mandatory rather than advisable. It is the one control that fails closed regardless of who or what wrote the change.

None of that is AI-specific advice. It is the advice that was always correct and that teams got away with skipping because the volume was low enough for careful reading to work.

The rule worth remembering

Config is what you want, state is what the tool remembers, reality is what is true — and the plan is the only place you find out how far apart they have drifted.

Read the plan. Apply the plan you read. Detect drift on a schedule rather than discovering it during an unrelated change. Split state along blast-radius lines. Everything else about IaC is syntax you can look up.