Terraform in CI/CD: Plans as Pull Request Comments, and Why Nobody Should EVER Run Apply from a…

Terraform in CI/CD: Plans as Pull Request Comments, and Why Nobody Should EVER Run Apply from a Laptop

by Gary Worthington, More Than Monkeys

A new engineer joins the team. Sensible question, week one: “how do we deploy infrastructure changes?”

“Dave runs apply.”

Not a pipeline. or a process. Dave — a specific human being, with production credentials in a dotfile, a terminal history that constitutes the company’s entire audit trail, and a laptop that is now, whether anyone has said it out loud or not, a tier-one piece of production infrastructure with stickers on it.

Every team goes through a Dave phase (c’mon, we all know a Dave). It’s how Terraform adoption starts: one or two people learn the tool, they run it from wherever they are, and it works, so nothing forces the question. This article is about answering the question before it’s asked at 2am — how to move Terraform into CI/CD properly, with plans reviewed on pull requests, applies run by machinery, and production behind a gate. Practically, with working YAML.

A quick recap

This is part four of a series. Part one covered how Terraform works and why state is the artefact your career depends on. Part two built environments as directories with separate state, separate AWS accounts, and separate credentials. Part three filled the modules/ directory with small, opinionated modules.

The thread running through all three: make the important things visible, and make the dangerous things hard to do by accident. CI/CD is where that thread pays off, because everything so far — thin roots, account boundaries, small state files — was quietly making this article possible.

What’s actually wrong with the laptop

It’s worth being precise, because “best practice says use CI” convinces nobody worth convincing.

The first problem is the gap between what was reviewed and what was applied. Your colleague approved a pull request. What got applied was whatever was sitting in Dave’s working directory at the time — which was probably the branch, mostly, plus an uncommitted tweak from debugging, minus the file he forgot to save. Nobody knows, including Dave. The review approved a text file; reality received a different one.

The second is credentials. Applying from laptops means production-writing credentials living on laptops — plural, growing with the team, walking out of the building every evening. Each one is a breach waiting for a coffee shop.

The third is the audit trail, by which I mean there isn’t one. When something breaks, the investigation consists of asking Dave what he remembers about Tuesday.

None of this is about trust. Dave is excellent. The point is that a process which only works when a specific human is careful isn’t a process — it’s a performance, and performances have off nights.

The rule, and the motif for this whole article: the pipeline holds the pen. Humans review and approve; only machinery writes.

The shape of the workflow

The target looks like this, and every piece of tooling below is in service of it. When a pull request opens, CI runs the cheap checks, then runs terraform plan and posts the output on the PR as a comment. Reviewers read the plan — not just the code — because the plan is what's actually about to happen. When the PR merges, CI applies dev automatically, and higher environments apply in order behind explicit approval gates.

Two things about this shape before the YAML. The unit of review changes: you’re no longer approving “this HCL looks reasonable”, you’re approving “these fourteen specific changes to reality, including that -/+ on line forty". That's a materially better review, and it's the main prize here.

And the unit of deployment becomes the merge. No separate release step, no “who’s applying this?” — merged means it’s going, in environment order, with the plan you looked at.

Plan on pull request

Here’s the core of it, GitHub Actions flavoured, for the dev environment:

on:
  pull_request:
    paths: ["infrastructure/**"]

permissions:
  id-token: write
  contents: read
  pull-requests: write
jobs:
  plan-dev:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v4

And the steps that do the work:

- uses: aws-actions/configure-aws-credentials@v6
        with:
          role-to-assume: arn:aws:iam::111111111111:role/terraform-plan-dev
          aws-region: eu-west-2

- run: terraform init
        working-directory: infrastructure/environments/dev
      - run: terraform plan -no-color -out=tfplan
        working-directory: infrastructure/environments/dev

Then pipe the plan output into a PR comment — actions/github-script with a few lines of JavaScript, or one of the purpose-built actions that formats it with a collapsible section. The mechanics matter less than the contract: every PR that touches infrastructure carries, on its face, exactly what it will do to each environment. A reviewer who approves without opening that comment has signed something they didn't read, and now everyone can see that.

Run one plan job per environment the PR touches. Because part two gave each environment its own directory, working out which environments a PR affects is a paths filter, not a research project.

No keys, anywhere

Look back at that credentials step. There’s no access key in it — no AWS_SECRET_ACCESS_KEY sitting in the repository secrets, waiting to leak into a fork or a log line.

That id-token: write permission is GitHub's OIDC support: the workflow proves its identity to AWS with a short-lived signed token, AWS checks the claim against a trust policy — this repository, this branch, this environment — and hands back temporary credentials that evaporate when the job ends. Nothing to rotate, nothing to steal off a laptop, nothing for a leaver to take with them.

The trust policy is where part two’s account structure earns its keep again. The role that plans dev lives in the dev account and can be assumed by any PR. The role that applies prod lives in the prod account and its trust policy accepts exactly one claim: the main branch, via the production environment gate. A feature branch asking for prod credentials isn’t refused by convention or code review — it’s refused by AWS, which has never once been talked into anything by an engineer in a hurry.

Long-lived AWS keys in CI secrets were the standard advice for years, and plenty of tutorials still teach it. It’s the DynamoDB lock table of deployment advice: it worked, it’s outdated, and there’s now a plainly better answer.

Apply on merge, gate on prod

On merge to main, the apply workflow runs the environments in order. Dev applies automatically. Staging, if you keep one, applies automatically after dev succeeds. Prod waits:

apply-prod:
    needs: apply-staging
    environment: production   # required reviewers live here
    concurrency:
      group: terraform-prod
    runs-on: ubuntu-latest

Two lines deserve attention. The environment: production line attaches GitHub's required-reviewers gate — a named human clicks approve before the job gets credentials, and that click is logged forever, which is the audit trail Dave's memory never was. The concurrency group serialises applies so two merges can't race each other into the same state file. Terraform's own locking (part one) would catch the collision anyway, but a queue is politer than an error, and a failed apply job at the wrong moment is how people learn what "partial apply" means.

Should the gate re-plan, or apply the exact plan file saved at review time? Applying the saved artifact (terraform apply tfplan) guarantees that what was approved is what happens — but a plan is a snapshot, and if reality moved since Tuesday, Terraform refuses it as stale. Re-planning at apply time always works but opens a small window where what applies isn't verbatim what was reviewed. My take: re-plan on apply, and let the environment ordering protect you — dev and staging applied this same commit minutes ago, so a surprising prod plan means drift, and you want to be stopped by that. Teams in stricter regulatory settings should take the saved-plan discipline and accept the occasional stale-plan re-run as the cost.

The nightly plan

Once the pipeline is the only writer, a scheduled job becomes your drift alarm — the “drift is a bug” opinion from part one, mechanised:

on:
  schedule:
    - cron: "0 6 * * *"

Run terraform plan -detailed-exitcode against each environment. Exit code 0, all quiet; exit code 2, someone or something changed reality outside the pipeline, and a Slack message goes to the channel with the plan attached. Six hours after a console quick-fix beats six months after, when the fix has load-bearing dependents and its author has a new job.

Break glass honestly

Every team eventually hits the incident where the pipeline itself is the obstacle — CI is down, or the fix can’t wait for a runner queue. Pretending this never happens is how you get secret laptop applies again, only now with guilt.

So make the exception explicit. A break-glass role in each account that a named group can assume, an expectation that its use is announced in the incident channel, and — this is the part people skip — a follow-up task to reconcile: whatever was done by hand gets pulled back into code and confirmed by a clean nightly plan. CloudTrail will show the role was assumed either way; better that the process expected it.

The rule: emergencies are allowed to bypass the pipeline. They are not allowed to bypass it quietly.

The objections

“It’s slower than running it myself.” The honest version: it’s slower than running it yourself when nothing goes wrong. A plan-on-PR loop with small state files (part two, again) runs in a couple of minutes, and it’s a couple of minutes during which nobody can apply unreviewed changes, lose a laptop with prod keys on it, or apply their uncommitted debugging. You’re not buying speed, you’re buying the ability to stop thinking about a whole category of disaster. That trade pays for itself the first time it fires.

“We’re too small for this.” Two engineers and a dev account? Fine, run applies locally a while longer — part two’s structure keeps you safe-ish. But the moment there’s an environment you’d be embarrassed to break, the pipeline stops being enterprise ceremony and starts being the cheapest insurance you’ll ever configure. It’s an afternoon of YAML. Dave is worth more than that.

The point of all this

Strip the YAML away and the design is one sentence: humans decide, machines execute, and the two are connected by a plan someone actually read.

Everything else — OIDC, gates, concurrency groups, nightly drift runs — is plumbing in service of that sentence. It’s the same argument this series keeps making, because it keeps being true: the systems that survive aren’t the ones with the cleverest configuration, they’re the ones where the dangerous action requires a visible, deliberate, logged decision, made by someone who can see exactly what they’re deciding.

Dave, meanwhile, gets to be an engineer again instead of a deployment mechanism.

He was always too good for that job.

Gary Worthington is a software engineer, delivery consultant, and fractional CTO who helps teams move fast, learn faster, and scale when it matters. He writes about modern engineering, product thinking, and helping teams ship things that matter.

Through his consultancy, More Than Monkeys, Gary helps startups and scaleups improve how they build software — from tech strategy and agile delivery to product validation and team development.

Visit morethanmonkeys.co.uk to learn how we can help you build better, faster.

Follow Gary on LinkedIn for practical insights into engineering leadership, agile delivery, and team performance.