Terraform Secrets Done Properly: Keeping Passwords Out of State, Git, and the Incident Report

by Gary Worthington, More Than Monkeys

A security review I sat in on a while back found a production database password in four places. The team expected one of them — the secrets manager, where it belonged. They did not expect the other three: the Terraform state file, a plan artifact retained by CI for ninety days, and the state bucket’s version history, where every previous value of the password sat in tidy chronological order, like a museum exhibit of the company’s rotation policy.

Nothing had been “leaked” in the dramatic sense. Nobody had committed a password to git. Everyone had followed what, until recently, was standard Terraform advice.

That’s the uncomfortable bit: for most of Terraform’s life, handling secrets properly was somewhere between awkward and impossible, because of a design fact this series met in part one — state stores a full snapshot of every attribute of every resource, and it does not care which of those attributes would end a career if pasted into the wrong Slack channel. The tooling has finally, properly caught up. This article covers where secrets actually leak, why the old workarounds never really worked, and the current mechanisms — ephemeral values and write-only arguments — that fix the problem at the root.

A quick recap

The series so far, in one breath: state is the artefact that matters, environments are directories with separate accounts, modules encode decisions, the pipeline holds the pen, and existing estates get imported one verified plan at a time. Part one made a claim we now cash in: treat state like a credentials file, because that’s what it is.

Today’s question is how to stop it being one.

The leak map

Before fixing anything, know your enemy. A secret passing through Terraform can end up in six places, roughly in ascending order of surprise.

In your code, hardcoded — the one everybody knows not to do. In a terraform.tfvars file that was gitignored on laptops and then committed by exactly one person, once, which is all it takes. In the state file, as part one warned. In saved plan files — that tfplan artifact from part four's pipeline contains the values it plans to use, and CI systems retain artifacts with the enthusiasm of a hoarder. In CI logs, where a debug flag or a careless terraform output prints what the display layer normally hides. And in your state bucket's version history, which is the one that gets people: S3 versioning — the thing part one told you to switch on, correctly — means every value your state has ever held is still there, in perpetuity, one API call away.

Notice that only the first two are about carelessness. The other four are Terraform and its surroundings working exactly as designed.

What sensitive = true actually does

The first thing everyone reaches for. Mark a variable or output as sensitive and Terraform redacts it from plan output and the console: password = (sensitive value). Reassuring.

It’s a curtain, not a safe.

The value is still in state, in plaintext, byte for byte. It’s still in the plan file. sensitive controls what gets displayed, and nothing about what gets stored — its job is stopping secrets landing in terminal scrollback and CI logs, which is worth having, but if your threat model ends at "can't see it on screen", your threat model is a screen. Use it everywhere it applies. Just don't confuse it with protection.

Why the old workarounds never quite worked

Teams that understood the above spent years on workarounds, and it’s worth being honest about why each one fell short, because they’re all still being recommended by tutorials with 2021 in the URL.

Passing secrets in as variables — from gitignored tfvars, environment variables, CI secrets — keeps them out of git, which is real progress. But the variable’s journey ends at a resource attribute, the attribute ends up in state, and you’ve moved the secret from a file you controlled into a file you’d forgotten was a file.

The smarter-looking version reads the secret from AWS Secrets Manager with a data source, so no human handles it at all. Elegant — except a data source’s results are written to state like everything else, so the secret you carefully kept in the vault now also has a copy outside the vault, in JSON. The vault, at this point, is decorative.

The genuinely sound old advice was narrower: keep secrets out of Terraform entirely — create the database with a throwaway password and have something else set the real one; have applications fetch credentials from the secrets manager at runtime rather than receiving them through Terraform outputs. That advice still holds. It was just annoying, because some resources demand a secret at creation time, and until recently Terraform had no way to hand one over without keeping a copy.

Ephemeral values: secrets with no memory

Terraform 1.10 introduced the fix at the language level: ephemeral values, which exist during an operation and are deliberately never written to state or plan files. Where a data source reads and records, an ephemeral resource opens, uses, and forgets:

hcl

ephemeral "aws_secretsmanager_secret_version" "db_password" {
  secret_id = aws_secretsmanager_secret.db.id
}

The secret flows through the run in memory and leaves no residue. Provider support has filled in across the majors — the AWS provider covers Secrets Manager reads among others, the random provider can mint ephemeral passwords, and the Vault provider does what you’d hope. That’s half the problem solved: reading a secret no longer copies it.

The other half is giving a secret to a resource that needs one, which is where write-only arguments come in.

Write-only arguments: the hole in the wall

Since Terraform 1.11, resources can offer write-only versions of their sensitive arguments — conventionally suffixed _wo. A write-only argument is a one-way slot: Terraform passes the value to the provider during apply, the provider configures the resource, and the value is then discarded. Not stored in state, not stored in the plan, not retrievable afterwards even by Terraform itself.

Put the two mechanisms together and you get the pattern that should now be your default for anything with a password:

ephemeral "random_password" "db" {
  length = 32
}

resource "aws_db_instance" "main" {
  # ...
  password_wo         = ephemeral.random_password.db.result
  password_wo_version = 1
}

A password is generated in memory, handed through the one-way slot, set on the database, and forgotten by everything except the database itself. State contains no password. The plan artifact contains no password. The S3 version history accumulates no passwords. The four-places problem from the opening becomes a one-place problem, which is the correct number of places.

That _wo_version line is the rotation handle. Write-only values are invisible to Terraform's diffing — it can't compare what it can't remember — so you tell it a change happened by bumping the version. Increment it, and the value is regenerated and re-sent on the next apply. Rotation becomes a one-character code change, reviewed and applied by the pipeline like anything else.

For RDS specifically there’s an even shorter answer: manage_master_user_password = true hands the whole problem to AWS — the password is created, stored in Secrets Manager, and rotated without Terraform ever laying eyes on it. For the common case, the best secret-handling code is the code that never handles the secret.

The shape of the whole thing

Zoom out and the architecture is one sentence: Terraform provisions the safe; it doesn’t hold the jewels.

Terraform builds the Secrets Manager secret, the KMS keys, the IAM policies controlling who reads what — the container infrastructure, which is ordinary, unsensitive code that reviews well. Values flow through ephemeral resources and write-only arguments at the moments they must, or better still are managed entirely inside AWS. And applications fetch their credentials from the secrets manager at runtime, using the IAM role your module gave them — never from a Terraform output, which would drag the secret back through state on its way to a terraform output command in a CI log.

Every piece of this rides the series’ earlier decisions. The per-environment accounts from part two mean dev engineers can’t read prod secrets even at their laziest. The modules from part three make the safe-by-default pattern the path of least resistance. The OIDC pipeline from part four means no long-lived keys are guarding the machinery that guards the keys.

What still leaks, and the rule that follows

Honesty section. Write-only arguments require Terraform 1.11+ and provider support for each specific argument — most core resources have them now, but check before assuming, and a resource without a _wo variant still stores what you set on it. Older resources in long-lived estates were created the leaky way and their values are still in state history. Debug logging can still print more than you'd like. And no mechanism above helps if state bucket access is broad enough that reading old versions is easy.

Which brings us to the rule this article has been building to, and it’s absolute: a secret that has ever touched state is rotated, not tidied.

Not deleted from the current state and declared handled — the version history remembers. Not obscured by a refactor to write-only arguments — the old values are still in the old versions. Rotated: new value issued, old value dead, so that every historical copy becomes a historical curiosity. When you migrate a resource to the ephemeral pattern, the migration ends with a rotation, or it didn’t happen. The version history museum stays open; you just make sure every exhibit is a dud.

Final thought

Secrets handling is where this series’ theme stops being philosophy and becomes blunt arithmetic. Every copy of a secret is a place it can leak; the count of copies only ever goes up unless something is actively designed to keep it down; and for a decade, Terraform’s design quietly incremented that counter on every apply while the display layer told you everything was (sensitive value).

Ephemeral values and write-only arguments are the design finally taking a side — the correct number of copies of a secret is the smallest one the system can function with, and the tool now agrees.

The password from that security review lives in one place now. The database knows it, the safe holds it, and Terraform — which built the safe, the vault door, and the guard rota — has no idea what it is.

Which is exactly as much as your infrastructure tooling ever needed to know.

Gary Worthington is a software engineer, delivery consultant, and fractional CTO who helps teams move fast, learn faster, and scale when it matters. He writes about modern engineering, product thinking, and helping teams ship things that matter.

Through his consultancy, More Than Monkeys, Gary helps startups and scaleups improve how they build software — from tech strategy and agile delivery to product validation and team development.

Visit morethanmonkeys.co.uk to learn how we can help you build better, faster.

Follow Gary on LinkedIn for practical insights into engineering leadership, agile delivery, and team performance.