Terraform, Explained Properly: How It Works, What State Actually Is, and Why It Deserves Your…

Terraform, Explained Properly: How It Works, What State Actually Is, and Why It Deserves Your Respect

by Gary Worthington, More Than Monkeys

Every cloud engineer starts the same way: in the AWS console, clicking. You click to make a VPC, you click to make a subnet, you click through seventeen screens of options you don’t understand to launch an EC2 instance, and at the end of it something exists and you feel like a wizard.

There’s nothing wrong with this. It’s how you learn. The console is a brilliant place to explore what AWS can do. However, it’s also a terrible place to build anything you intend to keep.

This article is for cloud engineers who are new to infrastructure as code and new to Terraform. I want to get you past the marketing-slide version — “declare your infrastructure! reproducible environments!” — and into how the thing actually works: what happens when you run it, what the state file really is, why it matters more than anything else in your setup, and the handful of decisions that will save you from learning all of this the expensive way.

I’ve spent years building AWS platforms with Terraform, from startups through to national-scale systems in regulated environments, and almost every serious Terraform incident I’ve seen traces back to the same root cause: somebody treated state as an implementation detail instead of the single most important artefact in the system.

So we’re going to spend proper time on it.

The problem Terraform exists to solve

Imagine you’ve built your company’s staging environment by hand. A VPC, some subnets, a load balancer, an ECS cluster, an RDS instance, a few security groups, an S3 bucket or six. It took a couple of days of clicking and it works.

Now build production. Identical, please, but with bigger instances and stricter security groups.

You won’t get it identical. Nobody ever does. You’ll forget a route table entry, or a security group rule, or that one bucket policy you added at 4pm on a Friday to fix a thing you no longer remember. Three months later something works in staging and fails in production, and you’ll spend a day diffing two environments by eye, screen by screen, like some sort of infrastructure spot-the-difference puzzle.

Then someone asks the really fun questions. What exactly is running in this account? Who changed that security group? Why is there a NAT gateway in eu-west-2 when we don’t use eu-west-2? Nobody knows. The console doesn’t remember. The person who clicked the buttons left in March.

This is ClickOps, and it fails for the same reason all manual processes fail: humans are shit…inconsistent, memories are short, and there’s no record of intent. You know what exists (sort of), but not why, and you can’t reproduce any of it.

Infrastructure as code is the fix, and the idea is almost embarrassingly simple: describe your infrastructure in text files, keep those files in git, and use a tool to make reality match the files.

That’s it. Everything else — the review process, the reproducibility, the audit trail, the ability to tear down and rebuild an environment before lunch — falls out of that one decision. Your infrastructure gets the same treatment your application code has had for decades: version control, pull requests, code review, history. “Who changed that security group?” becomes a git log command rather than an archaeology project.

What Terraform actually is

Terraform is a tool that reads a description of the infrastructure you want, compares it with what actually exists, and works out the API calls needed to close the gap.

The important word in that sentence is declarative. You don’t write instructions — “create a VPC, then create a subnet inside it, then attach an internet gateway”. You write a description of the end state: “a VPC with this CIDR block exists; a subnet with these properties exists inside it”. Terraform works out the doing.

This is a bigger deal than it sounds. A script that creates a VPC will happily create a second VPC if you run it twice. A Terraform configuration describing one VPC results in one VPC no matter how many times you apply it — if it already exists and matches, Terraform does nothing. If it exists but the config has changed, Terraform changes it. If someone deleted it, Terraform puts it back.

The configuration language is called HCL, and it looks like this:

resource "aws_s3_bucket" "app_data" {
  bucket = "mtm-app-data-prod"
  tags = {
      Environment = "production"
      ManagedBy   = "terraform"
    }
  }
resource "aws_s3_bucket_versioning" "app_data" {
  bucket = aws_s3_bucket.app_data.id
  versioning_configuration {
    status = "Enabled"
  }
}

Two things worth noticing. First, it reads like a description, because it is one. Second, look at bucket = aws_s3_bucket.app_data.id — the versioning resource references the bucket resource. Terraform uses references like this to build a dependency graph of your entire infrastructure, so it knows the bucket must exist before the versioning config can be applied, and it can create unrelated resources in parallel. You never write "do this first, then that". The graph handles ordering.

Terraform itself doesn’t know anything about AWS, or Azure, or anything else. All the cloud-specific knowledge lives in providers — plugins that translate between Terraform’s resource model and a vendor’s APIs. The AWS provider knows how to create an S3 bucket; the Cloudflare provider knows how to create a DNS record; there are thousands of them. This is why Terraform became the default rather than a per-cloud tool like CloudFormation: one language, one workflow, any API that has a provider.

The workflow: init, plan, apply

Day-to-day Terraform is three commands, and it’s worth being precise about what each one actually does.

terraform init sets up your working directory. It reads your configuration, downloads the providers you need into a .terraform directory, configures where your state lives (more on that shortly — much more), and writes a .terraform.lock.hcl file pinning the exact provider versions so your colleague doesn't silently get a different AWS provider than you. You run it once per project and again whenever providers or backends change.

terraform plan is the diff. Terraform reads your configuration, refreshes its record of what actually exists by querying the real APIs, compares the two, and prints exactly what it would do: create this, change that, destroy the other. Nothing is touched. It's a dry run, and it's the single best habit-forming feature in the tool — you get to review the blast radius of your change before anything happens.

terraform apply executes the plan. It shows you the same diff, asks for confirmation, then makes the API calls in dependency order.

The output of plan looks like this, and you should learn to read it the way you’d read a code review diff:

Terraform will perform the following actions:

# aws_s3_bucket.app_data will be created
  + resource "aws_s3_bucket" "app_data" {
      + bucket = "mtm-app-data-prod"
      ...
    }
Plan: 1 to add, 0 to change, 0 to destroy.

+ means create. ~ means modify in place. - means destroy. And -/+ means destroy and recreate, which is the one that should make you sit up straight, because if the resource in question is a database, Terraform is quietly telling you it's about to delete your database and make a new empty one. Plenty of attribute changes force replacement — renaming an RDS instance, changing an EC2 AMI — and the plan tells you every time. People get bitten because they stop reading.

That’s the whole loop. Change the code, plan, read the plan, apply, commit. Infrastructure changes become pull requests. Reviews become meaningful because the reviewer sees exactly what will change in the real world, not a vague promise.

But there’s a question hiding in that description, and answering it properly is the difference between using Terraform and understanding it. When Terraform “compares your configuration with what exists” — how does it know what exists?

The state file: where Terraform keeps its memory

Here’s the problem Terraform has to solve internally. Your configuration says “an S3 bucket called mtm-app-data-prod should exist”. Fine. But which real-world resources is Terraform responsible for? If there are forty S3 buckets in the account, which one is yours? If you rename the resource in your code, how does Terraform know it’s the same bucket and not a new one? If your config describes a security group and one also exists in AWS, is that the one you created earlier, or something a colleague made by hand that happens to look similar?

The configuration alone can’t answer any of that. Configuration describes intent; it says nothing about which actual, concrete resources — with their actual AWS-assigned IDs — correspond to that intent.

So Terraform keeps a record. Every time it creates a resource, it writes down the mapping: the resource address in your code (aws_s3_bucket.app_data) maps to this real resource (the bucket with this ARN, these attributes, created at this time). That record is the state file — by default a local file called terraform.tfstate — and it's just JSON. Open one up sometime; there's no magic in there. A version number, a serial that increments with each change, a lineage ID, and a big list of every resource Terraform manages with a full snapshot of its attributes.

When you run plan, Terraform is actually doing a three-way comparison: your configuration (what you want), the state file (what Terraform believes exists), and the real world (what actually exists, checked by querying the APIs during refresh). The plan is the set of actions that reconciles all three.

Once you understand that, a lot of Terraform’s otherwise-confusing behaviour becomes obvious.

Why does renaming a resource in your code make Terraform want to destroy and recreate it? Because state maps addresses to real resources. Rename aws_s3_bucket.app_data to aws_s3_bucket.application_data and, as far as Terraform can tell, one tracked resource vanished from the config and a brand new one appeared. It plans accordingly: destroy the old, create the new. The resource is identical; only the label moved. This is what moved blocks are for — a way of telling Terraform "same resource, new name" so it updates the mapping instead of cycling the infrastructure. Every Terraform beginner discovers this the hard way exactly once, usually while refactoring, and ideally not on the production database.

Why does Terraform ignore resources it didn’t create? Because they’re not in state. Terraform’s worldview is “the resources in my state file, and nothing else”. That hand-made security group your colleague built? Invisible. Terraform won’t manage it, won’t fix it, won’t delete it. If you want Terraform to adopt an existing resource, you have to explicitly import it into state — which is a normal and useful thing to do when you’re migrating a ClickOps estate into code.

And why is everyone so paranoid about the state file? Two reasons.

First, state contains a full attribute snapshot of everything Terraform manages — and that includes sensitive values. Database passwords, access keys, connection strings: if a resource has a secret attribute, that secret is sitting in the state JSON in plain text. Terraform marks values as sensitive in output, but the state file itself is not encrypted by Terraform. Treat it like a credentials file, because that’s what it is. It should never, ever go in git. Add *.tfstate* to your .gitignore before you write your first resource, not after your first incident.

Second — and this is the one that keeps platform engineers up at night — if the state file is lost or corrupted, Terraform develops amnesia. Your infrastructure is still there, humming away in AWS, but Terraform no longer knows it owns any of it. Run plan against an empty state and Terraform will cheerfully offer to create everything again from scratch, duplicating your entire estate, or collide with resources that already exist and error out. Recovering means importing every resource back into state by hand, one at a time, at whatever pace your patience and the AWS APIs allow. For an estate of any size this is somewhere between a lost week and a career highlight you’ll be dining out on bitterly for years.

Remote state — eg. get it off your laptop

By default, state lives in a local file next to your code. For a solo learning project, fine. For anything real, local state fails on three fronts at once: it’s on one person’s machine so nobody else can run Terraform, it isn’t backed up so a lost laptop is a lost estate, and there’s nothing stopping two people who both have copies from applying at the same time and destroying each other’s changes.

The fix is a remote backend: state stored in shared, durable storage that every engineer and every CI pipeline uses. On AWS, that means an S3 bucket:

terraform {
  backend "s3" {
    bucket       = "mtm-terraform-state"
    key          = "prod/networking/terraform.tfstate"
    region       = "eu-west-2"
    encrypt      = true
    use_lockfile = true
  }
}

Turn on versioning on that bucket. This is non-negotiable and costs pennies — every state change becomes a recoverable version, which converts “we corrupted the state file” from an incident into an inconvenience.

That use_lockfile line deserves a paragraph, because it solves a problem you might not spot until it bites. Suppose two engineers — or an engineer and a CI pipeline — run apply at the same time. Both read state, both make changes, both write state back. The second write clobbers the first, and now the state file no longer reflects reality: resources exist that state doesn't know about. Congratulations, you've combined the two failure modes from the previous section into one.

Locking prevents this. Before any operation that writes state, Terraform takes a lock; anyone else who tries gets an error and waits their turn. With use_lockfile = true, the S3 backend does this natively using conditional writes — a small lock file appears next to your state for the duration of the operation.

One thing to watch: a large chunk of the Terraform tutorials on the internet will tell you that S3 state locking requires a DynamoDB table. It doesn’t any more. Native S3 locking became generally available in Terraform 1.11, and the DynamoDB approach is now deprecated. The old way still works, and you’ll see it in plenty of existing codebases, but if you’re starting fresh, use_lockfile = true is all you need. Half the medium articles you'll find were written before this changed, which is a nice early lesson in the shelf life of infrastructure advice.

The ways state goes wrong

Everything above is the theory. Here’s what it looks like when it meets people, deadlines, and Friday afternoons. Every one of these is a pattern I’ve seen in the wild, more than once, and each teaches you a rule.

The laptop estate. A contractor builds an entire client platform with Terraform. Beautiful code, sensible modules, local state. The contract ends, the laptop is wiped, and the client is left with a fleet of running infrastructure and a git repo that describes it — but no state connecting the two. Terraform knows nothing. Every future change is either done by hand (hello again, ClickOps) or preceded by a painstaking import of the entire estate. The rule: remote state from day one, even when it’s just you. Especially when it’s just you, because nobody’s checking.

The public repository. A team commits terraform.tfstate to git because it seemed harmless — it's just a JSON file, right? The state contains the RDS master password, in plain text, because that's what state does. The repo later gets made public. The rule: state is a secrets file that happens to have other uses. Gitignore it before your first commit, and rotate anything that ever leaks.

The concurrent apply. Two engineers, no locking, one deploy each, simultaneously. Both applies half-succeed, the state file records a fiction, and the following morning’s plan proposes a set of changes that make no sense to anybody. Untangling it takes longer than the original work. The rule: locking is not an optimisation, it’s a seatbelt.

The quick console fix. Production issue at 5pm, an engineer bumps a security group rule in the console, incident resolved, everyone goes home. The change never makes it into code. Weeks later a routine apply quietly reverts it — because as far as Terraform’s concerned, that rule is a deviation from the declared truth — and the original outage returns, only now it’s a mystery, because “nothing changed”. This is drift, and it’s poisonous precisely because each individual instance seems so harmless. On my current team we treat drift as a bug, and I’d encourage you to do the same: the code is the truth, and anything that disagrees with it is a defect in need of a fix — either code catches up with reality, or reality gets put back. Run plan on a schedule if you can; drift you notice within a day is trivial, drift you find after six months is archaeology.

The rename that nearly dropped prod. An engineer tidies up resource names in a refactor, doesn’t read the plan properly, and approves a -/+ on an RDS instance. That's a destroy-and-recreate. On the database. The apply got stopped in time, which is more luck than judgement. The rules here: read every line of a production plan, treat -/+ as a stop sign, use moved blocks when refactoring, and put prevent_destroy lifecycle rules on anything stateful whose loss would end up in an incident report.

When things do go sideways, Terraform gives you a set of surgical tools: terraform state list shows everything in state, terraform state rm makes Terraform forget a resource without touching the real thing, import adopts existing resources, and force-unlock clears a stuck lock after a crashed run. Learn what they do before you need them — nobody makes good decisions learning state surgery for the first time during an incident.

Workspaces are not environments

Sooner or later you’ll discover terraform workspace and think you've found the answer to managing dev, staging, and production. The name practically begs you to.

Resist.

Workspaces give you multiple state files for the same configuration and the same backend. Same code, same AWS account credentials flow, same everything — just a different state file depending on which workspace you’ve selected. For genuinely identical, disposable copies of infrastructure (a review environment per branch, say), they’re fine.

But dev and production are not identical copies. They differ in instance sizes, in scaling, in security posture, and — if you’re doing it properly — they live in different AWS accounts, because an account boundary is the only blast-radius control that actually means it. Workspaces handle none of that well; you end up with a thicket of conditionals (var.environment == "prod" ? ... : ...) polluting every module, and one invisible piece of session state — the currently selected workspace — deciding whether your terraform destroy lands on the dev environment or the real one. An engineer who thinks they're in dev and is actually in prod is not having a hypothetical bad day; it's one of the classic Terraform incident patterns.

My opinion, for what it’s worth after a lot of years doing this: use a directory per environment, and make the differences boring and visible.

infrastructure/
├── modules/
│   ├── networking/
│   ├── ecs-service/
│   └── rds/
├── environments/
│   ├── dev/
│   ├── staging/
│   └── prod/

Each environment directory is a thin root module: it has its own backend config, its own state, its own variable values, and it calls the shared modules with environment-appropriate inputs. Promoting a change means applying the same module change environment by environment, watching the plan each time. Which environment you’re touching is determined by which directory you’re standing in — visible in your prompt, visible in your PR, impossible to get wrong silently.

While you’re at it, don’t put an entire environment in one state file. One giant state means every plan takes ten minutes, every lock blocks every engineer, and every mistake has maximum possible blast radius. Split state along seams that change at different speeds — networking, data stores, each application service — so that the state file you’re touching for a routine app change physically cannot delete a VPC. Small state files are to infrastructure what small pull requests are to code: less to review, less to break, easier to reason about.

Final thought

Terraform’s pitch is simple — describe what you want, let the tool make it so — and for the first week that’s exactly how it feels. The learning curve isn’t the language; HCL is readable inside an afternoon. The learning curve is the mental model: understanding that there are always three things in play — your code, the state file, and reality — and that Terraform’s entire job is reconciling them.

Get that model right and everything else follows. You’ll read plans properly because you understand what the diff is actually comparing. You’ll protect state like it matters because you understand it’s the only thing connecting your code to your infrastructure. You’ll structure repos around blast radius because you understand what a state file can and can’t reach.

The engineers who get burned by Terraform aren’t the ones who don’t know enough HCL. They’re the ones who treated state as a boring implementation detail, right up until the day it became the most interesting thing in the building.

Start with remote state, versioning, and locking — before you build anything you care about. Read every plan like it’s about to do exactly what it says, because it is. And when the plan says -/+ next to a database, stop.

The console makes you feel like a wizard. Terraform makes you something more useful: an engineer whose infrastructure is written down.

Gary Worthington is a software engineer, delivery consultant, and fractional CTO who helps teams move fast, learn faster, and scale when it matters. He writes about modern engineering, product thinking, and helping teams ship things that matter.

Through his consultancy, More Than Monkeys, Gary helps startups and scaleups improve how they build software — from tech strategy and agile delivery to product validation and team development.

Visit morethanmonkeys.co.uk to learn how we can help you build better, faster.

Follow Gary on LinkedIn for practical insights into engineering leadership, agile delivery, and team performance.