IaC — State Corruption from Untagged S3 Buckets

Every modern software team has faced the same nightmare: a production server dies at 2am, and the engineer who built it left the company six months ago. Nobody wrote anything down. The replacement server gets rebuilt from memory, Slack messages, and guesswork — and it's never quite right. This isn't a people problem. It's a process problem, and Infrastructure as Code (IaC) exists specifically to eliminate it. When your infrastructure lives in code, it lives in Git, in pull requests, in code reviews, and in your CI/CD pipeline — just like the application it runs.

Before IaC, provisioning infrastructure meant logging into a cloud console, clicking through wizards, and hoping the person next to you was watching and taking notes. Every environment — dev, staging, production — drifted apart over time because human hands configured them differently. This 'configuration drift' is the silent killer of reliable deployments. IaC solves this by making infrastructure declarative and repeatable: you describe the desired state of your system, and a tool like Terraform or Ansible figures out how to get there. The same code that spins up your staging environment spins up production, byte for byte.

By the end of this article you'll understand why IaC exists at a systems level, know the difference between declarative and imperative approaches, and have a real working Terraform + GitHub Actions CI/CD pipeline you can adapt for your own projects. You'll also know the two mistakes that catch almost every intermediate engineer off guard when they go to use IaC in a team setting.

Why Infrastructure as Code Is a State Machine, Not a Script

Infrastructure as Code (IaC) is the practice of defining and managing infrastructure — servers, networks, databases — through machine-readable definition files, not manual CLI commands or click-ops. The core mechanic is declarative: you specify the desired end state (e.g., three EC2 instances, one RDS read replica), and the IaC tool computes the diff between current and desired state, then executes only the necessary create/update/delete operations. This turns infrastructure into a reproducible artifact, versioned alongside application code.

In practice, IaC tools like Terraform, CloudFormation, or Pulumi maintain a state file that maps real-world resources to your definitions. This state is the source of truth — it tracks resource IDs, dependencies, and metadata. When you run a plan, the tool compares your config against this state, not against live cloud APIs directly. That means state corruption (e.g., from manual changes or untagged resources) causes drift: the tool sees a resource as missing when it's actually present, or vice versa, leading to duplicate resources, deletion of production data, or failed applies.

Use IaC for any environment that outlives a single developer session — production, staging, even long-lived dev environments. The value compounds when you need to recreate an environment from scratch (disaster recovery, blue/green deployments) or audit changes across a team. Without IaC, you're one accidental click away from an irreproducible mess. With it, you get deterministic provisioning, change history, and the ability to roll back infrastructure changes like code.

Declarative vs Imperative IaC — Choosing the Right Mental Model

There are two ways to tell someone how to make a cup of coffee. The imperative way: 'Boil water. Measure 18g of beans. Grind them. Pour water at 94°C. Wait 4 minutes.' The declarative way: 'I want a black filter coffee in this cup.' The declarative approach lets the system figure out the steps.

This distinction is the most important conceptual split in IaC. Terraform is declarative — you describe what your infrastructure should look like, and Terraform calculates the diff between current state and desired state, then makes the changes. Ansible is imperative by default — you write a sequence of tasks that run top to bottom. Both are valid. The right choice depends on what you're managing.

Declarative tools shine for cloud resource provisioning: creating VPCs, EC2 instances, databases, and load balancers. You don't want to think about order of operations — you just want the result. Imperative tools shine for configuration management: installing packages, editing config files, restarting services. The order genuinely matters there.

In a mature DevOps pipeline you'll often use both: Terraform provisions the server, Ansible configures it. Understanding why they work differently stops you from fighting the tool when it doesn't behave the way you expect.

# main.tf — Terraform declarative configuration for a basic web server on AWS
# Run: terraform init && terraform plan && terraform apply

# Tell Terraform which cloud provider to use and which region
provider "aws" {
  region = "us-east-1"
}

# Declare the desired state: an EC2 instance running Ubuntu
# Terraform will CREATE this if it doesn't exist, UPDATE it if the spec changed,
# and do NOTHING if it already matches — this is idempotency in action
resource "aws_instance" "web_server" {
  ami           = "ami-0c02fb55956c7d316"  # Ubuntu 22.04 LTS in us-east-1
  instance_type = "t3.micro"               # Free-tier eligible size

  # Tag the resource so humans can find it in the AWS console
  tags = {
    Name        = "theforge-web-server"
    Environment = "staging"
    ManagedBy   = "terraform"  # Critical: tells the team this isn't manually managed
  }
}

# Output block: print the public IP after apply so we know where to connect
output "web_server_public_ip" {
  description = "The public IP address of our web server"
  value       = aws_instance.web_server.public_ip
}

IaC Tools Comparison: Terraform, Ansible, and AWS CDK

Choosing the right IaC tool is like choosing the right hammer — they look similar but each is designed for a specific nail. The three most popular tools each have distinct strengths: Terraform for declarative cloud provisioning, Ansible for imperative configuration management, and AWS CDK for developers who want to write infrastructure in familiar programming languages.

CDK (Cloud Development Kit) is unique because it lets you define AWS resources using general-purpose languages. Under the hood, CDK synthesises CloudFormation templates — so you get the safety of declarative state management with the expressiveness of code. This is increasingly popular in DevOps-heavy teams that already use TypeScript for backend services.

In production, pick Terraform if you need multi-cloud or a mature state management story. Pick Ansible if you're configuring servers and don't want to manage state files. Pick CDK if your team lives in AWS and writes TypeScript daily. Mixing Terraform (cloud) + Ansible (config) is the most common pattern among teams that need flexibility.

// cdk-stack.ts — AWS CDK example (TypeScript)
// This creates an S3 bucket with versioning enabled.
// Run: cdk synth && cdk deploy

import * as cdk from 'aws-cdk-lib';
import * as s3 from 'aws-cdk-lib/aws-s3';

export class MyStack extends cdk.Stack {
  constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // The bucket is created with versioning — idempotent because CloudFormation tracks it
    new s3.Bucket(this, 'TheForgeBucket', {
      versioned: true,
      bucketName: 'theforge-iac-demo-bucket',
      removalPolicy: cdk.RemovalPolicy.RETAIN,  // Prevent accidental deletion
    });
  }
}

const app = new cdk.App();
new MyStack(app, 'TheForgeStack');

Push vs Pull Model in IaC — How Agents and Agentless Architectures Compare

IaC tools fall into two operational models: push and pull. This distinction affects everything from security posture to network topology. Understanding it helps you choose the right tool for your environment — and debug failures when the model conflicts with your infrastructure.

Push model (Agentless): The orchestration server (or user's laptop) directly connects to target nodes via SSH or WinRM and executes commands. Ansible is the canonical example. No agent needs to be installed on the target — the orchestrator pushes configuration to the node. This is simple to set up initially but requires network connectivity from the orchestrator to every target. In production pipelines, this often means running Ansible from a CI runner that has SSH access to your fleet.

Pull model (Agent-based): An agent is installed on each target node. The agent periodically polls a central server (or service) for its desired configuration, pulls it down, and applies it locally. Chef and Puppet (in default mode) work this way. Pull models scale better at the cost of more complex initial setup. The agent handles retries, avoids single orchestrator bottlenecks, and works even when the target is behind NAT or a firewall.

The trade-off: push is simpler for small infrastructures, pull is more resilient at scale. Many mature DevOps teams adopt a hybrid approach: Terraform (push for provisioning) + Chef/Puppet (pull for ongoing config).

# Ansible (Push) — from a control node to managed hosts
# No agents required; uses SSH

- name: Ensure nginx is installed on web servers
  hosts: webservers
  tasks:
    - name: Install nginx
      ansible.builtin.package:
        name: nginx
        state: present

# Chef (Pull) — agent on each node polls Chef Server periodically
# Requires a chef-client agent and a Chef Server
# node.rb (client.rb)
chef_server_url  "https://chef-server.example.com/organizations/myorg"
node_name        "webserver1.example.com"
client_key       "/etc/chef/client.pem"

# A run-list on the Chef Server defines which recipes to apply
# The agent runs every 30 minutes (or triggered manually)

Idempotency vs Immutability — Two Pillars of Reliable Infrastructure

Two concepts are often confused but serve different purposes: idempotency and immutability. Both make infrastructure safer, but they achieve safety through different mechanisms.

Idempotency means running the same operation multiple times produces the same result as running it once. Terraform is idempotent: if you apply the same configuration twice, the second apply does nothing (no changes). Idempotent tools are forgiving — you can rerun them as often as needed without side effects. This is essential for CI/CD where multiple triggers may attempt an apply.

Immutability means you never modify a running resource. Instead, you replace it entirely. When a configuration change is needed, you build a new server (or container) and switch traffic to it, then destroy the old one. This guarantees that the running system always matches the artifact that was built in CI. No drift, no snowflake servers. Tools like Packer and Docker champion immutability — you bake the configuration into an image, then deploy the image.

In a mature production environment you'll leverage both: use immutable images for core infrastructure (AMI pipelines) and idempotent configuration management for runtime adjustments (config files, secret rotation).

# Immutable approach: Use an AMI built by Packer, not an inline user_data script
# This ensures the server is exactly what CI validated
resource "aws_instance" "immutable_server" {
  ami           = var.baked_ami_id  # AMI built in CI pipeline — never changed after launch
  instance_type = "t3.micro"
  # No provisioner — config is baked into the AMI
}

# Idempotent approach: Terraform ensures the instance exists and has certain tags
# But configuration inside the instance is managed by Ansible (idempotent tasks)
resource "aws_instance" "idempotent_instance" {
  ami           = data.aws_ami.ubuntu.id
  instance_type = "t3.micro"
  tags          = {
    Name = "idempotent-inst"
  }
  # user_data script runs once at launch; Terraform won't rerun it on re-apply
  user_data = file("setup.sh")
}

IaC in a Real CI/CD Pipeline — Automate the Infrastructure Itself

Knowing how to run Terraform locally is a starting point. But IaC's real power unlocks when it runs automatically inside your CI/CD pipeline. Think about it: your application code goes through automated testing before it deploys. Why should your infrastructure changes be any different? A pull request that adds a new RDS database should go through the same review process as a pull request that adds a new API endpoint.

The pattern that works in production is this: on every pull request, run terraform plan and post the output as a PR comment. This gives reviewers an exact, human-readable diff of what will change in the real cloud — before anyone approves it. On merge to main, run terraform apply automatically. No one runs Terraform from their laptop. Ever.

This approach solves three problems at once. It creates an audit trail (every infrastructure change is a Git commit with an author and a timestamp). It prevents 'works on my machine' infrastructure (the pipeline always runs from a clean state). And it forces infrastructure changes through code review, which catches mistakes before they hit production.

The GitHub Actions workflow below implements this exact pattern. It's the real thing — not a toy example.

# .github/workflows/terraform-ci-cd.yml
# This pipeline runs Terraform plan on PRs and Terraform apply on merge to main.
# Prerequisites: Store AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY as GitHub Secrets.

name: Terraform Infrastructure Pipeline

on:
  pull_request:
    branches: [main]        # Run plan on any PR targeting main
    paths: ['terraform/**'] # Only trigger when Terraform files actually changed
  push:
    branches: [main]        # Run apply when code lands on main
    paths: ['terraform/**']

env:
  TF_VERSION: '1.7.0'                     # Pin the version — never use 'latest' in CI
  TF_WORKING_DIR: './terraform'            # All .tf files live in this directory
  AWS_REGION: 'us-east-1'

jobs:
  terraform-plan:
    name: Plan Infrastructure Changes
    runs-on: ubuntu-latest
    # Only run the plan job on pull requests, not on direct pushes to main
    if: github.event_name == 'pull_request'

    steps:
      - name: Check out repository code
        uses: actions/checkout@v4

      - name: Configure AWS credentials from GitHub Secrets
        uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ${{ env.AWS_REGION }}

      - name: Install the pinned version of Terraform
        uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: ${{ env.TF_VERSION }}

      - name: Terraform Init — download providers and configure backend
        working-directory: ${{ env.TF_WORKING_DIR }}
        run: terraform init -input=false
        # -input=false prevents the pipeline hanging waiting for keyboard input

      - name: Terraform Validate — catch syntax errors before planning
        working-directory: ${{ env.TF_WORKING_DIR }}
        run: terraform validate

      - name: Terraform Plan — calculate what will change
        working-directory: ${{ env.TF_WORKING_DIR }}
        id: tf_plan
        # -no-color prevents ANSI escape codes from cluttering the GitHub PR comment
        run: terraform plan -no-color -out=tfplan.binary

      - name: Post plan output as a PR comment so reviewers can see the diff
        uses: actions/github-script@v7
        with:
          script: |
            const planOutput = `${{ steps.tf_plan.outputs.stdout }}`;
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: '### Terraform Plan\n```\n' + planOutput + '\n```'
            })

  terraform-apply:
    name: Apply Infrastructure Changes
    runs-on: ubuntu-latest
    # Only run apply when a PR is merged to main — never on open PRs
    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
    environment: production   # Requires a GitHub environment approval gate if configured

    steps:
      - name: Check out repository code
        uses: actions/checkout@v4

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ${{ env.AWS_REGION }}

      - name: Install Terraform
        uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: ${{ env.TF_VERSION }}

      - name: Terraform Init
        working-directory: ${{ env.TF_WORKING_DIR }}
        run: terraform init -input=false

      - name: Terraform Apply — make the changes in the real cloud
        working-directory: ${{ env.TF_WORKING_DIR }}
        # -auto-approve skips the interactive yes/no prompt — safe here because
        # this only runs after a human approved and merged the PR
        run: terraform apply -input=false -auto-approve

Remote State and Modules — The Patterns That Make IaC Scale

A single main.tf file works fine for a hobby project. It falls apart the moment you have two engineers, two environments, or two services. This is where two patterns become non-negotiable: remote state backends and modules.

Remote state is how Terraform remembers what it already built. Without it, every terraform apply is flying blind. With a remote backend — like an S3 bucket with a DynamoDB lock table — the state file lives in the cloud, is accessible to everyone on the team, and is locked during applies so two engineers can't run it simultaneously and corrupt each other's work.

Modules are reusable Terraform components. Think of them as functions for infrastructure. Instead of copy-pasting the same EC2 + security group + IAM role configuration for every service, you write it once as a module and call it with different variables for each service. This is the IaC equivalent of the DRY principle and it's what separates a professional IaC setup from a pile of disconnected config files.

Below is a minimal but real remote backend configuration alongside a module call pattern. This is the structure you'd actually find in a production repository.

# ─────────────────────────────────────────────
# FILE 1: backend.tf — remote state configuration
# This tells Terraform to store its state file in S3 instead of locally.
# DynamoDB provides a lock so two engineers can't apply simultaneously.
# ─────────────────────────────────────────────

terraform {
  required_version = ">= 1.7.0"  # Prevent older Terraform versions from running this config

  backend "s3" {
    bucket         = "theforge-terraform-state-prod"  # Must exist before you run terraform init
    key            = "services/web-app/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true  # Encrypt state at rest — critical because state contains secrets
    dynamodb_table = "theforge-terraform-locks"  # Table must have a partition key named 'LockID'
  }

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"  # ~> means 5.x but not 6.x — prevents breaking changes
    }
  }
}


# ─────────────────────────────────────────────
# FILE 2: modules/web_server/main.tf — a reusable module
# This module encapsulates all the resources needed for one web server.
# Call it multiple times with different variables for different services.
# ─────────────────────────────────────────────

variable "service_name" {
  description = "Name of the service this server runs, e.g. 'payments' or 'user-api'"
  type        = string
}

variable "environment" {
  description = "Deployment environment: dev, staging, or production"
  type        = string
  validation {
    # Terraform will refuse to apply if someone passes an unexpected value
    condition     = contains(["dev", "staging", "production"], var.environment)
    error_message = "Environment must be dev, staging, or production."
  }
}

variable "instance_type" {
  description = "EC2 instance size — use t3.micro for dev, t3.medium for production"
  type        = string
  default     = "t3.micro"
}

resource "aws_instance" "service_server" {
  ami           = "ami-0c02fb55956c7d316"
  instance_type = var.instance_type

  tags = {
    Name        = "${var.service_name}-${var.environment}"  # e.g. 'payments-production'
    Environment = var.environment
    Service     = var.service_name
    ManagedBy   = "terraform"
  }
}

output "server_id" {
  value = aws_instance.service_server.id
}


# ─────────────────────────────────────────────
# FILE 3: main.tf — calling the module twice for two different services
# This is the DRY pattern in action. Same infrastructure blueprint, different values.
# ─────────────────────────────────────────────

module "payments_server" {
  source        = "./modules/web_server"
  service_name  = "payments"
  environment   = "production"
  instance_type = "t3.medium"  # Payments needs more power than default
}

module "user_api_server" {
  source       = "./modules/web_server"
  service_name = "user-api"
  environment  = "production"
  # instance_type not specified — falls back to the default t3.micro
}

IaC Security and Secrets Management — Don't Leak Your Infrastructure's Keys

Infrastructure code often requires secrets: API keys, database passwords, cloud provider credentials. A common rookie mistake is hardcoding these in the IaC files. Terraform state files, in particular, store resource attributes in plaintext, which can include sensitive values like database passwords or IAM secret keys.

The rule: IaC code should never contain secrets. Instead, use environment variables, encrypted variables in your CI/CD platform, or a dedicated secrets manager like HashiCorp Vault, AWS Secrets Manager, or GitHub Actions Secrets. For Terraform, use the sensitive = true attribute on outputs, and avoid outputting secrets in plan output. Use data sources that read from a secrets manager rather than embedding values.

Another critical practice: never commit .tfstate files to version control. Even if you delete them later, secrets are permanently exposed in Git history. Add .tfstate and .tfstate.backup to .gitignore on day one. Use a remote backend with encryption at rest.

Below is an example of using environment variables with Terraform, and a pattern for reading secrets from AWS Secrets Manager.

# Use environment variables for provider credentials — never hardcode them
provider "aws" {
  region = var.aws_region
  # Credentials are sourced from environment variables AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
}

# Read a secret from AWS Secrets Manager at apply time
# This avoids storing the secret in code or state

data "aws_secretsmanager_secret" "db_password" {
  name = "prod/db/password"
}

data "aws_secretsmanager_secret_version" "db_password_version" {
  secret_id = data.aws_secretsmanager_secret.db_password.id
}

# Use the secret in a resource, but mark it sensitive
resource "aws_db_instance" "main" {
  identifier = "theforge-prod-db"
  engine     = "postgres"
  username   = "admin"
  password   = data.aws_secretsmanager_secret_version.db_password_version.secret_string
  
  # Prevent Terraform from showing this value in output
  # The state file still contains the password, but it's encrypted at rest
  # by the S3 backend's server-side encryption
}

output "db_endpoint" {
  value     = aws_db_instance.main.endpoint
  sensitive = false  # endpoint is not secret
}

output "db_password" {
  value     = aws_db_instance.main.password
  sensitive = true   # prevents Terraform from printing the value in plan output
}

IaC Testing and Validation — Ensure Your Infrastructure Works Before You Apply

Writing infrastructure code without testing is like deploying a microservice without unit tests. You're one typo away from deleting a production database. IaC testing isn't as mature as application testing, but it's evolving fast. Here are three levels of validation every IaC pipeline needs:

1. Syntax and static analysis: Use terraform validate to catch basic HCL errors. Use tools like tflint for style and potential bugs, and checkov or tfsec for security policy violations. Run these on every PR before the plan step.
2. Plan review: The manual step where a human reviews the terraform plan output. This catches logical mistakes — like changing a security group that breaks connectivity, or accidentally destroying a stateful resource.
3. Integration testing: Tools like Terratest let you write Go tests that deploy real infrastructure, run assertions against it, and then destroy it. This is the gold standard, but it's expensive and slow. Use it sparingly for critical resources.

Below is a minimal GitHub Actions step that runs static analysis before the plan. Integrate this into your workflow to catch issues early.

# Add this as a step in your terraform-plan job before running terraform plan
# It runs tflint and checkov on the Terraform code

- name: Run tflint (Terraform linter)
  uses: terraform-linters/setup-tflint@v3
  with:
    tflint_version: 'v0.50.0'

- name: Lint Terraform code
  run: tflint --format compact
  working-directory: ${{ env.TF_WORKING_DIR }}

- name: Run checkov (security scanning)
  id: checkov
  uses: bridgecrewio/checkov-action@v12
  with:
    directory: ${{ env.TF_WORKING_DIR }}
    framework: terraform
    # Fail the pipeline on any error in the security scan
    soft_fail: false

# On failure, print detailed report
- name: Display checkov results
  if: failure()
  run: cat results
  working-directory: ${{ env.TF_WORKING_DIR }}

Why Your Deployment Pipeline Should Fail When the State Lock Is Missing

Most teams treat state locking as optional. It's not. Without a lock, two engineers — or worse, two pipelines — can apply changes simultaneously. The result is state corruption, partial deployments, and an infrastructure that matches neither configuration. I've debugged a three-hour outage caused by concurrent Terraform applies fighting over the same S3 backend. The fix was one line: dynamodb_table = "terraform-lock". Treat state locking like a database transaction. If your CI/CD pipeline doesn't fail when the lock is missing, you're accepting risk. In production, configure your backend to reject concurrent operations. Check lock status before planning. If the lock exists, abort. Your staging environment isn't a rehearsal — it's the same code path. If you skip safety there, you'll skip it in production. Add a pre-flight check that verifies lock availability before any apply command.

# io.thecodeforge.state_lock_preflight
import boto3
import sys

def check_lock(bucket, key, region='us-east-1'):
    s3 = boto3.client('s3', region_name=region)
    try:
        resp = s3.get_object(Bucket=bucket, Key=key)
        lock_body = resp['Body'].read().decode()
        if lock_body:
            print(f'⚠️  State lock ACTIVE. Aborting apply.')
            sys.exit(1)
        else:
            print('✅ No active lock. Proceeding.')
    except s3.exceptions.NoSuchKey:
        print('✅ No state file. Proceeding.')

check_lock('my-terraform-state-bucket', 'env/prod/terraform.tfstate.lock')

Stop Patching Your VMs — Burn Them and Rebuild

Mutable infrastructure feels comfortable. You SSH in, run yum update, and move on. But after six months, that server has accumulated packages, configs, and cron jobs no one remembers. Configuration drift turns your 'pet' into a fragile mystery. Immutable infrastructure solves this. When you need to update a server, you don't patch it — you destroy it and create a fresh one from a golden image. Your CI/CD pipeline builds that image, runs security scans, and deploys it to staging. If tests pass, the image goes to production. The old server is terminated. This shifts your mindset: you're not managing servers, you're managing releases. Use tools like Packer to build AMIs or Azure Image Builder for VHDs. Store images in a registry with version tags. Never SSH into production. If something's wrong, roll back to the previous image — not a 'patch Tuesday' prayer.

# io.thecodeforge.immutable_ami
source "amazon-ebs" "web" {
  ami_name      = "web-app-${formatdate("YYYYMMDDhhmm", timestamp())}"
  instance_type = "t3.micro"
  source_ami    = "ami-0c55b159cbfafe1f0"
  ssh_username  = "ubuntu"
  tags = {
    Environment = "production"
    Immutable   = "true"
  }
}

build {
  sources = ["source.amazon-ebs.web"]
  provisioner "shell" {
    inline = [
      "sudo apt-get update",
      "sudo apt-get install -y nginx",
      "sudo systemctl enable nginx"
    ]
  }
}