Ansible state:latest — One Task Broke Payments for 47 Min

Every cloud infrastructure beyond a certain size hits the same wall. Someone on the team is spending their Friday afternoon manually SSH-ing into 30 servers, running the same five commands in sequence, and quietly hoping they didn't typo on server 24. It's slow. It's completely unauditable. And it doesn't scale — not to 100 servers, not to three environments, certainly not to a team of ten engineers who all have slightly different opinions about how to run that one sed command.

Scale that manual process to hundreds of EC2 instances or GCP VMs and the problem stops being annoying and starts being a business risk. An outage caused by configuration drift — two servers out of forty that silently diverged from the others — is nearly impossible to diagnose if you have no record of what was changed, when, and by whom.

That's the exact world Ansible was built to fix. It enforces a declared, version-controlled state across every machine in your fleet simultaneously, starting from nothing more than an SSH key and a YAML file checked into Git.

But Ansible has traps that aren't obvious from the documentation. state: latest looks safe until it upgrades Nginx on a Monday morning and changes a default TLS cipher suite. Handlers look optional until you realize your playbook has been restarting your app server on every run for three months. Roles look like bureaucratic overhead until your playbook hits 300 lines and two engineers are editing conflicting sections.

By the end of this article you'll understand not just how to write playbooks, but why they're structured the way they are. You'll know how inventory files map to real cloud environments, how roles package automation that other teams can actually reuse, and how handlers restart services only when config genuinely changed — not on every run. You'll leave with the mental model and the production lessons that take most engineers two or three incidents to learn the hard way.

Why Ansible Basics Are Not Optional

Ansible is an agentless automation tool that uses SSH (or WinRM) to push declarative state to remote hosts. You write YAML playbooks describing the desired state—package installed, service running, file present—and Ansible idempotently converges the system to match. No agents, no daemons, no persistent connections: it spins up, executes, and tears down. The core mechanic is a push-based, stateless model where each playbook run is a fresh transaction against the target inventory. This means zero overhead on managed nodes, but also zero tolerance for drift between runs.

In practice, Ansible evaluates tasks sequentially, gathering facts first, then applying changes in order. Idempotency is not magic—it depends on modules checking current state before acting (e.g., yum module checks if package is already installed). If a module doesn't support check mode or fails to detect state correctly, you get unintended side effects. The control machine does all the heavy lifting; targets only need Python and SSH. This makes Ansible fast to adopt but slow at scale—running 1000 hosts serially takes O(n) time without tuning forks or async.

Use Ansible when you need to bootstrap, configure, or enforce state across a fleet without installing permanent infrastructure. It excels at one-shot provisioning, compliance checks, and ad-hoc fixes. But it is not a real-time configuration daemon—if you need continuous enforcement, pair it with a pull-based tool like Chef or a GitOps operator. The moment you treat Ansible as a live monitoring system, you will miss drift until the next playbook run.

Step-by-Step Ansible Installation on Ubuntu 22.04 — Control Node Setup That Lasts

Before you write a single playbook, you need a control node. This is the machine from which you'll run all Ansible commands. It can be your laptop, a dedicated jump box, or a CI runner. The installation method you choose has operational consequences for upgrade cycles and environment consistency.

Three installation methods compete for your attention. The Python package manager (pip) is the most flexible and lets you pin exact versions. The distribution's apt repository gives you system integration and automatic updates. The newer pipx method isolates Ansible in its own virtual environment and is the official Python Packaging Authority (PyPA) recommendation for installing CLI tools.

For production control nodes — dedicated VMs or CI runners — pip installation inside a Python virtual environment is the standard. It gives you version pinning (critical for consistency), isolation from the system Python, and easy upgrades via requirements files. The following sequence sets up Ansible in a virtual environment under /opt/ansible, with a symlink in /usr/local/bin for global access.

After installation, you configure the control node's SSH access. Ansible needs to reach every managed server via SSH with key-based authentication. The common failure point is SSH host key checking. When you connect to a server for the first time, Ansible's default behavior is to verify the host key against ~/.ssh/known_hosts. In dynamic cloud environments where IPs are recycled, this causes prompt blocks. The production fix is to manage known_hosts via a pre-seeded file or use host_key_checking=False in ansible.cfg with an understanding of the security trade-off.

The inventory file is your first configuration file. It lists your managed nodes and groups them logically. For testing, a one-line inventory with a single server is enough. In production, you'll use dynamic inventory plugins that query cloud APIs.

To verify installation, run ansible all -i 'localhost,' -m ping -c local. This pings the control node itself without SSH, confirming the Ansible engine works.

# ── Option A: pip in a virtual environment (recommended for dedicated control nodes) ──
sudo apt update && sudo apt install python3-venv python3-pip -y
sudo python3 -m venv /opt/ansible
/opt/ansible/bin/pip install --upgrade pip
/opt/ansible/bin/pip install ansible-core==2.17.4
sudo ln -s /opt/ansible/bin/ansible* /usr/local/bin/

# Verify
ansible --version

# ── Option B: apt (simple but version lags) ──
sudo apt update
sudo apt install software-properties-common -y
sudo add-apt-repository --yes --update ppa:ansible/ansible
sudo apt install ansible -y

# ── Option C: pipx (isolated CLI tool, good for laptops) ──
sudo apt install pipx -y
pipx ensurepath
pipx install ansible-core==2.17.4

# ── After any method: configure ansible.cfg ──
mkdir -p ~/ansible
cat > ~/ansible/ansible.cfg << 'EOF'
[defaults]
inventory = inventory
host_key_checking = False
pipelining = True
forks = 25
EOF

# ── Create a test inventory ──
echo 'localhost ansible_connection=local' > ~/ansible/inventory

# ── Test the ping module ──
cd ~/ansible
ansible all -i inventory -m ping

Ansible vs Chef vs Puppet vs SaltStack — Choosing the Right Configuration Management Tool

If you're new to configuration management, the first question is not 'how do I use Ansible' but 'should I use Ansible at all?' Chef, Puppet, and SaltStack are the three other major players, and each has strengths that match different operational philosophies. The right choice depends on your team's existing skills, infrastructure scale, and whether you prefer a push or pull model.

Ansible is the only major tool that is agentless — it connects to managed nodes over SSH (or WinRM) and executes tasks without installing any software. This makes initial setup trivial: if you have SSH keys, you have Ansible. The trade-off is that every playbook run opens new SSH connections, which creates overhead at scale. Ansible uses YAML for its playbooks, which is the easiest language for non-programmers to read and write. Its push model means you initiate changes from a central control node, which is natural for ad-hoc operations and CI/CD pipelines.

Chef uses a pull model: a client agent on each node periodically fetches the desired state from a Chef server. This is more resilient in environments where nodes are behind firewalls or have intermittent connectivity. Chef uses Ruby DSL for its cookbooks, which is more powerful but has a steeper learning curve. Its test-kitchen testing framework is the most mature in the CM space. Chef is strong for organizations that need a full audit trail and have dedicated platform teams.

Puppet also uses a pull model with an agent and is the oldest of the four. It has its own declarative language (Puppet DSL) that is designed for idempotency from the ground up. Puppet's module ecosystem (Puppet Forge) is vast, and its reporting capabilities are excellent. The downside is the complexity of running a Puppet server and the agent overhead on each node.

SaltStack (Salt) offers both push and pull modes via its ZeroMQ message bus, making it extremely fast at scale — it can manage thousands of nodes in seconds. It uses YAML or Python for its states (SLS files) and includes a powerful event-driven reactor system. Salt's master-minion architecture requires an agent and a master, but the agent is lightweight. It's popular in high-performance computing and environments that need real-time command execution.

The table below summarizes the key differences so you can make an informed decision based on your team's context.

<table>
  <thead>
    <tr>
      <th>Feature</th>
      <th>Ansible</th>
      <th>Chef</th>
      <th>Puppet</th>
      <th>SaltStack</th>
    </tr>
  </thead>
  <tbody>
    <tr><td>Architecture</td><td>Agentless (push)</td><td>Agent (pull)</td><td>Agent (pull)</td><td>Agent (push/pull)</td></tr>
    <tr><td>Language</td><td>YAML</td><td>Ruby DSL</td><td>Puppet DSL</td><td>YAML / Python</td></tr>
    <tr><td>Setup complexity</td><td>Low (SSH only)</td><td>High (server + agent)</td><td>High (server + agent)</td><td>Medium (master-minion)</td></tr>
    <tr><td>Learning curve</td><td>Lowest</td><td>Medium-High</td><td>Medium</td><td>Medium</td></tr>
    <tr><td>Scale performance</td><td>Moderate (SSH bottleneck)</td><td>Good (background agent)</td><td>Good</td><td>Excellent (ZeroMQ)</td></tr>
    <tr><td>Idempotency enforcement</td><td>Module-dependent</td><td>Built-in resource model</td><td>Built-in</td><td>Module-dependent</td></tr>
    <tr><td>Testing ecosystem</td><td>Molecule (evolving)</td><td>Test Kitchen (mature)</td><td>Beaker (mature)</td><td>Molecule (via Salt-Nostalgic)</td></tr>
    <tr><td>Best for</td><td>Small-medium fleets, CI/CD, ad-hoc</td><td>Large enterprises, audit-heavy</td><td>Large fleets with strict compliance</td><td>Massive fleets, HPC, real-time</td></tr>
  </tbody>
]</table>

Ansible Architecture — How the Control Node, Inventory, and Managed Nodes Interact

Understanding Ansible's architecture is the foundation for debugging connection issues, scaling automation, and choosing the right deployment model. The architecture is deceptively simple: a control node runs Ansible, reads an inventory, and connects to managed nodes via SSH. But the simplicity hides a few sharp edges that only show up in production at scale.

The control node is any machine with Ansible installed — your laptop, a build server, a dedicated jump host. It's the single point of failure in the architecture. If your control node goes down, you cannot run any automation until it's restored. This is why production setups use multiple control nodes in a load-balanced fashion or rely on a CI/CD platform that can re-run jobs from any agent.

The inventory is the source of truth for which nodes exist and how they're grouped. Static inventory files map hostnames to IP addresses. Dynamic inventory plugins query cloud provider APIs and build the host list at runtime. The inventory also holds variables that travel with hosts into playbooks.

Managed nodes are the target servers. They need SSH access from the control node and Python 3 installed. That's it. No agent, no daemon, no open ports beyond SSH. This is the biggest architectural advantage over agent-based tools: you can manage any server that's reachable over SSH, including on-premise machines, cloud VMs, containers (via Docker exec), and even Windows via WinRM.

Ansible's execution model is push-based. You run a command on the control node, Ansible opens SSH connections to each managed node in parallel (controlled by the forks setting), copies the Python module code over, executes it, collects JSON results, and closes the connection. There is no persistent connection. This simplicity means Ansible is stateless from the managed node's perspective, but it also means every playbook run pays the SSH connection overhead.

The following diagram visualizes the flow during a typical playbook run. The control node reads the playbook and inventory, resolves variables, then fans out tasks to each managed node group sequentially (play by play) but within a play, tasks run on all hosts in parallel up to forks concurrent connections.

┌─────────────────────────────────────────────────────────────────────┐
│                     CONTROL NODE (Ansible CLI)                      │
│  ┌─────────────┐   ┌──────────────┐   ┌────────────────────────┐   │
│  │  Playbook    │   │  Inventory    │   │  ansible.cfg          │   │
│  │  (YAML)      │   │  (static/    │   │  forks=25             │   │
│  │  - hosts     │──▶│   dynamic)    │   │  pipelining=True      │   │
│  │  - tasks     │   │  - groups    │   │  host_key_checking=no │   │
│  │  - handlers  │   │  - variables │   └────────────────────────┘   │
│  └──────┬───────┘   └──────────────┘                                │
│         │                                                            │
│         │  Ansible compiles task list per host                       │
│         ▼                                                            │
│  ┌──────────────────────────────────┐                                │
│  │  SSH Connection Pool (forks=25) │                                │
│  │  (Manages up to 25 parallel SSH)│                                │
│  └────────────┬─────────────────────┘                                │
└────────────────┼─────────────────────────────────────────────────────┘
                 │
                 │  SSH (port 22) + Python execution
                 ▼
┌─────────────────────────────────────────────────────────────────────┐
│                      MANAGED NODES                                  │
│                                                                     │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐     ┌──────────┐        │
│  │ Web-01   │  │ Web-02   │  │ DB-01    │     │ DB-02    │        │
│  │ 10.0.1.1 │  │ 10.0.1.2 │  │ 10.0.2.1 │     │ 10.0.2.2 │        │
│  │ Python 3 │  │ Python 3 │  │ Python 3 │     │ Python 3 │        │
│  │ SSH key  │  │ SSH key  │  │ SSH key  │     │ SSH key  │        │
│  └──────────┘  └──────────┘  └──────────┘     └──────────┘        │
│                                                                     │
│  On each node:                                                      │
│  1. Ansible copies Python script (module)                           │
│  2. Executes with given arguments                                   │
│  3. Returns JSON result (changed, ok, failed)                       │
│  4. Temporary files are cleaned up                                  │
└─────────────────────────────────────────────────────────────────────┘

Ad-Hoc Ansible Commands — Quick Operations Without Writing a Playbook

Not every task deserves a playbook. Sometimes you need to check the uptime on 50 servers, copy a configuration file to a specific server, or restart a service immediately during an incident. Ad-hoc commands are single-module operations you run directly from the command line without a playbook file. They're ideal for read-only queries, one-off changes, and emergency responses.

The pattern is always: ansible <host-pattern> -m <module> -a '<module arguments>'. The host pattern matches inventory groups, wildcards, or specific hostnames. The module name is the Ansible module to use. The -a argument string depends on the module.

Three modules dominate ad-hoc usage. The ping module tests SSH connectivity and Python availability — it's the first command you run after setting up a new inventory. The command module runs any shell command with arguments directly, but it always reports 'changed' and is not idempotent. In ad-hoc mode, that's usually fine because you're making a one-off change. The shell module is similar but runs through /bin/sh and supports shell operators like pipes and redirects.

For copying files, the copy module is idempotent even in ad-hoc mode: it only transfers the file if the source and destination differ. This makes it safe to use for emergency configuration pushes without worrying about overwriting an identical file unnecessarily.

Ad-hoc commands are powerful but leave no audit trail unless you log them. Every ad-hoc change should be logged with script or tee and followed up with a permanent playbook change. If you find yourself running the same ad-hoc command twice, it's a sign that operation should be a playbook.

A practical production use case: a security vulnerability requires updating a package version across the fleet immediately. You cannot wait for the CI pipeline. ansible all -m ansible.builtin.apt -a 'name=openssl state=latest update_cache=yes' --become patches all servers in one command. After the emergency, you pin the intended version in your main playbook and remove the state=latest usage.

# ── Ping all servers in the 'production' group ─────────────────────
ansible production -i inventory -m ping

# ── Run uptime on web servers ───────────────────────────────────────
ansible web_servers -i inventory -m command -a 'uptime'

# ── Check disk usage on all servers ─────────────────────────────────
ansible all -i inventory -m shell -a 'df -h / | tail -1'

# ── Copy a configuration file to one server ─────────────────────────
ansible app-01 -i inventory -m copy -a 'src=./nginx.conf dest=/etc/nginx/nginx.conf owner=root group=root mode=0644'

# ── Restart Nginx service on all web servers ────────────────────────
ansible web_servers -i inventory -m service -a 'name=nginx state=restarted' --become

# ── Gather facts for a specific host ─────────────────────────────────
ansible db-primary -i inventory -m setup -a 'gather_subset=network'

# ── Install a package everywhere (emergency) ─────────────────────────
ansible all -i inventory -m apt -a 'name=openssl state=latest update_cache=yes' --become

# ── Check if a service is running with piped shell command ───────────
ansible all -i inventory -m shell -a 'systemctl is-active nginx && echo "active" || echo "inactive"'

Ansible Modules Quick-Reference — The 15 Most Common Modules and When to Use Them

Modules are the building blocks of every Ansible task. Each module is a small Python script that performs a specific operation — installing a package, copying a file, managing a service. The art of writing good playbooks is knowing which module to use for which job. Using the wrong module (especially shell or command when a dedicated module exists) breaks idempotency, disables check mode, and makes your playbooks unreliable.

The table below lists the 15 most commonly used modules in production environments, along with their primary use case and a quick example. Master these and you can automate roughly 90% of infrastructure tasks.

The most important production rule: always prefer the dedicated module over command or shell. If you find yourself writing shell: apt-get install, replace it with the apt module. The dedicated module provides idempotency, check mode support, and proper change detection. The only legitimate use for command/shell is when no module exists for the operation, or in ad-hoc emergency commands.

---
# Example: Using the top 5 modules in a real playbook snippet
- name: Install and configure Nginx
  hosts: web_servers
  become: true
  tasks:
    - name: Ensure Nginx is installed (apt module)
      ansible.builtin.apt:
        name: nginx=1.24.*
        state: present
        update_cache: true

    - name: Deploy Nginx configuration (template module)
      ansible.builtin.template:
        src: nginx.conf.j2
        dest: /etc/nginx/nginx.conf
      notify: Reload Nginx

    - name: Create document root (file module)
      ansible.builtin.file:
        path: /var/www/html
        state: directory
        owner: www-data
        group: www-data

    - name: Copy index page (copy module)
      ansible.builtin.copy:
        src: index.html
        dest: /var/www/html/index.html

    - name: Ensure Nginx is running (service module)
      ansible.builtin.service:
        name: nginx
        state: started
        enabled: yes

  handlers:
    - name: Reload Nginx
      ansible.builtin.service:
        name: nginx
        state: reloaded

Your First Ansible Playbook — Install Apache on a Web Server Cluster

A playbook is a YAML file that describes the desired state of a set of hosts. It's the core unit of automation in Ansible. Writing your first playbook reinforces the mental model of declaring 'what should be true' rather than scripting 'what commands to run'.

The canonical first playbook installs and configures Apache on a group of web servers. It exercises the three most common modules: apt for package management, template for configuration files, and service for daemon management. It also introduces handlers by restarting Apache only when the configuration file actually changes.

Create an inventory file with one or two test servers, or use localhost with ansible_connection=local for a safe first run. The playbook below assumes an inventory group called web_servers that you define. It becomes root via become: true because installing packages and starting services requires superuser privileges.

The playbook has four tasks plus a handler. The first task installs Apache at the version provided by the distribution's default repositories. Using state: present ensures it's installed but won't upgrade it unexpectedly. The second task creates a custom index.html using the copy module with content directly — avoids a template file for this simple example. The third task copies an Apache virtualhost configuration from a file on the control node. The fourth task enables the site and ensures Apache runs on boot. The handler restarts Apache only when the virtualhost config changes.

Running the playbook the first time changes state (installs, writes, restarts). Running it again reports 'ok' for all tasks because the desired state is already in place — this is idempotency in action.

Use --check --diff before the first real run to see what would change without actually changing anything. This is covered in the next section.

# ── First playbook: install and configure Apache ──────────────────
# Run with: ansible-playbook -i inventory apache.yml
# Run dry-run: ansible-playbook -i inventory apache.yml --check --diff

- name: Install and configure Apache on web servers
  hosts: web_servers
  become: true
  gather_facts: false

  vars:
    # Default index.html content
    welcome_message: "Welcome to TheCodeForge demo!"
    # Apache virtualhost port (can be overridden per host via inventory)
    apache_port: 80

  handlers:
    - name: Restart Apache
      ansible.builtin.service:
        name: apache2
        state: restarted
      listen: "apache config changed"

  tasks:
    - name: Install Apache2 package
      ansible.builtin.apt:
        name: apache2
        state: present
        update_cache: true
        cache_valid_time: 3600

    - name: Create custom index.html
      ansible.builtin.copy:
        content: |
          <html>
          <head><title>TheCodeForge Demo</title></head>
          <body><h1>{{ welcome_message }}</h1></body>
          </html>
        dest: /var/www/html/index.html
        owner: www-data
        group: www-data
        mode: '0644'

    - name: Deploy virtualhost configuration
      ansible.builtin.copy:
        src: files/apache-site.conf
        dest: /etc/apache2/sites-available/001-webapp.conf
        owner: root
        group: root
        mode: '0644'
      notify: "apache config changed"
      # The handler only fires if this task reports 'changed'

    - name: Ensure Apache is enabled and running
      ansible.builtin.service:
        name: apache2
        state: started
        enabled: true

---
# files/apache-site.conf
# Place this file alongside the playbook:
# <VirtualHost *:{{ apache_port }}>
#     DocumentRoot /var/www/html
#     ServerName localhost
# </VirtualHost>

# ── Sample output on first run ───────────────────────────────────────
# PLAY [Install and configure Apache on web servers] ********************
# TASK [Install Apache2 package] ***************************************
# changed: [web-01]
# TASK [Create custom index.html] ***************************************
# changed: [web-01]
# TASK [Deploy virtualhost configuration] *******************************
# changed: [web-01]
# TASK [Ensure Apache is enabled and running] ***************************
# changed: [web-01]
# RUNNING HANDLER [Restart Apache] **************************************
# changed: [web-01]
# PLAY RECAP ************************************************************
# web-01   : ok=5   changed=4   unreachable=0   failed=0   skipped=0

Playbook Check Mode and Diff — Validate Before You Change

The --check flag runs a playbook in 'dry-run' mode: Ansible evaluates every task's condition and reports what would change without actually making any changes. Combined with --diff, it shows the exact content differences for template, copy, and other modules that manage file content. This combination is the closest thing Ansible has to a pre-deployment validation step.

Check mode is not a simulation. Modules that support it (most built-in modules) check their current state and report 'changed' or 'ok' based on whether the task would alter the system. Modules that don't support check mode run partially or report that they would change, reducing confidence. The shell and command modules, for example, always report 'changed' in check mode because they cannot predict their outcome. This is another reason to prefer dedicated modules.

The --diff flag shows the before-and-after content for files managed by copy, template, file (with content), and others. It also shows which lines in configuration files would be added, removed, or modified. You review this output to catch mistakes like a typo in a template variable or an incorrect file permission before they reach production.

In production CI pipelines, every playbook run that targets staging or production should first execute a check-diff run. If the playbook would change more than expected (e.g., 200 files changed when you only expected 2), the pipeline should halt and alert a human. This is a classic 'change validation' pattern.

One caveat: check mode does not execute handlers, even if they would be notified. It also does not run command or shell tasks, so if your playbook relies on those for idempotency, check mode gives less reliable output. The rule of thumb: the more dedicated modules you use, the more accurate your dry-run results will be.

# ── Dry-run a playbook against staging ─────────────────────────────
ansible-playbook -i staging site.yml --check --diff

# ── Dry-run against a single host ────────────────────────────────────
ansible-playbook -i production site.yml --limit app-01 --check --diff

# ── Capture diff output for review ───────────────────────────────────
ansible-playbook -i staging site.yml --check --diff 2>&1 | tee /tmp/dry-run-$(date +%Y%m%d-%H%M).log

# ── Grep for any tasks that would change in dry-run ───────────────────
grep -E '(changed|TASK)' /tmp/dry-run-*.log | grep -v 'ok:'

# ── Run in check mode with increased verbosity for detailed output ───
ansible-playbook -i production site.yml --check -vv

# ── In a CI pipeline, fail if check mode shows unexpected changes ─────
# After dry-run, if the output contains 'changed:', and the expected
# change count is > X, the pipeline should fail.
ansible-playbook -i staging site.yml --check --diff 2>&1 | \n  awk '/^TASK/ { task=$0 } /changed/ { print task, $0 }' | \n  wc -l
# (examine count, set threshold in pipeline logic)

Ansible Variables and Precedence — The 22 Levels That Bite You

Variable precedence is the most common source of 'why is my playbook using the wrong value?' in production. Ansible has 22 different places where a variable can be defined, with a specific order of precedence. When the same variable name appears in multiple places, the highest-precedence definition wins. Misunderstanding this order leads to subtle bugs that only appear in certain environments.

The precedence order from lowest to highest (later overrides earlier): - role defaults (defaults/main.yml) - inventory group_vars/all - inventory group_vars/groupname - inventory host_vars/hostname - playbook group_vars/all - playbook group_vars/groupname - playbook host_vars/hostname - vars in playbook (vars block) - vars files included via include_vars - role vars (vars/main.yml) - block vars (within a block) - task vars (vars on a task) - set_fact (at runtime) - register (variable from task output) - extra vars (-e, highest priority)

In practice, the conflict you'll encounter most often is between group_vars/all (low) and --extra-vars (high). A stray -e in a CI pipeline can override everything else in your inventory, causing the wrong environment to be configured. Another common trap: using set_fact inside a loop — it overwrites the variable each iteration instead of accumulating.

To debug variable precedence, use the debug module to print the variable value at different points in the playbook. The -v flag also shows variable interpolation. When you need to merge lists or dictionaries instead of overriding, use the combine filter with the recursive=true option.

# Example showing variable precedence and debugging
- name: Demonstrate variable precedence
  hosts: localhost
  gather_facts: no
  vars:
    app_port: 8080          # playbook vars
  tasks:
    - name: Set a fact (runtime, higher precedence)
      set_fact:
        app_port: 9090

    - name: Print variable – will show 9090, not 8080
      debug:
        msg: "app_port is {{ app_port }}"

    - name: Demonstrate extra-vars override (run with -e app_port=9999)
      debug:
        msg: "If passed -e, this will show 9999"

    - name: Merge dictionaries instead of overriding
      set_fact:
        config: "{{ config | default({}) | combine({'timeout': 30}, recursive=True) }}"

Ansible Roles — Stop Copy-Pasting Playbooks Like an Amateur

Roles are how you stop treating Ansible like a scripting language and start treating it like infrastructure code you’d ship to production. Without roles, your playbooks become a tangled mess of tasks, handlers, and variables that only you understand — until you don't. Roles enforce a filesystem contract: tasks go in tasks/, handlers in handlers/, defaults in defaults/, templates in templates/. This isn't bureaucracy; it's survival. When a junior onboards and your playbook has 500 lines of unorganized YAML, they will break something. Roles give you modular, reusable, testable units. Use ansible-galaxy init to scaffold one. Wire it into a playbook with a single include_role call. That's it. Your production deploys should be a composition of roles, not a novel.

// io.thecodeforge — devops tutorial

// Scafold the role structure once
// ansible-galaxy init web_server

- name: Deploy web application to production cluster
  hosts: webservers
  become: yes
  roles:
    - role: common_packages         # Installs curl, htop, etc. from common_packages role
    - role: web_server               # Installs and configures nginx
      vars:
        nginx_port: 8443             # Override default from roles/web_server/defaults/main.yml
        ssl_cert_path: /etc/ssl/certs/server.crt
    - role: monitoring_agent         # Installs and configures datadog agent
      when: env == "production"      # Only run this role in production, not staging

Ansible vs Terraform — Two Different Hammers, One Toolbox

Here’s what nobody tells you: Ansible and Terraform are not competitors. They solve different layers of the same problem. Terraform is a provisioning tool. It talks to cloud APIs to create infrastructure — VMs, networks, load balancers. It cares about state, drift detection, and idempotent resource creation. Ansible is a configuration tool. It logs into that provisioned machine and installs software, tweaks config files, starts services. Terraform asks 'Does this VM exist?' Ansible asks 'Is Apache running?' Use Terraform to build the house. Use Ansible to furnish it. The moment you try to use Ansible to create an AWS EC2 instance, you’re fighting the tool. The moment you use Terraform to configure nginx inside that instance, you’re fighting the tool. Know the boundary. Your CI/CD pipeline should call Terraform first, then Ansible. Every senior engineer I know does this.

// io.thecodeforge — devops tutorial

// Example CI/CD pipeline stage (GitLab CI / GitHub Actions equivalent)

# Step 1: Terraform provisions the infrastructure
# terraform apply -auto-approve

# Step 2: Ansible inventory is generated from Terraform output
# terraform output -json ansible_inventory > inventory.json

- name: Configure provisioned EC2 instances
  hosts: all
  become: yes
  vars:
    # Terraform outputs these IPs; inject via --extra-vars or dynamic inventory script
    app_version: "1.4.2"
    db_connection_string: "{{ lookup('env', 'DB_CONN') }}"  # Fetch from CI/CD secrets
  tasks:
    - name: Install application binary
      ansible.builtin.copy:
        src: "/artifacts/myapp-{{ app_version }}.bin"
        dest: "/usr/local/bin/myapp"
        mode: '0755'
      notify: restart app

  handlers:
    - name: restart app
      ansible.builtin.systemd:
        name: myapp
        state: restarted
        daemon_reload: yes

Ansible Tower (AWX) — Centralized Execution Without the Spreadsheet Mayhem

If you're still ssh-ing into your control node and running ansible-playbook by hand, you're one fat-finger away from taking down production. Ansible Tower (or its open-source upstream, AWX) gives you a web UI, RBAC, job scheduling, and — most importantly — an audit trail. When the VP asks 'Who ran that playbook at 3 AM?', you don't shrug. You pull up the job log. Tower also solves the 'works on my machine' problem. Playbooks run on Tower's execution environment, not your laptop with the experimental Python 3.12. Use it to segment environments: developers get 'Run' access to staging, read-only to production. Operations gets full control. No more shared passwords on a sticky note. The real power? The REST API. Your CI/CD can POST to Tower instead of SSH-ing into a jump box. That way, every deploy is recorded, every failure is logged, and every success is celebrated.

// io.thecodeforge — devops tutorial

// Trigger a Tower job template from CI/CD pipeline
// curl -X POST https://tower.example.com/api/v2/job_templates/7/launch/ \
//   -H "Authorization: Bearer $(get_tower_token)" \
//   -H "Content-Type: application/json" \
//   -d '{"extra_vars": {"env": "production", "version": "1.4.2"}}'

# Inside the job template, the playbook looks like:
- name: Deploy app from Tower job
  hosts: "{{ env }}"                # Environment passed as extra_var
  become: yes
  roles:
    - role: deploy_app
      vars:
        app_version: "{{ version }}" # Version passed from CI/CD
  
  # Tower automatically captures:
  # - who triggered the job
  # - when it ran
  # - output of every task
  # - exit code and failures