Ansible Best Practices: Project Structure That Survives Production

You've been there. A 2000-line playbook that 'works on my machine' but takes 45 minutes to run in prod and nobody dares touch. The variables are scattered across five files, half of them unused. The inventory is a flat list of IPs that someone updates by hand. This is the Ansible equivalent of a Jenga tower — one wrong move and everything collapses. I've seen this bring down a deployment pipeline when a junior dev accidentally overwrote a critical variable because it was defined in three places with no precedence rules understood.

This isn't about writing Ansible that works. It's about writing Ansible that survives. Survives onboarding new team members. Survives scaling from 10 servers to 1000. Survives the 3am pager when a config change breaks production. The problem with most Ansible code is that it's written like a shell script — linear, fragile, and impossible to reuse. You need a structure that enforces separation of concerns, makes variables predictable, and lets you reason about what a playbook does without reading every line.

By the end of this, you'll be able to design an Ansible project that a new hire can understand in 10 minutes, that CI can validate automatically, and that you can confidently run against production without holding your breath. You'll know exactly where to put variables, how to structure roles, and when to break the rules. This is the structure I've used across three companies and hundreds of servers — it's not theoretical, it's battle-tested.

Why Flat Playbooks Are a Production Liability

Every Ansible project starts innocent. A single playbook, a few tasks. Then you add a role. Then another. Before you know it, you have a 500-line playbook with include statements pointing to files in three different directories. The problem isn't the lines of code — it's the lack of boundaries. When everything is in one file, you can't test a single component without running the whole thing. You can't reuse a configuration across projects. You can't even understand what the playbook does without scrolling through every task.

Flat playbooks also encourage hardcoding. You write hosts: webservers directly in the playbook instead of using an inventory. You embed apt: name=nginx state=present without considering that different environments might need different versions. This works until you need to deploy to a second environment. Then you copy the playbook, change a few lines, and now you have two files to maintain. Multiply by ten environments and you're in maintenance hell.

The fix is simple: use roles. Roles are self-contained units of automation. Each role has its own tasks, handlers, templates, files, and variables. A role should do one thing and do it well. For example, an nginx role installs and configures nginx. It doesn't also install PHP. That's a separate role. This separation lets you compose playbooks like building blocks: webserver.yml includes the nginx, php, and firewall roles. You can test each role independently, reuse them across projects, and swap out implementations without rewriting everything.

# io.thecodeforge — DevOps tutorial
# Bad: flat playbook with hardcoded values
---
- hosts: webservers
  become: yes
  tasks:
    - name: Install nginx
      apt:
        name: nginx
        state: present
    - name: Copy config
      copy:
        src: /home/user/nginx.conf
        dest: /etc/nginx/nginx.conf
    - name: Restart nginx
      service:
        name: nginx
        state: restarted

The Official Best Practices Directory Layout

Ansible's official documentation recommends a specific directory layout. Don't ignore it. It's not perfect, but it's the closest thing to a standard that exists. Here's the layout I use in production:

`` production/ ├── ansible.cfg ├── inventories/ │ ├── production/ │ │ ├── hosts │ │ └── group_vars/ │ │ ├── all.yml │ │ ├── webservers.yml │ │ └── databases.yml │ └── staging/ │ ├── hosts │ └── group_vars/ │ ├── all.yml │ └── webservers.yml ├── roles/ │ ├── common/ │ │ ├── tasks/ │ │ │ └── main.yml │ │ ├── handlers/ │ │ │ └── main.yml │ │ ├── templates/ │ │ ├── files/ │ │ ├── vars/ │ │ │ └── main.yml │ │ └── defaults/ │ │ └── main.yml │ ├── nginx/ │ └── postgres/ ├── playbooks/ │ ├── site.yml │ ├── webservers.yml │ └── databases.yml └── group_vars/ └── all.yml ``

Key points: Each environment gets its own inventory directory with its own group_vars. The top-level group_vars/ is for variables shared across all environments — use it sparingly. Roles are in roles/, playbooks in playbooks/. The ansible.cfg at the root sets project-wide defaults. This layout makes it clear where to find anything. When a new team member asks 'where's the nginx config template?', the answer is always 'in the nginx role's templates directory'.

# io.thecodeforge — DevOps tutorial
# ansible.cfg - project-level configuration
[defaults]
# Use project-specific roles path
roles_path = roles
# Inventory per environment - specify with -i
inventory = inventories/production/hosts
# Enable pipelining for speed (requires SSH pipelining enabled on target)
pipelining = True
# Number of parallel forks
forks = 20
# Host key checking off for dynamic environments
host_key_checking = False
# Log to file for debugging
log_path = ansible.log
# Retry files for failed hosts
retry_files_enabled = True
retry_files_save_path = .retry
# Don't gather facts unless needed
gathering = smart
fact_caching = jsonfile
fact_caching_connection = /tmp/ansible_facts
fact_caching_timeout = 3600

Variable Precedence: The Only Rule That Matters

Ansible has a variable precedence order that determines which value wins when the same variable is defined in multiple places. Ignore this at your peril. The order from lowest to highest priority is:

1. Role defaults (roles/role_name/defaults/main.yml)
2. Inventory vars (inventories/environment/group_vars/all.yml)
3. Inventory group_vars per group
4. Inventory host_vars
5. Playbook vars (vars: in playbook)
6. Playbook vars_files
7. Role vars (roles/role_name/vars/main.yml)
8. Block vars (via vars: on a block)
9. Task vars (via vars: on a task)
10. Extra vars (-e on command line)

The critical insight: role defaults have the LOWEST precedence. That means they are meant to be overridden. Use them for sensible defaults that work in development. Then override them in inventory group_vars for each environment. Never put environment-specific values in role vars — that's what group_vars is for. And never, ever use -e to pass secrets in production (they end up in logs). Use Ansible Vault instead.

I've seen a team spend three days debugging a variable that was defined in role defaults, group_vars, and playbook vars — each with a different value. The playbook vars won, but nobody knew because they assumed group_vars was the source of truth. The fix was to delete the playbook vars and move the value to group_vars, then add a CI check that fails if any variable is defined in more than two places.

# io.thecodeforge — DevOps tutorial
# Role defaults - lowest precedence, meant to be overridden
---
nginx_port: 80
nginx_user: www-data
nginx_worker_processes: 4
nginx_ssl_enabled: false
nginx_ssl_cert_path: /etc/nginx/ssl/cert.pem
nginx_ssl_key_path: /etc/nginx/ssl/key.pem

Designing Roles That Don't Suck

A good role is like a function: it takes inputs (variables), does one thing, and produces a predictable state. A bad role is a kitchen sink that installs nginx, configures PHP, sets up a firewall, and deploys the application code. Don't do that. Each role should have a single responsibility.

Here's how I structure a role:

• tasks/main.yml: The main list of tasks. Keep it short. Use include_tasks to break up complex logic into separate files (e.g., install.yml, configure.yml, service.yml).
• handlers/main.yml: Handlers for restarting services. Only define handlers that are notified by tasks in this role.
• templates/: Jinja2 templates for config files. Name them after the destination file (e.g., nginx.conf.j2).
• files/: Static files to copy. Avoid if possible — use templates instead.
• vars/main.yml: Internal variables that should NOT be overridden. Use sparingly.
• defaults/main.yml: Overridable defaults. This is where most variables go.
• meta/main.yml: Role metadata including dependencies. Use dependencies sparingly — they create coupling.

A common mistake is putting too much logic in tasks/main.yml. If your role has more than 20 tasks, split it. For example, an nginx role might have install.yml, configure.yml, vhosts.yml, and ssl.yml. Then main.yml just includes them in order. This makes it easy to skip parts by overriding variables (e.g., nginx_install: false).

# io.thecodeforge — DevOps tutorial
# Main tasks file for nginx role
---
- name: Include OS-specific variables
  include_vars: "{{ ansible_os_family }}.yml"

- name: Install nginx
  include_tasks: install.yml
  when: nginx_install | default(true)

- name: Configure nginx
  include_tasks: configure.yml
  when: nginx_configure | default(true)

- name: Manage vhosts
  include_tasks: vhosts.yml
  when: nginx_vhosts | length > 0

- name: Setup SSL
  include_tasks: ssl.yml
  when: nginx_ssl_enabled

- name: Ensure nginx is running
  service:
    name: nginx
    state: started
    enabled: yes

Inventory Organization: Beyond Flat Files

The inventory is your source of truth for what servers exist and what groups they belong to. For small projects, a flat hosts file works. But once you have more than 50 servers, you need dynamic inventory. Ansible supports dynamic inventory scripts that query cloud APIs (AWS EC2, GCP, Azure) or CMDBs. This eliminates the manual step of updating a file every time a server is added or removed.

But even with static inventory, structure matters. Use groups to represent both function (e.g., webservers, databases) and environment (e.g., production, staging). Then use group_vars to set environment-specific variables. Never put IP addresses in playbooks — always reference groups.

Here's a pattern I use for multi-environment inventory:

``` [production:children] prod-webservers prod-databases

[prod-webservers] web01.example.com web02.example.com

[prod-databases] db01.example.com

[staging:children] staging-webservers staging-databases

[staging-webservers] staging-web01.example.com

[staging-databases] staging-db01.example.com ```

Then in group_vars/production.yml, you set variables that apply to all production hosts. In group_vars/prod-webservers.yml, you set variables specific to production web servers. This hierarchy mirrors the group structure and makes variable lookup predictable.

# io.thecodeforge — DevOps tutorial
# Production inventory with group hierarchy
[prod-webservers]
web01 ansible_host=10.0.1.10
web02 ansible_host=10.0.1.11

[prod-databases]
db01 ansible_host=10.0.2.10
db02 ansible_host=10.0.2.11

[production:children]
prod-webservers
prod-databases

[all:vars]
ansible_user=deploy
ansible_ssh_private_key_file=/path/to/prod-key.pem

Playbook Design: Orchestration, Not Implementation

A playbook should be a thin orchestrator that applies roles to groups. It should not contain tasks directly. Think of playbooks as the 'conductor' and roles as the 'musicians'. The playbook decides which roles run on which hosts and in what order.

Example: site.yml might apply the common role to all hosts, then webservers.yml applies nginx and php to web servers, and databases.yml applies postgres to database servers. Each playbook is a single file that includes roles via the roles: keyword.

Use pre_tasks and post_tasks sparingly. They're useful for things like taking a host out of a load balancer before updating it, but don't put business logic there. Keep the playbook clean.

Another pattern: use import_playbook to compose larger playbooks from smaller ones. For example, site.yml might import common.yml, webservers.yml, and databases.yml. This lets you run individual playbooks for targeted updates or the full site playbook for a complete deployment.

# io.thecodeforge — DevOps tutorial
# Site-wide playbook that orchestrates all roles
---
- name: Apply common configuration to all hosts
  hosts: all
  roles:
    - common

- name: Configure web servers
  hosts: webservers
  roles:
    - nginx
    - php

- name: Configure database servers
  hosts: databases
  roles:
    - postgres

- name: Deploy application
  hosts: webservers
  roles:
    - app-deploy

Secrets Management: Ansible Vault and Beyond

Hardcoding passwords in playbooks is a fireable offense. Use Ansible Vault to encrypt sensitive variables. The workflow: create a vault-encrypted file (e.g., group_vars/production/vault.yml) that contains secrets like db_password: !vault | .... Then reference db_password in your playbooks normally. Ansible decrypts it at runtime if you provide the vault password.

But vault has limitations. It's slow for large files. It doesn't integrate with secret rotation. For production, consider using a secrets manager like HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault. Ansible can fetch secrets from these at runtime using lookup plugins (e.g., hashi_vault, aws_secret). This avoids storing encrypted files in your repo and enables dynamic secret rotation.

Never use -e to pass secrets on the command line. They appear in process lists and logs. If you must use extra vars, use --extra-vars @secrets.yml where secrets.yml is vault-encrypted.

A pattern I use: store a single vault password in a CI/CD secret (e.g., GitHub Actions secret). The CI job uses that password to decrypt vault files during deployment. The vault files themselves are committed to the repo, but only the CI has the password. This is a reasonable trade-off for small teams. For larger teams, use a secrets manager.

# io.thecodeforge — DevOps tutorial
# Vault-encrypted file containing secrets
# Encrypt with: ansible-vault encrypt group_vars/production/vault.yml
---
vault_db_password: !vault |
          $ANSIBLE_VAULT;1.1;AES256
          66386439653236336...
vault_api_key: !vault |
          $ANSIBLE_VAULT;1.1;AES256
          34326634363534363...

Testing and CI: Catch Failures Before They Hit Prod

Ansible has no built-in testing framework. That's terrifying. You need to validate your playbooks before they run against production. Here's my minimum testing pipeline:

1. Syntax check: ansible-playbook --syntax-check playbooks/site.yml — catches YAML errors and missing variables.
2. Dry run: ansible-playbook --check playbooks/site.yml — simulates changes without applying them. Not perfect (modules may not support check mode), but catches most issues.
3. Linting: Use ansible-lint to enforce best practices. It catches things like hardcoded IPs, missing tags, and privilege escalation issues.
4. Unit testing with Molecule: Molecule lets you test roles in Docker containers or VMs. You write test scenarios that verify the role produces the expected state. This is the closest you get to a proper test suite.
5. Integration testing: Run your playbooks against a staging environment that mirrors production. This catches environment-specific issues that unit tests miss.

In CI (GitHub Actions, GitLab CI, Jenkins), run syntax check, lint, and Molecule tests on every commit. Run dry-run against staging on merge to main. Run full integration tests before deployment to production.

I've seen teams skip testing and then discover that a playbook works on Ubuntu 20.04 but fails on 22.04 because a package name changed. Molecule with multiple Docker images would have caught that.

# io.thecodeforge — DevOps tutorial
# GitHub Actions workflow for Ansible CI
name: Ansible CI
on: [push, pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Install dependencies
        run: |
          pip install ansible ansible-lint molecule molecule-docker
      - name: Syntax check
        run: ansible-playbook --syntax-check playbooks/site.yml
      - name: Lint
        run: ansible-lint
      - name: Molecule test
        run: molecule test
        working-directory: roles/nginx

When to Break the Rules

Every best practice has exceptions. Here's when to deviate:

• Small teams, small infrastructure: If you have 5 servers and 2 environments, a flat playbook with a few roles is fine. The overhead of the full directory layout isn't worth it.
• Prototyping: When you're exploring a new tool, don't worry about structure. Write a quick playbook to test the concept. Refactor later.
• One-off tasks: For ad-hoc operations (e.g., 'restart all web servers'), use ansible command directly, not a playbook.
• Tight coupling: Sometimes two roles are so tightly coupled that separating them adds complexity. For example, an app role that depends on a specific nginx config. In that case, consider merging them or using role dependencies.
• Performance: If you have thousands of hosts, the overhead of fact caching and pipelining might not be enough. Consider using ansible-pull for pull-based configuration, or switch to a different tool like SaltStack or Terraform for certain tasks.

The key is to know the rules so you know when to break them. If you're breaking a rule, document why in a README or inline comment. Future you will thank you.