Ansible Configuration Drift: Stop Servers Going Rogue with Idempotent Playbooks

Configuration drift is the silent killer of reliable infrastructure. I've seen a single manual chmod 777 on a config file cascade into a 3-hour outage because the monitoring agent stopped reporting. Drift is not a matter of if, but when — and it's almost always the result of someone 'just this once' SSHing in to fix something.

The problem is that most teams treat configuration as a one-time deployment. You push a playbook, servers look good, and you move on. But servers are living systems — patches, reboots, manual fixes, and time itself change them. Without continuous enforcement, your infrastructure becomes a snowflake farm where no two boxes are alike.

By the end of this article, you'll be able to detect configuration drift in your Ansible-managed servers, write playbooks that self-heal on every run, and build a compliance pipeline that alerts you the moment a server goes out of spec. No more mystery outages. No more 'it works on my machine'.

Why Drift Happens: The Three Enemies of Idempotence

Drift doesn't appear out of nowhere. It's caused by three things: manual intervention (someone SSHes in and tweaks a file), time-based configs (like SSL certs that expire), and failed automation runs (a playbook stops halfway, leaving partial state). Each of these breaks the idempotent promise — that running the same playbook twice yields the same result.

Without idempotence, you can't trust your automation. A playbook that works on Monday might break on Tuesday because a previous run left a file in an unexpected state. The fix is to write every task so it checks the current state before making changes. Ansible modules like copy, template, and lineinfile are idempotent by design — but only if you use them correctly. Never use shell or command to do something a module can do. That's where drift hides.

# io.thecodeforge — DevOps tutorial
---
- name: Detect and fix SSH config drift
  hosts: all
  tasks:
    - name: Ensure SSH config has correct permissions
      ansible.builtin.file:
        path: /etc/ssh/sshd_config
        owner: root
        group: root
        mode: '0600'
      # This is idempotent — file module checks current mode before changing

    - name: Enforce PermitRootLogin no
      ansible.builtin.lineinfile:
        path: /etc/ssh/sshd_config
        regexp: '^PermitRootLogin'
        line: 'PermitRootLogin no'
      notify: restart sshd
      # lineinfile is idempotent — only changes if line doesn't match

  handlers:
    - name: restart sshd
      ansible.builtin.service:
        name: sshd
        state: restarted

Detecting Drift with Check Mode and Diff

Before you fix drift, you need to find it. Ansible's --check mode runs the playbook without making changes. It reports what would change. Combine it with --diff to see the exact lines that would be added, removed, or modified. This is your first line of defense against surprise changes.

But `--check` has a dirty secret: not all modules support it. The command and shell modules always report 'changed' in check mode because they can't predict the outcome. That's another reason to avoid them. Stick to modules that implement check mode properly — copy, template, file, lineinfile, service, package, user, group.

For continuous drift detection, run --check on a schedule (e.g., every 15 minutes via cron) and alert if any changes are reported. This catches manual tweaks before they cause issues.

# io.thecodeforge — DevOps tutorial
---
- name: Check for drift in NTP config
  hosts: all
  tasks:
    - name: Verify NTP config file
      ansible.builtin.template:
        src: ntp.conf.j2
        dest: /etc/ntp.conf
        owner: root
        group: root
        mode: '0644'
      check_mode: yes  # This task always runs in check mode, even if playbook doesn't use --check

    - name: Ensure NTP service is running
      ansible.builtin.service:
        name: ntpd
        state: started
        enabled: yes
      check_mode: yes

Continuous Compliance with Assert Modules

Check mode tells you what would change, but it doesn't enforce anything. For real compliance, you need tasks that validate the current state and fail if it doesn't match. Ansible's assert module is perfect for this. You can check file contents, package versions, user existence, service status — anything you can express as a condition.

Combine assert with fail to stop the playbook if a server is out of compliance. This is useful for audit trails: every run either passes (all asserts true) or fails with a clear message. You can also use assert in a separate compliance playbook that runs on a schedule and alerts on failure.

But be careful: assert only checks, it doesn't fix. For self-healing, use assert to detect drift and then run a remediation playbook. Or combine both in one playbook — check first, then fix.

# io.thecodeforge — DevOps tutorial
---
- name: Compliance audit
  hosts: all
  tasks:
    - name: Check that /etc/hosts.deny is empty
      ansible.builtin.assert:
        that:
          - lookup('file', '/etc/hosts.deny') == ''
        fail_msg: "COMPLIANCE FAIL: /etc/hosts.deny is not empty on {{ inventory_hostname }}"
        success_msg: "OK: /etc/hosts.deny is empty"

    - name: Check that no users have UID 0 except root
      ansible.builtin.assert:
        that:
          - ansible_facts.getent_passwd | dict2items | selectattr('value.1', 'equalto', '0') | list | length == 1
        fail_msg: "COMPLIANCE FAIL: Multiple users with UID 0 on {{ inventory_hostname }}"

    - name: Ensure ntpd is running
      ansible.builtin.assert:
        that:
          - ansible_facts.services['ntpd'].state == 'running'
        fail_msg: "COMPLIANCE FAIL: ntpd not running on {{ inventory_hostname }}"

Self-Healing Playbooks: Remediate Drift Automatically

Detection is useless without remediation. A self-healing playbook runs on a schedule (every hour, every day) and fixes any drift it finds. The key is idempotence: the playbook should only change what's wrong and leave everything else alone.

Design your playbooks in two phases: first, gather facts about the current state. Second, use modules that compare desired vs. actual state and make the minimum change. For example, lineinfile with regexp replaces only the matching line. template with validate checks syntax before overwriting.

But self-healing has a dark side: if the playbook itself has a bug, it can 'fix' servers into a broken state. Always run a new playbook in check mode on a canary server first. And version-control everything — you need to roll back a playbook change, not just a config change.

# io.thecodeforge — DevOps tutorial
---
- name: Self-healing playbook for web servers
  hosts: web_servers
  vars:
    desired_max_clients: 150
  tasks:
    - name: Gather current Apache config
      ansible.builtin.slurp:
        src: /etc/httpd/conf/httpd.conf
      register: httpd_config

    - name: Check MaxClients setting
      ansible.builtin.set_fact:
        current_max_clients: "{{ (httpd_config.content | b64decode).split('\n') | select('match', '^MaxClients') | first | regex_replace('.* (\\d+).*', '\\1') }}"

    - name: Fix MaxClients if wrong
      ansible.builtin.lineinfile:
        path: /etc/httpd/conf/httpd.conf
        regexp: '^MaxClients'
        line: 'MaxClients {{ desired_max_clients }}'
      when: current_max_clients | int != desired_max_clients
      notify: restart httpd

  handlers:
    - name: restart httpd
      ansible.builtin.service:
        name: httpd
        state: restarted

Handling Time-Based Drift: Certificates and Passwords

Some drift is inevitable — SSL certificates expire, passwords rotate, API keys get revoked. These are time-based drifts that no amount of idempotence can prevent. The solution is to manage them as variables that change over time, and trigger playbook runs when they change.

For SSL certs, use Ansible's openssl_certificate module to generate self-signed certs or fetch from a CA. Set a notify handler to restart the web server only when the cert changes. For passwords, use password_hash filter to store hashed passwords in variables, and run the playbook whenever the variable changes (e.g., via a CI/CD pipeline).

But the real trick is monitoring: use openssl_certificate_info to check expiry dates and alert before they expire. Don't wait for the playbook to fail because the cert is invalid.

# io.thecodeforge — DevOps tutorial
---
- name: Check and renew SSL certificates
  hosts: web_servers
  tasks:
    - name: Get certificate info
      community.crypto.openssl_certificate_info:
        path: /etc/ssl/certs/example.com.pem
      register: cert_info

    - name: Fail if cert expires in less than 30 days
      ansible.builtin.assert:
        that:
          - cert_info.not_after > ansible_date_time.epoch + (30 * 24 * 3600)
        fail_msg: "SSL cert for example.com expires on {{ cert_info.not_after | ansible.builtin.from_epoch | ansible.builtin.to_date }}"

    - name: Renew cert if needed (using acme.sh or similar)
      ansible.builtin.command:
        cmd: /usr/local/bin/acme.sh --renew -d example.com
      when: cert_info.not_after < ansible_date_time.epoch + (30 * 24 * 3600)
      notify: restart nginx

  handlers:
    - name: restart nginx
      ansible.builtin.service:
        name: nginx
        state: restarted

When Not to Use Ansible for Drift Management

Ansible is great for configuration drift, but it's not a silver bullet. If you need real-time compliance enforcement (millisecond-level), Ansible's push model is too slow. Use a tool like OPA (Open Policy Agent) or a service mesh sidecar that enforces policies at the request level.

Also, Ansible doesn't handle stateful services well. If a database's configuration drifts, you can't just overwrite the config file and restart — you might lose data. For databases, use dedicated tools like pt-config-diff or mysql_config_editor.

Finally, if your infrastructure is ephemeral (containers that live minutes), drift is less of a problem because you rebuild from scratch. Focus on image compliance instead of runtime drift.