Event-Driven Ansible: Automating Incident Response Without Polling

Most infrastructure automation is reactive polling. You set a cron job to run every 5 minutes, check if something is wrong, and fix it. That's wasteful and slow. Event-Driven Ansible flips the model: it listens for events and reacts instantly. No more wasted cycles checking for problems that don't exist. By the end of this article, you'll be able to set up an event-driven rule that automatically restarts a failed service, scales a deployment based on load, or even triggers a rollback when a deployment fails — all without a single cron job.

Why Polling Is a Waste of Resources

Cron jobs are the duct tape of automation. They run regardless of whether anything needs to happen. In production, that means thousands of unnecessary API calls, wasted CPU cycles, and delayed responses. Event-Driven Ansible eliminates the polling tax. Instead of asking 'Is it broken yet?' every 5 minutes, it waits for the answer to come to it. The result: faster reaction times, lower resource usage, and simpler code. No more managing cron schedules or worrying about overlapping runs.

# io.thecodeforge — DevOps tutorial

# Polling approach (bad):
# cron: */5 * * * * ansible-playbook check_and_fix.yml

# Event-driven approach (good):
# Rulebook listens for 'service_down' event
- name: React to service failure
  hosts: localhost
  sources:
    - ansible.eda.alertmanager:
        host: 0.0.0.0
        port: 5000
  rules:
    - name: Restart failed service
      condition: event.alert.status == 'firing' and event.alert.labels.service == 'nginx'
      action:
        run_playbook:
          name: restart_service.yml

Setting Up Your First Event Source

Event sources are the ears of Event-Driven Ansible. They can be webhooks, message queues, cloud events, or monitoring alerts. The most common production setup is a webhook endpoint that receives POST requests from your monitoring system or CI/CD pipeline. Here's how to configure a simple webhook source that listens for JSON payloads on port 5000.

# io.thecodeforge — DevOps tutorial

- name: Webhook listener for deployment events
  hosts: localhost
  sources:
    - ansible.eda.webhook:
        host: 0.0.0.0
        port: 5000
  rules:
    - name: Deploy new version
      condition: event.payload.action == 'deploy' and event.payload.environment == 'production'
      action:
        run_playbook:
          name: deploy_app.yml
          vars:
            version: "{{ event.payload.version }}"

Writing Rulebooks: The Brain of the Operation

Rulebooks define the logic: when an event arrives, what condition must be true, and what action to take. Conditions can check event fields, nested values, or even run Jinja2 expressions. Actions typically run a playbook, but can also run a module directly or send a notification. Keep rulebooks simple — one rule per logical action. Complex conditions lead to debugging nightmares.

# io.thecodeforge — DevOps tutorial

- name: Production incident response
  hosts: localhost
  sources:
    - ansible.eda.alertmanager:
        host: 0.0.0.0
        port: 5000
  rules:
    - name: Restart high-load service
      condition: event.alert.labels.severity == 'critical' and event.alert.labels.alertname == 'HighCPU'
      action:
        run_playbook:
          name: restart_service.yml
          vars:
            service: "{{ event.alert.labels.instance }}"
    - name: Notify on warning
      condition: event.alert.labels.severity == 'warning'
      action:
        run_module:
          name: ansible.builtin.debug
          args:
            msg: "Warning alert: {{ event.alert.annotations.summary }}"

Running the EDA Controller in Production

The EDA controller is a long-running process that listens for events. In production, run it as a systemd service or in a container with restart policies. Never run it as a background job in a terminal — it will die when you log out. Use a dedicated user with minimal permissions. Monitor the controller itself: if it goes down, your automation goes deaf.

# io.thecodeforge — DevOps tutorial

# Create systemd service for EDA controller
sudo cat > /etc/systemd/system/eda-controller.service << 'EOF'
[Unit]
Description=Event-Driven Ansible Controller
After=network.target

[Service]
Type=simple
User=ansible-eda
ExecStart=/usr/bin/ansible-rulebook --rulebook /etc/eda/rules.yml --controller
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable eda-controller
sudo systemctl start eda-controller

# Check logs
journalctl -u eda-controller -f

Handling Event Deduplication and Idempotency

Events can arrive multiple times (e.g., retries from monitoring). Your playbooks must be idempotent — running them twice should have the same effect as running them once. Use Ansible's state parameters and creates/removes to ensure idempotency. For deduplication, track event IDs in a Redis cache and skip already-processed events.

# io.thecodeforge — DevOps tutorial

# Playbook that checks if an event was already processed
- name: Restart service with dedup
  hosts: all
  vars:
    event_id: "{{ lookup('env', 'EVENT_ID') }}"
  tasks:
    - name: Check if event already processed
      ansible.builtin.slurp:
        src: "/var/log/eda/processed/{{ event_id }}"
      register: processed
      failed_when: false
    - name: Skip if already done
      ansible.builtin.meta: end_play
      when: processed.content is defined
    - name: Restart the service
      ansible.builtin.service:
        name: nginx
        state: restarted
    - name: Mark event as processed
      ansible.builtin.file:
        path: "/var/log/eda/processed/{{ event_id }}"
        state: touch

Scaling Event-Driven Ansible Horizontally

A single EDA controller can handle hundreds of events per second. Beyond that, run multiple controllers behind a load balancer, each with the same rulebook. Use a shared event source (like Kafka or RabbitMQ) that partitions events by a key (e.g., hostname) to ensure related events go to the same controller. This avoids race conditions on the same host.

# io.thecodeforge — DevOps tutorial

- name: Kafka event source for scaled setup
  hosts: localhost
  sources:
    - ansible.eda.kafka:
        host: kafka-broker:9092
        topic: ansible-events
        group_id: eda-group-1
        partition: 0
  rules:
    - name: Handle host-specific event
      condition: event.host == inventory_hostname
      action:
        run_playbook:
          name: handle_event.yml

When Not to Use Event-Driven Ansible

Event-Driven Ansible is overkill for simple scheduled tasks like daily backups or log rotation. Stick with cron for those. Also avoid it for high-frequency events (thousands per second) — the rulebook evaluation overhead adds latency. Use a stream processor like Flink for that. Finally, don't use it for critical safety systems (e.g., emergency shutdown) where you need guaranteed delivery and ordering — use a dedicated industrial controller.