Ansible Azure Automation: Production-Grade Infrastructure as Code That Won't Fail at 3 AM

I've seen Ansible Azure automation fail spectacularly at 2 AM when a misconfigured service principal expired and a critical VM scale set couldn't be updated. The runbook errored silently, the team assumed success, and the production deployment went sideways. The root cause? A 30-second TTL on the cached token that nobody accounted for.

This isn't a tutorial on 'how to create a VM with Ansible.' You can find that on any blog. This is about making Ansible Azure automation production-hardened: handling API rate limits, managing state across hundreds of resources, and debugging failures when the Azure API returns a 429 or a cryptic AuthorizationFailed.

By the end of this, you'll be able to design Ansible playbooks that survive token expiry, throttle gracefully, and integrate with Azure DevOps pipelines without causing a stampede on the Azure Resource Manager API. You'll also know the exact commands to run when your automation breaks at 3 AM.

Why Ansible for Azure? The Problem with ARM Templates and CLI Scripts

Before Ansible, Azure automation meant ARM templates (JSON nightmares) or brittle bash scripts with az commands. ARM templates are declarative but lack idempotency checks—you can't easily say 'ensure this VM exists with these settings' without blowing away the old one. CLI scripts are imperative and fail halfway, leaving resources in inconsistent states.

Ansible solves this with its azure_rm_* modules that are idempotent: they compare the current state of the Azure resource against the desired state in your playbook and only make API calls if something changed. This is critical for production where you can't afford to recreate a database server just because you ran a playbook twice.

But idempotency isn't free. The modules must query Azure's API to get current state before deciding what to change. That means every playbook run incurs API calls just for 'check mode'—and Azure Resource Manager has rate limits (12,000 reads per hour per subscription). You'll hit those limits if you manage hundreds of resources naively.

# io.thecodeforge — DevOps tutorial
# Production playbook to ensure a VM exists with specific configuration
- name: Ensure production VM exists
  hosts: localhost
  connection: local
  tasks:
    - name: Create or update VM
      azure_rm_virtualmachine:
        resource_group: "prod-rg"
        name: "web-vm-01"
        vm_size: Standard_DS2_v2
        admin_username: azureuser
        ssh_password_enabled: false
        ssh_public_keys:
          - path: /home/azureuser/.ssh/authorized_keys
            key_data: "ssh-rsa AAAAB3NzaC1yc2E..."
        image:
          offer: UbuntuServer
          publisher: Canonical
          sku: '18.04-LTS'
          version: latest
        # Idempotent: if VM exists with same config, no API call
        state: present
      register: vm_output

    - name: Debug VM creation status
      debug:
        msg: "VM {{ vm_output.state.name }} is {{ 'created' if vm_output.changed else 'already exists' }}"

Authentication: Service Principals, Managed Identities, and Token Expiry

The most common production failure with Ansible Azure automation is authentication. You set up a service principal, store the secret in Ansible Vault, and it works for a week. Then the playbook fails with AuthenticationFailedError: The client secret expired.

Service principal secrets have a max lifetime of 2 years (or less if your org rotates them). If you're not rotating secrets, your automation will break. The fix is to use managed identities for Azure resources when running Ansible from an Azure VM or Azure DevOps agent. Managed identities auto-rotate credentials and eliminate secret management.

For local development or external runners, use Azure AD application with certificate-based authentication—certificates can be rotated via automation. Never hardcode secrets. Use Ansible Vault with a vault password stored in Azure Key Vault, retrieved at runtime via the azure_rm_keyvault_secret module.

Token expiry is another silent killer. The Azure SDK caches tokens for 30 seconds by default. If your playbook takes longer than that between tasks, the token may expire mid-playbook. The symptom is intermittent AuthorizationFailed errors. Mitigation: set AZURE_TOKEN_CACHE_EXPIRATION_SECONDS to 300 or use azure_rm_resource with api_version to force a fresh token.

# io.thecodeforge — DevOps tutorial
# Using managed identity for Azure authentication (runs on Azure VM or Azure DevOps agent)
- name: Deploy resources using managed identity
  hosts: localhost
  connection: local
  vars:
    # No client_id, secret, or tenant needed—managed identity handles it
    resource_group: "prod-rg"
  tasks:
    - name: Ensure resource group exists
      azure_rm_resourcegroup:
        name: "{{ resource_group }}"
        location: eastus
        state: present
      # Module automatically uses managed identity if environment has MSI_ENDPOINT

    - name: Create storage account
      azure_rm_storageaccount:
        resource_group: "{{ resource_group }}"
        name: "prodsa{{ 100 | random }}"
        type: Standard_LRS
        state: present

Idempotency and State Management: The Hidden API Cost

Ansible's azure_rm_* modules are idempotent, but they achieve this by making GET requests to Azure before every PUT/POST/DELETE. For a playbook that manages 100 resources, that's 100 GET calls just to check state—plus the actual mutation calls. At scale, this burns through your API rate limit.

Worse, some modules like azure_rm_virtualmachine don't support partial updates. If you change only the VM size, the module may still call CreateOrUpdate with the full VM configuration, which can trigger a reboot. That's not idempotent in practice—it's destructive.

Solution: Use azure_rm_resource module for fine-grained control. It lets you call specific Azure REST API endpoints with PUT, PATCH, or GET. For example, to update only the VM size without rebooting, use PATCH on the Microsoft.Compute/virtualMachines resource. This reduces API calls and avoids unintended side effects.

Another pattern: Use azure_rm_resource_facts to gather current state of all resources in a single API call (list operation), then compare locally. This cuts the number of GET calls from N to 1.

# io.thecodeforge — DevOps tutorial
# Update VM size without triggering full update (PATCH instead of PUT)
- name: Update VM size idempotently
  hosts: localhost
  connection: local
  vars:
    vm_name: "web-vm-01"
    resource_group: "prod-rg"
    new_size: "Standard_DS3_v2"
  tasks:
    - name: Get current VM state
      azure_rm_resource_facts:
        resource_group: "{{ resource_group }}"
        provider: compute
        resource_type: virtualmachines
        resource_name: "{{ vm_name }}"
        api_version: '2021-07-01'
      register: vm_facts

    - name: Update VM size only if different
      azure_rm_resource:
        resource_group: "{{ resource_group }}"
        provider: compute
        resource_type: virtualmachines
        resource_name: "{{ vm_name }}"
        api_version: '2021-07-01'
        method: PATCH
        body:
          properties:
            hardwareProfile:
              vmSize: "{{ new_size }}"
      when: vm_facts.response.properties.hardwareProfile.vmSize != new_size
      register: patch_result

    - name: Notify change
      debug:
        msg: "VM size updated to {{ new_size }}"
      when: patch_result.changed

Error Handling and Retry Logic for Azure API Failures

Azure APIs are reliable, but they do throttle (HTTP 429) and occasionally return 5xx errors. Ansible's default behavior is to fail immediately—which is fine for a lab, but in production you need retries with exponential backoff.

Ansible's until loop combined with retries and delay gives you control. But there's a gotcha: the until condition checks the result of the task, and if the task fails, the result variable may not be defined. You need to use failed_when: false to prevent the task from failing before the loop.

Also, not all errors are retryable. A 400 Bad Request won't become a 200 on retry. So check the HTTP status code before retrying. Use response.status_code from the module's return value.

Here's a pattern I use in production: retry on 429 and 5xx, fail fast on 4xx (except 429). Also, log the retry count to a file for monitoring.

# io.thecodeforge — DevOps tutorial
# Retry logic for Azure API calls with exponential backoff
- name: Create resource group with retry
  hosts: localhost
  connection: local
  vars:
    resource_group: "prod-rg"
  tasks:
    - name: Ensure resource group exists (with retry)
      azure_rm_resourcegroup:
        name: "{{ resource_group }}"
        location: eastus
        state: present
      register: result
      until: result is success or (result.response is defined and result.response.status_code not in [429, 500, 502, 503])
      retries: 5
      delay: "{{ 10 * (2 ** (ansible_loop.index - 1)) }}"  # Exponential backoff: 10, 20, 40, 80, 160
      failed_when: false  # Prevent immediate failure
      
    - name: Fail if still not successful after retries
      fail:
        msg: "Failed to create resource group after retries: {{ result.msg }}"
      when: result is failed

    - name: Log success
      debug:
        msg: "Resource group {{ resource_group }} created/verified after {{ result.attempts }} attempts"

Scaling Ansible for Large Azure Deployments: Throttling and Parallelism

Running Ansible against 500 Azure resources sequentially takes hours. But running them all in parallel will hit API rate limits and saturate your network. The sweet spot is to control concurrency with Ansible's serial and throttle directives.

serial controls how many hosts are processed at once in a play. For Azure, you're often running against localhost (since Ansible runs locally and calls Azure APIs), so serial doesn't help directly. Instead, use throttle on individual tasks to limit how many concurrent API calls a task makes.

But throttle is per-task, not global. If you have 10 tasks, each can have its own throttle, but the total API calls can still spike. Better approach: use azure_rm_resource with batch operations where possible. For example, to create 100 VMs, use a single ARM template deployment via azure_rm_deployment instead of 100 individual azure_rm_virtualmachine calls.

Another pattern: use include_tasks with loop and throttle to process resources in batches. This gives you fine-grained control over the API call rate.

# io.thecodeforge — DevOps tutorial
# Batch create VMs using ARM template deployment to reduce API calls
- name: Deploy multiple VMs via ARM template
  hosts: localhost
  connection: local
  vars:
    resource_group: "prod-rg"
    vm_count: 10
  tasks:
    - name: Generate VM names
      set_fact:
        vm_names: "{{ range(1, vm_count + 1) | map('regex_replace', '^', 'web-vm-') | list }}"

    - name: Deploy VMs using ARM template
      azure_rm_deployment:
        resource_group: "{{ resource_group }}"
        name: "batch-vm-deploy-{{ ansible_date_time.epoch }}"
        template:
          $schema: "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#"
          contentVersion: "1.0.0.0"
          parameters:
            vmNames:
              type: array
              defaultValue: "{{ vm_names }}"
          resources:
            - name: "[parameters('vmNames')[copyIndex()]]"
              type: Microsoft.Compute/virtualMachines
              apiVersion: '2021-07-01'
              location: eastus
              copy:
                name: vmCopy
                count: "{{ vm_count }}"
              properties:
                hardwareProfile:
                  vmSize: Standard_DS2_v2
                # ... other properties
        parameters:
          vmNames:
            value: "{{ vm_names }}"
        state: present
      register: deploy_result

Integrating with Azure DevOps Pipelines: Secrets and Dynamic Inventory

Running Ansible from Azure DevOps is common, but it introduces two problems: secret management and dynamic inventory. For secrets, use Azure Key Vault task to fetch secrets and pass them as environment variables. Never store service principal secrets in pipeline variables—they're not encrypted at rest in classic pipelines.

For dynamic inventory, use the azure_rm.py inventory script (now deprecated) or the azure_rm plugin. The plugin queries Azure for VMs and returns them as hosts. But beware: the plugin fetches all VMs in a subscription by default, which can be slow and expose unintended hosts. Use include_vm_resource_groups to limit scope.

Also, the plugin doesn't handle Azure VMSS instances well—it returns the scale set name, not individual instances. For VMSS, use azure_rm_virtualmachinescaleset module directly.

Here's a production pipeline snippet that uses managed identity (Azure DevOps agent running on an Azure VM) and dynamic inventory.

# io.thecodeforge — DevOps tutorial
# Azure DevOps pipeline for Ansible Azure automation
# Assumes agent is Azure VM with managed identity

trigger:
  branches:
    include:
      - main

pool:
  vmImage: 'ubuntu-latest'

variables:
  - group: 'Ansible-Azure-Variables'  # Contains ANSIBLE_HOST_KEY_CHECKING, etc.

steps:
  - task: UsePythonVersion@0
    inputs:
      versionSpec: '3.8'

  - script: |
      pip install ansible[azure]
    displayName: 'Install Ansible and Azure modules'

  - script: |
      ansible-playbook -i azure_rm.yaml deploy.yml
    displayName: 'Run Ansible playbook'
    env:
      AZURE_CLIENT_ID: $(AZURE_CLIENT_ID)  # From variable group
      AZURE_SECRET: $(AZURE_SECRET)        # From variable group (use Key Vault)
      AZURE_SUBSCRIPTION_ID: $(AZURE_SUBSCRIPTION_ID)
      AZURE_TENANT: $(AZURE_TENANT)

When NOT to Use Ansible for Azure Automation

Ansible is great for configuration management and simple provisioning, but it's not the right tool for every job. Avoid Ansible for:

• Stateful infrastructure with complex dependencies: If you need to manage a multi-tier app with rolling updates, health checks, and canary deployments, use Terraform or Bicep for provisioning and Kubernetes for orchestration.
• High-frequency updates: Ansible's pull-based model (SSH or local execution) doesn't scale for real-time resource changes. Use Azure Functions or Event Grid for event-driven automation.
• Teams that don't know YAML: If your team is more comfortable with Python or TypeScript, consider Pulumi or Azure SDK directly.
• Environments with strict compliance logging: Ansible's default logging is minimal. You'd need to implement custom logging and audit trails, which is easier with Azure Policy or Azure Blueprints.

My rule of thumb: Use Ansible when you need to 'bootstrap and configure'—install software, set configs, ensure a VM is in a known state. Use Terraform when you need to 'provision and version'—create resource groups, networks, and managed services. Use both together (Terraform for infra, Ansible for config) for a robust pipeline.