Ansible Dynamic Inventory: AWS EC2 Plugin Gotchas and Production Patterns

I still remember the Monday morning when our deployment pipeline failed for the second week in a row. The playbook ran fine on Friday, but on Monday it couldn't find any EC2 instances. The error was cryptic: SKIPPED: No hosts matched. After an hour of head-scratching, I discovered that the legacy dynamic inventory script we were using had a hardcoded AWS region that no longer existed. That was the day I decided to switch to inventory plugins.

Historically, Ansible dynamic inventory was done via executable scripts (Python, shell) that output JSON. They worked, but were brittle: no caching, no native error handling, and each script was a snowflake. The Ansible team introduced inventory plugins in Ansible 2.4, and they became the recommended approach by 2.9. Plugins are faster, support caching out of the box, and integrate deeply with Ansible's group and variable logic.

This article covers everything you need to use dynamic inventory in production: from the AWS EC2 plugin configuration to tagging strategies, caching, and debugging. We'll also touch on GCP and Azure plugins. I'll share real incidents and patterns from running Ansible at scale across hundreds of instances.

Inventory Plugins vs Scripts: Why Plugins Win in Production

Legacy dynamic inventory scripts are standalone executables (Python, shell, etc.) that output JSON to stdout. They work, but they have no standard way to handle caching, errors, or configuration. Ansible 2.4 introduced inventory plugins, which are Python modules that integrate with Ansible's internals. They support caching, configuration via YAML, and advanced features like keyed_groups and compose.

To use a plugin, create a YAML file (e.g., aws_ec2.yml) with: ``yaml plugin: amazon.aws.aws_ec2 regions: - us-east-1 - us-west-2 filters: instance-state-name: running keyed_groups: - prefix: tag key: tags.Name compose: ansible_host: public_ip_address ` Then reference it with -i aws_ec2.yml`.

Migration: If you have legacy scripts, wrap them in a plugin using ansible.builtin.script plugin? No, that's not a plugin. Instead, rewrite the logic as a custom plugin or use the constructed plugin to add groups. But honestly, just use the cloud-specific plugin.

Configuring amazon.aws.aws_ec2 Plugin for Production

The amazon.aws.aws_ec2 plugin is the gold standard for AWS dynamic inventory. Install the collection: ansible-galaxy collection install amazon.aws. Then create a YAML file, typically named aws_ec2.yml.

Production example: ``yaml plugin: amazon.aws.aws_ec2 regions: - us-east-1 - eu-west-1 filters: instance-state-name: running tag:Environment: production hostnames: - dns-name - private-dns-name keyed_groups: - prefix: tag key: tags.Name - prefix: env key: tags.Environment compose: ansible_host: public_ip_address cache_plugin: jsonfile cache_timeout: 300 strict: false ``

Gotcha: The hostnames list order matters. If you use dns-name first, but the instance has no public DNS, it falls back to the next. Always include private-dns-name for VPC instances.

Authentication: The plugin uses boto3. Ensure AWS credentials are available via environment variables, IAM role, or ~/.aws/credentials. Use aws sts get-caller-identity to verify.

Tagging Strategy for Dynamic Groups with keyed_groups

Tags are the backbone of dynamic grouping. With keyed_groups, you can automatically create Ansible groups based on EC2 tags, instance attributes, or any metadata.

Syntax: ``yaml keyed_groups: - prefix: tag key: tags.Environment separator: '' ` This creates groups like tag_production, tag_staging. The prefix is prepended, then the tag value. Use separator: ''` to avoid double underscores.

Production tagging strategy: Use consistent tag keys across all instances. Common tags: - Name: instance name (often unique) - Environment: production, staging, development - Role: web, database, cache - Tier: frontend, backend - Project: project name

Then configure groups: ``yaml keyed_groups: - prefix: env key: tags.Environment separator: '' - prefix: role key: tags.Role - prefix: tag key: tags.Name parent_group: all ``

Nested groups: You can nest groups by combining tags. Use keyed_groups with parent_group to create hierarchy. Example: instances with tags.Environment=prod and tags.Role=web could be in groups env_prod and role_web. To group all prod web servers, use group: "{{ tags.Environment }}_{{ tags.Role }}" via the constructed plugin, but that's more complex.

Gotcha: Tag values with spaces or special characters become invalid Ansible group names. Set strict: false to ignore them, or sanitize with compose.

Best practice: Use lowercase tag values and avoid spaces. E.g., Environment: production not Environment: Production.

Advanced Grouping with keyed_groups and compose

The keyed_groups directive can create groups from any key returned by the plugin, not just tags. For example, you can group by instance type, region, or VPC ID.

``yaml keyed_groups: - prefix: instance_type key: instance_type - prefix: region key: placement.region ``

But keyed_groups only creates groups based on exact values. For more complex logic, use compose to create custom variables, then group on those.

Example: Group instances by whether they have a public IP. ``yaml compose: has_public_ip: public_ip_address is defined keyed_groups: - key: has_public_ip prefix: public ` This creates groups public_True and public_False`.

Another pattern: Create groups based on multiple tags. Use Jinja2 expressions: ``yaml compose: env_role: "{{ tags.Environment | default('unknown') }}_{{ tags.Role | default('unknown') }}" keyed_groups: - key: env_role prefix: '' ` This creates groups like production_web`.

Gotcha: compose runs after keyed_groups? Actually, compose is evaluated before keyed_groups, so you can use composed variables in keyed_groups keys. But be careful with order: compose sets host variables, then keyed_groups uses them.

Performance: Complex compose expressions are evaluated for each host. For large inventories (thousands of hosts), keep expressions simple. Avoid expensive filters like regex_replace.

Caching Inventory to Avoid API Rate Limits

Without caching, each ansible-inventory call hits the AWS API. For large environments, this can trigger rate limits (e.g., RequestLimitExceeded). Caching stores the inventory locally and refreshes it periodically.

Configure caching in the plugin YAML: ``yaml cache_plugin: jsonfile cache_timeout: 300 cache_connection: ~/.ansible/tmp/inventory_cache ` Or set globally in ansible.cfg: `ini [inventory] cache_plugin = jsonfile cache_timeout = 300 cache_connection = ~/.ansible/tmp/inventory_cache ``

Cache plugins: jsonfile (default), redis, memcached, sqlite. For single-controller setups, jsonfile is fine. For multi-controller, use redis to share cache.

Cache invalidation: The cache is invalidated after cache_timeout seconds. To force refresh, delete the cache directory: rm -rf ~/.ansible/tmp/inventory_cache/. Or run with --flush-cache: ``bash ansible-inventory -i aws_ec2.yml --list --flush-cache ``

Production pattern: Set cache_timeout based on how often your infrastructure changes. For auto-scaling groups, set to 60 seconds. For static environments, 300 seconds is fine.

Gotcha: If using cache_plugin: jsonfile, ensure the cache directory is writable. Also, the cache file can become corrupt if multiple processes write simultaneously. Use redis for concurrent access.

Debugging cache: Check the cache file: ``bash cat ~/.ansible/tmp/inventory_cache/ansible_inventory_cache | jq '. | keys' ``

Testing Dynamic Inventory with ansible-inventory --list

The ansible-inventory command is your best friend for debugging. Use --list to output the full inventory as JSON.

``bash ansible-inventory -i aws_ec2.yml --list | jq '."tag_Name_web"' ``

Testing specific groups: ``bash ansible-inventory -i aws_ec2.yml --list | jq '.env_production' ``

Testing host variables: ``bash ansible-inventory -i aws_ec2.yml --list | jq '._meta.hostvars["i-12345"]' ``

Using with a playbook: ``bash ansible-playbook -i aws_ec2.yml site.yml --list-hosts ``

Gotcha: The --list output includes _meta with hostvars. Use --export to omit it if you want a pure inventory JSON.

Automated testing: In CI, run: ``bash ansible-inventory -i aws_ec2.yml --list > /dev/null && echo "Inventory valid" ``

Pro tip: Pipe through jq to extract specific fields. For example, list all hostnames: ``bash ansible-inventory -i aws_ec2.yml --list | jq '._meta.hostvars | keys' ``

GCP Inventory Plugin: gcp_compute

For Google Cloud Platform, use the gcp_compute plugin from the google.cloud collection. Install: ansible-galaxy collection install google.cloud.

Configuration example (`gcp_compute.yml`): ``yaml plugin: gcp_compute projects: - my-project zones: - us-central1-a - us-east1-b filters: - status = RUNNING keyed_groups: - prefix: gcp key: labels.environment compose: ansible_host: networkInterfaces[0].accessConfigs[0].natIP hostnames: - name - networkInterfaces[0].networkIP cache_plugin: jsonfile cache_timeout: 300 ``

Authentication: Use application default credentials or service account JSON file via GCP_SERVICE_ACCOUNT_FILE environment variable.

Gotcha: The gcp_compute plugin does not support hostnames with fallback like AWS. You must specify a list; the first match is used. If the instance has no public IP, natIP will be undefined, and the hostname will be empty.

Production pattern: Use compose to set ansible_host to the internal IP if public IP is missing: ``yaml compose: ansible_host: "{{ networkInterfaces[0].accessConfigs[0].natIP | default(networkInterfaces[0].networkIP) }}" ``

Azure Inventory Plugin: azure_rm

For Microsoft Azure, use the azure_rm plugin from the azure.azcollection collection. Install: ansible-galaxy collection install azure.azcollection.

Configuration example (`azure_rm.yml`): ``yaml plugin: azure_rm include_vm_resource_groups: - my-resource-group - another-rg auth_source: auto keyed_groups: - prefix: azure key: tags.environment - prefix: location key: location compose: ansible_host: public_ip_address | default(private_ip_address) hostnames: - name - private_ip_address cache_plugin: jsonfile cache_timeout: 300 ``

Authentication: Use az login or service principal via AZURE_CLIENT_ID, AZURE_SECRET, AZURE_TENANT environment variables.

Gotcha: The azure_rm plugin can be slow for large subscriptions. Use include_vm_resource_groups to limit scope. Also, the plugin does not support hostnames fallback as elegantly; you may need to use compose.

Production pattern: Use tags to organize VMs. Example: ``yaml keyed_groups: - prefix: environment key: tags.Environment - prefix: role key: tags.Role ``

Performance: For large Azure environments, consider using azure_rm with cache_plugin: redis to share cache across controllers.

Combining Multiple Inventory Sources

In production, you often need hosts from multiple clouds or sources. Ansible supports multiple inventory sources by specifying a directory with multiple files, or by listing multiple -i flags.

Using a directory: Place all inventory YAML files in a directory (e.g., inventory/). Then run: ``bash ansible-playbook -i inventory/ site.yml `` Ansible will merge all inventories. Hosts with the same name are merged (variables from later sources override earlier ones).

Using multiple -i flags: ``bash ansible-playbook -i aws_ec2.yml -i gcp_compute.yml -i azure_rm.yml site.yml ``

Merging logic: Groups from all sources are combined. If a host appears in multiple sources, its variables are merged (last source wins). To avoid conflicts, ensure hostnames are unique across clouds (e.g., use instance ID or FQDN).

Using constructed plugin: The constructed plugin can add groups and variables based on existing inventory data. Example: ``yaml plugin: constructed strict: false keyed_groups: - prefix: cloud key: cloud_type compose: cloud_type: "{{ 'aws' if 'ec2' in group_names else 'gcp' }}" ` But this requires setting cloud_type` first.

Production pattern: Use separate inventory files per cloud, and a directory to combine them. Then use a playbook-level hosts: all to target all.

Custom Inventory Plugins: When and How

Sometimes the built-in plugins don't meet your needs (e.g., custom API, on-premise servers). You can write a custom inventory plugin. This is advanced but powerful.

Structure: A plugin is a Python module in a collection. It must inherit from ansible.plugins.inventory.BaseInventoryPlugin and implement parse(), verify_file(), and optionally get_option().

Minimal example: ```python from ansible.plugins.inventory import BaseInventoryPlugin

DOCUMENTATION = ''' name: my_custom plugin_type: inventory options: my_option: description: Example option required: true type: str '''

class InventoryModule(BaseInventoryPlugin): NAME = 'my_custom'

def verify_file(self, path): return path.endswith('.my.yml')

def parse(self, inventory, loader, path, cache=True): super().parse(inventory, loader, path, cache) self.set_options() # Fetch data from external source hosts = [{'name': 'host1', 'groups': ['web'], 'vars': {'ansible_host': '10.0.0.1'}}] for host in hosts: inventory.add_host(host['name']) for group in host['groups']: inventory.add_group(group) inventory.add_host_to_group(host['name'], group) for k, v in host['vars'].items(): inventory.set_variable(host['name'], k, v) ```

Otherwise, stick with existing plugins.

Common Pitfalls and How to Avoid Them

Here are the most common mistakes I've seen with dynamic inventory:

1. Not setting ansible_host Without compose: { ansible_host: public_ip_address }, Ansible uses the hostname as the connection address. If the hostname is not resolvable, SSH fails.

2. Over-filtering Using filters that are too restrictive (e.g., tag:Environment: production) but forgetting to tag new instances. Always include a fallback group like ungrouped.

3. Ignoring cache Not enabling caching leads to slow runs and API rate limits. Always set cache_plugin and cache_timeout.

4. Using deprecated inventory_script Legacy scripts still work but are deprecated. They lack caching and error handling. Migrate to plugins.

5. Not testing with --list Skipping ansible-inventory --list leads to surprises in production. Always test after changes.

6. Group name collisions If two inventory sources create groups with the same name, they merge. This can cause unexpected host membership.

7. Missing dependencies For AWS, install amazon.aws collection and boto3 and botocore. For GCP, install google.cloud collection and google-auth. For Azure, install azure.azcollection and azure-cli.

8. Not handling missing tags If a tag is missing, keyed_groups will fail (if strict: true) or skip. Use default filter in compose to provide defaults.

Production Deployment: Putting It All Together

Here's a production-grade setup for dynamic inventory:

Directory structure: `` inventory/ aws_ec2.yml gcp_compute.yml azure_rm.yml group_vars/ all.yml env_production.yml role_web.yml ``

ansible.cfg: ```ini [defaults] inventory = inventory/ host_key_checking = False

[inventory] cache_plugin = jsonfile cache_timeout = 300 cache_connection = ~/.ansible/tmp/inventory_cache ```

CI/CD pipeline: ```bash # Install collections ansible-galaxy collection install amazon.aws google.cloud azure.azcollection

# Clear cache and test ansible-inventory -i inventory/ --list --flush-cache > /dev/null

# Run playbook ansible-playbook -i inventory/ site.yml ```

Monitoring: Use ansible-inventory --list to export inventory to a monitoring system. For example, dump to JSON and push to Prometheus.