Ansible AWS Automation: Production Patterns & Gotchas with amazon.aws Collection

I still remember the 3 AM wake-up call. Our production deployment had been running smoothly for months, but that night, a seemingly innocuous change to an Ansible playbook caused a 45-minute outage. The root cause? I had used the deprecated ec2 module instead of ec2_instance, and the module didn't handle idempotency correctly — it terminated all existing instances and created new ones, thinking they were 'extra'. That incident taught me the hard way that Ansible AWS automation requires deep understanding of module behavior, API consistency, and cloud state management.

Historically, Ansible's AWS support started with basic modules like ec2 and s3, which were monolithic and often inconsistent. The community developed workarounds, but the real game-changer was the amazon.aws collection (introduced in Ansible 2.9, now the standard). This collection provides focused, idempotent modules like ec2_instance, s3_bucket, iam_role, and rds_instance, designed to work with the AWS API's eventual consistency model.

In this article, I'll share production patterns I've developed over years of managing thousands of AWS resources with Ansible. We'll cover the essential modules from the amazon.aws collection, dynamic inventory with the aws_ec2 plugin, handling idempotency and eventual consistency, and securing secrets with AWS SSM Parameter Store. Every code example is battle-tested in production environments.

By the end, you'll have a practical playbook (pun intended) for building robust, scalable AWS automation with Ansible that won't cause 3 AM phone calls.

Setting Up the amazon.aws Collection for Production

The first step is ensuring you have the correct collection version. The amazon.aws collection is the replacement for the deprecated community.aws collection. Install it with:

``bash ansible-galaxy collection install amazon.aws:==5.0.0 ``

Pin the version in requirements.yml:

``yaml --- collections: - name: amazon.aws version: '>=5.0.0,<6.0.0' ``

Then run ansible-galaxy collection install -r requirements.yml. The collection requires boto3 and botocore >= 1.21.0. On the control node:

``bash pip install 'boto3>=1.21.0' 'botocore>=1.24.0' ``

For authentication, use IAM instance profiles on EC2 or environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY). In production, avoid hardcoding secrets; use Ansible Vault or SSM Parameter Store (covered later).

A typical group_vars/all.yml:

``yaml --- ansible_aws_region: us-east-1 ansible_aws_retry_max_attempts: 10 ansible_aws_retry_delay: 5 ``

These variables control retry behavior for all modules. The default retries are often insufficient for eventual consistency.

---
collections:
  - name: amazon.aws
    version: '>=6.0.0,<7.0.0'
  - name: community.aws
    version: '>=5.0.0,<6.0.0'

Managing EC2 Instances with ec2_instance Module

The ec2_instance module is the modern way to manage EC2 instances. It supports exact_count for idempotent instance management. Here's a production playbook snippet:

``yaml - name: Launch web servers amazon.aws.ec2_instance: name: "web-{{ item }}" instance_type: t3.medium image_id: ami-0abcdef1234567890 key_name: my-key security_group: "sg-xxxx" vpc_subnet_id: "subnet-xxxx" exact_count: 3 instance_role: Name: my-instance-profile network: assign_public_ip: true tags: Environment: production Role: web wait: yes wait_timeout: 600 loop: "{{ range(1, 4) | list }}" ``

Gotcha: The exact_count parameter works by filtering instances based on name tag and other filters you provide. If you don't set name, it may count unrelated instances. Always set name and tags to scope the count.

For updating instances (e.g., change instance type), use state: running and modify parameters. However, not all attributes are updatable in place; some require replacement. Use instance_ids to target specific instances for operations like stop/start.

---
- name: Launch EC2 instance
  amazon.aws.ec2_instance:
    name: "prod-web-{{ env }}"
    instance_type: t3.medium
    image_id: ami-0abcdef1234567890
    key_name: prod-key
    vpc_subnet_id: subnet-12345678
    security_groups:
      - sg-12345678
    network:
      assign_public_ip: true
    tags:
      Environment: "{{ env }}"
      Role: web
    state: running
    wait: yes
  register: ec2_result

Creating S3 Buckets and Objects with Idempotency

The s3_bucket and s3_object modules manage S3 resources. s3_bucket is idempotent by default: if the bucket exists and is owned by you, it reports ok. If it exists but is owned by another account, it fails with BucketAlreadyExists. For production:

``yaml - name: Create application bucket amazon.aws.s3_bucket: name: my-app-bucket state: present region: "{{ ansible_aws_region }}" permission: private versioning: yes tags: Environment: production ``

• permission: Default is private. Avoid public-read or public-read-write without explicit need.
• versioning: Enable for critical data.

For s3_object:

``yaml - name: Upload configuration file amazon.aws.s3_object: bucket: my-app-bucket object: /config/app.conf src: /local/path/app.conf mode: put permission: bucket-owner-full-control ``

Gotcha: The s3_bucket module does not manage bucket policies. Use a separate task with aws_s3_bucket_policy or iam_policy.

Idempotency for objects: The s3_object module with mode: put will upload the file every time unless you use force: false (default). To avoid unnecessary uploads, use overwrite: different (new in amazon.aws 5.0.0) which compares MD5 checksums:

``yaml - name: Upload config only if changed amazon.aws.s3_object: bucket: my-app-bucket object: /config/app.conf src: /local/path/app.conf mode: put overwrite: different ``

---
- name: Create S3 bucket with versioning
  amazon.aws.s3_bucket:
    name: "my-app-bucket-{{ env }}"
    state: present
    versioning: yes
    tags:
      Environment: "{{ env }}"
  register: bucket_result

- name: Upload object
  amazon.aws.s3_object:
    bucket: "my-app-bucket-{{ env }}"
    object: /config/app.conf
    content: "{{ lookup('file', 'app.conf') }}"
    overwrite: different
  register: upload_result

Creating IAM Roles and Instance Profiles

IAM role management is critical for security. The iam_role module creates roles and attaches policies. Production example:

``yaml - name: Create EC2 service role amazon.aws.iam_role: name: ec2-service-role assume_role_policy_document: "{{ lookup('file', 'assume-role-policy.json') }}" managed_policies: - arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess - arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy state: present create_instance_profile: yes ``

assume_role_policy_document must be a valid JSON string. Use lookup('file', ...) to load from file. Example assume-role-policy.json:

``json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "ec2.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } ``

Gotcha: The iam_role module is idempotent only if the assume_role_policy_document is exactly the same. If you change the file, the module updates the role. However, managed_policies are additive — if you remove a policy from the list, the module does NOT detach it. To manage policies precisely, use iam_policy module or set managed_policies: [] and manage separately.

For instance profiles, create_instance_profile: yes creates a profile with the same name as the role. To attach the profile to an EC2 instance, use ec2_instance with instance_role parameter.

---
- name: Create IAM role for EC2
  amazon.aws.iam_role:
    name: "ec2-app-role-{{ env }}"
    assume_role_policy_document:
      Version: '2012-10-17'
      Statement:
        - Effect: Allow
          Principal:
            Service: ec2.amazonaws.com
          Action: sts:AssumeRole
    managed_policies:
      - arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess
    state: present
  register: role_result

- name: Create instance profile
  amazon.aws.iam_instance_profile:
    name: "ec2-app-profile-{{ env }}"
    roles:
      - "ec2-app-role-{{ env }}"
    state: present

Provisioning RDS Instances with rds_instance

The rds_instance module manages RDS databases. Production example:

``yaml - name: Create PostgreSQL RDS instance amazon.aws.rds_instance: db_instance_identifier: mydb engine: postgres engine_version: "14.6" db_instance_class: db.t3.medium allocated_storage: 100 storage_type: gp3 master_username: "{{ db_master_username }}" master_user_password: "{{ db_master_password }}" vpc_security_group_ids: - sg-xxxx db_subnet_group_name: my-db-subnet-group publicly_accessible: no storage_encrypted: yes backup_retention_period: 7 skip_final_snapshot: no final_snapshot_identifier: mydb-final-{{ ansible_date_time.epoch }} wait: yes wait_timeout: 1200 state: present ``

Gotcha: The module does not support changing master_username after creation. If you need to change, you must delete and recreate.

Idempotency: The module checks for an existing instance with the same db_instance_identifier. If found, it compares parameters and updates if necessary. Not all parameters are updatable in place; some require replacement.

---
- name: Provision RDS MySQL instance
  amazon.aws.rds_instance:
    id: "prod-db-{{ env }}"
    engine: mysql
    engine_version: '8.0'
    db_instance_class: db.t3.small
    allocated_storage: 20
    master_username: admin
    master_user_password: "{{ vault_db_password }}"
    db_name: myapp
    vpc_security_group_ids:
      - sg-12345678
    db_subnet_group_name: "prod-subnet-group"
    publicly_accessible: no
    skip_final_snapshot: yes
    state: present
    wait: yes
  register: rds_result

Building VPC Networks with aws_vpc and Subnet Modules

VPC modules allow you to define network infrastructure as code. Production example:

```yaml - name: Create VPC amazon.aws.ec2_vpc_net: name: my-vpc cidr_block: 10.0.0.0/16 tags: Environment: production state: present region: "{{ ansible_aws_region }}"

• name: Create public subnets
• amazon.aws.ec2_vpc_subnet:
• vpc_id: "{{ vpc_result.vpc.id }}"
• cidr: "10.0.{{ item }}.0/24"
• az: "{{ ansible_aws_region }}{{ item }}"
• tags:
• Name: "public-{{ item }}"
• Tier: public
• state: present
• loop:
• - a
• - b
• - c
• register: subnet_results
• ```

Idempotency: These modules use tags and cidr_block to identify existing resources. If you change the CIDR, it creates a new resource (the old one is not deleted). To delete, set state: absent.

Gotcha: The ec2_vpc_subnet module requires vpc_id or vpc_name. Using vpc_name is convenient but can be ambiguous if multiple VPCs have the same name. Prefer vpc_id.

For internet gateway and route tables:

```yaml - name: Create Internet Gateway amazon.aws.ec2_vpc_igw: vpc_id: "{{ vpc_result.vpc.id }}" tags: Name: my-igw state: present

• name: Create public route table
• amazon.aws.ec2_vpc_route_table:
• vpc_id: "{{ vpc_result.vpc.id }}"
• tags:
• Name: public-rt
• subnets:
• - "{{ subnet_results.results[0].subnet.id }}"
• - "{{ subnet_results.results[1].subnet.id }}"
• - "{{ subnet_results.results[2].subnet.id }}"
• routes:
• - dest: 0.0.0.0/0
• gateway_id: "{{ igw_result.gateway_id }}"
• state: present
• ```

Production insight: Use ec2_vpc_route_table with subnets list to associate subnets. The module is idempotent: if routes and associations match, it does nothing.

---
- name: Create VPC
  amazon.aws.ec2_vpc_net:
    name: "prod-vpc-{{ env }}"
    cidr_block: 10.0.0.0/16
    tags:
      Environment: "{{ env }}"
    state: present
  register: vpc_result

- name: Create subnet
  amazon.aws.ec2_vpc_subnet:
    vpc_id: "{{ vpc_result.vpc.id }}"
    cidr: 10.0.1.0/24
    az: "{{ aws_region }}a"
    tags:
      Name: "prod-subnet-{{ env }}"
    state: present
  register: subnet_result

Using Dynamic Inventory with AWS EC2 Plugin

The aws_ec2 inventory plugin dynamically builds inventory from AWS EC2 instances. Configure it in inventory/aws_ec2.yml:

``yaml plugin: amazon.aws.aws_ec2 regions: - us-east-1 - us-west-2 filters: tag:Environment: production instance-state-name: running hostnames: - tag:Name - dns-name - private-dns-name compose: ansible_host: public_ip_address keyed_groups: - key: tags.Role prefix: role - key: placement.region prefix: aws_region cache: yes cache_plugin: jsonfile cache_timeout: 3600 ``

Run the playbook with:

``bash ansible-playbook -i inventory/aws_ec2.yml playbook.yml ``

Gotcha: The plugin caches inventory. If you create new instances, they won't appear until cache expires. Force refresh with --flush-cache.

Production pattern: Use cache: yes but set a low cache_timeout (e.g., 300) during deployments, then increase for normal operations.

plugin: amazon.aws.aws_ec2
regions:
  - us-east-1
filters:
  tag:Environment: "{{ env }}"
  instance-state-name: running
keyed_groups:
  - key: tags.Role
    prefix: role
hostnames:
  - dns-name
compose:
  ansible_host: private_ip_address

Ensuring Idempotency in Cloud Modules

Idempotency means running a playbook multiple times produces the same result. AWS modules in amazon.aws are designed to be idempotent, but there are pitfalls.

Pattern 1: Use state: present with unique identifiers. For example, ec2_instance with name and tags uniquely identifies instances. Without name, the module may create duplicates.

Pattern 2: Use exact_count for EC2. This ensures exactly N instances exist. Without it, each run creates a new instance.

Pattern 3: Use force: false (default) on s3_bucket to avoid recreating.

Pattern 4: For IAM roles, the assume_role_policy_document must match exactly. If you use a template that changes every run (e.g., with timestamps), the role will be updated every time. Avoid dynamic content in policy documents.

Pattern 5: Use register and when to skip tasks if resource already exists. For example:

```yaml - name: Check if bucket exists amazon.aws.aws_s3_bucket_info: name: my-bucket register: bucket_info ignore_errors: yes

• name: Create bucket if not exists
• amazon.aws.s3_bucket:
• name: my-bucket
• state: present
• when: bucket_info is failed
• ```

This pattern is useful for modules that are not fully idempotent (e.g., some community modules).

Gotcha: Some modules like ec2_instance with exact_count can be slow because they query all instances matching filters. Use specific filters to limit scope.

---
- name: Ensure security group exists
  amazon.aws.ec2_security_group:
    name: "web-sg-{{ env }}"
    description: Security group for web servers
    vpc_id: vpc-12345678
    rules:
      - proto: tcp
        ports:
          - 80
          - 443
        cidr_ip: 0.0.0.0/0
    rules_egress:
      - proto: all
        cidr_ip: 0.0.0.0/0
    state: present
    purge_rules: yes
    purge_rules_egress: yes
  register: sg_result

Handling Eventual Consistency with Retries

AWS APIs are eventually consistent — after creating a resource, it may not be immediately visible in other APIs. This causes failures in subsequent tasks. The amazon.aws collection provides retries and delay parameters.

Global retry settings: Set in group_vars/all.yml:

``yaml ansible_aws_retry_max_attempts: 10 ansible_aws_retry_delay: 5 ``

Per-task retries: Override with module parameters:

``yaml - name: Wait for EC2 instance to be ready amazon.aws.ec2_instance_info: instance_ids: - i-xxxx region: "{{ ansible_aws_region }}" register: instance_info retries: 15 delay: 10 until: instance_info.instances[0].state.name == "running" ``

Gotcha: The until condition must be a boolean expression. Use | default('') to handle missing keys.

Production pattern: Use a custom retry wrapper:

``yaml - name: Retry until resource is found amazon.aws.ec2_vpc_net_info: filters: "tag:Name": my-vpc register: vpc_info retries: 10 delay: 5 until: vpc_info.vpcs | length > 0 ``

---
- name: Wait for IAM role propagation
  amazon.aws.iam_role_info:
    name: "ec2-app-role-{{ env }}"
  register: role_info
  until: role_info.iam_roles | length > 0
  retries: 10
  delay: 5
  ignore_errors: yes

- name: Attach policy after propagation
  amazon.aws.iam_policy:
    iam_type: role
    iam_name: "ec2-app-role-{{ env }}"
    policy_name: S3ReadOnly
    policy_document:
      Version: '2012-10-17'
      Statement:
        - Effect: Allow
          Action: s3:GetObject
          Resource: '*'
    state: present

Storing Secrets with AWS SSM Parameter Store

Never hardcode secrets in playbooks. Use AWS SSM Parameter Store with the aws_ssm_parameter module to manage parameters, and lookup to retrieve them securely.

Storing a secret:

``yaml - name: Store database password in SSM amazon.aws.aws_ssm_parameter: name: /myapp/dbpassword value: "{{ db_password }}" type: SecureString overwrite: yes region: "{{ ansible_aws_region }}" no_log: true ``

no_log: true prevents the value from being logged.

Retrieving a secret in a playbook:

``yaml - name: Get database password from SSM set_fact: db_password: "{{ lookup('aws_ssm_parameter', '/myapp/dbpassword', decrypt=True, region=ansible_aws_region) }}" no_log: true ``

Then use {{ db_password }} in subsequent tasks.

Gotcha: The lookup plugin returns the value as a string. If the parameter is SecureString, you must set decrypt=True and have permission to decrypt.

Production pattern: Use environment-specific paths:

``yaml - name: Get environment-specific secret set_fact: db_password: "{{ lookup('aws_ssm_parameter', '/myapp/' + env + '/dbpassword', decrypt=True, region=ansible_aws_region) }}" ``

Permissions: The IAM role executing Ansible must have ssm:GetParameter and kms:Decrypt (if using KMS) permissions.

Alternative: Use Ansible Vault, but SSM is better for centralized secret management across teams.

---
- name: Store secret in SSM
  amazon.aws.ssm_parameter:
    name: "/prod/app/db_password"
    description: Database password
    type: SecureString
    value: "{{ vault_db_password }}"
    overwrite: yes
    state: present
  register: ssm_result

- name: Retrieve secret
  amazon.aws.ssm_parameter:
    name: "/prod/app/db_password"
    state: present
    with_decryption: yes
  register: db_password_retrieved
  no_log: true

Advanced: Combining Modules for Multi-Tier Deployments

Production applications often require multiple AWS resources. Here's a playbook that creates a VPC, subnets, security groups, RDS, and EC2 instances in a coordinated way.

```yaml --- - name: Provision multi-tier application hosts: localhost connection: local gather_facts: no vars: vpc_cidr: 10.0.0.0/16 public_subnets: - cidr: 10.0.1.0/24 az: "{{ ansible_aws_region }}a" - cidr: 10.0.2.0/24 az: "{{ ansible_aws_region }}b" private_subnets: - cidr: 10.0.10.0/24 az: "{{ ansible_aws_region }}a" - cidr: 10.0.11.0/24 az: "{{ ansible_aws_region }}b" tasks: - name: Create VPC amazon.aws.ec2_vpc_net: name: myapp-vpc cidr_block: "{{ vpc_cidr }}" tags: Environment: production state: present register: vpc

• name: Create subnets
• amazon.aws.ec2_vpc_subnet:
• vpc_id: "{{ vpc.vpc.id }}"
• cidr: "{{ item.cidr }}"
• az: "{{ item.az }}"
• tags:
• Name: "{{ item.name }}"
• state: present
• loop:
• - { cidr: "{{ public_subnets[0].cidr }}", az: "{{ public_subnets[0].az }}", name: "public-a" }
• - { cidr: "{{ public_subnets[1].cidr }}", az: "{{ public_subnets[1].az }}", name: "public-b" }
• - { cidr: "{{ private_subnets[0].cidr }}", az: "{{ private_subnets[0].az }}", name: "private-a" }
• - { cidr: "{{ private_subnets[1].cidr }}", az: "{{ private_subnets[1].az }}", name: "private-b" }
• register: subnets

• name: Create security group for web
• amazon.aws.ec2_security_group:
• name: web-sg
• description: Security group for web servers
• vpc_id: "{{ vpc.vpc.id }}"
• rules:
• - proto: tcp
• ports: 80
• cidr_ip: 0.0.0.0/0
• - proto: tcp
• ports: 443
• cidr_ip: 0.0.0.0/0
• tags:
• Name: web-sg
• state: present

• name: Create RDS subnet group
• amazon.aws.rds_subnet_group:
• name: myapp-db-subnet
• description: Subnet group for RDS
• subnet_ids:
• - "{{ subnets.results[2].subnet.id }}"
• - "{{ subnets.results[3].subnet.id }}"
• state: present

• name: Create RDS instance
• amazon.aws.rds_instance:
• db_instance_identifier: myapp-db
• engine: postgres
• engine_version: "14.6"
• db_instance_class: db.t3.medium
• allocated_storage: 100
• master_username: "{{ db_user }}"
• master_user_password: "{{ db_password }}"
• db_subnet_group_name: myapp-db-subnet
• vpc_security_group_ids:
• - "{{ sg_result.group_id }}"
• wait: yes
• wait_timeout: 1200
• state: present

• name: Launch EC2 instances
• amazon.aws.ec2_instance:
• name: web-{{ item }}
• instance_type: t3.medium
• image_id: ami-0abcdef1234567890
• vpc_subnet_id: "{{ subnets.results[0].subnet.id }}"
• security_group: web-sg
• exact_count: 2
• tags:
• Environment: production
• Role: web
• wait: yes
• loop: "{{ range(1, 3) | list }}"
• ```

---
- name: Deploy multi-tier app
  hosts: localhost
  gather_facts: no
  tasks:
    - name: Create VPC
      amazon.aws.ec2_vpc_net:
        name: "multi-tier-vpc"
        cidr_block: 10.0.0.0/16
      register: vpc
    - name: Create public subnet
      amazon.aws.ec2_vpc_subnet:
        vpc_id: "{{ vpc.vpc.id }}"
        cidr: 10.0.1.0/24
        az: "{{ aws_region }}a"
        tags:
          Tier: public
    - name: Create private subnet
      amazon.aws.ec2_vpc_subnet:
        vpc_id: "{{ vpc.vpc.id }}"
        cidr: 10.0.2.0/24
        az: "{{ aws_region }}b"
        tags:
          Tier: private
    - name: Launch web instance
      amazon.aws.ec2_instance:
        name: web
        instance_type: t3.micro
        image_id: ami-0abcdef1234567890
        vpc_subnet_id: "{{ public_subnet.subnet.id }}"
        state: running

Testing and Validating AWS Playbooks Locally

Testing AWS playbooks without affecting real infrastructure is crucial. Use --check mode and --diff to preview changes.

``bash ansible-playbook -i inventory/aws_ec2.yml playbook.yml --check --diff ``

Limitations: --check mode does not actually call AWS APIs for creation tasks; it simulates. Some modules return 'changed' even in check mode. To validate syntax:

``bash ansible-playbook playbook.yml --syntax-check ``

Unit testing with Molecule: Use the molecule tool with the ec2 driver to spin up temporary instances for testing. Example molecule.yml:

``yaml --- dependency: name: galaxy driver: name: ec2 region: us-east-1 instance_type: t2.micro image_id: ami-0abcdef1234567890 vpc_subnet_id: subnet-xxxx security_group: sg-xxxx platforms: - name: instance groups: - web provisioner: name: ansible inventory: group_vars: all: ansible_aws_region: us-east-1 verifier: name: ansible ``

Run molecule test to create, test, and destroy instances.

Gotcha: Molecule with EC2 driver incurs costs. Use --destroy=never during development to keep instances for debugging.

Production pattern: Use a separate AWS account for testing. Implement check_mode: yes in playbooks with conditional logic to skip destructive tasks.

``yaml - name: Create EC2 instance (check mode safe) amazon.aws.ec2_instance: ... when: not ansible_check_mode ``

#!/bin/bash
# Test playbook locally with moto
pip install moto[server] ansible
moto_server ec2 &>/dev/null &
MOTO_PID=$!
sleep 2
export AWS_ACCESS_KEY_ID=fake
export AWS_SECRET_ACCESS_KEY=fake
export AWS_DEFAULT_REGION=us-east-1
ansible-playbook -i localhost, site.yml -e "env=test" --connection=local
kill $MOTO_PID