Ansible AWX & Tower: Production Automation Without the Fire Drills

You've automated your infrastructure with Ansible. Now you have 50 playbooks, 10 engineers, and a compliance officer breathing down your neck. Running playbooks from the CLI is a disaster waiting to happen — someone runs the wrong playbook against production at 3 PM on a Friday. AWX (or its enterprise sibling Tower) is the answer. It gives you a GUI, RBAC, job scheduling, and an audit trail. But it's not a magic bullet. I've seen teams deploy AWX and still have jobs fail because they didn't understand how the worker pool works. This guide covers the production setup, common pitfalls, and how to debug when things go sideways. By the end, you'll be able to deploy AWX, configure it for multi-team use, and troubleshoot job failures without panicking.

Why You Need AWX (And When You Don't)

If you're running Ansible from a single laptop, skip this. AWX shines when you have multiple teams, need audit trails, or want to schedule jobs. The problem it solves: chaos. Without AWX, you have playbooks scattered across repos, credentials in plain text, and no way to know who ran what. I've seen a junior engineer run a playbook that wiped a staging database because they had the wrong inventory selected. AWX gives you guardrails: RBAC, credential management, and job history. But it adds complexity. For a small team with 10 playbooks, plain Ansible with a CI/CD pipeline might be simpler. Don't over-engineer.

// io.thecodeforge — DevOps tutorial

# Install AWX on minikube for testing
minikube start --cpus=4 --memory=8192
kubectl create namespace awx
kubectl config set-context --current --namespace=awx

# Install AWX Operator (latest version)
kubectl apply -f https://raw.githubusercontent.com/ansible/awx-operator/release-2.5.0/deploy/awx-operator.yaml

# Wait for operator to be ready
kubectl wait --for=condition=available deployment/awx-operator --timeout=120s

# Create AWX instance
cat <<EOF | kubectl apply -f -
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx-demo
spec:
  service_type: nodeport
  ingress_type: none
  postgres_storage_class: standard
  postgres_storage_requirements:
    requests:
      storage: 8Gi
EOF

# Wait for deployment
kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=awx-demo --timeout=300s

# Get admin password
kubectl get secret awx-demo-admin-password -o jsonpath='{.data.password}' | base64 --decode

Architecture: The Parts That Matter

AWX is a Django app with a Celery worker pool and a PostgreSQL database. The web frontend talks to the API, which queues jobs in RabbitMQ/Redis. Celery workers pick up jobs and execute Ansible playbooks. The database stores job history, inventories, credentials, and templates. The critical piece: the worker pool. If you have more concurrent jobs than workers, jobs queue up. If you have too many workers, you saturate the database. I've seen teams set AWX_TASK_CONCURRENCY=20 on a 4-core VM — the database couldn't keep up and jobs timed out. Rule of thumb: start with 2 workers per CPU core, then monitor.

// io.thecodeforge — DevOps tutorial

# AWX deployment overrides for production
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx-prod
spec:
  # Scale web and task replicas
  web_replicas: 2
  task_replicas: 3
  
  # Resource limits (adjust based on load)
  web_resource_requirements:
    requests:
      cpu: 500m
      memory: 1Gi
    limits:
      cpu: 1
      memory: 2Gi
  task_resource_requirements:
    requests:
      cpu: 1
      memory: 2Gi
    limits:
      cpu: 2
      memory: 4Gi
  
  # Celery worker concurrency
  task_extra_settings:
    - setting: AWX_TASK_CONCURRENCY
      value: "6"  # 2 per CPU core
  
  # PostgreSQL settings
  postgres_resource_requirements:
    requests:
      cpu: 500m
      memory: 2Gi
    limits:
      cpu: 1
      memory: 4Gi

Inventory Management: Don't Hardcode Hosts

The worst AWX setups I've seen have static inventories with IP addresses hardcoded. That's not automation, that's a glorified spreadsheet. Use dynamic inventories: AWS EC2, Azure, GCP, or custom scripts. AWX supports inventory plugins and custom inventory scripts. The key: keep your inventory source of truth external. When a new VM spins up, AWX should pick it up automatically. I once consulted for a company where the ops team manually added hosts to AWX — they missed a new server and a deployment failed. Use the ansible-inventory command to test your dynamic inventory script before plugging it into AWX.

// io.thecodeforge — DevOps tutorial

# Dynamic inventory for AWS EC2
plugin: amazon.aws.aws_ec2
regions:
  - us-east-1
  - us-west-2
filters:
  tag:Environment:
    - production
    - staging
hostnames:
  - dns-name
  - private-dns-name
compose:
  ansible_host: private_ip_address
keyed_groups:
  - key: tags.Environment
    prefix: env_
  - key: tags.Role
    prefix: role_

# Test with: ansible-inventory -i aws_ec2_inventory.yml --list

Credentials: The Art of Not Leaking Secrets

AWX has a built-in credential store. Use it. Never put passwords in playbook variables. The problem: AWX encrypts credentials at rest, but they're decrypted when passed to the playbook. If a playbook task outputs the variable (e.g., debug: var=my_secret), the secret appears in job logs. Always set no_log: true on tasks that handle secrets. Also, use the 'Hide sensitive data' option in job templates — it redacts output from the web UI. But beware: this only hides from the UI, not the API. If you have API access, you can still see raw output. For true secrets management, integrate with HashiCorp Vault or CyberArk.

// io.thecodeforge — DevOps tutorial

---
- name: Deploy application
  hosts: all
  vars:
    db_password: "{{ lookup('awx', 'my_db_password') }}"  # from AWX credential
  tasks:
    - name: Configure database
      ansible.builtin.template:
        src: db_config.j2
        dest: /etc/myapp/db.conf
      no_log: true  # prevents password from appearing in logs
    - name: Restart service
      ansible.builtin.systemd:
        name: myapp
        state: restarted
      no_log: false

Job Templates: The Right Way to Parameterize

Job templates are the core of AWX — they define what playbook to run, against which inventory, with which credentials. The mistake: hardcoding variables in the template. Use survey questions to prompt for variables at runtime. But surveys have a downside: they're static. If you need dynamic options (e.g., list of available branches from Git), use the API to update the survey. Another trap: extra vars. You can pass JSON extra vars in the template, but they override everything. I've seen a team pass { "host_key_checking": false } as extra vars — that's a security risk. Use the 'Ask for extra vars' option sparingly. For complex workflows, use workflow templates.

// io.thecodeforge — DevOps tutorial

// Example survey for a job template (via API)
{
  "name": "Deploy version",
  "description": "Select version to deploy",
  "spec": [
    {
      "question_name": "Application version",
      "question_description": "Git tag or branch to deploy",
      "required": true,
      "type": "text",
      "variable": "app_version",
      "min": 1,
      "max": 128,
      "default": "v1.0.0"
    },
    {
      "question_name": "Environment",
      "question_description": "Target environment",
      "required": true,
      "type": "multiplechoice",
      "variable": "target_env",
      "choices": ["staging", "production"],
      "default": "staging"
    }
  ]
}

Scaling Workers: When One Isn't Enough

AWX uses Celery workers to run jobs. By default, it runs one worker process with multiple threads. For high concurrency, you need more worker pods. The gotcha: each worker consumes database connections. If you scale to 10 workers, you might hit PostgreSQL's max_connections limit. Also, workers share the same RabbitMQ queue — if a job is long-running, it blocks others. Solution: use separate queues for different job types. AWX supports custom queues via the instance_groups feature. Assign high-priority jobs to a dedicated queue with more workers.

// io.thecodeforge — DevOps tutorial

# Create an instance group for high-priority jobs
# Via AWX CLI (awx-cli)
awx instance_groups create --name "high_priority" --credential 1

# Add a worker instance to the group
awx instance_groups associate --instance worker-1 --instance_group high_priority

# Assign a job template to the group
awx job_templates associate --instance_group high_priority --job_template 5

# Verify
awx instance_groups list

Monitoring and Alerting: Know Before Your Users

AWX has a built-in metrics endpoint at /api/v2/metrics/ (Prometheus format). Expose it and scrape with Prometheus. Key metrics: awx_job_count (total jobs), awx_job_status (success/failure), awx_worker_pool_size. Set alerts for job failures and queue depth. I've seen teams ignore job failures for days because they didn't have monitoring. Also, monitor the AWX database: slow queries can cause timeouts. Use pg_stat_statements to find slow queries. Another blind spot: the AWX web UI itself. If the web pod goes down, users can't launch jobs. Set up a health check endpoint and alert on 5xx responses.

// io.thecodeforge — DevOps tutorial

# Prometheus alert rules for AWX
groups:
  - name: awx
    rules:
      - alert: AWXJobFailureRateHigh
        expr: rate(awx_job_status{status="failed"}[5m]) > 0.1
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High job failure rate in AWX"
      - alert: AWXQueueDepthHigh
        expr: awx_queue_depth > 10
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "AWX job queue depth > 10"
      - alert: AWXWebDown
        expr: up{job="awx-web"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "AWX web UI is down"

Backup and Disaster Recovery: Because It Will Fail

AWX stores everything in PostgreSQL. Back up the database. But also back up the secrets encryption key (SECRET_KEY in the AWX deployment). Without it, you can't restore encrypted credentials. I've seen a team lose their entire AWX config because they backed up the database but not the secret key. Use the AWX operator's built-in backup feature or a cron job with pg_dump. For Kubernetes deployments, use Velero or similar to backup PVs. Test your restore process quarterly. The worst time to discover your backup is corrupt is during an outage.

// io.thecodeforge — DevOps tutorial

# Kubernetes CronJob to backup AWX database
apiVersion: batch/v1
kind: CronJob
metadata:
  name: awx-db-backup
  namespace: awx
spec:
  schedule: "0 2 * * *"  # daily at 2 AM
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: pgdump
            image: postgres:15
            command:
            - /bin/sh
            - -c
            - |
              PGPASSWORD=$DB_PASSWORD pg_dump -h $DB_HOST -U $DB_USER awx > /backup/awx-$(date +%Y%m%d).sql
            env:
            - name: DB_HOST
              value: awx-postgres
            - name: DB_USER
              value: awx
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: awx-demo-postgres-configuration
                  key: password
            volumeMounts:
            - name: backup
              mountPath: /backup
          restartPolicy: OnFailure
          volumes:
          - name: backup
            persistentVolumeClaim:
              claimName: awx-backup-pvc