Ansible Vault Secrets: The Production Gotchas That Will Burn You

I still remember the 3 AM incident vividly. Our Ansible deploy had been running smoothly for months, then suddenly every playbook failed with 'Vault password incorrect' errors. The team was panicking – we had a production release scheduled in two hours. After digging through logs, we discovered that someone had accidentally committed the vault password file to a shared repo, and the security team had rotated it. But nobody had told the automation team. The vault password file on the CI server was stale. That night, we learned the hard way that secrets management isn't just about encryption; it's about key distribution, rotation, and lifecycle management.

Ansible Vault has been around since Ansible 1.5, introduced to solve the problem of storing sensitive data like passwords, API keys, and SSH keys in playbooks. Before that, teams either kept secrets in plaintext (a security nightmare) or used external tools with complex integrations. Ansible Vault provided a simple, built-in way to encrypt files and strings using symmetric AES-256 encryption. But simplicity can be deceptive. Over the years, I've seen countless production issues caused by misunderstandings: using the wrong vault ID, encrypting the wrong thing, or failing to integrate with CI/CD properly.

In this article, I'll cover everything you need to know to use Ansible Vault in production: from basic encrypt/decrypt/rekey operations to advanced patterns like vault-id labels, encrypting individual strings vs whole files, managing vault passwords in CI/CD pipelines, and integrating with HashiCorp Vault for dynamic secrets. I'll also share real incidents from my own experience and from colleagues, so you can avoid the same mistakes.

By the end, you'll be able to architect a secrets management strategy that is both secure and maintainable, whether you're a team of one or a hundred.

Encrypting and Decrypting Files: The Basics with Production Patterns

The most fundamental operation is encrypting a file. Use ansible-vault encrypt to encrypt an existing plaintext file:

``bash ansible-vault encrypt --vault-id prod@~/.vault_pass_prod group_vars/all/secrets.yml ``

This replaces the file with its encrypted version. Always use --vault-id with a label (like prod) instead of the deprecated --ask-vault-pass. The label is embedded in the encrypted file header: $ANSIBLE_VAULT;1.1;AES256;prod.

To decrypt:

``bash ansible-vault decrypt --vault-id prod@~/.vault_pass_prod group_vars/all/secrets.yml ``

But in production, you rarely decrypt to disk. Instead, use view to peek:

``bash ansible-vault view --vault-id prod@~/.vault_pass_prod group_vars/all/secrets.yml ``

A common gotcha: if you encrypt a file that's already encrypted, ansible-vault will prompt to overwrite. Use --force if you're sure, but better to avoid this.

Production pattern: never encrypt files that are actively being edited. Instead, keep a plaintext template in a secure location (like a password manager) and encrypt a copy for the repo. Or use ansible-vault encrypt_string for individual secrets (see next section).

Another pattern: use ansible-vault rekey to change the password without decrypting to disk:

``bash ansible-vault rekey --vault-id prod@~/.vault_pass_prod --new-vault-id prod@~/.vault_pass_prod_new group_vars/all/secrets.yml ``

This is essential when rotating vault passwords. But beware: rekey only changes the password, not the encryption algorithm or iterations. If you need to upgrade the encryption, you must decrypt and re-encrypt.

Encrypting Individual Strings vs Whole Files: When to Use Which

Ansible Vault supports two modes: encrypting entire files and encrypting individual strings (using encrypt_string). The choice has significant implications for manageability and security.

Whole-file encryption: Use ansible-vault encrypt to encrypt a YAML file containing multiple variables. Pros: simple, single password to manage. Cons: to change one secret, you must decrypt the entire file, edit, and re-encrypt. This is error-prone and exposes all secrets during the edit.

String encryption: Use ansible-vault encrypt_string to encrypt a single value:

``bash ansible-vault encrypt_string --vault-id prod@~/.vault_pass_prod 'my_secret_value' --name 'db_password' ``

This outputs:

``yaml db_password: !vault | $ANSIBLE_VAULT;1.1;AES256;prod 363932613338... ``

You then paste this into your vars file. Pros: you can change one secret without touching others, and the file remains mostly readable. Cons: requires managing multiple encrypted blobs.

Production recommendation: Use string encryption for secrets that change frequently (e.g., API tokens) and whole-file encryption for static secrets that rarely change (e.g., SSH keys). Never mix both in the same file without clear labeling.

A common mistake: encrypting a string that contains YAML special characters (like :, {, }). The encrypted output is safe, but if you manually edit the surrounding YAML, you might break the structure. Always use the !vault tag format as output by encrypt_string.

Another pitfall: using encrypt_string without --name results in an unlabeled encrypted value that's hard to reference. Always provide a name.

Mastering Vault-ID Labels for Multi-Environment Secrets

Vault-id labels allow you to use different vault passwords for different environments (dev, staging, prod) or different teams. The label is embedded in the encrypted file header and must match the --vault-id flag used to decrypt.

Creating a vault-id labeled file:

``bash ansible-vault encrypt --vault-id dev@~/.vault_pass_dev group_vars/dev/secrets.yml ``

The file header now contains dev as the label.

Using multiple vault-ids in a playbook:

``bash ansible-playbook --vault-id dev@~/.vault_pass_dev --vault-id prod@~/.vault_pass_prod site.yml ``

Ansible will try each vault-id in order until it finds one that can decrypt the file. The order matters: put the most specific label first.

Why labels matter: Without labels, if you have two vault passwords, Ansible tries them all. But if two files use different passwords, the wrong order can cause failures. Labels ensure the correct password is used for each file.

Production pattern: Use a naming convention like env/role for labels. For example, prod_web and prod_db for secrets specific to web and database tiers.

Gotcha: If you encrypt a file with label prod but then run ansible-playbook with --vault-id prod@new_password, the file will fail to decrypt if the password changed. Always rekey files after password rotation.

Integrating with CI/CD: In a pipeline, you might have multiple vault passwords stored as CI secrets. Use a script to create the password files at runtime:

``bash echo "$VAULT_PASS_PROD" > /tmp/vault_pass_prod ansible-playbook --vault-id prod@/tmp/vault_pass_prod playbook.yml ``

But be careful: writing secrets to disk can leave traces. Use mktemp and clean up in a finally block.

Vault Password Files and Environment Variables: Secure Distribution

The vault password is the single point of failure. If it's compromised, all secrets are exposed. Therefore, how you store and distribute the vault password is critical.

Vault password files: Store the password in a plaintext file with permissions 0600. The file should contain only the password (no newline at the end is best, but Ansible handles it). Example:

``bash # ~/.vault_pass_prod my_secure_vault_password ``

Then reference it:

``bash ansible-playbook --vault-id prod@~/.vault_pass_prod playbook.yml ``

Environment variable: You can set ANSIBLE_VAULT_PASSWORD_FILE to point to the file:

``bash export ANSIBLE_VAULT_PASSWORD_FILE=~/.vault_pass_prod ansible-playbook playbook.yml ``

This is useful in CI/CD where you can set the environment variable in the pipeline configuration.

Avoid hardcoding: Never put the password in the playbook or in a file in the repo. Instead, use a secrets manager like HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault to store the vault password itself. Then, in your CI/CD, retrieve the vault password and write it to a temporary file (or pass it via stdin).

Production pattern: Use a wrapper script that fetches the vault password from a secure source and passes it to Ansible:

``bash #!/bin/bash # fetch_vault_pass.sh vault_password=$(vault kv get -field=password secret/ansible/vault-pass-prod) echo "$vault_password" ``

Then use:

``bash ansible-playbook --vault-id prod@./fetch_vault_pass.sh playbook.yml ``

Note: Ansible can execute a script to get the vault password if the script is executable and outputs the password to stdout. This is a powerful pattern because the password never sits in a file on disk.

Gotcha: If using a script, ensure it's executable (chmod +x) and returns the password exactly (no trailing newline, unless your password ends with a newline). Test with ./fetch_vault_pass.sh | ansible-vault view.

Rekeying: Rotating Vault Passwords Without Decrypting Everything

Rekeying is the process of changing the vault password on an encrypted file without decrypting it to disk. This is essential for password rotation, especially when you have many encrypted files.

Command:

``bash ansible-vault rekey --vault-id prod@~/.vault_pass_old --new-vault-id prod@~/.vault_pass_new group_vars/all/secrets.yml ``

This decrypts the file in memory using the old password, then re-encrypts it with the new password. The file is never written to disk in plaintext.

Bulk rekeying: Use a shell loop:

``bash for f in group_vars/*/.yml; do ansible-vault rekey --vault-id prod@~/.vault_pass_old --new-vault-id prod@~/.vault_pass_new "$f" done ``

But be careful: if any file is not encrypted, the command will fail. Add a check:

``bash for f in group_vars/*/.yml; do if head -1 "$f" | grep -q '^\$ANSIBLE_VAULT'; then ansible-vault rekey ... "$f" fi done ``

Production incident: I once ran a rekey on a directory that included a plaintext file. The command failed midway, leaving some files rekeyed and others not. We had to restore from backup. Lesson: always test rekey on a single file first, and have a backup.

Rekey with multiple vault-ids: If you have files encrypted with different labels, you must rekey each with the correct old and new vault-id. You cannot change the label during rekey; the label remains the same. To change the label, you must decrypt and re-encrypt.

Password rotation policy: Rotate vault passwords at least every 90 days, or immediately after a security incident. Use a script that generates a new password, updates the vault password file, and runs rekey on all encrypted files.

Using Ansible Vault in CI/CD Pipelines: Best Practices

Integrating Ansible Vault into CI/CD requires careful handling of the vault password. Here are patterns for Jenkins, GitLab CI, GitHub Actions, and others.

General approach: 1. Store the vault password as a secret in your CI/CD platform (e.g., Jenkins secret text, GitLab CI variable, GitHub Actions secret). 2. In the pipeline, retrieve the secret and write it to a temporary file (or use a script that outputs it). 3. Pass the file to ansible-playbook via --vault-id. 4. Clean up the temporary file after the run.

GitLab CI example:

```yaml variables: VAULT_PASS_PROD: $VAULT_PASS_PROD # defined in CI/CD settings

before_script: - echo "$VAULT_PASS_PROD" > /tmp/vault_pass_prod - chmod 600 /tmp/vault_pass_prod

deploy: script: - ansible-playbook --vault-id prod@/tmp/vault_pass_prod site.yml after_script: - rm -f /tmp/vault_pass_prod ```

GitHub Actions example:

``yaml - name: Run Ansible playbook env: VAULT_PASS: ${{ secrets.VAULT_PASS_PROD }} run: | echo "$VAULT_PASS" > /tmp/vault_pass_prod chmod 600 /tmp/vault_pass_prod ansible-playbook --vault-id prod@/tmp/vault_pass_prod site.yml rm -f /tmp/vault_pass_prod ``

Using a script to avoid writing to disk: If your CI runner has a secure tmpfs, you can use that. Or use a script that reads the password from an environment variable and outputs it:

``bash #!/bin/bash # vault_pass_script.sh echo "$ANSIBLE_VAULT_PASSWORD" ``

Then in the pipeline:

``yaml - name: Run playbook env: ANSIBLE_VAULT_PASSWORD: ${{ secrets.VAULT_PASS_PROD }} run: | chmod +x vault_pass_script.sh ansible-playbook --vault-id prod@./vault_pass_script.sh site.yml ``

Gotcha: If the script outputs a newline, it's fine. But if the password itself contains a newline, you must handle it carefully. Best to generate passwords without newlines.

Security: Ensure the pipeline logs do not expose the vault password. Mask the variable in CI/CD settings. Also, avoid echoing the password in build logs.

Integrating with HashiCorp Vault Lookup Plugin

Ansible's hashi_vault lookup plugin allows you to fetch secrets directly from HashiCorp Vault at runtime, eliminating the need for static encrypted files. This is the modern approach for dynamic secrets management.

Installation: The plugin is included in community.hashi_vault collection:

``bash ansible-galaxy collection install community.hashi_vault ``

Prerequisites: You need a Vault server, a token or approle with appropriate policies, and the hvac Python library:

``bash pip install hvac ``

Usage in a playbook:

``yaml - name: Get secret from Vault debug: msg: "{{ lookup('community.hashi_vault.hashi_vault', 'secret/data/ansible:token={{ vault_token }} url=https://vault.example.com:8200') }}" ``

But hardcoding the token is not ideal. Instead, use environment variables or Ansible variables set from CI/CD.

Better pattern: Use a Vault agent or approle for authentication:

``yaml - name: Fetch secret using approle set_fact: db_password: "{{ lookup('community.hashi_vault.hashi_vault', 'secret/data/db', role_id='my_role_id', secret_id='my_secret_id', url='https://vault.example.com:8200') }}" ``

Production integration with Ansible Vault: You can combine both: use HashiCorp Vault to store the Ansible vault password itself. For example, have a script that retrieves the vault password from Vault and passes it to ansible-playbook. This way, you never store the vault password in a file.

Example script:

``bash #!/bin/bash # vault_pass_fetcher.sh VAULT_ADDR="https://vault.example.com:8200" VAULT_TOKEN="$(cat /etc/vault_token)" vault kv get -field=password secret/ansible/vault-pass-prod ``

Then:

``bash ansible-playbook --vault-id prod@./vault_pass_fetcher.sh site.yml ``

Gotcha: The lookup plugin returns the secret as a dictionary. You need to access the correct key. For Vault KV v2, the data is nested under data.data.

Caching: The lookup plugin does not cache by default, so every task that uses it will make a request to Vault. This can be slow. Use the cache parameter or set ansible_facts to avoid repeated lookups.

Security: Ensure Vault communication uses TLS. Set validate_certs: true (default). Use short-lived tokens or approle secret IDs.

Advanced: Using Multiple Vault Passwords and Labels in Complex Environments

In large organizations, you may have multiple teams managing different parts of the infrastructure, each with their own vault password. Vault-id labels allow you to manage this cleanly.

Scenario: You have a shared inventory with secrets for different applications. Each application team has its own vault password. You can encrypt each application's vars file with a unique label:

``bash ansible-vault encrypt --vault-id app1@~/.vault_pass_app1 group_vars/app1/secrets.yml ansible-vault encrypt --vault-id app2@~/.vault_pass_app2 group_vars/app2/secrets.yml ``

Then run the playbook with both vault-ids:

``bash ansible-playbook --vault-id app1@~/.vault_pass_app1 --vault-id app2@~/.vault_pass_app2 site.yml ``

Default label: If you don't specify a label, Ansible uses the label default. This is fine for simple setups, but avoid it in multi-team environments.

Nested labels: You can also use labels for different tiers within the same application. For example, app1_dev and app1_prod.

Automation: Use a wrapper script that reads a configuration file mapping labels to password sources (file, script, env var). This centralizes vault password management.

Auditing: Since the label is in the file header, you can audit which password was used to encrypt a file. This is useful for compliance.

Gotcha: If you forget to pass a vault-id for a label, Ansible will fail with 'Decryption failed'. Always ensure all labels used in encrypted files are provided.

Production pattern: Use a vault.yml file in each group_vars directory that defines the vault-id label for that group. This makes it self-documenting.

Common Pitfalls with Encrypting Strings: YAML Formatting and Special Characters

When using encrypt_string, the output is a YAML block scalar with the !vault tag. It's crucial to preserve the exact formatting to avoid parsing errors.

Correct format:

``yaml password: !vault | $ANSIBLE_VAULT;1.1;AES256;prod 363932613338... ``

The | indicates a literal block scalar. The encrypted value must be indented correctly (usually 2 spaces relative to the key).

Common mistake: Adding extra spaces or tabs before the encrypted value. This breaks YAML parsing.

Special characters in the plaintext secret: If your secret contains characters that are special in YAML (like :, #, {, }, [, ], ,, &, *, ?, |, -, <, >, =, !, %, @, ` `), encrypt_string` handles them correctly because it encrypts the entire string. However, if you manually edit the encrypted block, you might introduce errors.

Multiline secrets: If your secret contains newlines, encrypt_string will encrypt the entire string including newlines. When decrypted, it will have newlines. This is fine for SSH private keys, for example.

Testing: Always test your encrypted string by using ansible-vault view on the file or running a playbook with --syntax-check.

Production pattern: Use a script to encrypt strings and automatically paste them into the correct var file. This reduces manual errors.

``bash #!/bin/bash # encrypt_var.sh read -sp "Enter secret: " secret echo encrypted=$(ansible-vault encrypt_string --vault-id prod@~/.vault_pass_prod "$secret" --name "$1") echo "$encrypted" >> group_vars/all/secrets.yml ``

Gotcha: If you encrypt a string that is empty, encrypt_string will still produce an encrypted blob, but it will decrypt to an empty string. This is valid but may cause issues if your playbook expects a non-empty value.

Debugging Vault Decryption Failures: A Systematic Approach

When ansible-playbook fails with vault-related errors, follow this systematic approach:

1. Check the error message: Common messages include:
2. - ERROR! Decryption failed (no vault secrets found that could decrypt)
3. - ERROR! Vault password file not found
4. - ERROR! input file is not encrypted
5. Verify the file is encrypted: Use head -1 file.yml. If it doesn't start with $ANSIBLE_VAULT, the file is not encrypted.
6. Check the vault-id label: The header shows the label (e.g., $ANSIBLE_VAULT;1.1;AES256;prod). Ensure you're using --vault-id prod@password_file.
7. Test decryption manually:
8. ```bash
9. ansible-vault view --vault-id prod@password_file file.yml
10. ```
11. If this fails, the password is wrong.
12. Check password file: Ensure the password file exists, is readable, and contains exactly the password (no extra whitespace). Use cat -A password_file to see hidden characters.
13. Check environment variable: If using ANSIBLE_VAULT_PASSWORD_FILE, ensure it's set correctly and the file exists.
14. Check multiple vault-ids: If you have multiple vault-ids, ensure the correct one is used for each file. Ansible tries them in order; if the first one fails, it tries the next. But if a file is encrypted with a label not in the list, it fails.
15. Check for corrupted files: If the file was partially encrypted or edited, it may be corrupted. Try re-encrypting from a backup.

Production incident: We had a file that was encrypted with an older version of Ansible (1.1 format vs 1.2). The newer Ansible version could still decrypt it, but we had to update the file header to match the new format. We used ansible-vault rekey to rewrite the file.

Logging: Enable verbose mode with -vvv to see which vault-ids are being tried and which files are being decrypted.

Performance Considerations: Decryption Overhead and Caching

Decrypting vault files adds overhead to playbook runs. For large files or many encrypted files, this can be significant.

Decryption cost: Each encrypted file is decrypted once when it's first accessed. The decryption is done in memory and the plaintext is cached for the duration of the playbook run. So the overhead is mainly I/O and CPU for the initial decryption.

Large files: If you have a 10MB encrypted file, decryption takes noticeable time (a few seconds). Avoid encrypting large files; instead, encrypt only the secrets.

Many small files: If you have hundreds of encrypted var files, the cumulative decryption time can add up. Consider consolidating secrets into fewer files.

Vault lookup plugin performance: The hashi_vault lookup makes an HTTP request to Vault for each lookup. If you have many lookups, this can be slow. Use the cache parameter to cache results within a playbook run:

``yaml - name: Get secret with caching set_fact: db_password: "{{ lookup('community.hashi_vault.hashi_vault', 'secret/data/db', cache=True, cache_timeout=3600) }}" ``

Ansible fact caching: If you use ansible_facts to store secrets, they are cached for the duration of the playbook. But be careful not to expose secrets in fact caching to other hosts.

Production pattern: Use --tags to limit which parts of the playbook run, reducing the number of files that need to be decrypted.

Benchmark: In one project, we had 50 encrypted var files totaling 500KB. Decryption added about 2 seconds to the playbook startup. Acceptable, but for larger deployments, consider using HashiCorp Vault for dynamic secrets to reduce the number of encrypted files.

Security Best Practices: Beyond Basic Encryption

Ansible Vault encryption is strong, but the overall security depends on how you manage the vault password and the encrypted files.

1. Use strong, unique vault passwords: Generate random passwords with at least 20 characters, using a mix of letters, numbers, and symbols. Use a password manager to store them.

2. Rotate vault passwords regularly: Set a rotation policy (e.g., every 90 days) and automate rekeying.

3. Limit access to vault password files: Only the automation system and authorized admins should have access. Use filesystem permissions (0600) and access control lists.

4. Audit encrypted files: Use ansible-vault view to periodically inspect the contents of encrypted files for any unintended secrets.

5. Use separate vault passwords per environment: Never reuse the same vault password for dev, staging, and prod. If dev is compromised, prod remains secure.

6. Integrate with a secrets manager: Store the vault password itself in HashiCorp Vault or AWS Secrets Manager, and retrieve it at runtime.

7. Encrypt sensitive data at rest and in transit: Ensure your CI/CD pipeline uses HTTPS for Vault communication and that the vault password file is never transmitted over insecure channels.

8. Monitor for unauthorized access: Log all attempts to decrypt vault files. Ansible doesn't have built-in audit logging, but you can wrap the ansible-playbook command with logging.

9. Use .gitignore and pre-commit hooks: Prevent accidental commits of vault password files or plaintext secrets.

10. Consider using Ansible Vault with GPG: For additional security, you can encrypt the vault password file itself with GPG, and only decrypt it when needed.

Production incident: A developer accidentally pushed a vault password file to a public GitHub repo. The password was rotated within minutes, but the damage was done. We implemented a pre-commit hook that scans for files matching vaultpass* and blocks the commit.