AWS KMS: Key Management and Encryption

You're encrypting S3 buckets with AES-256, but your security team just flagged a critical finding: your encryption keys are stored in plaintext in a Git repository. This is the reality for teams that roll their own encryption without a managed key service. AWS KMS isn't just a key vault—it's the backbone of data protection in AWS, handling key generation, rotation, and access control so you don't have to. But misconfigure it, and you'll face outages, permission errors, or worse, data leaks. In this article, we'll cover KMS key types, IAM policies, key rotation, and real-world failure modes to avoid. By the end, you'll know how to encrypt production workloads without shooting yourself in the foot.

Why AWS KMS Exists: The Key Problem in Cloud Encryption

Encryption is easy. Key management is hard. In the cloud, you can't afford to store keys alongside data—that's like locking your house and leaving the key under the mat. AWS KMS (Key Management Service) solves this by providing a centralized, hardware-backed key store that integrates with nearly every AWS service. It handles key generation, rotation, auditing, and access control, so you don't have to build your own HSM cluster. The core concept is the Customer Master Key (CMK)—a logical key that never leaves KMS. You use KMS API calls to encrypt and decrypt data, and KMS enforces IAM policies and key policies. This separation of duties is critical: developers can encrypt data without ever seeing the plaintext key. In production, this means you can grant an application permission to encrypt but not decrypt, or vice versa, reducing blast radius if a service is compromised.

aws kms create-key --description "Production encryption key" --key-usage ENCRYPT_DECRYPT --origin AWS_KMS --tags Key=Environment,Value=Production
# Output:
# {
#     "KeyMetadata": {
#         "KeyId": "1234abcd-12ab-34cd-56ef-1234567890ab",
#         "Arn": "arn:aws:kms:us-east-1:123456789012:key/1234abcd-12ab-34cd-56ef-1234567890ab",
#         "CreationDate": "2025-01-01T00:00:00+00:00",
#         "Enabled": true,
#         "KeyUsage": "ENCRYPT_DECRYPT",
#         "KeyState": "Enabled",
#         "Origin": "AWS_KMS",
#         "KeyManager": "CUSTOMER"
#     }
# }

Key Policies vs IAM Policies: Who Controls Access?

KMS has a dual authorization model: key policies and IAM policies. A key policy is attached directly to the CMK and defines who can use the key and under what conditions. IAM policies grant permissions to users/roles, but they only take effect if the key policy also allows it. This means you can lock down a key so that even an admin with full IAM access cannot use it unless the key policy explicitly permits. In production, this is a safety net: you can create a key policy that requires encryption operations to come from a specific VPC or include a condition like kms:ViaService to restrict usage to certain AWS services. For example, you can allow S3 to use the key for SSE-KMS but deny direct API calls from EC2 instances. Always start with a restrictive key policy and grant IAM permissions only as needed. A common mistake is leaving the default key policy (which gives full access to the root user) and then wondering why a role can't decrypt.

{
  "Version": "2012-10-17",
  "Id": "key-consolepolicy-3",
  "Statement": [
    {
      "Sid": "Enable IAM User Permissions",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:root"
      },
      "Action": "kms:*",
      "Resource": "*"
    },
    {
      "Sid": "Allow use of the key by S3",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:role/S3EncryptionRole"
      },
      "Action": [
        "kms:Encrypt",
        "kms:Decrypt",
        "kms:ReEncrypt*",
        "kms:GenerateDataKey*",
        "kms:DescribeKey"
      ],
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "kms:ViaService": "s3.us-east-1.amazonaws.com"
        }
      }
    }
  ]
}

Encrypting Data at Rest with KMS: S3, EBS, RDS

KMS integrates deeply with AWS storage services. For S3, you enable Server-Side Encryption with KMS (SSE-KMS) by specifying the CMK in the bucket policy or during object upload. This gives you an audit trail of every encryption/decryption operation via CloudTrail. For EBS, you can encrypt volumes at creation using a KMS key, and the data is transparently encrypted on the physical disk. RDS supports encryption at rest using KMS for all database engines. The key insight: when you use KMS-managed keys, the service calls KMS on your behalf. This means you pay per API call, and there are rate limits. In production, you must monitor KMS API usage to avoid throttling. For example, if you have a large EBS snapshot restore, it may generate thousands of GenerateDataKeyWithoutPlaintext calls. Pre-warm your KMS keys or request a limit increase if needed. Also, note that cross-account access requires the key policy to grant permissions to the external account's root, and then that account's IAM policies can delegate further.

import boto3

s3 = boto3.client('s3')
kms_key_id = '1234abcd-12ab-34cd-56ef-1234567890ab'

# Upload with SSE-KMS
s3.put_object(
    Bucket='my-production-bucket',
    Key='secrets/config.json',
    Body=b'{"api_key": "secret"}',
    ServerSideEncryption='aws:kms',
    SSEKMSKeyId=kms_key_id
)

# Download (decryption happens automatically if role has kms:Decrypt)
response = s3.get_object(Bucket='my-production-bucket', Key='secrets/config.json')
print(response['Body'].read().decode())

Envelope Encryption: How KMS Handles Large Data

KMS has a 1 MB limit on direct encrypt/decrypt calls. For larger data, you must use envelope encryption: generate a data key from KMS, encrypt your data locally with that data key, then store the encrypted data key alongside the ciphertext. The data key is a symmetric key (256-bit) that KMS returns in two forms: plaintext (for encryption) and ciphertext (for storage). You use the plaintext key to encrypt your data with a fast algorithm like AES-256-GCM, then discard it. To decrypt, you send the ciphertext data key back to KMS to get the plaintext key, then decrypt locally. This pattern is used by S3, EBS, and most AWS services internally. In production, you must protect the plaintext data key in memory—never log it or write it to disk. Use secure memory wiping (e.g., sodium_memzero in C, or array.clear() in Python). Also, consider key hierarchy: you can have a master key that encrypts multiple data keys, each for different purposes.

import boto3
from cryptography.fernet import Fernet

kms = boto3.client('kms')
key_id = '1234abcd-12ab-34cd-56ef-1234567890ab'

# Generate data key
response = kms.generate_data_key(KeyId=key_id, KeySpec='AES_256')
plaintext_key = response['Plaintext']
ciphertext_key = response['CiphertextBlob']

# Encrypt data locally
fernet = Fernet(plaintext_key)
data = b'This is a large payload over 1MB...' * 1000
encrypted_data = fernet.encrypt(data)

# Store ciphertext_key + encrypted_data
# To decrypt:
response = kms.decrypt(CiphertextBlob=ciphertext_key)
plaintext_key = response['Plaintext']
fernet = Fernet(plaintext_key)
decrypted_data = fernet.decrypt(encrypted_data)
print(decrypted_data[:50])

Key Rotation: Automatic vs Manual

AWS KMS supports automatic key rotation for symmetric CMKs. When enabled, KMS rotates the backing key annually—new cryptographic material is generated, but the key ID and metadata remain the same. Old data encrypted with previous backing keys remains decryptable because KMS keeps the old material. This is transparent to applications. However, automatic rotation only applies to customer-managed keys, not AWS-managed keys. For asymmetric keys, you must rotate manually by creating a new key and updating references. In production, enable automatic rotation for all symmetric CMKs used for encryption. But beware: rotation does not re-encrypt existing data. If you need to re-encrypt data with a new key, you must do that yourself (e.g., by reading and rewriting S3 objects). Also, consider key rotation for compliance (e.g., PCI-DSS requires annual rotation). For high-security environments, you can use custom key stores (CloudHSM) to control the rotation schedule.

aws kms enable-key-rotation --key-id 1234abcd-12ab-34cd-56ef-1234567890ab
# Verify rotation status
aws kms get-key-rotation-status --key-id 1234abcd-12ab-34cd-56ef-1234567890ab
# Output:
# {
#     "KeyRotationEnabled": true
# }

Auditing KMS Usage with CloudTrail and Metrics

Every KMS API call is logged in CloudTrail, including the key ID, caller identity, source IP, and whether the operation succeeded or failed. This is invaluable for security audits and incident response. You can create CloudWatch alarms on metrics like KMS.SuccessfulDecrypt or KMS.ThrottledCount to detect anomalies. For example, a sudden spike in Decrypt calls might indicate a data exfiltration attempt. In production, set up a CloudTrail trail that delivers logs to a centralized S3 bucket with encryption enabled (using a different KMS key). Also, use CloudWatch Logs Insights to query for specific patterns, such as decryption attempts from unknown IPs. Remember that KMS generates a lot of logs; consider using selective logging or log filtering to reduce costs. Additionally, you can use AWS Config rules to enforce that keys have rotation enabled or that key policies are not overly permissive.

fields @timestamp, @message
| filter eventSource = 'kms.amazonaws.com'
| filter eventName = 'Decrypt'
| filter errorCode exists
| stats count() by userIdentity.arn, sourceIPAddress
| sort count desc
| limit 10

Multi-Region Keys: Disaster Recovery and Cross-Region Access

AWS KMS now supports multi-region keys (MRK), which are symmetric CMKs that exist in multiple regions with the same key ID and material. This simplifies cross-region encryption: you can encrypt in one region and decrypt in another without needing to re-encrypt. MRKs are ideal for disaster recovery setups where you have a primary and secondary region. However, they are not automatically replicated; you must create replicas in each region. The key material is shared, but each region has its own key resource. In production, use MRKs for global applications that need to encrypt data in one region and decrypt in another (e.g., cross-region S3 replication). But be aware of latency: decrypting in a different region adds network round-trips. Also, MRKs have the same rate limits per region, so you might need to increase limits in each region. For compliance, ensure that using MRKs does not violate data residency requirements—key material may be stored in multiple regions.

# Create primary MRK in us-east-1
aws kms create-key --multi-region --description "Global MRK" --region us-east-1
# Output includes MultiRegion: true

# Create replica in eu-west-1
aws kms replicate-key --key-id mrk-1234abcd12ab34cd56ef1234567890ab --replica-region eu-west-1 --region us-east-1
# Output:
# {
#     "ReplicaKeyMetadata": {
#         "KeyId": "mrk-1234abcd12ab34cd56ef1234567890ab",
#         "Arn": "arn:aws:kms:eu-west-1:123456789012:key/mrk-1234abcd12ab34cd56ef1234567890ab",
#         "MultiRegion": true,
#         "MultiRegionConfiguration": {
#             "MultiRegionKeyType": "REPLICA",
#             "PrimaryKey": {
#                 "Arn": "arn:aws:kms:us-east-1:123456789012:key/mrk-1234abcd12ab34cd56ef1234567890ab",
#                 "Region": "us-east-1"
#             },
#             "ReplicaKeys": []
#         }
#     }
# }

Custom Key Stores: When AWS-Managed HSMs Aren't Enough

For organizations that need to control the HSM that stores key material, AWS offers custom key stores backed by AWS CloudHSM. This gives you direct control over the HSM cluster, including the ability to delete keys (which is not possible with standard KMS). You can also meet compliance requirements that mandate keys be stored in a dedicated HSM under your control. However, custom key stores add operational complexity: you must manage the CloudHSM cluster, including backups, scaling, and patching. If the HSM cluster becomes unavailable, all KMS operations on those keys fail. In production, use custom key stores only when required by compliance (e.g., FIPS 140-2 Level 3, or specific regulatory mandates). For most workloads, standard KMS is sufficient and more resilient. If you do use custom key stores, ensure you have a disaster recovery plan for the HSM cluster, and monitor its health with CloudWatch alarms.

# Prerequisite: Create CloudHSM cluster and initialize
aws kms create-custom-key-store --custom-key-store-name MyHSMStore --cloud-hsm-cluster-id cluster-abc123 --trust-anchor-certificate file://customerCA.pem --key-store-password MyPassword123
# Output:
# {
#     "CustomKeyStoreId": "cks-1234567890abcdef0"
# }

# Create a CMK in the custom store
aws kms create-key --custom-key-store-id cks-1234567890abcdef0 --description "Key in my HSM"

Cost Optimization: KMS Pricing and Throttling

KMS pricing is based on the number of keys and API calls. Each CMK costs $1/month (pro-rated). API calls cost $0.03 per 10,000 requests (encrypt, decrypt, etc.). For high-volume workloads, costs can add up. For example, if you encrypt each S3 object individually with a new data key, you pay per object. To reduce costs, use S3 Bucket Keys (which cache a data key at the bucket level) or batch encrypt/decrypt operations. Also, consider using AWS-managed keys (free) for services that don't need customer control. However, AWS-managed keys have limited audit and no rotation control. In production, monitor KMS spending with Cost Explorer and set budgets. Throttling is another concern: KMS has default limits of 5,500 requests per second per region (for symmetric keys). You can request increases, but they are not guaranteed. Use exponential backoff in your applications and consider caching data keys locally (with a short TTL) to reduce API calls.

import boto3
from cachetools import TTLCache

kms = boto3.client('kms')
key_id = '1234abcd-12ab-34cd-56ef-1234567890ab'
cache = TTLCache(maxsize=100, ttl=300)  # 5 min TTL

def get_data_key():
    if key_id in cache:
        return cache[key_id]
    response = kms.generate_data_key(KeyId=key_id, KeySpec='AES_256')
    cache[key_id] = (response['Plaintext'], response['CiphertextBlob'])
    return cache[key_id]

# Use cached key for encryption
plaintext_key, ciphertext_key = get_data_key()

Common Pitfalls and How to Avoid Them

Even experienced teams make mistakes with KMS. Here are the top failures we've seen: (1) Using the default key policy without restrictions, allowing any IAM user with KMS permissions to use the key. Always scope key policies to specific roles and conditions. (2) Forgetting to grant kms:Decrypt permission to the role that reads encrypted data. This causes silent failures or 403 errors. (3) Hitting KMS rate limits during bulk operations. Use exponential backoff and request quota increases proactively. (4) Storing plaintext data keys in environment variables or config files. Always use envelope encryption and discard plaintext keys. (5) Not enabling key rotation and then failing compliance audits. Enable rotation by default. (6) Using the same key for encryption and signing (if using asymmetric keys). Use separate keys for different purposes. (7) Cross-account access misconfiguration: the key policy must grant access to the external account's root, and then that account's IAM must allow the action. Test with the kms:ViaService condition to limit exposure.

# Test if a role can decrypt using the key
aws kms decrypt --ciphertext-blob fileb://encrypted_data.bin --key-id 1234abcd-12ab-34cd-56ef-1234567890ab --profile myapp
# If AccessDeniedException, check key policy and IAM policy
# Common fix: add kms:Decrypt to the role's IAM policy
aws iam put-role-policy --role-name MyAppRole --policy-name KMSDecrypt --policy-document '{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "kms:Decrypt",
      "Resource": "arn:aws:kms:us-east-1:123456789012:key/1234abcd-12ab-34cd-56ef-1234567890ab"
    }
  ]
}'

KMS in CI/CD: Automating Key Management

Integrating KMS into CI/CD pipelines requires careful handling of secrets. Never store plaintext keys in code repositories. Instead, use KMS to encrypt secrets at rest and decrypt them at deploy time. For example, you can encrypt environment variables with KMS and store the ciphertext in your repo. During deployment, the CI/CD role (e.g., CodeBuild) must have kms:Decrypt permission to decrypt them. Alternatively, use AWS Secrets Manager or Parameter Store with KMS encryption. For infrastructure as code (e.g., Terraform, CloudFormation), you can create KMS keys and policies declaratively. But beware of circular dependencies: if your KMS key is used to encrypt a resource that is created in the same stack, you may need to use a pre-existing key. In production, use a separate KMS key for CI/CD secrets, and rotate it frequently. Also, ensure that the CI/CD role has a tight key policy that only allows decryption from the specific VPC or IP range of the build environment.

resource "aws_kms_key" "cicd" {
  description             = "KMS key for CI/CD secrets"
  deletion_window_in_days = 7
  enable_key_rotation     = true
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid    = "Enable IAM User Permissions"
        Effect = "Allow"
        Principal = {
          AWS = "arn:aws:iam::123456789012:root"
        }
        Action   = "kms:*"
        Resource = "*"
      },
      {
        Sid    = "Allow CodeBuild to decrypt"
        Effect = "Allow"
        Principal = {
          AWS = "arn:aws:iam::123456789012:role/CodeBuildRole"
        }
        Action   = "kms:Decrypt"
        Resource = "*"
        Condition = {
          StringEquals = {
            "kms:ViaService": "codebuild.us-east-1.amazonaws.com"
          }
        }
      }
    ]
  })
}

resource "aws_kms_alias" "cicd" {
  name          = "alias/cicd-key"
  target_key_id = aws_kms_key.cicd.key_id
}

Disaster Recovery: Key Deletion and Backup Strategies

KMS keys are critical infrastructure. If you delete a key, all data encrypted with it becomes permanently unrecoverable. AWS enforces a waiting period (7-30 days) before deletion, during which you can cancel. In production, never delete keys that are still in use. Instead, disable them and monitor for any decryption attempts. For disaster recovery, you should back up key metadata (key ID, policy) but note that you cannot export the key material from KMS (unless using custom key stores). For multi-region keys, the replica can serve as a backup. For standard keys, you must re-encrypt data with a new key if the original is lost. Therefore, always have a process to rotate keys and re-encrypt data before decommissioning a key. Also, consider using AWS Backup to snapshot KMS keys? No, that's not supported. Instead, document your key hierarchy and ensure that key administrators have a recovery plan. Test key recovery by simulating a key deletion scenario in a non-production environment.

# Schedule key deletion (cannot be undone after waiting period)
aws kms schedule-key-deletion --key-id 1234abcd-12ab-34cd-56ef-1234567890ab --pending-window-in-days 7
# Output:
# {
#     "KeyId": "1234abcd-12ab-34cd-56ef-1234567890ab",
#     "DeletionDate": "2025-01-08T00:00:00+00:00",
#     "KeyState": "PendingDeletion"
# }

# Cancel deletion
aws kms cancel-key-deletion --key-id 1234abcd-12ab-34cd-56ef-1234567890ab