AWS Secrets Manager: Secure Credential Rotation

You just pushed a code change that rotated database credentials. Now your production app is down because the old password was cached in a connection pool. This is the reality of manual rotation — it's error-prone and risky. AWS Secrets Manager solves this by automating the entire lifecycle: creation, rotation, and deletion of secrets. But here's the kicker: most teams use it wrong. They enable automatic rotation without understanding the underlying Lambda execution model, leading to race conditions and expired credentials during rotation windows. In this post, I'll show you how to configure rotation correctly, handle edge cases like multi-AZ failovers, and avoid the pitfalls that take down production.

Why Manual Credential Rotation Fails in Production

Manual credential rotation is a ticking time bomb. In production, teams often rely on shared secrets stored in config files, environment variables, or even hardcoded in source code. When a credential expires or leaks, the scramble to rotate it manually leads to downtime, inconsistent state across services, and security gaps. I've seen incidents where a database password expired at 3 AM, taking down a critical microservice because no one remembered to update the secret in the CI/CD pipeline. The root cause: no automated rotation. AWS Secrets Manager solves this by centralizing secret storage and enabling automatic rotation via Lambda functions. But the real challenge is designing rotation that doesn't break your applications. You need to ensure that during rotation, both old and new credentials are valid simultaneously (grace period), and that clients refresh secrets without restarting. This section sets the stage for why you need a robust rotation strategy, not just a tool.

# Simulating a manual rotation failure
# Old secret in env
DB_PASSWORD="old_password_123"
# New secret set manually
DB_PASSWORD="new_password_456"
# Application still uses old connection pool
# => Connection errors until restart
# No audit trail, no rollback plan

AWS Secrets Manager: Architecture and Key Concepts

AWS Secrets Manager is a service for securely storing, retrieving, and rotating secrets. At its core, it stores secrets as encrypted blobs in a dedicated Secrets Manager instance, with encryption handled by AWS KMS. Each secret has a name, description, and rotation configuration. The service integrates with AWS Lambda for custom rotation logic. Key concepts: secret versions (staged labels like AWSCURRENT, AWSPREVIOUS, AWSPENDING), automatic rotation schedule (e.g., every 30 days), and resource-based policies for cross-account access. The rotation process works by creating a new version (AWSPENDING), updating the credential in the target service, testing it, and then marking it as AWSCURRENT while deprecating the old one. This ensures zero-downtime rotation if implemented correctly. However, the default rotation templates (e.g., for RDS) may not fit all use cases—you often need custom Lambda functions for non-standard services or complex validation. Understanding these concepts is critical before writing rotation code.

aws secretsmanager create-secret \
    --name prod/myapp/db-password \
    --description "Database password for MyApp production" \
    --secret-string '{"username":"myapp","password":"InitialPassword123!"}' \
    --kms-key-id alias/aws/secretsmanager \
    --tags Key=Environment,Value=prod

Designing a Custom Rotation Lambda for Zero Downtime

The default rotation templates work for simple cases, but production often requires custom logic. For example, rotating a database password for a multi-region deployment or a service that doesn't support concurrent credentials. The Lambda function must implement four steps: createSecret, setSecret, testSecret, and finishSecret. In createSecret, you generate a new password and store it as AWSPENDING. In setSecret, you update the target service (e.g., RDS) with the new password. In testSecret, you verify the new credential works (e.g., by making a test query). In finishSecret, you move AWSPENDING to AWSCURRENT and deprecate the old one. Critical design decisions: ensure the target service supports two active passwords during rotation (e.g., RDS allows multiple users with different passwords). If not, you need a different strategy, like updating all clients before finalizing. Also, handle failures gracefully—if setSecret fails, don't proceed to finishSecret. Use idempotent operations to avoid partial updates. The Lambda should have minimal permissions: only access to the specific secret and the target service.

import boto3
import json
import logging
import os

logger = logging.getLogger()
logger.setLevel(logging.INFO)

def lambda_handler(event, context):
    arn = event['SecretId']
    token = event['ClientRequestToken']
    step = event['Step']
    
    secrets_client = boto3.client('secretsmanager')
    
    try:
        metadata = secrets_client.describe_secret(SecretId=arn)
        if not metadata['RotationEnabled']:
            logger.error("Secret %s is not enabled for rotation", arn)
            raise ValueError("Rotation not enabled")
        
        versions = metadata['VersionIdsToStages']
        if token not in versions:
            logger.error("Token %s not found in versions", token)
            raise ValueError("Invalid token")
        if 'AWSPENDING' in versions[token]:
            logger.info("Token %s is already AWSPENDING, skipping", token)
            return
        
        if step == 'createSecret':
            create_secret(secrets_client, arn, token)
        elif step == 'setSecret':
            set_secret(secrets_client, arn, token)
        elif step == 'testSecret':
            test_secret(secrets_client, arn, token)
        elif step == 'finishSecret':
            finish_secret(secrets_client, arn, token)
        else:
            raise ValueError("Invalid step %s" % step)
    except Exception as e:
        logger.error("Rotation failed: %s", str(e))
        raise

def create_secret(client, arn, token):
    # Generate new password
    new_password = generate_random_password()
    # Store as AWSPENDING
    client.put_secret_value(
        SecretId=arn,
        ClientRequestToken=token,
        SecretString=json.dumps({'username': 'myapp', 'password': new_password}),
        VersionStages=['AWSPENDING']
    )
    logger.info("Created AWSPENDING version for %s", arn)

def set_secret(client, arn, token):
    # Retrieve pending secret
    pending = client.get_secret_value(
        SecretId=arn,
        VersionId=token,
        VersionStage='AWSPENDING'
    )
    creds = json.loads(pending['SecretString'])
    # Update RDS password
    rds_client = boto3.client('rds')
    rds_client.modify_db_instance(
        DBInstanceIdentifier='mydb',
        MasterUserPassword=creds['password']
    )
    logger.info("Set new password on RDS instance")

def test_secret(client, arn, token):
    # Retrieve pending secret
    pending = client.get_secret_value(
        SecretId=arn,
        VersionId=token,
        VersionStage='AWSPENDING'
    )
    creds = json.loads(pending['SecretString'])
    # Test connection
    import pymysql
    try:
        conn = pymysql.connect(
            host='mydb.xxxxx.us-east-1.rds.amazonaws.com',
            user=creds['username'],
            password=creds['password'],
            database='mydb',
            connect_timeout=5
        )
        conn.close()
        logger.info("Test connection successful")
    except Exception as e:
        logger.error("Test connection failed: %s", str(e))
        raise

def finish_secret(client, arn, token):
    # Move AWSPENDING to AWSCURRENT
    client.update_secret_version_stage(
        SecretId=arn,
        VersionStage='AWSCURRENT',
        MoveToVersionId=token,
        RemoveFromVersionId='AWSPREVIOUS'
    )
    logger.info("Finished rotation for %s", arn)

def generate_random_password():
    import secrets
    import string
    alphabet = string.ascii_letters + string.digits + '!@#$%^&*()'
    return ''.join(secrets.choice(alphabet) for _ in range(32))

Integrating Secrets Manager with Your Application

Once secrets are stored and rotated, your application needs to fetch them dynamically. The worst approach is to read the secret once at startup and cache it indefinitely—you'll miss rotations. Instead, use the AWS SDK to fetch the secret on demand or cache with a short TTL (e.g., 5 minutes). For high-throughput applications, consider using a secrets caching library like the AWS Secrets Manager Java Client Caching or the Python-based cache with refresh. The key is to handle rotation gracefully: if a secret is rotated while your app holds a stale connection, you need to detect authentication failures and retry with the new secret. Implement a retry mechanism that re-fetches the secret on failure. Also, ensure your application doesn't start if the secret is missing—fail fast. For containerized environments, inject secrets as environment variables at deployment time, but use a sidecar or init container to refresh them periodically. Avoid storing secrets in volumes or config maps.

import boto3
import json
from cachetools import cached, TTLCache

cache = TTLCache(maxsize=100, ttl=300)  # 5 minutes

def get_secret(secret_name, region_name='us-east-1'):
    session = boto3.session.Session()
    client = session.client(
        service_name='secretsmanager',
        region_name=region_name
    )
    try:
        response = client.get_secret_value(SecretId=secret_name)
        return json.loads(response['SecretString'])
    except Exception as e:
        print(f"Error fetching secret {secret_name}: {e}")
        raise

@cached(cache)
def get_cached_secret(secret_name):
    return get_secret(secret_name)

# Usage in database connection
def get_db_connection():
    secret = get_cached_secret('prod/myapp/db-password')
    # Use secret to connect
    # If connection fails due to auth, invalidate cache and retry
    try:
        conn = create_connection(secret['username'], secret['password'])
        return conn
    except pymysql.err.OperationalError as e:
        if e.args[0] == 1045:  # Access denied
            cache.pop('prod/myapp/db-password', None)
            secret = get_secret('prod/myapp/db-password')  # Force refresh
            return create_connection(secret['username'], secret['password'])
        raise

Monitoring and Alerting for Rotation Failures

Rotation can fail silently. Common failure modes: Lambda timeout, insufficient IAM permissions, target service unavailability, or network issues. You need monitoring to detect failures immediately. Enable CloudTrail to log all Secrets Manager API calls. Set up CloudWatch alarms on the Lambda function's error count and duration. Also, monitor the secret's last rotation timestamp—if it exceeds the rotation interval, alert. Use AWS Config rules to ensure rotation is enabled on all secrets. For critical secrets, implement a canary that periodically tests the secret's validity (e.g., a Lambda that tries to authenticate with the current secret). If the canary fails, trigger an incident. Also, log rotation events with context: which secret, which step failed, and the error message. This helps in post-mortem analysis. Finally, have a manual rollback procedure: if rotation fails, you can manually set the secret back to AWSPREVIOUS using the AWS CLI.

# Check last rotation timestamp
aws secretsmanager describe-secret --secret-id prod/myapp/db-password --query 'LastRotatedDate'

# CloudWatch alarm for Lambda errors
aws cloudwatch put-metric-alarm \
    --alarm-name rotation-lambda-errors \
    --alarm-description "Alarm when rotation Lambda errors > 0" \
    --metric-name Errors \
    --namespace AWS/Lambda \
    --statistic Sum \
    --period 300 \
    --evaluation-periods 1 \
    --threshold 0 \
    --comparison-operator GreaterThanThreshold \
    --dimensions Name=FunctionName,Value=rotation-lambda \
    --alarm-actions arn:aws:sns:us-east-1:123456789012:rotation-alerts

# Manual rollback to previous version
aws secretsmanager update-secret-version-stage \
    --secret-id prod/myapp/db-password \
    --version-stage AWSCURRENT \
    --move-to-version-id $(aws secretsmanager list-secret-version-ids --secret-id prod/myapp/db-password --query "VersionIdsToStages[?contains(@, 'AWSPREVIOUS')]" --output text)

Handling Multi-Region and Cross-Account Secrets

In multi-region architectures, you need to replicate secrets across regions for disaster recovery. Secrets Manager supports replication via the replicate-secret-to-regions API. However, rotation must be handled carefully: if you rotate a secret in the primary region, the replica becomes stale. You have two options: either rotate in each region independently (with separate Lambda functions) or use a global rotation strategy where the primary region rotates and then pushes the new secret to replicas. The latter is simpler but introduces latency. For cross-account access, use resource-based policies on the secret to grant access to other accounts. For example, a central secrets account can share secrets with multiple workload accounts. Ensure that the rotation Lambda in the central account can update the target service in the workload account (via cross-account IAM roles). This adds complexity but centralizes management. Also, consider using AWS Organizations to automate secret sharing.

# Replicate secret to another region
aws secretsmanager replicate-secret-to-regions \
    --secret-id prod/myapp/db-password \
    --add-replica-regions Region=eu-west-1

# Cross-account access policy (attached to secret)
aws secretsmanager put-resource-policy \
    --secret-id prod/myapp/db-password \
    --resource-policy '{
        "Version": "2012-10-17",
        "Statement": [{
            "Effect": "Allow",
            "Principal": {"AWS": "arn:aws:iam::123456789012:root"},
            "Action": "secretsmanager:GetSecretValue",
            "Resource": "*"
        }]
    }'

Cost Optimization and Secret Organization

Secrets Manager costs $0.40 per secret per month plus $0.05 per 10,000 API calls. For large deployments, costs can add up. Optimize by consolidating related secrets into a single secret with a JSON structure (e.g., all database credentials for one service). Use tags to organize secrets by environment, team, or application. Implement lifecycle policies: delete unused secrets after a retention period (e.g., 30 days). Use AWS Backup to take snapshots of secrets for disaster recovery. Also, consider using Parameter Store for non-sensitive configuration (e.g., feature flags) to reduce costs. For secrets that are rotated frequently, the API call cost can be significant—batch updates if possible. Monitor usage with Cost Explorer and set budgets.

# Tag secrets for cost allocation
aws secretsmanager tag-resource \
    --secret-id prod/myapp/db-password \
    --tags Key=Environment,Value=prod Key=Team,Value=platform

# List secrets with no rotation (potential cost waste)
aws secretsmanager list-secrets --filters Key=rotation-enabled,Values=false --query 'SecretList[*].Name'

# Delete unused secret (after ensuring no dependencies)
aws secretsmanager delete-secret --secret-id old/unused-secret --force-delete-without-recovery

Testing Rotation in a Staging Environment

Never test rotation in production first. Set up a staging environment that mirrors production: same target services, same IAM roles, same network configuration. Use a separate secret (e.g., staging/myapp/db-password) with a short rotation interval (e.g., 1 hour) to test frequently. Automate testing with a script that triggers rotation and validates the outcome: check that the new secret works, the old secret still works (during grace period), and that applications using the secret reconnect without errors. Simulate failure scenarios: make the target service unavailable, revoke Lambda permissions, or corrupt the secret. Verify that the rotation Lambda handles these gracefully and that alerts fire. Also, test rollback: manually set the secret back to AWSPREVIOUS and confirm applications recover. Document the test results and update the rotation Lambda accordingly.

# Trigger rotation manually
aws secretsmanager rotate-secret --secret-id staging/myapp/db-password

# Wait for rotation to complete
sleep 60

# Verify new secret works
aws secretsmanager get-secret-value --secret-id staging/myapp/db-password --version-stage AWSCURRENT

# Verify old secret still works (AWSPREVIOUS)
aws secretsmanager get-secret-value --secret-id staging/myapp/db-password --version-stage AWSPREVIOUS

# Test rollback
aws secretsmanager update-secret-version-stage \
    --secret-id staging/myapp/db-password \
    --version-stage AWSCURRENT \
    --move-to-version-id $(aws secretsmanager list-secret-version-ids --secret-id staging/myapp/db-password --query "VersionIdsToStages[?contains(@, 'AWSPREVIOUS')]" --output text)

Common Pitfalls and How to Avoid Them

Even with careful design, rotation can fail. Common pitfalls: 1) Not handling the grace period—if the target service doesn't support two active passwords, you must update all clients before finalizing. 2) Lambda permissions too broad—use least privilege. 3) Not testing with actual application traffic—a secret may work in isolation but fail under load. 4) Forgetting to update connection strings in application code—use dynamic fetching. 5) Rotating secrets that are used by multiple services simultaneously—coordinate via a distributed lock or stagger rotations. 6) Not monitoring rotation failures—set alerts. 7) Hardcoding secret names in Lambda—use environment variables. 8) Not planning for region failure—replicate secrets. 9) Using default KMS key—use a custom key for better control. 10) Not documenting the rotation process—team members need to know how to respond to failures. Avoid these by following AWS best practices and conducting regular drills.

# Pitfall: Not handling grace period
# If target service doesn't support two passwords,
# you must update all clients before finishSecret.
# Example: Redis password rotation
# Solution: Use a two-phase approach:
# 1. Set new password but keep old one valid
# 2. Update all clients
# 3. Remove old password
# This requires custom logic in setSecret and finishSecret.

Production Rollout Strategy

Rolling out automated rotation to production requires a phased approach. Start with non-critical secrets (e.g., API keys for internal tools) and monitor for a week. Then move to low-impact secrets (e.g., read-only database users). Finally, rotate critical secrets (e.g., admin credentials) during maintenance windows. Use feature flags to enable rotation per secret. Have a rollback plan: if rotation causes issues, you can disable rotation and revert to manual management temporarily. Communicate the rollout to all teams: share the rotation schedule, expected behavior, and incident response procedures. Provide a runbook for handling rotation failures. After rollout, conduct a post-mortem to identify improvements. Also, consider using AWS Secrets Manager's automatic rotation with a custom Lambda that logs all steps to CloudWatch for auditability. Finally, enforce rotation via AWS Config rules: require all secrets to have rotation enabled within 30 days of creation.

# Phase 1: Enable rotation on non-critical secret
aws secretsmanager rotate-secret --secret-id dev/internal-api-key

# Phase 2: Monitor for 1 week
aws cloudwatch get-metric-statistics --namespace AWS/Lambda --metric-name Errors --dimensions Name=FunctionName,Value=rotation-lambda --start-time 2024-01-01T00:00:00 --end-time 2024-01-08T00:00:00 --period 3600 --statistics Sum

# Phase 3: Enable rotation on critical secret with maintenance window
aws secretsmanager rotate-secret --secret-id prod/myapp/db-password --rotation-window-hours 2

# Rollback if needed: disable rotation
aws secretsmanager cancel-rotate-secret --secret-id prod/myapp/db-password

Beyond Passwords: Rotating Other Credential Types

Secrets Manager isn't just for passwords. You can rotate SSH keys, API tokens, OAuth tokens, and even X.509 certificates. For SSH keys, store the private key as the secret and rotate by generating a new key pair and updating authorized_keys on servers. For API tokens, the rotation Lambda must call the third-party API to generate a new token and invalidate the old one. For OAuth tokens, you need to handle refresh tokens and access tokens separately. For certificates, use ACM for automatic renewal, but if you have custom certificate authorities, store the certificate and private key in Secrets Manager and rotate before expiry. Each credential type requires a custom rotation strategy. The key is to understand the lifecycle of the credential: can it be rotated without downtime? Does it support multiple active versions? Always test with the actual service before deploying.

import boto3
import requests

def rotate_api_token(secret_arn):
    client = boto3.client('secretsmanager')
    # Get current token
    current = client.get_secret_value(SecretId=secret_arn)
    old_token = current['SecretString']
    
    # Generate new token via API
    response = requests.post(
        'https://api.example.com/token/refresh',
        headers={'Authorization': f'Bearer {old_token}'},
        json={'grant_type': 'refresh_token'}
    )
    new_token = response.json()['access_token']
    
    # Store new token as AWSPENDING
    client.put_secret_value(
        SecretId=secret_arn,
        SecretString=new_token,
        VersionStages=['AWSPENDING']
    )
    
    # Test new token
    test_response = requests.get(
        'https://api.example.com/me',
        headers={'Authorization': f'Bearer {new_token}'}
    )
    if test_response.status_code != 200:
        raise Exception('New token invalid')
    
    # Finalize
    client.update_secret_version_stage(
        SecretId=secret_arn,
        VersionStage='AWSCURRENT',
        MoveToVersionId=current['VersionId']
    )

The Future: Secrets Manager with IAM Roles Anywhere and EKS

For workloads outside AWS (e.g., on-premises or other clouds), you can use IAM Roles Anywhere to fetch secrets without long-lived credentials. This extends Secrets Manager's rotation benefits to hybrid environments. For EKS, use the Secrets Store CSI Driver to mount secrets as volumes, with automatic rotation via driver's rotation reconciler. This eliminates the need for application code changes. The driver can sync secrets from Secrets Manager to Kubernetes secrets, and pods get updated automatically. However, be aware of latency: the driver polls for changes every few minutes. For critical secrets, use the driver's rotation with a sidecar that forces refresh. Also, consider using External Secrets Operator for more advanced scenarios. The trend is towards zero-touch secret management where applications don't even know secrets exist—they're injected by the platform.

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: aws-secrets
spec:
  provider: aws
  parameters:
    objects: |
      - objectName: "prod/myapp/db-password"
        objectType: "secretsmanager"
        jmesPath:
          - path: "username"
            objectAlias: "db-username"
          - path: "password"
            objectAlias: "db-password"
  secretObjects:
    - secretName: db-secret
      type: Opaque
      data:
        - objectName: "db-username"
          key: username
        - objectName: "db-password"
          key: password