Secret Manager: Storage, Rotation, and Integration with Compute/GKE

A fintech startup lost $1.2M when a developer pushed a .env file to a public repo. The API key inside gave attackers full access to their payment processor. This wasn't a hack—it was negligence. Yet most teams still treat secrets as config files. Secret managers exist to make this impossible. They encrypt secrets at rest and in transit, enforce access controls, and rotate credentials automatically. If you're not using one, you're one commit away from a breach. This article covers storage, rotation, and integration with compute and GKE—the three pillars of production secret management.

What is a Secret Manager?

A secret manager is a dedicated service for storing and managing sensitive data separately from application code and configuration. Unlike environment variables or config files, secret managers provide encryption, access control, audit logging, and automatic rotation. They are essential for compliance with standards like SOC 2, PCI-DSS, and HIPAA. In production, you should never store secrets in code, config maps, or even in CI/CD variables—always use a secret manager.

gcloud secrets create my-api-key --replication-policy="automatic"
echo -n "sk-1234abcd" | gcloud secrets versions add my-api-key --data-file=-

Secret Storage: Encryption at Rest and in Transit

Secrets must be encrypted at rest using AES-256 or similar, and in transit via TLS. Most cloud secret managers (AWS Secrets Manager, GCP Secret Manager, Azure Key Vault) handle this automatically. However, you must ensure that encryption keys are managed separately—ideally using a hardware security module (HSM) or a key management service (KMS). Never store encryption keys in the same system as the secrets. For on-premise solutions like HashiCorp Vault, you need to configure encryption manually using a seal mechanism (e.g., AWS KMS or Azure Key Vault as the seal).

seal "awskms" {
  region     = "us-west-2"
  kms_key_id = "arn:aws:kms:us-west-2:123456789012:key/abc123"
}

storage "consul" {
  address = "127.0.0.1:8500"
  path    = "vault/"
}

listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_disable = false
  tls_cert_file = "/etc/vault/certs/cert.pem"
  tls_key_file  = "/etc/vault/certs/key.pem"
}

Access Control: Who Can Read What?

Access control is the most critical part of secret management. Use IAM roles or policies to grant least-privilege access. For example, in GCP, you can grant roles/secretmanager.secretAccessor to a service account. Never use broad roles like roles/editor. For HashiCorp Vault, use policies with path-based restrictions. Always audit access logs—unexpected access patterns often indicate a breach. Implement time-bound access for emergency scenarios (break-glass).

path "secret/data/myapp/*" {
  capabilities = ["read"]
}

path "secret/data/myapp/db-password" {
  capabilities = ["read", "list"]
}

path "sys/leases/revoke" {
  capabilities = ["sudo"]
}

Secret Rotation: Why and How

Secrets must be rotated regularly to limit the window of exposure if a secret is compromised. Rotation can be manual or automatic. Manual rotation is error-prone and often forgotten. Automatic rotation is preferred: set a schedule (e.g., every 90 days) and use a rotation function that generates a new secret, updates the target service, and deactivates the old version. Cloud secret managers often provide built-in rotation for common services like RDS. For custom secrets, you need to implement a rotation lambda or cloud function.

import boto3
import json

def rotate_secret(event, context):
    client = boto3.client('secretsmanager')
    secret_id = event['SecretId']
    
    # Generate new password
    new_password = secrets.generate_random_password()
    
    # Update the secret
    client.put_secret_value(
        SecretId=secret_id,
        SecretString=json.dumps({'password': new_password}),
        VersionStages=['AWSCURRENT']
    )
    
    # Update the database (example)
    rds = boto3.client('rds')
    rds.modify_db_instance(
        DBInstanceIdentifier='mydb',
        MasterUserPassword=new_password
    )
    
    return {'status': 'rotated'}

Integrating Secret Manager with Compute (GCE)

On Google Compute Engine, the recommended way to access secrets is via the Secret Manager API using a service account attached to the VM. Never store secrets on the VM's disk or in environment variables. Use the Google Cloud SDK or client libraries to fetch secrets at startup or on-demand. For high-performance scenarios, cache the secret in memory with a TTL. Ensure the VM's service account has roles/secretmanager.secretAccessor.

from google.cloud import secretmanager

def access_secret(project_id, secret_id, version_id="latest"):
    client = secretmanager.SecretManagerServiceClient()
    name = f"projects/{project_id}/secrets/{secret_id}/versions/{version_id}"
    response = client.access_secret_version(request={"name": name})
    return response.payload.data.decode('UTF-8')

# Usage
api_key = access_secret("my-project", "my-api-key")
print(f"API key: {api_key}")

Integrating Secret Manager with GKE

In GKE, the best practice is to use the Secret Manager CSI driver or the Workload Identity with the Secret Manager API. The CSI driver mounts secrets as volumes, so pods can read them as files. Workload Identity allows pods to authenticate as a service account and call the API directly. Avoid using Kubernetes Secrets for sensitive data—they are only base64-encoded, not encrypted. For production, always use external secret management.

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
  - name: my-container
    image: nginx
    volumeMounts:
    - name: secrets
      mountPath: "/etc/secrets"
      readOnly: true
  volumes:
  - name: secrets
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: "my-secret-provider"

Secret Manager with Workload Identity

Workload Identity allows GKE pods to impersonate a Google service account. This is the most secure way to access Secret Manager from GKE because it avoids static credentials. You create a Kubernetes service account, bind it to a Google service account, and grant that Google service account access to secrets. Pods then authenticate automatically via the node's metadata server.

# Create Google service account
gcloud iam service-accounts create my-app-sa --display-name="My App SA"

# Grant secret access
gcloud secrets add-iam-policy-binding my-api-key \
    --member="serviceAccount:my-app-sa@project.iam.gserviceaccount.com" \
    --role="roles/secretmanager.secretAccessor"

# Create Kubernetes service account
kubectl create serviceaccount my-app-ksa

# Annotate KSA with GSA
kubectl annotate serviceaccount my-app-ksa \
    iam.gke.io/gcp-service-account=my-app-sa@project.iam.gserviceaccount.com

# Create pod with KSA
kubectl run my-pod --image=nginx --serviceaccount=my-app-ksa

Auditing and Monitoring Secret Access

Auditing is crucial for detecting unauthorized access and for compliance. Enable audit logs for your secret manager. In GCP, this is done via Cloud Audit Logs. Monitor for unusual patterns: access from unexpected IPs, high frequency of reads, or access to secrets that are not used by the caller. Set up alerts for these patterns. Also, regularly review who has access to secrets and remove unused permissions.

# Query audit logs for secret access
gcloud logging read "resource.type=secretmanager.googleapis.com/Secret AND protoPayload.methodName=google.cloud.secretmanager.v1.SecretManagerService.AccessSecretVersion" --limit 10

Disaster Recovery: Secret Backup and Restore

Secrets are critical for system recovery. If your secret manager goes down, you need a backup plan. For cloud secret managers, enable cross-region replication. For Vault, use a secondary cluster with replication. Regularly export secrets to an encrypted backup (e.g., using Vault's raft snapshot or cloud export). Test restoration procedures. Remember: if you lose your secrets, you lose access to your infrastructure.

# Take a snapshot of Vault storage (Consul backend)
consul snapshot save /tmp/vault-backup.snap

# Encrypt the snapshot
gpg --symmetric --cipher-algo AES256 /tmp/vault-backup.snap

# Upload to cloud storage
gsutil cp /tmp/vault-backup.snap.gpg gs://my-backup-bucket/vault/

Cost Considerations and Performance

Secret managers charge per secret and per API call. For high-throughput applications, API calls can become expensive. Cache secrets in memory with a TTL to reduce calls. For example, cache for 1 hour and refresh in the background. Also, batch secret retrieval if possible. On GCP, Secret Manager costs $0.06 per secret per month and $0.03 per 10,000 access operations. For 100 secrets accessed 1000 times per hour, that's ~$260/month. Consider using a local cache or a sidecar proxy to reduce costs.

import time
from google.cloud import secretmanager

class SecretCache:
    def __init__(self, ttl=3600):
        self.cache = {}
        self.ttl = ttl
        self.client = secretmanager.SecretManagerServiceClient()
    
    def get(self, project_id, secret_id):
        now = time.time()
        key = f"{project_id}/{secret_id}"
        if key in self.cache and now - self.cache[key]['timestamp'] < self.ttl:
            return self.cache[key]['value']
        name = f"projects/{project_id}/secrets/{secret_id}/versions/latest"
        response = self.client.access_secret_version(request={"name": name})
        value = response.payload.data.decode('UTF-8')
        self.cache[key] = {'value': value, 'timestamp': now}
        return value

Multi-Cloud and Hybrid Secret Management

If you run workloads across multiple clouds or on-premises, you need a unified secret management strategy. HashiCorp Vault is the most popular choice for multi-cloud. It can replicate secrets across regions and clouds. Alternatively, use a cloud-agnostic approach with a sidecar that fetches secrets from the local cloud's secret manager. Avoid vendor lock-in by abstracting secret access behind an interface.

path "secret/*" {
  capabilities = ["read", "list"]
}

# Enable replication
vault write -f sys/replication/performance/primary/enable

Common Pitfalls and How to Avoid Them

Common mistakes include: storing secrets in environment variables, using Kubernetes Secrets without encryption, not rotating secrets, over-permissioning service accounts, and not auditing access. To avoid these: always use a secret manager, enable encryption at rest and in transit, implement least-privilege IAM, automate rotation, and monitor access logs. Also, never log secrets—use structured logging and redact sensitive fields.

import logging
import re

class SecretRedactingFilter(logging.Filter):
    def filter(self, record):
        # Redact common secret patterns
        record.msg = re.sub(r'(api_key|password|secret)=[^&\s]+', r'\1=***', record.msg)
        return True

logger = logging.getLogger(__name__)
logger.addFilter(SecretRedactingFilter())

Gradual Secret Rollout: Binding to Specific Versions Instead of Latest

Using the 'latest' alias for secret version resolution is convenient but dangerous in production. A bad secret value pushed to latest can immediately break all services consuming it. Instead, bind your application to a specific secret version resolved at deployment time. During rotation, resolve the new version name (e.g., 'projects/my-project/secrets/my-key/versions/5'), store it in your deployment configuration, and roll it out gradually. This enables canary deployments, automatic rollback by reverting the config, and prevents a bad rotation from causing a service-wide outage. For applications that need dynamic updates, resolve 'latest' at startup only and cache it for the application's lifetime—this avoids continuous polling while still picking up rotations on restart. Avoid continuous polling approaches (resolving 'latest' on every request) as they risk immediate, non-gradual exposure to bad secrets.

from google.cloud import secretmanager

def access_secret_version(project_id, secret_id, version_id):
    """Access a specific version of a secret, not 'latest'."""
    client = secretmanager.SecretManagerServiceClient()
    name = f"projects/{project_id}/secrets/{secret_id}/versions/{version_id}"
    response = client.access_secret_version(request={"name": name})
    return response.payload.data.decode('UTF-8')

# In deployment config, store the resolved version:
# DB_PASSWORD_VERSION=5
# Then at startup:
db_password = access_secret_version(
    "my-project", "db-password",
    os.environ.get("DB_PASSWORD_VERSION", "latest")
)

Setting Up Rotation Schedules with Pub/Sub Notifications

Secret Manager supports automatic rotation schedules using Pub/Sub notifications. Configure a rotation period (e.g., 30 days) and a Pub/Sub topic on your secret. When rotation is due, Secret Manager publishes a message to the topic. A Cloud Run function or Cloud Function subscribes to this topic, generates a new secret value, updates the external service (e.g., database), and creates a new secret version. The rotation function must be reentrant—able to restart from where it left off if interrupted. Use secret labels to track rotation state (e.g., ROTATING_TO_NEW_VERSION_NUMBER=3) and etags for optimistic concurrency control to prevent concurrent rotation executions from conflicting. After rotation, gradually roll out the new version, disable the old version after verification, then destroy it. The default Cloud Run service account has Editor role—create a custom service account with minimal Secret Manager roles (e.g., secretmanager.secretVersionAdder) following least-privilege principles.

# Create a Pub/Sub topic for rotation notifications
gcloud pubsub topics create my-secret-rotation

# Create a secret with rotation schedule
gcloud secrets create my-db-password \
  --replication-policy="automatic" \
  --rotation-period="7776000s" \
  --next-rotation-time="2026-08-11T00:00:00Z" \
  --topics="my-secret-rotation"

# Label the secret for rotation tracking
gcloud secrets update my-db-password \
  --update-labels="rotation_status=pending"

# View the rotation config
gcloud secrets describe my-db-password --format="json(rotation,topics)"