Advanced 3 min · March 06, 2026

Kubernetes Secret Env Var Snapshot Causes Connection Storm

Secret env vars are snapshots — all pods restarted at once after update, causing DB connection storms and 500s.

N
Naren · Founder
Plain-English first. Then code. Then the interview question.
About
 ● Production Incident 🔎 Debug Guide
⚡Quick Answer
  • ConfigMap: Stores non-sensitive key-value pairs (feature flags, URLs, config files).
  • Secret: Stores sensitive data (passwords, tokens, TLS certs). Values are base64-encoded, NOT encrypted by default.
  • Both can be injected as environment variables OR mounted as files into a pod.
  • Storage: Both are stored in etcd. Secrets have an additional encryption-at-rest layer (envelope encryption) that must be explicitly enabled.
  • Env vars: Simple, but require pod restart for updates. Visible in pod spec and process environment.
  • Volume mounts: Support live rotation without restart. File permissions can be restricted.
  • Assuming base64 encoding equals encryption. It does not. Anyone with API access can decode a Secret. Envelope encryption must be configured separately via EncryptionConfiguration.
Plain-English First

Imagine your app is a coffee machine. The machine itself is the same every time — but the recipe card telling it how strong to brew, and the locked safe holding the Wi-Fi password, those change depending on the kitchen it's in. ConfigMaps are the recipe card: non-sensitive settings anyone can read. Secrets are the locked safe: passwords and keys that only authorized hands should touch. Kubernetes separates these two so you never accidentally display your database password in plain sight on a Post-it note.

Every production Kubernetes cluster eventually hits the same wall: the app image is beautifully immutable, but the configuration it needs — database URLs, feature flags, TLS certificates, API keys — changes constantly across environments. Baking config into the image means rebuilding for a one-line change. Passing it ad-hoc at runtime means no audit trail and no consistency across hundreds of pods.

Kubernetes solves this with two first-class API objects: ConfigMaps for ordinary configuration data and Secrets for sensitive credentials. Both decouple config from the container image, but they have fundamentally different storage mechanisms, access controls, and risk profiles. Understanding that difference — not just syntactically, but at the etcd and kubelet level — is what separates engineers who use these safely from engineers who accidentally expose credentials in pod specs.

This is not a syntax reference. It is for engineers who need to understand how ConfigMaps and Secrets are persisted, why base64 is NOT encryption, when to mount as a file versus inject as an environment variable and why it matters for rotation, how to enable envelope encryption at rest, and the RBAC patterns that keep least-privilege real in a multi-team cluster.

What is Kubernetes ConfigMaps and Secrets?

Kubernetes ConfigMaps and Secrets are core API objects that externalize configuration from container images. ConfigMaps hold non-sensitive data: environment-specific URLs, feature flags, configuration files. Secrets hold sensitive data: passwords, API tokens, TLS private keys. Both are namespaced resources, stored in etcd, and can be consumed by pods as environment variables or mounted volumes.

io/thecodeforge/kubernetes/configmap-example.yamlYAML
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
# ConfigMap: Non-sensitive configuration
# Package: io.thecodeforge.kubernetes
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
  namespace: production
data:
  DATABASE_HOST: "postgres.internal.svc.cluster.local"
  LOG_LEVEL: "info"
  FEATURE_FLAG_NEW_UI: "true"
binaryData:
  logo.png: "iVBORw0KGgoAAAANSUhEUg..."  # binary data supported
---
# Secret: Sensitive credentials
apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
  namespace: production
type: Opaque  # default type
data:
  username: cG9zdGdyZXM=    # base64-encoded 'postgres'
  password: c3VwZXJzZWNyZXQ=  # base64-encoded 'supersecret'
stringData:
  connection-string: "postgresql://postgres:supersecret@postgres:5432/appdb"  # plain text, auto-encoded
Output
ConfigMap and Secret created in 'production' namespace.
Base64 Is Not Encryption
  • base64 is reversible with a single command: echo <value> | base64 -d.
  • Security relies on RBAC: who can get or list Secrets.
  • Encryption at rest (envelope encryption) protects against etcd data theft, not API access.
  • For real encryption, use external secret stores (Vault, AWS Secrets Manager) with the Secrets Store CSI Driver.
Production Insight
Every Secret in the cluster is readable by any entity with get permission on the secrets resource in that namespace. In practice, this means: cluster admins can see all secrets, CI/CD service accounts often have broad secret access, and compromised pods with mounted service account tokens can potentially read other secrets. Mitigate with: dedicated service accounts per workload, minimal RBAC roles, and disabling automounting of service account tokens (automountServiceAccountToken: false).
Key Takeaway
ConfigMaps and Secrets solve the configuration management problem by decoupling config from images. But they are not a security boundary. The real security comes from RBAC, network policies, and optionally envelope encryption. Treat Secret access as a privilege, not a default.

Storage Internals: How etcd Handles ConfigMaps and Secrets

Both ConfigMaps and Secrets are stored as key-value pairs in etcd through the API Server. The API Server is the only component that talks to etcd directly. When you create a ConfigMap or Secret, the API Server validates it, serializes it, and writes it to etcd. When a kubelet needs to mount a Secret into a pod, it fetches the data from the API Server (which reads from etcd).

io/thecodeforge/kubernetes/encryption-config.yamlYAML
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# Envelope Encryption Configuration for Secrets at Rest
# This file goes on the API Server as --encryption-provider-config
# Package: io.thecodeforge.kubernetes

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets       # Encrypt Secrets at rest
      - configmaps    # Optionally encrypt ConfigMaps too
    providers:
      - aescbc:       # Local encryption (AES-CBC)
          keys:
            - name: key1
              secret: <base64-encoded-32-byte-key>
      - identity: {}  # Fallback: no encryption (reads old unencrypted data)
Output
API Server configured to encrypt Secrets at rest in etcd.
Envelope Encryption: What It Actually Does
  • Default: Secrets are stored in plain text in etcd.
  • Envelope encryption is opt-in via --encryption-provider-config flag on the API Server.
  • Key rotation: Add a new key to the config, restart API Server, then run kubectl get secrets -A -o json | kubectl replace -f - to re-encrypt.
  • Cloud KMS integration (e.g., AWS KMS) means you never handle the KEK directly.
Production Insight
Without envelope encryption, a compromised etcd backup or a disk from a decommissioned etcd node exposes every Secret in the cluster history. This is not theoretical — it is a common finding in security audits. Enable envelope encryption on day one. Use cloud KMS for the KEK so key rotation is handled by the cloud provider. Monitor etcd's etcd_debugging_mvcc_db_total_size_in_bytes metric; large Secrets (e.g., multi-megabyte TLS certs) inflate etcd and degrade performance.
Key Takeaway
etcd is the persistence layer for all Kubernetes objects, including Secrets. By default, Secrets are stored unencrypted. Envelope encryption with a KMS-backed KEK is the production standard. Treat etcd backups with the same sensitivity as the Secrets themselves.

Env Var vs Volume Mount: The Rotation Trade-off

ConfigMaps and Secrets can be consumed by pods in two ways: as environment variables or as mounted files. This choice has significant implications for secret rotation, visibility, and application architecture.

io/thecodeforge/kubernetes/consumption-patterns.yamlYAML
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
# Pattern 1: Environment Variable Injection (Static)
# Values are set at pod creation time. Do NOT update on Secret change.
apiVersion: v1
kind: Pod
metadata:
  name: app-env-injection
spec:
  containers:
    - name: app
      image: io.thecodeforge/app:1.0
      envFrom:
        - configMapRef:
            name: app-config
        - secretRef:
            name: db-credentials
      env:
        - name: SPECIFIC_KEY
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: password
---
# Pattern 2: Volume Mount (Dynamic Rotation Support)
# Files update when source Secret/ConfigMap changes (kubelet sync interval).
apiVersion: v1
kind: Pod
metadata:
  name: app-volume-mount
spec:
  containers:
    - name: app
      image: io.thecodeforge/app:1.0
      volumeMounts:
        - name: config-volume
          mountPath: /etc/config
          readOnly: true
        - name: secret-volume
          mountPath: /etc/secrets
          readOnly: true
          # subPath mounts do NOT auto-update!
      # subPath example (static, no rotation):
      # - name: secret-volume
      #   mountPath: /etc/secrets/password.txt
      #   subPath: password
  volumes:
    - name: config-volume
      configMap:
        name: app-config
    - name: secret-volume
      secret:
        secretName: db-credentials
        defaultMode: 0440  # read-only for owner and group
Output
Pods configured with both env-var and volume-mount patterns.
The Rotation Problem
  • Env vars: Simple, but require pod restart for updates. Values visible in kubectl describe pod.
  • Volume mounts: Support live rotation (~60s kubelet sync). Values NOT visible in pod spec.
  • subPath mounts: Do NOT auto-update. Avoid for rotating secrets.
  • Application must watch or periodically re-read mounted files to detect changes.
Production Insight
The kubelet's Secret/ConfigMap sync interval is controlled by --sync-frequency (default 1 minute). This means there is up to a 60-second window where a pod has stale credentials after a Secret update. For security-critical rotations (e.g., revoking a compromised key), this delay is unacceptable. In those cases, perform a rolling pod restart immediately after updating the Secret, rather than relying on the kubelet sync.
Key Takeaway
Choose volume mounts for any credential that rotates. Choose env vars only for truly static configuration. Never use subPath for rotating secrets. Design your application to re-read mounted files, not just read them once at startup.

RBAC and Least Privilege for Secrets

Secret access must be tightly controlled via RBAC. In a multi-team cluster, the default behavior — where any pod's service account can potentially read any Secret in its namespace — is a security anti-pattern.

io/thecodeforge/kubernetes/rbac-secret-isolation.yamlYAML
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
# Service Account: Dedicated per workload
apiVersion: v1
kind: ServiceAccount
metadata:
  name: payment-service-account
  namespace: payments
automountServiceAccountToken: false  # Disable unless the app needs API access
---
# Role: Read only specific Secrets
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: secret-reader
  namespace: payments
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["payment-gateway-key", "db-credentials"]  # Only specific secrets
    verbs: ["get"]  # NOT 'list' — prevents enumeration
---
# RoleBinding: Bind SA to Role
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: payment-secret-access
  namespace: payments
subjects:
  - kind: ServiceAccount
    name: payment-service-account
    namespace: payments
roleRef:
  kind: Role
  name: secret-reader
  apiGroup: rbac.authorization.k8s.io
Output
Least-privilege RBAC for Secret access configured.
The Service Account Token Problem
  • Default service account has broad permissions in many clusters.
  • Compromised pod + mounted token = potential Secret enumeration.
  • Set automountServiceAccountToken: false on all pods that do not need API access.
  • Use resourceNames in RBAC to restrict which specific Secrets a service account can read.
  • Avoid list verb on Secrets — it allows enumeration of all Secret names.
Production Insight
In multi-tenant clusters, consider using network policies to block pod-to-API-Server traffic for pods that do not need it. Combine this with OPA/Gatekeeper or Kyverno policies that enforce automountServiceAccountToken: false as a cluster-wide default. Audit RBAC regularly: kubectl auth can-i --list --as=system:serviceaccount:<ns>:<sa> shows exactly what a service account can do.
Key Takeaway
RBAC is the real security boundary for Secrets. Use dedicated service accounts per workload, restrict to specific Secret names, disable automounting, and never grant list on Secrets unless absolutely necessary. Treat every service account token as a potential attack vector.

External Secret Stores: Beyond Native Kubernetes Secrets

For organizations with mature secret management practices, native Kubernetes Secrets are often insufficient. External secret stores like HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, and Azure Key Vault provide centralized management, automatic rotation, audit logging, and fine-grained access policies that Kubernetes RBAC alone cannot match.

io/thecodeforge/kubernetes/secrets-store-csi.yamlYAML
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
# Secrets Store CSI Driver: Mount external secrets as volumes
# Requires: secrets-store-csi-driver installed in cluster
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: vault-db-creds
  namespace: production
spec:
  provider: vault  # or aws, gcp, azure
  parameters:
    vaultAddress: "https://vault.internal:8200"
    roleName: "payment-service"
    objects: |
      - objectName: "db-password"
        secretPath: "secret/data/production/db"
        secretKey: "password"
---
# Pod consuming the external secret
apiVersion: v1
kind: Pod
metadata:
  name: payment-service
spec:
  containers:
    - name: app
      image: io.thecodeforge/payment-service:2.1
      volumeMounts:
        - name: secrets-store
          mountPath: /mnt/secrets
          readOnly: true
  volumes:
    - name: secrets-store
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "vault-db-creds"
Output
External secret from Vault mounted into pod via CSI driver.
When to Use External Secret Stores
  • Native Secrets: Simple, no external dependencies, but limited rotation and audit capabilities.
  • External stores: Centralized lifecycle, audit logs, automatic rotation, cross-cluster sharing.
  • Secrets Store CSI Driver: The bridge. Mounts external secrets as files without storing them in etcd.
  • External Secrets Operator: Alternative approach. Syncs external secrets INTO native Kubernetes Secrets.
Production Insight
The Secrets Store CSI Driver mounts secrets directly from the external store into the pod as files. The secret never touches etcd. This eliminates the etcd encryption concern entirely. However, it adds a runtime dependency: if Vault is down, new pods cannot start. Mitigate with: Vault HA clusters, local caching in the CSI driver (--set syncSecret.enabled=true), and fallback logic in your application.
Key Takeaway
External secret stores are the production standard for mature organizations. The Secrets Store CSI Driver is the integration layer. The trade-off is operational complexity and a new failure mode (external store availability). Plan for it.
● Production incidentPOST-MORTEMseverity: high

Secret Rotation Failure Causes Database Connection Storm

Symptom
All application pods restarted simultaneously after a Secret update. Pods logged 'authentication failed' errors. Database connection pool exhausted. Customer-facing 500 errors spiked.
Assumption
The new password in the Secret was incorrect or the database had not been updated yet.
Root cause
The database password was injected as an environment variable. Kubernetes does not update environment variables in running pods when a Secret changes — it only updates mounted volume files. The ops team updated the Secret in Kubernetes AND the database simultaneously. However, the pods were not restarted, so they kept using the old password from their environment. When the pods were eventually restarted (either manually or by a deployment trigger), all pods restarted at once because there was no rolling restart strategy, causing a connection storm against the database.
Fix
1. Switched from env-var injection to volume mount for the Secret containing the database password. The kubelet updates mounted Secret files within a configurable sync interval (default ~60 seconds). 2. Implemented application-level graceful credential reload (watching the mounted file for changes) instead of reading env vars once at startup. 3. Added a rolling restart strategy using kubectl rollout restart deployment with maxUnavailable: 1 to avoid simultaneous restarts. 4. Configured the application's connection pool to retry authentication with exponential backoff.
Key lesson
  • Env vars are a snapshot at pod start time. They do NOT update when the source Secret changes.
  • Volume-mounted Secrets support live rotation, but the application must be designed to re-read the file.
  • Never restart all pods simultaneously during credential rotation. Use rolling restarts.
  • Test your rotation runbook in staging before executing in production.
Production debug guideSymptom-first investigation path for configuration and credential failures in Kubernetes.5 entries
Symptom · 01
Pod fails to start with 'ConfigMap/Secret not found' error.
→
Fix
Verify the ConfigMap/Secret exists in the SAME namespace as the pod. Check for typos in the name. Ensure it was created before the pod was scheduled.
Symptom · 02
Pod starts but application reads stale configuration values.
→
Fix
Check if config is injected as env var (stale) or volume mount (potentially live). If mounted, check kubelet sync interval. If env var, pod restart is required.
Symptom · 03
Secret values visible in kubectl describe pod output.
→
Fix
This is expected for env-var-injected Secrets. The value is base64-decoded and shown in the pod spec. Switch to volume mount to limit exposure. Review RBAC to restrict who can describe pods.
Symptom · 04
Pod gets 'permission denied' when reading a mounted Secret file.
→
Fix
Check the defaultMode field in the volume mount. Verify the pod's securityContext.runAsUser can read the file. Default mode is 0644 but can be overridden.
Symptom · 05
etcd storage growing unexpectedly large.
→
Fix
Check for large ConfigMaps/Secrets (e.g., binary data, large JSON blobs). Use kubectl get secret <name> -o json | wc -c to measure size. Consider external secret stores for large payloads.
★ ConfigMap and Secret Triage CommandsRapid commands to isolate configuration and secret issues.
Pod cannot find a ConfigMap or Secret.−
Immediate action
Verify existence and namespace.
Commands
kubectl get configmap <name> -n <namespace>
kubectl get secret <name> -n <namespace> -o yaml
Fix now
If missing, create it. If in wrong namespace, move the resource or update the pod spec.
Secret value appears wrong or empty.+
Immediate action
Decode and inspect the Secret data.
Commands
kubectl get secret <name> -o jsonpath='{.data.<key>}' | base64 -d
kubectl describe secret <name>
Fix now
If the decoded value is wrong, update the Secret. If empty, check the key name in both the Secret and the pod spec.
Mounted ConfigMap file not updating after ConfigMap change.+
Immediate action
Check kubelet sync delay and mount type.
Commands
kubectl exec <pod> -- cat /path/to/config
kubectl get configmap <name> -o yaml | grep -A 5 data
Fix now
Wait up to 60s for kubelet sync. If still stale, check if the mount uses subPath (subPath mounts do NOT auto-update).
Application logs show 'access denied' to Secret file.+
Immediate action
Check file permissions and security context.
Commands
kubectl exec <pod> -- ls -la /path/to/secret
kubectl get pod <pod> -o jsonpath='{.spec.securityContext}'
Fix now
Set defaultMode: 0440 on the Secret volume. Ensure runAsUser matches the application's expected UID.
ConfigMap vs Secret: Key Differences
AspectConfigMapSecret
PurposeNon-sensitive configuration dataSensitive credentials, keys, tokens
EncodingPlain text (UTF-8)Base64-encoded (NOT encrypted by default)
Size Limit1 MiB per ConfigMap1 MiB per Secret
Encryption at RestNot encrypted by default (can be enabled)Not encrypted by default (envelope encryption opt-in)
Env Var InjectionSupported (visible in pod spec)Supported (values visible in kubectl describe pod)
Volume MountSupported (auto-updates on change)Supported (auto-updates on change)
subPath MountStatic (no auto-update)Static (no auto-update)
RBAC ConsiderationGenerally low-risk to exposeCritical: restrict get and list verbs
Use CaseFeature flags, URLs, config filesPasswords, API keys, TLS certs, tokens

Key takeaways

1
ConfigMaps and Secrets decouple configuration from container images, but they have fundamentally different security profiles.
2
Base64 is not encryption. Enable envelope encryption for Secrets at rest. Use external secret stores for mature environments.
3
Volume mounts support live rotation; env vars do not. Design your application to re-read mounted files.
4
RBAC is the real security boundary. Use dedicated service accounts, restrict to specific Secret names, and disable automounting.
5
Never use subPath for rotating secrets. Never grant list on Secrets unless absolutely required.
6
Test your rotation runbook. The gap between 'Secret updated' and 'pod using new credentials' is where outages happen.
INTERVIEW PREP · PRACTICE MODE

Interview Questions on This Topic

FAQ · 5 QUESTIONS

Frequently Asked Questions

01
What is the difference between ConfigMaps and Secrets in Kubernetes?
02
Is base64 encoding in Secrets secure?
03
How do I rotate a Secret without downtime?
04
What is envelope encryption and should I enable it?
05
When should I use an external secret store instead of native Kubernetes Secrets?
Related Next Steps
→Kubernetes Pods and DeploymentsPod lifecycle, ReplicaSets, rolling updates, rollbacks, resource requests/limits, and health probes.→Kubernetes Network Policies: Zero-Trust Segmentation in ProductionIngress/egress rules, namespace isolation, CNI behaviour, and building a zero-trust network inside Kubernetes.→Helm Charts for KubernetesPackaging, versioning, and deploying Kubernetes apps with Helm — chart structure, values files, and release management.
🔥

That's Kubernetes. Mark it forged?

3 min read · try the examples if you haven't

←
Previous
Kubernetes Services and Ingress
4 / 12 · Kubernetes
Next
Kubernetes StatefulSets
→