DevOps Advanced

Kubernetes ConfigMaps and Secrets: Internals, Pitfalls, and Production Patterns

📅 March 06, 2026 ⏱ 3 min read 🎯 Advanced

Where developers are forged. · Structured learning · Free forever.

📍 Part of: Kubernetes → Topic 4 of 12

Kubernetes ConfigMaps and Secrets deep-dive: how they're stored in etcd, mounted vs env-var tradeoffs, rotation strategies, and production gotchas you won't find elsewhere.

🔥 Advanced — solid DevOps foundation required

In this tutorial, you'll learn

Kubernetes ConfigMaps and Secrets deep-dive: how they're stored in etcd, mounted vs env-var tradeoffs, rotation strategies, and production gotchas you won't find elsewhere.

ConfigMaps and Secrets decouple configuration from container images, but they have fundamentally different security profiles.
Base64 is not encryption. Enable envelope encryption for Secrets at rest. Use external secret stores for mature environments.
Volume mounts support live rotation; env vars do not. Design your application to re-read mounted files.

✦ Plain-English analogy ✦ Real code with output ✦ Interview questions

⚡Quick Answer

ConfigMap: Stores non-sensitive key-value pairs (feature flags, URLs, config files).
Secret: Stores sensitive data (passwords, tokens, TLS certs). Values are base64-encoded, NOT encrypted by default.
Both can be injected as environment variables OR mounted as files into a pod.
Storage: Both are stored in etcd. Secrets have an additional encryption-at-rest layer (envelope encryption) that must be explicitly enabled.
Env vars: Simple, but require pod restart for updates. Visible in pod spec and process environment.
Volume mounts: Support live rotation without restart. File permissions can be restricted.
Assuming base64 encoding equals encryption. It does not. Anyone with API access can decode a Secret. Envelope encryption must be configured separately via EncryptionConfiguration.

🚨 START HERE

ConfigMap and Secret Triage Commands

Rapid commands to isolate configuration and secret issues.

🟡Pod cannot find a ConfigMap or Secret.

Immediate ActionVerify existence and namespace.

Commands

kubectl get configmap <name> -n <namespace>

kubectl get secret <name> -n <namespace> -o yaml

Fix NowIf missing, create it. If in wrong namespace, move the resource or update the pod spec.

🟡Secret value appears wrong or empty.

Immediate ActionDecode and inspect the Secret data.

Commands

kubectl get secret <name> -o jsonpath='{.data.<key>}' | base64 -d

kubectl describe secret <name>

Fix NowIf the decoded value is wrong, update the Secret. If empty, check the key name in both the Secret and the pod spec.

🟡Mounted ConfigMap file not updating after ConfigMap change.

Immediate ActionCheck kubelet sync delay and mount type.

Commands

kubectl exec <pod> -- cat /path/to/config

kubectl get configmap <name> -o yaml | grep -A 5 data

Fix NowWait up to 60s for kubelet sync. If still stale, check if the mount uses subPath (subPath mounts do NOT auto-update).

🟡Application logs show 'access denied' to Secret file.

Immediate ActionCheck file permissions and security context.

Commands

kubectl exec <pod> -- ls -la /path/to/secret

kubectl get pod <pod> -o jsonpath='{.spec.securityContext}'

Fix NowSet `defaultMode: 0440` on the Secret volume. Ensure `runAsUser` matches the application's expected UID.

Production IncidentSecret Rotation Failure Causes Database Connection StormAfter rotating a database password stored in a Kubernetes Secret, all application pods simultaneously lost database connectivity and entered CrashLoopBackOff, causing a 14-minute outage.

SymptomAll application pods restarted simultaneously after a Secret update. Pods logged 'authentication failed' errors. Database connection pool exhausted. Customer-facing 500 errors spiked.

AssumptionThe new password in the Secret was incorrect or the database had not been updated yet.

Root causeThe database password was injected as an environment variable. Kubernetes does not update environment variables in running pods when a Secret changes — it only updates mounted volume files. The ops team updated the Secret in Kubernetes AND the database simultaneously. However, the pods were not restarted, so they kept using the old password from their environment. When the pods were eventually restarted (either manually or by a deployment trigger), all pods restarted at once because there was no rolling restart strategy, causing a connection storm against the database.

Fix1. Switched from env-var injection to volume mount for the Secret containing the database password. The kubelet updates mounted Secret files within a configurable sync interval (default ~60 seconds). 2. Implemented application-level graceful credential reload (watching the mounted file for changes) instead of reading env vars once at startup. 3. Added a rolling restart strategy using kubectl rollout restart deployment with maxUnavailable: 1 to avoid simultaneous restarts. 4. Configured the application's connection pool to retry authentication with exponential backoff.

Key Lesson

Env vars are a snapshot at pod start time. They do NOT update when the source Secret changes.Volume-mounted Secrets support live rotation, but the application must be designed to re-read the file.Never restart all pods simultaneously during credential rotation. Use rolling restarts.Test your rotation runbook in staging before executing in production.

Production Debug GuideSymptom-first investigation path for configuration and credential failures in Kubernetes.

Pod fails to start with 'ConfigMap/Secret not found' error.→Verify the ConfigMap/Secret exists in the SAME namespace as the pod. Check for typos in the name. Ensure it was created before the pod was scheduled.

Pod starts but application reads stale configuration values.→Check if config is injected as env var (stale) or volume mount (potentially live). If mounted, check kubelet sync interval. If env var, pod restart is required.

Secret values visible in kubectl describe pod output.→This is expected for env-var-injected Secrets. The value is base64-decoded and shown in the pod spec. Switch to volume mount to limit exposure. Review RBAC to restrict who can describe pods.

Pod gets 'permission denied' when reading a mounted Secret file.→Check the defaultMode field in the volume mount. Verify the pod's securityContext.runAsUser can read the file. Default mode is 0644 but can be overridden.

etcd storage growing unexpectedly large.→Check for large ConfigMaps/Secrets (e.g., binary data, large JSON blobs). Use kubectl get secret <name> -o json | wc -c to measure size. Consider external secret stores for large payloads.

Every production Kubernetes cluster eventually hits the same wall: the app image is beautifully immutable, but the configuration it needs — database URLs, feature flags, TLS certificates, API keys — changes constantly across environments. Baking config into the image means rebuilding for a one-line change. Passing it ad-hoc at runtime means no audit trail and no consistency across hundreds of pods.

Kubernetes solves this with two first-class API objects: ConfigMaps for ordinary configuration data and Secrets for sensitive credentials. Both decouple config from the container image, but they have fundamentally different storage mechanisms, access controls, and risk profiles. Understanding that difference — not just syntactically, but at the etcd and kubelet level — is what separates engineers who use these safely from engineers who accidentally expose credentials in pod specs.

This is not a syntax reference. It is for engineers who need to understand how ConfigMaps and Secrets are persisted, why base64 is NOT encryption, when to mount as a file versus inject as an environment variable and why it matters for rotation, how to enable envelope encryption at rest, and the RBAC patterns that keep least-privilege real in a multi-team cluster.

What is Kubernetes ConfigMaps and Secrets?

Kubernetes ConfigMaps and Secrets are core API objects that externalize configuration from container images. ConfigMaps hold non-sensitive data: environment-specific URLs, feature flags, configuration files. Secrets hold sensitive data: passwords, API tokens, TLS private keys. Both are namespaced resources, stored in etcd, and can be consumed by pods as environment variables or mounted volumes.

io/thecodeforge/kubernetes/configmap-example.yaml · YAML

1234567891011121314151617181920212223242526

# ConfigMap: Non-sensitive configuration
# Package: io.thecodeforge.kubernetes
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
  namespace: production
data:
  DATABASE_HOST: "postgres.internal.svc.cluster.local"
  LOG_LEVEL: "info"
  FEATURE_FLAG_NEW_UI: "true"
binaryData:
  logo.png: "iVBORw0KGgoAAAANSUhEUg..."  # binary data supported
---
# Secret: Sensitive credentials
apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
  namespace: production
type: Opaque  # default type
data:
  username: cG9zdGdyZXM=    # base64-encoded 'postgres'
  password: c3VwZXJzZWNyZXQ=  # base64-encoded 'supersecret'
stringData:
  connection-string: "postgresql://postgres:supersecret@postgres:5432/appdb"  # plain text, auto-encoded

▶ Output

ConfigMap and Secret created in 'production' namespace.

Mental Model

Base64 Is Not Encryption

Think of base64 like putting a letter in an envelope. Anyone who gets the envelope can open it. The security comes from controlling who receives the envelope.

base64 is reversible with a single command: echo <value> | base64 -d.
Security relies on RBAC: who can get or list Secrets.
Encryption at rest (envelope encryption) protects against etcd data theft, not API access.
For real encryption, use external secret stores (Vault, AWS Secrets Manager) with the Secrets Store CSI Driver.

📊 Production Insight

Every Secret in the cluster is readable by any entity with get permission on the secrets resource in that namespace. In practice, this means: cluster admins can see all secrets, CI/CD service accounts often have broad secret access, and compromised pods with mounted service account tokens can potentially read other secrets. Mitigate with: dedicated service accounts per workload, minimal RBAC roles, and disabling automounting of service account tokens (automountServiceAccountToken: false).

🎯 Key Takeaway

ConfigMaps and Secrets solve the configuration management problem by decoupling config from images. But they are not a security boundary. The real security comes from RBAC, network policies, and optionally envelope encryption. Treat Secret access as a privilege, not a default.

Storage Internals: How etcd Handles ConfigMaps and Secrets

Both ConfigMaps and Secrets are stored as key-value pairs in etcd through the API Server. The API Server is the only component that talks to etcd directly. When you create a ConfigMap or Secret, the API Server validates it, serializes it, and writes it to etcd. When a kubelet needs to mount a Secret into a pod, it fetches the data from the API Server (which reads from etcd).

io/thecodeforge/kubernetes/encryption-config.yaml · YAML

12345678910111213141516

# Envelope Encryption Configuration for Secrets at Rest
# This file goes on the API Server as --encryption-provider-config
# Package: io.thecodeforge.kubernetes

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets       # Encrypt Secrets at rest
      - configmaps    # Optionally encrypt ConfigMaps too
    providers:
      - aescbc:       # Local encryption (AES-CBC)
          keys:
            - name: key1
              secret: <base64-encoded-32-byte-key>
      - identity: {}  # Fallback: no encryption (reads old unencrypted data)

▶ Output

API Server configured to encrypt Secrets at rest in etcd.

Mental Model

Envelope Encryption: What It Actually Does

Without envelope encryption, anyone with filesystem access to etcd nodes can read Secrets in plain text.

Default: Secrets are stored in plain text in etcd.
Envelope encryption is opt-in via --encryption-provider-config flag on the API Server.
Key rotation: Add a new key to the config, restart API Server, then run kubectl get secrets -A -o json | kubectl replace -f - to re-encrypt.
Cloud KMS integration (e.g., AWS KMS) means you never handle the KEK directly.

📊 Production Insight

Without envelope encryption, a compromised etcd backup or a disk from a decommissioned etcd node exposes every Secret in the cluster history. This is not theoretical — it is a common finding in security audits. Enable envelope encryption on day one. Use cloud KMS for the KEK so key rotation is handled by the cloud provider. Monitor etcd's etcd_debugging_mvcc_db_total_size_in_bytes metric; large Secrets (e.g., multi-megabyte TLS certs) inflate etcd and degrade performance.

🎯 Key Takeaway

etcd is the persistence layer for all Kubernetes objects, including Secrets. By default, Secrets are stored unencrypted. Envelope encryption with a KMS-backed KEK is the production standard. Treat etcd backups with the same sensitivity as the Secrets themselves.

Env Var vs Volume Mount: The Rotation Trade-off

ConfigMaps and Secrets can be consumed by pods in two ways: as environment variables or as mounted files. This choice has significant implications for secret rotation, visibility, and application architecture.

io/thecodeforge/kubernetes/consumption-patterns.yaml · YAML

12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152

# Pattern 1: Environment Variable Injection (Static)
# Values are set at pod creation time. Do NOT update on Secret change.
apiVersion: v1
kind: Pod
metadata:
  name: app-env-injection
spec:
  containers:
    - name: app
      image: io.thecodeforge/app:1.0
      envFrom:
        - configMapRef:
            name: app-config
        - secretRef:
            name: db-credentials
      env:
        - name: SPECIFIC_KEY
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: password
---
# Pattern 2: Volume Mount (Dynamic Rotation Support)
# Files update when source Secret/ConfigMap changes (kubelet sync interval).
apiVersion: v1
kind: Pod
metadata:
  name: app-volume-mount
spec:
  containers:
    - name: app
      image: io.thecodeforge/app:1.0
      volumeMounts:
        - name: config-volume
          mountPath: /etc/config
          readOnly: true
        - name: secret-volume
          mountPath: /etc/secrets
          readOnly: true
          # subPath mounts do NOT auto-update!
      # subPath example (static, no rotation):
      # - name: secret-volume
      #   mountPath: /etc/secrets/password.txt
      #   subPath: password
  volumes:
    - name: config-volume
      configMap:
        name: app-config
    - name: secret-volume
      secret:
        secretName: db-credentials
        defaultMode: 0440  # read-only for owner and group

▶ Output

Pods configured with both env-var and volume-mount patterns.

Mental Model

The Rotation Problem

This is why volume mounts are the preferred pattern for credentials that rotate frequently.

Env vars: Simple, but require pod restart for updates. Values visible in kubectl describe pod.
Volume mounts: Support live rotation (~60s kubelet sync). Values NOT visible in pod spec.
subPath mounts: Do NOT auto-update. Avoid for rotating secrets.
Application must watch or periodically re-read mounted files to detect changes.

📊 Production Insight

The kubelet's Secret/ConfigMap sync interval is controlled by --sync-frequency (default 1 minute). This means there is up to a 60-second window where a pod has stale credentials after a Secret update. For security-critical rotations (e.g., revoking a compromised key), this delay is unacceptable. In those cases, perform a rolling pod restart immediately after updating the Secret, rather than relying on the kubelet sync.

🎯 Key Takeaway

Choose volume mounts for any credential that rotates. Choose env vars only for truly static configuration. Never use subPath for rotating secrets. Design your application to re-read mounted files, not just read them once at startup.

RBAC and Least Privilege for Secrets

Secret access must be tightly controlled via RBAC. In a multi-team cluster, the default behavior — where any pod's service account can potentially read any Secret in its namespace — is a security anti-pattern.

io/thecodeforge/kubernetes/rbac-secret-isolation.yaml · YAML

12345678910111213141516171819202122232425262728293031323334

# Service Account: Dedicated per workload
apiVersion: v1
kind: ServiceAccount
metadata:
  name: payment-service-account
  namespace: payments
automountServiceAccountToken: false  # Disable unless the app needs API access
---
# Role: Read only specific Secrets
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: secret-reader
  namespace: payments
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["payment-gateway-key", "db-credentials"]  # Only specific secrets
    verbs: ["get"]  # NOT 'list' — prevents enumeration
---
# RoleBinding: Bind SA to Role
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: payment-secret-access
  namespace: payments
subjects:
  - kind: ServiceAccount
    name: payment-service-account
    namespace: payments
roleRef:
  kind: Role
  name: secret-reader
  apiGroup: rbac.authorization.k8s.io

▶ Output

Least-privilege RBAC for Secret access configured.

Mental Model

The Service Account Token Problem

This is why automountServiceAccountToken: false should be your default unless the pod explicitly needs Kubernetes API access.

Default service account has broad permissions in many clusters.
Compromised pod + mounted token = potential Secret enumeration.
Set automountServiceAccountToken: false on all pods that do not need API access.
Use resourceNames in RBAC to restrict which specific Secrets a service account can read.
Avoid list verb on Secrets — it allows enumeration of all Secret names.

📊 Production Insight

In multi-tenant clusters, consider using network policies to block pod-to-API-Server traffic for pods that do not need it. Combine this with OPA/Gatekeeper or Kyverno policies that enforce automountServiceAccountToken: false as a cluster-wide default. Audit RBAC regularly: kubectl auth can-i --list --as=system:serviceaccount:<ns>:<sa> shows exactly what a service account can do.

🎯 Key Takeaway

RBAC is the real security boundary for Secrets. Use dedicated service accounts per workload, restrict to specific Secret names, disable automounting, and never grant list on Secrets unless absolutely necessary. Treat every service account token as a potential attack vector.

External Secret Stores: Beyond Native Kubernetes Secrets

For organizations with mature secret management practices, native Kubernetes Secrets are often insufficient. External secret stores like HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, and Azure Key Vault provide centralized management, automatic rotation, audit logging, and fine-grained access policies that Kubernetes RBAC alone cannot match.

io/thecodeforge/kubernetes/secrets-store-csi.yaml · YAML

12345678910111213141516171819202122232425262728293031323334353637

# Secrets Store CSI Driver: Mount external secrets as volumes
# Requires: secrets-store-csi-driver installed in cluster
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: vault-db-creds
  namespace: production
spec:
  provider: vault  # or aws, gcp, azure
  parameters:
    vaultAddress: "https://vault.internal:8200"
    roleName: "payment-service"
    objects: |
      - objectName: "db-password"
        secretPath: "secret/data/production/db"
        secretKey: "password"
---
# Pod consuming the external secret
apiVersion: v1
kind: Pod
metadata:
  name: payment-service
spec:
  containers:
    - name: app
      image: io.thecodeforge/payment-service:2.1
      volumeMounts:
        - name: secrets-store
          mountPath: /mnt/secrets
          readOnly: true
  volumes:
    - name: secrets-store
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "vault-db-creds"

▶ Output

External secret from Vault mounted into pod via CSI driver.

Mental Model

When to Use External Secret Stores

The decision is not 'native vs external' — it is about where your secret lifecycle management should live.

Native Secrets: Simple, no external dependencies, but limited rotation and audit capabilities.
External stores: Centralized lifecycle, audit logs, automatic rotation, cross-cluster sharing.
Secrets Store CSI Driver: The bridge. Mounts external secrets as files without storing them in etcd.
External Secrets Operator: Alternative approach. Syncs external secrets INTO native Kubernetes Secrets.

📊 Production Insight

The Secrets Store CSI Driver mounts secrets directly from the external store into the pod as files. The secret never touches etcd. This eliminates the etcd encryption concern entirely. However, it adds a runtime dependency: if Vault is down, new pods cannot start. Mitigate with: Vault HA clusters, local caching in the CSI driver (--set syncSecret.enabled=true), and fallback logic in your application.

🎯 Key Takeaway

External secret stores are the production standard for mature organizations. The Secrets Store CSI Driver is the integration layer. The trade-off is operational complexity and a new failure mode (external store availability). Plan for it.

🗂 ConfigMap vs Secret: Key Differences

Understanding the operational and security differences between these two API objects.

Aspect	ConfigMap	Secret
Purpose	Non-sensitive configuration data	Sensitive credentials, keys, tokens
Encoding	Plain text (UTF-8)	Base64-encoded (NOT encrypted by default)
Size Limit	1 MiB per ConfigMap	1 MiB per Secret
Encryption at Rest	Not encrypted by default (can be enabled)	Not encrypted by default (envelope encryption opt-in)
Env Var Injection	Supported (visible in pod spec)	Supported (values visible in `kubectl describe pod`)
Volume Mount	Supported (auto-updates on change)	Supported (auto-updates on change)
subPath Mount	Static (no auto-update)	Static (no auto-update)
RBAC Consideration	Generally low-risk to expose	Critical: restrict `get` and `list` verbs
Use Case	Feature flags, URLs, config files	Passwords, API keys, TLS certs, tokens

🎯 Key Takeaways

ConfigMaps and Secrets decouple configuration from container images, but they have fundamentally different security profiles.
Base64 is not encryption. Enable envelope encryption for Secrets at rest. Use external secret stores for mature environments.
Volume mounts support live rotation; env vars do not. Design your application to re-read mounted files.
RBAC is the real security boundary. Use dedicated service accounts, restrict to specific Secret names, and disable automounting.
Never use subPath for rotating secrets. Never grant list on Secrets unless absolutely required.
Test your rotation runbook. The gap between 'Secret updated' and 'pod using new credentials' is where outages happen.

⚠ Common Mistakes to Avoid

✕Assuming base64 encoding equals encryption. It does not. Anyone with API read access can decode Secrets.

✕Using environment variables for rotating credentials. Env vars do not update when the source Secret changes.

✕Using subPath for Secret volume mounts. subPath mounts do not auto-update when the Secret changes.

✕Granting `list` verb on Secrets. This allows enumeration of all Secret names in a namespace.

✕Not enabling envelope encryption for Secrets at rest in etcd. Unencrypted etcd backups are a security risk.

✕Storing large binary blobs (e.g., ML models, large JSON files) in ConfigMaps. The 1 MiB limit and etcd performance make this impractical.

✕Not setting `automountServiceAccountToken: false` on pods that do not need Kubernetes API access.

✕Restarting all pods simultaneously during Secret rotation, causing connection storms and outages.

✕Hardcoding ConfigMap/Secret names across environments instead of using Kustomize overlays or Helm values.

✕Not testing the Secret rotation runbook in staging before executing in production.

Interview Questions on This Topic

QExplain the difference between ConfigMaps and Secrets at the storage level. Where are they persisted and what are the security implications?
QA team injects database credentials as environment variables. What happens when they rotate the Secret? How would you change the architecture?
QWhat is envelope encryption in Kubernetes and why does it matter for Secrets?
QHow does the kubelet update mounted ConfigMaps and Secrets? What is the sync interval and what are the edge cases?
QDescribe a least-privilege RBAC setup for Secret access in a multi-team cluster.
QWhen would you recommend an external secret store (e.g., Vault) over native Kubernetes Secrets? What are the trade-offs?
QWhat is the subPath problem with Secret mounts and how do you avoid it?
QHow would you design a zero-downtime Secret rotation workflow?

Frequently Asked Questions

What is the difference between ConfigMaps and Secrets in Kubernetes?

ConfigMaps store non-sensitive configuration data (plain text). Secrets store sensitive data (base64-encoded by default, with optional envelope encryption at rest). Both can be injected as environment variables or mounted as volume files. The key difference is security: Secrets have additional RBAC and encryption considerations.

Is base64 encoding in Secrets secure?

No. Base64 is a serialization format, not encryption. Anyone with read access to the Secret resource can decode the values. Security comes from RBAC (who can access the Secret) and optionally from envelope encryption (protecting the data at rest in etcd).

How do I rotate a Secret without downtime?

Mount the Secret as a volume (not an env var). The kubelet will sync the updated file within ~60 seconds. Design your application to re-read the file. For critical rotations, perform a rolling restart of pods after updating the Secret, using kubectl rollout restart deployment with controlled maxUnavailable.

What is envelope encryption and should I enable it?

Envelope encryption encrypts Secret values before writing to etcd, using a local Data Encryption Key (DEK) which is itself encrypted by a Key Encryption Key (KEK). Yes, enable it in production. Without it, Secrets are stored in plain text in etcd. Use a cloud KMS (AWS KMS, GCP KMS) as the KEK provider for automated key management.

When should I use an external secret store instead of native Kubernetes Secrets?

Use external stores when you need: centralized secret management across clusters, automatic rotation policies, audit logging for compliance, or secrets shared across cloud accounts. The Secrets Store CSI Driver or External Secrets Operator are the integration layers. The trade-off is operational complexity and a runtime dependency on the external store.

🔥

Naren Founder & Author

Developer and founder of TheCodeForge. I built this site because I was tired of tutorials that explain what to type without explaining why it works. Every article here is written to make concepts actually click.

About Naren Get in touch

Forged with 🔥 at TheCodeForge.io — Where Developers Are Forged