Kustomize: Stop Copy-Pasting YAML — Declarative Config Management That Scales
Kustomize declarative configuration management for Kubernetes: stop templating, start patching.
20+ years shipping production infrastructure and CI/CD at scale. Lessons pulled from things that broke in production.
- ✓Basic Kubernetes YAML syntax
- ✓Understanding of Deployments, Services, ConfigMaps
- ✓Familiarity with kubectl apply
Kustomize lets you define a base Kubernetes config and then layer environment-specific patches without duplicating YAML. Run kubectl kustomize to render the final config. It's built into kubectl since 1.14.
Think of Kustomize like a recipe card. You write the base recipe (ingredients, cook time) once. Then for a dinner party you add a note: 'double the garlic, use red wine instead of white.' For a picnic you note: 'skip the sauce, pack cold.' You never rewrite the whole recipe. Kustomize does that for Kubernetes YAML — you keep one base, and overlays describe the differences.
You've been there: three environments — dev, staging, prod — each with its own copy of the same Deployment YAML. A replica count change means editing three files. A new label? Three files. Someone forgets to sync staging with prod, and now your canary is pointing at the wrong ConfigMap. This is the mess Kustomize was built to clean up. Not by introducing a templating language with loops and conditionals, but by letting you declare what changes per environment as pure YAML patches. No Helm charts, no Go templates, no custom scripts. Just kubectl kustomize . and you get exactly the config you need. After this article, you'll be able to structure a multi-environment Kubernetes repo using Kustomize overlays, apply patches without breaking a sweat, and know exactly when Kustomize is the right tool — and when it's not.
Why Kustomize Exists: The Pain of Copy-Paste YAML
Before Kustomize, teams managed environment-specific configs by duplicating YAML files. Dev, staging, prod — each had its own copy of the same Deployment, Service, ConfigMap. A simple change like adding a label meant editing N files. The result: config drift, human error, and late-night firefights. Helm tried to solve this with templates, but templates introduce complexity: conditionals, loops, and a steep learning curve. Kustomize takes a different approach. It says: keep one base config, and describe only the differences per environment. No templates, no scripting. Just YAML patching. This is declarative configuration management at its purest — you declare what you want, not how to get there.
kustomize build . to see the fully rendered YAML. Pipe it to diff against the previous version in CI to catch silent patch failures.Kustomize Structure: Base + Overlays
The core pattern is a directory tree with a base/ and overlays/ directory. The base contains the canonical YAML for your application — Deployment, Service, ConfigMap — plus a kustomization.yaml that lists them. Each overlay (dev, staging, prod) has its own kustomization.yaml that references the base and adds patches or customizations. This structure mirrors how you think about environments: dev is like prod but with fewer replicas, different resource limits, maybe a different ConfigMap. The overlay only describes the delta. This keeps your base clean and your environments consistent.
namePrefix or nameSuffix, remember that Services select Pods by label. The prefix changes the Service name but not the selector labels. Your Service might point to nothing. Always verify selectors after applying a prefix.Patches: Strategic Merge vs. JSON Patch
Kustomize supports two patch types. patchesStrategicMerge is the most common: you write a partial YAML that gets merged into the target resource. It works like a three-way merge — you only specify the fields you want to change. patchesJson6902 uses JSON Patch (RFC 6902) operations: op: replace, add, remove. Use strategic merge for simple field overrides (replicas, image tag). Use JSON patch when you need to modify lists (like container args) or remove fields. Strategic merge can't delete a field; JSON patch can. Both are declared in kustomization.yaml.
apiVersion, kind, metadata.name. If the name doesn't match, the patch is silently ignored. Always check that your patch's metadata.name matches the base resource name.ConfigMap and Secret Generation
One of Kustomize's killer features is generating ConfigMaps and Secrets from literals or files. Instead of writing a ConfigMap YAML by hand, you declare configMapGenerator with key-value pairs or file references. Kustomize creates the ConfigMap with a content-hash suffix, so any change triggers a rolling update of your Deployments. This solves the classic problem: you update a ConfigMap but Pods don't pick it up because the name hasn't changed. With Kustomize, the name changes automatically when content changes. Same for Secrets. Use behavior: merge to add keys from an overlay, or behavior: replace to completely replace the base ConfigMap.
literals in your repo. Use files or envs pointing to encrypted files (e.g., sops-encrypted). Or use an external secrets operator. secretGenerator with literals in plaintext is a security incident waiting to happen.Kustomize Components: Reusable Patches
Components are a newer feature (stable in Kustomize v4) that let you define reusable patches that can be included by multiple overlays. Think of them as mixins. For example, a component that adds a sidecar container for logging, or a component that attaches a persistent volume. Components are declared in their own directory with a kustomization.yaml using kind: Component. Overlays include them via components: field. This is great for enforcing cross-cutting concerns like monitoring, logging, or security sidecars across all environments.
Kustomize vs. Helm: When to Use Which
This is the debate that won't die. Here's the truth: Kustomize is better when your configs are mostly static with small per-environment variations. It's pure YAML — no learning curve beyond Kubernetes itself. Helm is better when you need to package and distribute an application with configurable parameters, especially if you're publishing a chart for others to use. Helm's templating allows complex logic (loops, conditionals) that Kustomize can't do. But that complexity is also its weakness: Helm charts are harder to debug, and template errors can produce invalid YAML. My rule: if you control the deployment and just need env-specific tweaks, use Kustomize. If you're building a reusable package for a community, use Helm. Or use both — Helm to package, Kustomize to customize per environment.
--set to override values in production. It's not auditable. Instead, commit a values.yaml per environment and use Kustomize to apply environment-specific patches on top of the Helm chart output.Production Patterns: Multi-Tenant and Multi-Cluster
In production, you often need to manage configs for multiple tenants or clusters. Kustomize scales well here. For multi-tenant, create an overlay per tenant with its own namespace, name prefix, and ConfigMap. For multi-cluster, you can have a top-level directory per cluster (us-east, eu-west) with overlays per environment inside. Use kustomize build with --load_restrictor none if you need to reference resources outside the kustomization root (but be careful — this can break reproducibility). Another pattern: use kustomize edit in CI to programmatically modify the kustomization.yaml, then build and apply. This is how you automate canary deployments or feature flags.
namePrefix, remember that metadata.name of all resources gets prefixed, but spec.selector.matchLabels in Deployments and Services do NOT get prefixed. If two overlays share the same base, they might have conflicting selectors. Always set commonLabels in each overlay to include a unique label like tenant: alpha.Kustomize in CI/CD: GitOps and ArgoCD
Kustomize is a natural fit for GitOps workflows. Tools like ArgoCD and Flux natively support Kustomize. You point ArgoCD at a Git repo with a kustomization.yaml, and it automatically syncs the rendered config. This means your Git repo is the single source of truth — no manual kubectl apply. In CI, you can run kustomize build to validate the config and even diff it against the live cluster. Use kustomize build . | kubectl diff -f - to see what would change. This catches errors before they hit production. For multi-environment pipelines, use separate branches or directories. I prefer directories: clusters/prod/, clusters/staging/ — each with its own kustomization.yaml that references a shared base.
path: overlays/prod in your Application spec. ArgoCD will run kustomize build and sync the result. No need to commit the rendered YAML.When Kustomize Breaks: Limitations and Workarounds
Kustomize is not a silver bullet. It struggles with complex transformations that require conditionals or loops. For example, you can't say 'if environment is prod, add this sidecar, else skip it' — you'd need two overlays. Also, Kustomize's patch matching is fragile: it matches on apiVersion, kind, and name. If you rename a resource in the base, all patches break silently. Another pain point: patchesStrategicMerge can't delete a field; you need JSON patch for that. And JSON patch with array indices is brittle. For very complex configs, consider using kustomize with helm via helm template piped into kustomize. Or use a tool like jsonnet or yq for heavy lifting. Know when to walk away.
Debugging Kustomize: Common Errors and Fixes
The most common error: error: json: unknown field "patchesStrategicMerge" — you're using an old Kustomize version. Upgrade to v4+. Another: error: no matches for Id — your patch target doesn't match any resource. Check apiVersion, kind, and name. Use kustomize build to see the rendered resources and their names. If you see error: accumulating resources: accumulation err='...', it's usually a YAML syntax error in one of your files. Run yamllint on all YAML files. For namePrefix issues, use kustomize build and grep for the prefix to verify it's applied. And always, always run kustomize build before applying — never apply a kustomization without reviewing the output.
validate: kustomize build overlays/prod > /dev/null && yamllint overlays/*/.yaml. Run it in CI before every deploy.The 4GB Container That Kept Dying
resources.requests.memory: 512Mi and limits.memory: 1Gi. The prod overlay was supposed to bump limits to 4Gi but the patch used patchesStrategicMerge with an incorrect indentation — it never applied. The pod ran with 1Gi limit and OOM'd under load.kustomize build step in CI that diffs the rendered output against expected resource limits. Now every PR shows the final memory limit in the diff.- Always render and review the final Kustomize output before applying.
- A mis-indented patch silently fails.
kustomize build overlays/prod | grep -A 10 'resources:' to verify limits are applied. 2. Check for indentation errors in patch YAML. 3. Ensure patch metadata.name matches base resource name.kubectl get svc -n <namespace> and note the Service name. 2. Run kubectl get pods -n <namespace> --show-labels and check if Pod labels match Service selector. 3. Add commonLabels to overlay to ensure label consistency.configMapGenerator (has hash suffix). 2. Verify Deployment references the ConfigMap by name (not hardcoded). 3. Run kustomize build and confirm the ConfigMap name changed.kustomize build overlays/prod | grep -A 5 'kind: Deployment'cat overlays/prod/deployment_patch.yamlmetadata.name in patch matches base resource name (including any namePrefix).| File | Command / Code | Purpose |
|---|---|---|
| base | apiVersion: kustomize.config.k8s.io/v1beta1 | Why Kustomize Exists |
| overlays | apiVersion: kustomize.config.k8s.io/v1beta1 | Kustomize Structure |
| overlays | apiVersion: apps/v1 | Patches |
| overlays | apiVersion: kustomize.config.k8s.io/v1beta1 | ConfigMap and Secret Generation |
| components | apiVersion: kustomize.config.k8s.io/v1alpha1 | Kustomize Components |
| overlays | apiVersion: kustomize.config.k8s.io/v1beta1 | Production Patterns |
| ci-pipeline.sh | set -euo pipefail | Kustomize in CI/CD |
| workaround-json-patch.yaml | patchesJson6902: | When Kustomize Breaks |
| debug-commands.sh | kustomize build overlays/prod | less | Debugging Kustomize |
Key takeaways
kustomize build and review the output before applying. A mis-indented patch silently fails.configMapGenerator and secretGenerator to get automatic rolling updates when config changes.Interview Questions on This Topic
How does Kustomize handle resource ordering when applying multiple resources?
kubectl apply with multiple files or a tool like kapp.Frequently Asked Questions
20+ years shipping production infrastructure and CI/CD at scale. Lessons pulled from things that broke in production.
That's Kubernetes. Mark it forged?
4 min read · try the examples if you haven't