Helm Upgrade Deleted Production ConfigMap — Ownership

Kubernetes solves the hard problem of running containers at scale, but it hands you a new problem in return: you're now managing dozens of YAML files per application, each tightly coupled to an environment, a team convention, or a one-off config decision made eighteen months ago by someone who has since left. At five services this is annoying. At fifty, it's a liability. Helm exists precisely because the Kubernetes community hit this wall and needed a package manager — the same reason Linux got apt and Node got npm.

Helm solves three distinct problems in one tool. First, it packages all Kubernetes manifests for an application into a versioned, distributable artifact. Second, it gives you a templating layer so a single chart can target dev, staging, and production with different values rather than copy-pasted YAML. Third, it tracks release state in the cluster itself, enabling atomic upgrades and rollbacks without external tooling. These three capabilities together are why Helm became the de-facto standard for application delivery on Kubernetes.

By the end of this article you'll understand how Helm's rendering pipeline actually works under the hood, how to write charts that are safe for production (not just tutorials), how hooks let you orchestrate pre- and post-install logic, and where Helm's design choices create real operational risk if you're not paying attention. We'll build a realistic multi-environment chart from scratch, examine the Release object internals, and walk through the patterns that separate hobby charts from charts you'd trust in a regulated, high-availability environment.

Chart Structure and Anatomy

A Helm chart is a directory with a standardised layout. At the root you'll find Chart.yaml (metadata), values.yaml (default configuration), and a templates/ folder containing Go template files that expand to Kubernetes manifests. Optional directories include charts/ (for dependencies) and crds/ (for CRDs that must be installed before the templates).

Understanding this structure is the first step to writing production-grade charts. Every file has a purpose: Chart.yaml holds version, appVersion, and dependencies; values.yaml defines the config surface your team will override; and the templates/ folder contains the logic that turns those values into Deployments, Services, and everything else. The secret to maintainable charts is keeping templates thin and making values.yaml expressive. Don't hide logic in templates — expose it as knobs in values.yaml and document each one.

The chart version is separate from the application version. Use semantic versioning for the chart and appVersion for the underlying software. Helm's release system uses the chart version to determine upgrade paths, so bumping the chart version even for trivial changes is critical for traceability.

# TheCodeForge - Sample Chart.yaml
apiVersion: v2
name: myapp
description: A production-grade web application
version: 1.2.0
appVersion: "3.0.1"
type: application
dependencies:
  - name: redis
    version: "~17.0.0"
    repository: "https://charts.bitnami.com/bitnami"
    condition: redis.enabled
  - name: postgresql
    version: "^12.x"
    repository: "https://charts.bitnami.com/bitnami"
    condition: postgresql.enabled

Helm Chart Directory Structure Visual

A clear mental model of the chart directory tree is essential for debugging and team collaboration. Below is the canonical directory layout for a Helm chart named mychart. Every production chart should follow this structure strictly — deviations confuse operators and break dependency management.

The root directory contains three mandatory items: Chart.yaml (chart metadata), values.yaml (default configuration values), and the templates/ directory (Kubernetes manifest templates). Optionally, a charts/ directory holds packaged subchart dependencies (downloaded via helm dependency update), a crds/ directory for CustomResourceDefinitions (installed before templates), and a README.md for documentation. The templates/ directory may contain a _helpers.tpl file for reusable template functions and tests/ for test manifests (run with helm test).

Understanding this layout at a glance helps you quickly locate the source of a misconfigured resource: if a ConfigMap is missing, check whether the template file exists under templates/ and whether it's gated by an if condition in the template logic. If a dependency isn't installed, check charts/ or Chart.yaml dependencies block.

Template Syntax (Go Templates) Basics

Helm uses Go's text/template engine to turn template files into manifest YAML. If you've never written a Go template, the syntax can look cryptic. This section gives you the foundational constructs you need to read and write chart templates confidently.

The core syntax is {{ ... }} — anything inside double curly braces is evaluated as a Go template expression. Plain text outside the braces is output as-is. The dot (.) represents the current context, which is the root of the merged values object at the top level. You access values with .Values.<key>. For example, {{ .Values.replicaCount }} outputs the replica count from values.yaml.

Variables: assign values with $var :=. Useful to cache complex expressions: {{ $port := .Values.service.port | default 8080 }}.

Functions: Helm includes all Sprig functions (e.g., upper, default, quote, toYaml, b64enc). The pipe syntax chains functions: {{ .Values.name | upper | quote }}.

The include function is critical for reusing template blocks. It takes a template name and a context: {{ include "mychart.labels" . }} renders the _helpers.tpl block called "mychart.labels" with the current dot.

Escaping and whitespace: {{- trims leading whitespace; -}} trims trailing whitespace. Always use {{- and -}} inside control structures to avoid blank lines in rendered YAML.

Built-in objects: Besides .Values, Helm provides .Release (Name, Namespace, Service, Revision), .Chart (metadata from Chart.yaml), .Files (access to chart files outside templates), and .Capabilities (cluster API versions).

# TheCodeForge - Go Template syntax examples in Helm
apiVersion: v1
kind: ConfigMap
metadata:
  # include a helper template, trim leading whitespace
  name: {{- include "mychart.fullname" . }}-config
  labels:
    {{- include "mychart.labels" . | nindent 4 }}
data:
  # default function provides fallback
  app_port: "{{ .Values.service.port | default 8080 | quote }}"
  # range loop over a list
  allowed_origins: |
    {{- range .Values.allowedOrigins }}
    - {{ . | quote }}
    {{- end }}
  # with scoping — . refers to .Values.database inside
  {{- with .Values.database }}
  db_host: {{ .host | required "db.host is required" | quote }}
  db_port: {{ .port | default 5432 | quote }}
  {{- end }}
  # inline if condition
  feature_debug: {{ .Values.debug | default false | quote }}

Templating Engine Internals: How Values and Templates Merge

Helm uses Go's text/template engine with an extended function set from Sprig and a few Helm-specific functions (include, required, toYaml, etc.). The rendering pipeline works in three phases: parse, evaluate, and produce YAML.

1. Parse: Helm reads all template files (.yaml, .tpl) from the templates/ directory and the values from values.yaml + any --set or --values flags. Values are merged in order: built-in defaults, values.yaml, parent chart values, user-supplied values. Later overrides win.
2. Evaluate: Go templates are executed. The context (dot) starts as the root of the merged values object. You access values with .Values.someKey. Nested keys use dot notation. Templates can use control structures (if/else, range, with), variables ($val := .Values.someKey), and pipelines. The 'required' function is a common safety net: required "Port is mandatory" .Values.service.port.
3. Produce: The evaluated templates are concatenated into a single YAML document. If an evaluated template produces multiple YAML documents (separated by ---), each is treated as a separate resource. Validation then runs against the cluster's API before anything is applied.

A common trap: scope. The 'range' instruction inside a template changes the dot to the current iteration item. If you need to access the top-level Values inside a range, save it beforehand with $root := ..

# TheCodeForge - A well-structured ConfigMap template
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ include "myapp.fullname" . }}-config
  labels:
    {{- include "myapp.labels" . | nindent 4 }}
data:
  # Tolerates missing value by falling back to a sensible default
  app_port: "{{ .Values.service.port | default 8080 | quote }}"
  log_level: "{{ .Values.logging.level | default "info" }}"
  feature_flags: |
    {{- .Values.featureFlags | toYaml | nindent 4 }}

Release Lifecycle: Install, Upgrade, Rollback, Delete

Helm tracks releases via Secrets in the target namespace (v3). Each release revision gets a Secret named sh.helm.release.v1.<RELEASE-NAME>.v<REVISION>. The Secret contains the rendered manifests, values, and metadata. This in-cluster storage is the source of truth for all Helm operations.

• Install: Creates the first revision. Helm renders templates and applies them to the cluster. If any resource fails, the install is rolled back (the release is marked FAILED, not deleted). You can then inspect the failure and retry.
• Upgrade: Creates a new revision. By default, Helm does a three-way strategic merge patch: it compares the current live state, the previous release state, and the desired state. This allows it to detect and preserve manual changes (but also creates risk if resources are removed from templates — see production incident).
• Rollback: Creates a new revision that is a copy of a previous revision's manifests. Rollback is itself an upgrade — it increments the revision number. This means you can rollback a rollback.
• Delete: Removes the release record and deletes all resources that were created during the install. By default, the history is preserved so you can still see release names. Pass '--keep-history' to retain release secrets for audit purposes.

Understanding the upgrade strategy is vital. The three-way merge is powerful but can produce surprising results when coupled with hooks, custom resource definitions, or manually patched resources.

# TheCodeForge - Safe upgrade with dry-run and diff
helm diff upgrade my-release ./mychart --values values-prod.yaml
if [ $? -eq 0 ]; then
  echo "No changes detected, skipping."
elif [ $? -eq 2 ]; then
  echo "Changes detected. Proceeding with dry-run..."
  helm upgrade my-release ./mychart --values values-prod.yaml --dry-run
  read -p "Apply? (y/N) " -n 1 -r
  if [[ $REPLY =~ ^[Yy]$ ]]; then
    helm upgrade my-release ./mychart --values values-prod.yaml --atomic --timeout 10m
  fi
else
  echo "Diff failed, check Helm version and cluster connectivity."
  exit 1
fi

Hooks: Pre- and Post-Operations

Helm hooks allow you to run actions at specific points in the release lifecycle — before install, after install, before upgrade, after upgrade, before delete, after delete. Hooks are defined by annotating a template with "helm.sh/hook": <hook-name> (e.g., pre-install, post-upgrade).

Hooks run as Kubernetes Jobs, and their pods must complete successfully for the lifecycle phase to proceed. If a pre-install hook fails, the install fails. Hooks can also have a weight to control execution order. Lower weight runs first; same weight runs in alphabetical order.

A hidden danger: hook resources are NOT part of the release's managed resource set. If you delete a release, hook resources (if they completed) are not automatically deleted. You must clean them up manually or use the hook-delete-policy annotation.

# TheCodeForge - Database migration hook
apiVersion: batch/v1
kind: Job
metadata:
  name: {{ include "myapp.fullname" . }}-migration
  annotations:
    "helm.sh/hook": pre-upgrade
    "helm.sh/hook-weight": "-5"
    "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: migration
        image: {{ .Values.migration.image | quote }}
        env:
        - name: DB_HOST
          value: {{ required "A valid .Values.db.host is required" .Values.db.host | quote }}

Helm Hooks Lifecycle Reference Table

Hooks are triggered at specific points during the Helm release lifecycle. The table below summarises all available hook annotations, when they fire, and whether they support weights. Use this as a quick reference when designing hook-based workflows.

Weights range from negative to positive integers; lower numbers run first. Hooks with the same weight run in alphabetical order by resource name. The hook-delete-policy annotation controls cleanup: before-hook-creation (remove previous hook before new one), hook-succeeded (delete after success), hook-failed (keep failed hook for debugging).

A common pattern: use a single pre-install hook with hook-delete-policy: before-hook-creation,hook-succeeded to ensure only one completed job remains in the namespace at any time. For critical migrations, set hook-succeeded only (keep for audit), but monitor namespace resource usage.

Production Patterns: Multi-Environment, Dependencies, and Library Charts

Production Helm usage demands patterns that separate environments, manage interdependencies, and share common code across teams.

Multi-environment values: Maintain separate values files: values-dev.yaml, values-staging.yaml, values-prod.yaml. Use a base values.yaml for defaults and override only the differences. Avoid copying the entire values structure; use granular overrides. Helm's value merging picks user-supplied files in order, and later files override earlier ones. A common CI pattern: helm upgrade --values values.yaml --values values-$(ENV).yaml.

Dependencies: Charts can depend on other charts. Declare them in Chart.yaml's dependencies block, then run 'helm dependency update' to download them into charts/. Bitnami charts are commonly used for databases and middleware. The condition field allows enabling/disabling subcharts based on a values flag: condition: redis.enabled.

Library charts: When multiple application charts share helper templates (e.g., labels, ingress helpers, secret generation), extract those into a library chart. Library charts have type: library and are pulled as dependencies. They never create releases; they only export templates. This reduces duplication and ensures consistency.

Subchart scope: Subcharts have their own scope. If your main chart depends on a Redis chart, you configure it under .Values.redis in the parent values. The subchart receives only its own subtree. To share values across subcharts, use global values under the global key in the parent values.

# TheCodeForge - Production values override
environment: production
replicaCount: 5
service:
  port: 443
  type: ClusterIP
resources:
  requests:
    memory: "512Mi"
    cpu: "500m"
  limits:
    memory: "1Gi"
    cpu: "1"
ingress:
  enabled: true
  host: app.example.com
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
redis:
  enabled: true
  architecture: replication
  auth:
    enabled: true
    password: ""  # Should be provided via external secrets or --set
postgresql:
  enabled: false

global:
  imageRegistry: "my-registry.internal:5000"