Kubernetes Admission Controllers

A single misconfigured Pod can bring down a production cluster. In 2023, a major outage at a financial services firm was traced back to a developer deploying a container with privileged: true and hostNetwork: true, bypassing all existing security controls. The fix? Admission controllers. These plugins sit between the API server and etcd, intercepting every create, update, or delete request. They are the gatekeepers that can mutate, validate, or reject resources before they become reality. Unlike RBAC or network policies, admission controllers operate at the object level, giving you surgical control over what enters your cluster. If you're not using them, you're running a cluster with no safety net.

What Are Admission Controllers?

Admission controllers are compiled-in or webhook-based plugins that intercept Kubernetes API requests after authentication and authorization, but before the object is persisted in etcd. They can modify (mutating) or reject (validating) the request. Kubernetes ships with a set of built-in admission controllers (e.g., NamespaceLifecycle, LimitRanger, PodSecurityPolicy), but the real power comes from dynamic admission control via MutatingAdmissionWebhook and ValidatingAdmissionWebhook. These allow you to run custom logic in external HTTP servers. Every request to the API server—kubectl apply, a Deployment controller reconciling, a ServiceAccount token mount—passes through the configured chain of admission controllers. Understanding this pipeline is critical for debugging why a resource was rejected or modified unexpectedly.

kubectl api-resources --verbs=list -o name | xargs -I {} sh -c 'kubectl get --raw /api/v1/namespaces/default/{} 2>/dev/null | head -c 100'

Mutating vs Validating: The Two Flavors

Mutating admission controllers modify the incoming object before it is stored. They run first, in a chain, and can inject sidecars, set default resource limits, add labels, or patch security contexts. Validating admission controllers run after all mutating controllers, and can reject the request based on the final state of the object. The separation is crucial: mutating controllers can fix common omissions (e.g., adding a default PodSecurityContext), while validating controllers enforce hard rules (e.g., no privileged containers). Both are implemented as webhooks, but mutating webhooks must be idempotent—they may be called multiple times for the same request if the object is modified by another mutating webhook. A common pitfall is writing a mutating webhook that is not idempotent, leading to infinite loops or corrupted objects.

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: sidecar-injector.example.com
webhooks:
- name: sidecar-injector.example.com
  clientConfig:
    service:
      name: sidecar-injector
      namespace: default
      path: /mutate
    caBundle: <base64-encoded-ca>
  rules:
  - operations: ["CREATE"]
    apiGroups: [""]
    apiVersions: ["v1"]
    resources: ["pods"]
  admissionReviewVersions: ["v1"]
  sideEffects: None
  timeoutSeconds: 5
  reinvocationPolicy: IfNeeded

Building a Validating Webhook: Deny Privileged Containers

Let's build a simple validating webhook that rejects any Pod with 'securityContext.privileged: true'. The webhook is an HTTP server that receives AdmissionReview requests and returns AdmissionReview responses. We'll use Go with the 'k8s.io/api/admission/v1' package. The server listens on /validate, decodes the request, checks the Pod spec, and returns 'allowed: false' with a message if privileged is set. Deploy it as a Service and register it via ValidatingWebhookConfiguration. This is a common pattern for enforcing security policies without modifying the cluster's built-in controllers.

package main

import (
	"encoding/json"
	"fmt"
	"log"
	"net/http"

	admissionv1 "k8s.io/api/admission/v1"
	corev1 "k8s.io/api/core/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

func validate(w http.ResponseWriter, r *http.Request) {
	var review admissionv1.AdmissionReview
	if err := json.NewDecoder(r.Body).Decode(&review); err != nil {
		http.Error(w, err.Error(), http.StatusBadRequest)
		return
	}

	var pod corev1.Pod
	if err := json.Unmarshal(review.Request.Object.Raw, &pod); err != nil {
		http.Error(w, err.Error(), http.StatusBadRequest)
		return
	}

	allowed := true
	message := ""
	if pod.Spec.SecurityContext != nil && pod.Spec.SecurityContext.Privileged != nil && *pod.Spec.SecurityContext.Privileged {
		allowed = false
		message = "Privileged containers are not allowed"
	}

	review.Response = &admissionv1.AdmissionResponse{
		UID:     review.Request.UID,
		Allowed: allowed,
		Result:  &metav1.Status{Message: message},
	}

	w.Header().Set("Content-Type", "application/json")
	json.NewEncoder(w).Encode(review)
}

func main() {
	http.HandleFunc("/validate", validate)
	log.Fatal(http.ListenAndServeTLS(":443", "/etc/certs/tls.crt", "/etc/certs/tls.key", nil))
}

Building a Mutating Webhook: Inject a Sidecar Proxy

Mutating webhooks can automatically inject sidecar containers, such as a logging agent or service mesh proxy. The webhook patches the Pod spec to add a container, volume mounts, and annotations. The key is to use JSON Patch (RFC 6902) operations. For example, to inject an Envoy sidecar, you add a container to 'spec.containers' and a volume mount for the Envoy config. The webhook must be idempotent: check for an annotation like 'sidecar-injected: true' to avoid double injection. This pattern is used by Istio, Linkerd, and many internal platforms.

func mutate(w http.ResponseWriter, r *http.Request) {
	var review admissionv1.AdmissionReview
	json.NewDecoder(r.Body).Decode(&review)

	var pod corev1.Pod
	json.Unmarshal(review.Request.Object.Raw, &pod)

	// Check if already injected
	if pod.Annotations["sidecar-injected"] == "true" {
		// Already injected, skip
		review.Response = &admissionv1.AdmissionResponse{
			UID:     review.Request.UID,
			Allowed: true,
			Patch:   nil,
			PatchType: func() *admissionv1.PatchType { pt := admissionv1.PatchTypeJSONPatch; return &pt }(),
		}
		json.NewEncoder(w).Encode(review)
		return
	}

	// Build JSON Patch to add sidecar container
	patch := []map[string]interface{}{
		{
			"op":    "add",
			"path":  "/spec/containers/-",
			"value": map[string]interface{}{
				"name":  "envoy",
				"image": "envoyproxy/envoy:v1.28-latest",
				"ports": []map[string]interface{}{{"containerPort": 9901}},
			},
		},
		{
			"op":    "add",
			"path":  "/metadata/annotations/sidecar-injected",
			"value": "true",
		},
	}

	patchBytes, _ := json.Marshal(patch)

	review.Response = &admissionv1.AdmissionResponse{
		UID:     review.Request.UID,
		Allowed: true,
		Patch:   patchBytes,
		PatchType: func() *admissionv1.PatchType { pt := admissionv1.PatchTypeJSONPatch; return &pt }(),
	}

	json.NewEncoder(w).Encode(review)
}

Webhook Configuration: Rules, Failure Policy, and Timeouts

A webhook configuration defines which resources trigger the webhook via 'rules' (operations, apiGroups, apiVersions, resources, scope). You can also set 'failurePolicy' to 'Fail' (default) or 'Ignore'. 'Fail' means if the webhook is unreachable, the request is rejected—use for security-critical webhooks. 'Ignore' allows the request to proceed if the webhook is down—use for non-critical mutations like sidecar injection. 'timeoutSeconds' should be set low (e.g., 5-10) to avoid blocking the API server. 'sideEffects' must be 'None' or 'NoneOnDryRun' to indicate the webhook has no side effects beyond admission. 'reinvocationPolicy' controls whether the webhook is called again after other mutating webhooks. Proper configuration prevents cluster-wide outages.

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: deny-privileged.example.com
webhooks:
- name: deny-privileged.example.com
  clientConfig:
    service:
      name: deny-privileged
      namespace: default
      path: /validate
    caBundle: <base64-ca>
  rules:
  - operations: ["CREATE", "UPDATE"]
    apiGroups: [""]
    apiVersions: ["v1"]
    resources: ["pods"]
    scope: "Namespaced"
  failurePolicy: Fail
  matchPolicy: Equivalent
  timeoutSeconds: 5
  sideEffects: None
  admissionReviewVersions: ["v1"]

Debugging Admission Webhooks: Common Pitfalls

Admission webhooks can be tricky to debug. Common issues include: TLS certificate mismatches (CA bundle not matching server cert), webhook service unreachable (wrong namespace or port), infinite loops (mutating webhook modifies an object that triggers itself), and timeouts. Use 'kubectl describe validatingwebhookconfiguration' to see the configuration. Check the API server logs (often in /var/log/kube-apiserver.log) for admission errors. Enable verbose logging on your webhook server. A common mistake is forgetting to set 'sideEffects: None'—without it, the API server will not call the webhook on dry-run requests. Also, ensure your webhook handles both CREATE and UPDATE operations if needed.

# Check webhook configuration
kubectl get validatingwebhookconfiguration -o yaml

# Check API server logs for admission errors
kubectl logs -n kube-system kube-apiserver-controlplane | grep -i admission

# Test webhook with dry-run
kubectl apply -f pod.yaml --dry-run=server

# Simulate an admission request with curl
curl -k -X POST https://webhook-service:443/validate \
  -H "Content-Type: application/json" \
  -d @admission-review.json

Built-in Admission Controllers You Should Know

Kubernetes ships with several built-in admission controllers that are enabled by default in most distributions. 'NamespaceLifecycle' prevents creation of resources in terminating namespaces. 'LimitRanger' enforces default and maximum resource limits per namespace. 'ResourceQuota' enforces namespace-level resource quotas. 'PodSecurity' (replaces PodSecurityPolicy) enforces Pod Security Standards (privileged, baseline, restricted). 'NodeRestriction' limits kubelet's self-modification capabilities. 'AlwaysPullImages' ensures images are always pulled, preventing use of stale or malicious images. Understanding these built-in controllers helps you avoid reinventing the wheel. For example, instead of writing a custom webhook to enforce resource limits, use LimitRanger and ResourceQuota.

# List enabled admission controllers (kube-apiserver flags)
kubectl get pods -n kube-system -l component=kube-apiserver -o jsonpath="{.items[0].spec.containers[0].command}" | tr ' ' '\n' | grep enable-admission-plugins

# Example output:
# --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,PodSecurity,DefaultStorageClass,DefaultTolerationSeconds,ResourceQuota

Admission Controllers and Pod Security Standards

Pod Security Standards (PSS) define three policies: privileged, baseline, and restricted. The 'PodSecurity' admission controller (GA since v1.25) enforces these policies at the namespace level via labels. You can set 'pod-security.kubernetes.io/enforce', 'pod-security.kubernetes.io/audit', and 'pod-security.kubernetes.io/warn' labels on namespaces. This replaces the deprecated PodSecurityPolicy (PSP). PSS is simpler and more maintainable. However, for fine-grained control, you may still need custom webhooks. For example, PSS does not allow you to block specific image registries or enforce specific seccomp profiles. In those cases, combine PSS with a validating webhook.

apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: latest
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

Performance Considerations and Best Practices

Admission webhooks add latency to every API request. A slow webhook can cripple the API server. Best practices: keep webhooks stateless and fast (sub-millisecond response times). Use caching for expensive operations (e.g., fetching data from external APIs). Set 'timeoutSeconds' low (5s) and implement a circuit breaker to fail fast. Use 'failurePolicy: Ignore' for non-critical webhooks. Deploy multiple replicas behind a Service for high availability. Monitor webhook latency and error rates with Prometheus. Avoid making network calls to external services inside the webhook—they can fail and block the cluster. If you must, use a local cache or a sidecar that pre-fetches data.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: admission-webhook
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: admission-webhook
  template:
    metadata:
      labels:
        app: admission-webhook
    spec:
      containers:
      - name: webhook
        image: myregistry/admission-webhook:v1.0
        ports:
        - containerPort: 443
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 500m
            memory: 256Mi
        livenessProbe:
          httpGet:
            path: /healthz
            port: 443
            scheme: HTTPS
          initialDelaySeconds: 5
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /readyz
            port: 443
            scheme: HTTPS
          initialDelaySeconds: 5
          periodSeconds: 10

Advanced Patterns: Chaining and Dynamic Policies

Multiple admission webhooks can be chained. The order of execution is determined by the webhook name (alphabetical) for mutating webhooks, and then validating webhooks run in parallel. You can use 'reinvocationPolicy' to re-run mutating webhooks after others have run. This allows patterns like: first webhook adds default labels, second webhook injects sidecar, third webhook validates final state. Dynamic policies can be implemented by having the webhook fetch policy from a ConfigMap or an external policy engine (e.g., OPA/Gatekeeper). Gatekeeper is a popular project that uses admission webhooks with a declarative policy language (Rego). It provides a framework for creating, managing, and auditing policies.

package k8s.admission

import data.k8s.pods

violation[{"msg": msg}] {
    input.request.kind.kind == "Pod"
    container := input.request.object.spec.containers[_]
    container.securityContext.privileged == true
    msg := sprintf("Container %v is privileged", [container.name])
}

Testing Admission Webhooks Locally

Before deploying to a cluster, test your webhook locally. Use 'kubectl create --dry-run=server' to simulate an admission request. You can also use the 'admission-webhook-bootstrapper' tool or write unit tests that send AdmissionReview JSON to your server. For integration testing, set up a local kind cluster and deploy the webhook. Use 'curl' to send test requests directly to the webhook endpoint. Ensure your webhook handles edge cases: empty objects, missing fields, and multiple containers. Test both allowed and denied scenarios. Automate these tests in CI/CD to prevent regressions.

# Create a test AdmissionReview request
cat > test-request.json <<EOF
{
  "apiVersion": "admission.k8s.io/v1",
  "kind": "AdmissionReview",
  "request": {
    "uid": "test-uid",
    "operation": "CREATE",
    "object": {
      "apiVersion": "v1",
      "kind": "Pod",
      "metadata": {
        "name": "test-pod"
      },
      "spec": {
        "containers": [{
          "name": "test",
          "image": "nginx",
          "securityContext": {
            "privileged": true
          }
        }]
      }
    }
  }
}
EOF

# Send to webhook
curl -k -X POST https://localhost:8443/validate \
  -H "Content-Type: application/json" \
  -d @test-request.json

Production Deployment: Certificates, Monitoring, and Rollback

Deploying admission webhooks in production requires careful planning. Use cert-manager to automate TLS certificate provisioning and rotation. Monitor webhook latency, error rates, and request volume with Prometheus. Set up alerts for high latency or error spikes. Implement a circuit breaker: if the webhook fails consecutively, stop calling it and fall back to 'failurePolicy: Ignore' temporarily. Have a rollback plan: if a webhook causes issues, delete the webhook configuration to disable it. Use feature flags or toggles to enable/disable webhooks without redeploying. Document the webhook's purpose, failure mode, and rollback procedure.

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: selfsigned-issuer
  namespace: default
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: admission-webhook-cert
  namespace: default
spec:
  secretName: admission-webhook-tls
  dnsNames:
  - admission-webhook.default.svc
  issuerRef:
    name: selfsigned-issuer
    kind: Issuer