OPA Gatekeeper and Kyverno — Policy Engines

Welcome to OPA Gatekeeper and Kyverno — Policy Engines. We'll break this down from first principles.

Why Policy Engines Are Non-Negotiable in Production

Kubernetes clusters are complex systems with hundreds of resources. Without policy enforcement, a single misconfigured Pod can expose secrets, run as root, or bypass network policies. Policy engines intercept admission requests (create, update, delete) and either allow or deny them based on rules. OPA Gatekeeper and Kyverno are the two dominant open-source solutions. Gatekeeper is the OPA-integrated Kubernetes admission controller; Kyverno is a purpose-built Kubernetes policy engine. Both run as validating/mutating webhooks. In production, you need policy engines to enforce security standards (e.g., Pod Security Standards), compliance (e.g., PCI-DSS), and operational consistency (e.g., required labels). Without them, you rely on manual reviews or post-deployment scanning, which are too slow and error-prone.

apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8srequiredlabels
spec:
  crd:
    spec:
      names:
        kind: K8sRequiredLabels
      validation:
        openAPIV3Schema:
          type: object
          properties:
            labels:
              type: array
              items:
                type: string
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8srequiredlabels
        violation[{"msg": msg}] {
          provided := {label | input.review.object.metadata.labels[label]}
          required := {label | label := input.parameters.labels[_]}
          missing := required - provided
          count(missing) > 0
          msg := sprintf("missing required labels: %v", [missing])
        }

OPA Gatekeeper: Rego-Powered Policy as Code

OPA (Open Policy Agent) is a general-purpose policy engine that decouples policy from software. Gatekeeper integrates OPA into Kubernetes as an admission controller. Policies are written in Rego, a declarative query language. Gatekeeper introduces two CRDs: ConstraintTemplate (defines reusable policy logic) and Constraint (instantiates a template with parameters). This separation allows platform teams to define templates and app teams to create constraints. Rego is powerful but has a steep learning curve. Gatekeeper also supports mutation (via mutation CRDs) and audit (periodic evaluation of existing resources). In production, Gatekeeper is ideal for complex policies that require logic beyond simple label checks, such as cross-resource validation or complex data transformations.

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
  name: require-app-label
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
    namespaces: ["production"]
  parameters:
    labels: ["app"]

Kyverno: Kubernetes-Native Policy Engine

Kyverno is designed specifically for Kubernetes. Policies are written in YAML, using familiar Kubernetes resource structures. Kyverno supports validate, mutate, generate, and verifyImages rules. It can also generate new resources (e.g., create a NetworkPolicy when a Namespace is created). Kyverno's learning curve is lower because it uses YAML and JSON patches. It also has built-in support for Pod Security Standards (PSS) and a large library of ready-to-use policies. In production, Kyverno is excellent for teams that want to move fast without learning Rego. However, for very complex logic (e.g., cross-resource validation with conditions), Kyverno can become verbose.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-app-label
spec:
  validationFailureAction: Enforce
  rules:
    - name: check-label
      match:
        any:
          - resources:
              kinds:
                - Pod
              namespaces:
                - production
      validate:
        message: "Label 'app' is required"
        pattern:
          metadata:
            labels:
              app: "*"

Comparing Gatekeeper and Kyverno: When to Use Which

Both tools solve the same problem but have different trade-offs. Gatekeeper is better for complex, cross-resource policies that require advanced logic (e.g., 'no two Ingresses can have the same host'). Kyverno is better for simple validation, mutation, and generation tasks. Gatekeeper's Rego is Turing-complete; Kyverno's YAML is not. If your team already knows Rego, stick with Gatekeeper. If you want to onboard quickly, choose Kyverno. Gatekeeper has a stronger separation of concerns (template vs constraint), which is beneficial for large organizations. Kyverno's policies are self-contained, which is simpler. Both support dry-run mode, audit, and metrics. In production, we run both: Gatekeeper for complex security policies, Kyverno for operational policies like label enforcement.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: add-safe-to-evict
spec:
  rules:
    - name: mutate-pod
      match:
        any:
          - resources:
              kinds:
                - Pod
      mutate:
        patchStrategicMerge:
          metadata:
            annotations:
              cluster-autoscaler.kubernetes.io/safe-to-evict: "true"

Installing Gatekeeper and Kyverno

Both tools are installed via Helm or manifests. Gatekeeper requires OPA constraint framework CRDs. Kyverno is a single deployment. For production, use Helm with version pinning and custom resource limits. Gatekeeper's default resource requests are low; we had to increase them to handle large clusters. Kyverno's default works well but we recommend setting resource limits to prevent OOM. Both support high availability with multiple replicas. After installation, verify the webhook is serving by checking the ValidatingWebhookConfiguration. For Gatekeeper, also check the constraint templates are installed. For Kyverno, check the ClusterPolicy status. Always test with a dry-run policy first.

helm repo add gatekeeper https://open-policy-agent.github.io/gatekeeper/charts
helm install gatekeeper/gatekeeper --name-template=gatekeeper --namespace=gatekeeper-system --create-namespace \
  --set replicas=2 \
  --set resources.limits.cpu=500m \
  --set resources.limits.memory=512Mi

Writing Policies: Gatekeeper Rego Example

Gatekeeper policies consist of a ConstraintTemplate (Rego logic) and a Constraint (instantiation). The Rego code must define a 'violation' rule that returns a message when the policy is violated. Gatekeeper provides input.review.object for the resource under review. Parameters are accessed via input.parameters. Rego supports sets, comprehensions, and rules. Common patterns: check labels, container security context, resource limits, and forbidden registries. Always use 'deny' or 'violation' rules. For mutation, Gatekeeper uses a separate mutation framework. Debug Rego with --enable-print and print() statements.

package k8srequiredlabels

violation[{"msg": msg}] {
  provided := {label | input.review.object.metadata.labels[label]}
  required := {label | label := input.parameters.labels[_]}
  missing := required - provided
  count(missing) > 0
  msg := sprintf("missing required labels: %v", [missing])
}

Writing Policies: Kyverno YAML Example

Kyverno policies are YAML with match, exclude, and validate/mutate/generate blocks. The validate block uses pattern or anyPattern to define allowed values. Kyverno supports JSON patches for mutation. It also has a rich set of built-in variables like {{ request.object.metadata.name }}. Kyverno can validate across resources using context variables (e.g., fetch another resource). For complex logic, use deny rules with conditions. Kyverno's policy library (https://kyverno.io/policies/) provides ready-to-use policies. Always set validationFailureAction: Enforce for blocking, or Audit for dry-run.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-privileged-containers
spec:
  validationFailureAction: Enforce
  rules:
    - name: check-privileged
      match:
        any:
          - resources:
              kinds:
                - Pod
      validate:
        message: "Privileged containers are not allowed"
        pattern:
          spec:
            containers:
              - securityContext:
                  privileged: false

Testing Policies in CI/CD

Both tools support testing policies before deployment. For Gatekeeper, use opa test for Rego unit tests and gator test for integration tests. For Kyverno, use kyverno apply to test policies against YAML manifests. Integrate these into CI pipelines to catch policy violations early. We run policy tests on every pull request that modifies policies. Also test against a real cluster using dry-run mode. For Gatekeeper, set --enable-print to debug. For Kyverno, use --dry-run flag. Always version your policies and store them in Git.

# Gatekeeper test
opa test ./policies/gatekeeper/ -v

# Kyverno test
kyverno apply ./policies/kyverno/ --resource=./test-resources/pod.yaml --output=./results.json

Audit and Monitoring: Detecting Drift

Both engines provide audit capabilities. Gatekeeper's audit runs periodically and reports violations on existing resources via the status field of constraints. Kyverno's audit is built-in and reports via policy reports (a CRD). Monitor these reports to detect resources that violate policies but were created before the policy existed or bypassed the webhook. We use Prometheus metrics from both tools to track violation counts and webhook latency. Set up alerts for high violation rates. Also, use kubectl get constraint to see Gatekeeper audit results. For Kyverno, kubectl get policyreport.

# Gatekeeper audit
kubectl get k8srequiredlabels.constraints.gatekeeper.sh/require-app-label -o yaml | grep -A5 violations

# Kyverno audit
kubectl get policyreport -n production -o json | jq '.items[].results[] | select(.result=="fail")'

Performance Tuning and Resource Management

Policy engines add latency to every admission request. In production, tune resource limits, replica count, and webhook timeout. For Gatekeeper, increase --max-serving-threads and set --operation=mutate,validate to only intercept necessary operations. For Kyverno, adjust --webhookTimeout and --maxQueuedEvents. Use pod anti-affinity to spread replicas across nodes. Monitor webhook latency with Prometheus. If latency exceeds 1 second, consider simplifying policies or moving to Kyverno. Also, use failurePolicy: Fail for critical policies and Ignore for non-critical ones to avoid cluster-wide outages.

apiVersion: config.gatekeeper.sh/v1alpha1
kind: Config
metadata:
  name: config
  namespace: gatekeeper-system
spec:
  sync:
    syncOnly:
      - group: ""
        version: "v1"
        kind: "Namespace"
  validation:
    traces:
      - user: "system:serviceaccount:gatekeeper-system:gatekeeper-admin"
        kind: "Pod"
        version: "v1"

Migration from Gatekeeper to Kyverno (or Vice Versa)

Migrating between policy engines is non-trivial because policy syntax differs. Start by auditing existing policies and categorizing them by complexity. Simple label checks can be easily translated. Complex Rego logic may require redesign. Use Kyverno's --dry-run to test translated policies. For Gatekeeper, use --enable-print to debug. Run both engines in parallel during migration, with one in audit mode. Gradually shift enforcement from one to the other. Document all policies and their intent. We migrated from Gatekeeper to Kyverno for simpler policies but kept Gatekeeper for cross-resource validation.

# Run Kyverno in audit mode
kubectl annotate clusterpolicy/require-app-label --overwrite kyverno.io/validationFailureAction=Audit

# Check Kyverno audit results
kubectl get policyreport -n production -o json | jq '.items[].results[] | select(.result=="fail")'

Production Best Practices and Pitfalls

Start with a small set of policies and expand gradually. Use dry-run mode first. Monitor webhook latency and set alerts. Always have a break-glass mechanism: if the webhook fails, set failurePolicy: Ignore temporarily. Use policy as code with GitOps. Regularly audit existing resources. Avoid policies that require fetching many resources (e.g., list all Pods) as they can slow down the API server. Use Kyverno's generate rules sparingly. For Gatekeeper, avoid using data.inventory for large clusters. Test policies in a staging cluster before production. Document each policy's purpose and owner.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-resource-limits
spec:
  validationFailureAction: Enforce
  rules:
    - name: check-limits
      match:
        any:
          - resources:
              kinds:
                - Pod
      validate:
        message: "Resource limits are required"
        pattern:
          spec:
            containers:
              - resources:
                  limits:
                    memory: "?*"
                    cpu: "?*"