Kubernetes Pod Priority and Preemption

Imagine your production API server is about to crash because a CI job ate all the CPU. Without priority and preemption, you're watching the logs helplessly. Kubernetes Pod Priority and Preemption is the bouncer that kicks out the cheap seats when VIPs arrive. It's not about fairness—it's about survival. In a real incident, a misconfigured PriorityClass caused our monitoring stack to be evicted by a data pipeline, leading to a 45-minute outage. This feature is your last line of defense against resource starvation, but misused, it can cause chaos.

What Are PriorityClasses?

PriorityClasses are cluster-scoped resources that assign an integer value to pods. Higher values mean higher priority. When the scheduler can't place a pod, it preempts (evicts) lower-priority pods to free resources. PriorityClasses are defined once and referenced by pods. They are not namespaced, so they apply across the cluster. The default PriorityClass is 0; pods without explicit priority get this. Critical system pods (e.g., kube-system) often have very high priorities (e.g., 1000000000) to avoid preemption. You must create PriorityClasses before using them—there's no built-in except the default.

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 1000000
globalDefault: false
description: "For production-critical pods"

How Preemption Works

Preemption is the process where the scheduler evicts lower-priority pods to make room for a pending higher-priority pod. It only happens when the scheduler cannot find a node that meets the pod's resource requests. The scheduler selects a node where preempting pods would free enough resources, then picks the lowest-priority pods on that node to evict. Preemption respects PodDisruptionBudgets (PDBs) but can violate them if necessary—it will try to minimize disruption but may evict PDB-protected pods if no other option exists. Preemption is not graceful termination; pods get a SIGTERM and a brief grace period (default 30s).

apiVersion: v1
kind: Pod
metadata:
  name: critical-api
spec:
  priorityClassName: high-priority
  containers:
  - name: api
    image: nginx:latest
    resources:
      requests:
        memory: "512Mi"
        cpu: "500m"

Setting Up PriorityClasses in a Cluster

To use priority and preemption, you must create PriorityClasses. Best practice: define a hierarchy with clear value gaps. For example: system (1000000000), production (1000000), staging (100000), batch (10000), default (0). Avoid overlapping ranges. PriorityClasses are immutable after creation—you must delete and recreate to change the value. Use globalDefault: true on exactly one PriorityClass to set the default for all pods without a priorityClassName. This is risky: if you set a high default, all pods become high-priority, defeating the purpose.

kubectl create priorityclass system-priority --value=1000000000 --global-default=false --description="System pods"
kubectl create priorityclass prod-priority --value=1000000 --global-default=false --description="Production pods"
kubectl get priorityclass

Pod Priority and Preemption in Action

Let's simulate a scenario: a high-priority pod is pending because all nodes are full of low-priority pods. The scheduler will select a victim node and evict enough low-priority pods to fit the high-priority pod. The evicted pods are terminated and may be recreated by their controllers (e.g., Deployment) but will likely be preempted again if they remain low-priority. This can lead to a loop if resources are tight. To observe preemption, watch scheduler events with kubectl get events --watch. The scheduler emits events like 'Preempted pod low-priority-pod on node worker-1'.

kubectl create deployment low-priority --image=nginx --replicas=10 -- /bin/sh -c "sleep 3600"
kubectl set resources deployment low-priority --requests=cpu=1,memory=1Gi
# Now create a high-priority pod that needs more resources than available
kubectl run high-priority --image=nginx --requests=cpu=4,memory=4Gi --priority-class-name=high-priority
kubectl get events --watch | grep -i preempt

Preemption and PodDisruptionBudgets

PodDisruptionBudgets (PDBs) limit the number of voluntary disruptions a set of pods can tolerate. Preemption is considered a voluntary disruption (unlike node failure). The scheduler respects PDBs when choosing victims: it will try to avoid violating PDBs, but if no other option exists, it will violate them. This means a PDB does not guarantee protection from preemption. To protect critical pods, set their priority high enough that they are never victims. PDBs are more effective for controlling disruptions during voluntary actions like node drains.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: critical-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: critical

Priority Inversion and How to Avoid It

Priority inversion occurs when a low-priority pod holds a resource (e.g., a lock) needed by a high-priority pod, causing the high-priority pod to wait. In Kubernetes, this can happen if a low-priority pod is running and holding a mutex or a database connection pool, and a high-priority pod needs that same resource. Preemption doesn't solve this because the low-priority pod is already running and won't be preempted unless the high-priority pod is pending due to resource shortage. To avoid priority inversion, ensure that critical resources are not shared across priority tiers, or use priority-based resource quotas.

# Assume a low-priority pod holds a distributed lock
# High-priority pod tries to acquire the same lock
# No preemption occurs because the high-priority pod is not pending on resources
# Solution: use priority-based resource quotas or separate namespaces

Best Practices for Priority Values

Choose priority values with clear separation. Use a scheme like: system (1000000000), production (1000000), staging (100000), batch (10000), default (0). Leave gaps for future tiers. Never use values like 1000 and 1001—they are too close and make it hard to insert new tiers. Document the priority mapping in your cluster's runbook. Consider using an admission controller to enforce priority classes and prevent pods from using arbitrary values. Also, avoid setting very high priorities for non-critical pods, as they can preempt system components.

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: system
value: 1000000000
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: production
value: 1000000
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: staging
value: 100000
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: batch
value: 10000
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: default
value: 0
globalDefault: true

Monitoring and Debugging Preemption

Monitor preemption events via Kubernetes events, scheduler metrics, and logs. Key metrics: scheduler_preemption_attempts_total, scheduler_preemption_victims. Use kubectl describe pod to see the 'Preempted' condition. Enable scheduler profiling for detailed logs. Set up alerts for high preemption rates—they indicate resource pressure or misconfigured priorities. Tools like kube-state-metrics expose priority class info. Also, audit pod priority assignments regularly to catch anomalies.

# Get scheduler events
kubectl get events --field-selector reason=Preempted
# Describe a pod that was preempted
kubectl describe pod <preempted-pod> | grep -A5 Preempted
# Check scheduler metrics (if Prometheus is set up)
# promql: scheduler_preemption_attempts_total

Preemption and Cluster Autoscaler

Cluster Autoscaler (CA) can interact with preemption. When a high-priority pod can't be scheduled, CA may trigger scale-up instead of preemption, depending on configuration. CA respects pod priorities: it will try to scale up for pending pods, but if scale-up is not possible (e.g., cloud quota), preemption kicks in. To avoid unnecessary scale-ups, set --expander=priority and configure priority thresholds. CA also has a --skip-nodes-with-system-pods flag to avoid preempting system pods. Understand the interplay to avoid cost surprises.

# Cluster Autoscaler deployment with priority expander
command:
  - ./cluster-autoscaler
  - --expander=priority
  - --priority-config=/config/priority-expander-config.yaml
# priority-expander-config.yaml
priorities:
- priority: 1000000
  nodeGroup: production-nodes
- priority: 100000
  nodeGroup: staging-nodes

Common Pitfalls and How to Avoid Them

Pitfall 1: Setting all pods to the same priority. This disables preemption and defeats the purpose. Pitfall 2: Using too many priority tiers, causing confusion. Stick to 4-6 tiers. Pitfall 3: Not testing preemption in a staging environment. Always simulate resource pressure to see how your system behaves. Pitfall 4: Ignoring PDB interactions. Test preemption with PDBs to ensure they don't cause unexpected behavior. Pitfall 5: Forgetting that preemption only happens at scheduling time. Running pods are not preempted unless a higher-priority pod is pending.

# Simulate resource pressure in a test namespace
kubectl create ns test-priority
kubectl run low-priority --image=nginx --requests=cpu=1,memory=1Gi -n test-priority --priority-class-name=batch
# Fill nodes
# Then create high-priority pod
kubectl run high-priority --image=nginx --requests=cpu=4,memory=4Gi -n test-priority --priority-class-name=production
# Observe preemption
kubectl get events -n test-priority --watch

Advanced: Priority and Resource Quotas

ResourceQuotas can be scoped by priority class using scopeSelector. This allows you to limit total resources consumed by pods of a certain priority. For example, you can cap batch jobs to 10 CPUs total, preventing them from overwhelming the cluster. Combine with LimitRanges to enforce min/max resource requests per priority. This gives fine-grained control and prevents priority abuse. However, be careful: if a high-priority pod exceeds its quota, it will be rejected, not preempt lower-priority pods.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: batch-quota
spec:
  hard:
    requests.cpu: "10"
    requests.memory: 20Gi
  scopeSelector:
    matchExpressions:
    - operator: In
      scopeName: PriorityClass
      values: ["batch"]

When Not to Use Priority and Preemption

Not every cluster needs priority and preemption. If your cluster has ample resources and workloads are homogeneous, it adds complexity without benefit. Avoid it if you have a small cluster with few pods—preemption can cause unnecessary churn. Also, if your workloads are all critical, setting priorities is meaningless. In multi-tenant clusters, priority can be abused by tenants setting high priorities. Use admission controllers to enforce priority ranges per namespace. Finally, if you need guaranteed resources, consider using Guaranteed QoS class and node affinity instead.

# For clusters with ample resources, you may not need priority
# Instead, use resource requests and limits to ensure QoS
apiVersion: v1
kind: Pod
metadata:
  name: guaranteed-pod
spec:
  containers:
  - name: app
    image: nginx
    resources:
      requests:
        cpu: 1
        memory: 1Gi
      limits:
        cpu: 1
        memory: 1Gi