GPU Support and Specialized Hardware in Kubernetes

Welcome to GPU Support and Specialized Hardware in Kubernetes. We'll break this down from first principles.

Why GPU Support Matters in Kubernetes

Kubernetes was designed for stateless microservices, but AI/ML workloads demand specialized hardware. Without GPU support, you'd have to manually assign nodes and manage device contention. The Kubernetes ecosystem now provides device plugins and scheduler extensions to treat GPUs as first-class resources. This matters because it allows you to run training jobs, inference servers, and data processing pipelines on the same cluster, with the same orchestration benefits. When to use it: any time your workload requires CUDA, ROCm, or other GPU-accelerated libraries. Avoid it for CPU-only tasks—overhead isn't worth it.

apiVersion: v1
kind: Pod
metadata:
  name: gpu-pod
spec:
  containers:
  - name: cuda-vector-add
    image: nvidia/cuda:12.2.0-base-ubuntu22.04
    command: ["nvidia-smi"]
    resources:
      limits:
        nvidia.com/gpu: 1

Prerequisites: NVIDIA Drivers and Container Runtime

Before Kubernetes can schedule GPUs, each node must have the NVIDIA driver installed and a container runtime that supports GPU passthrough. The driver version must match the CUDA version your workloads expect. Use nvidia-smi to verify. For the runtime, install nvidia-container-toolkit and configure containerd or Docker to use it. This is a common failure point: mismatched driver/runtime versions cause pods to fail with 'cannot find CUDA driver'. Always pin driver versions in your node image.

# Install NVIDIA container toolkit on Ubuntu
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=containerd
sudo systemctl restart containerd

Installing the NVIDIA Device Plugin

The NVIDIA device plugin is a DaemonSet that advertises GPU resources to the kubelet. It watches for GPU devices and updates the node's capacity. Install it via Helm or raw YAML. The plugin also supports MIG (Multi-Instance GPU) partitioning for A100/H100 GPUs. Without it, nvidia.com/gpu won't appear as a schedulable resource. Common mistake: forgetting to set the nvidia.com/gpu resource in the pod spec, or requesting more GPUs than available.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nvidia-device-plugin-daemonset
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: nvidia-device-plugin-ds
  template:
    metadata:
      labels:
        name: nvidia-device-plugin-ds
    spec:
      tolerations:
      - operator: Exists
        effect: NoSchedule
      containers:
      - image: nvidia/k8s-device-plugin:v0.14.0
        name: nvidia-device-plugin-ctr
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop: ["ALL"]
        volumeMounts:
        - name: device-plugin
          mountPath: /var/lib/kubelet/device-plugins
      volumes:
      - name: device-plugin
        hostPath:
          path: /var/lib/kubelet/device-plugins

Scheduling GPU Workloads: Resource Requests and Limits

GPU resources are advertised as nvidia.com/gpu and must be specified as limits (requests are ignored by the scheduler). You can request fractional GPUs only if using MIG or time-slicing. The scheduler uses a bin-packing algorithm by default, but you can configure node selectors or taints to isolate GPU nodes. Important: GPU memory is not directly managed by Kubernetes—if your pod exceeds GPU memory, it will OOM and crash. Use CUDA's cudaSetDevice and memory monitoring tools.

apiVersion: batch/v1
kind: Job
metadata:
  name: gpu-job
spec:
  template:
    spec:
      containers:
      - name: cuda-work
        image: nvidia/cuda:12.2.0-runtime-ubuntu22.04
        command: ["python3", "-c", "import torch; print(torch.cuda.is_available())"]
        resources:
          limits:
            nvidia.com/gpu: 1
      restartPolicy: Never

Multi-Instance GPU (MIG) Partitioning

MIG allows partitioning a single A100/H100 into up to 7 smaller GPU instances, each with isolated memory and compute. This improves utilization for smaller workloads. To use MIG, enable it on the GPU (via nvidia-smi mig -cgi), then configure the device plugin with the mig-strategy flag. The plugin exposes each MIG slice as a separate nvidia.com/gpu resource. Without MIG, a single GPU can only be used by one pod at a time.

# Enable MIG on GPU 0
sudo nvidia-smi mig -cgi 19,19,19 -i 0
# Verify
nvidia-smi mig -lgi
# Output:
# GPU 0 Profile ID 19 Placements: {0,1,2}:1

GPU Time-Slicing for Oversubscription

Time-slicing allows multiple pods to share a single GPU by time-multiplexing compute. Unlike MIG, memory is not isolated—pods can interfere. Enable it via the device plugin's gpu-sharing-strategy flag. Useful for development or low-priority batch jobs, but risky for production because a memory-hungry pod can OOM others. Always set resource limits for CPU/memory to mitigate.

apiVersion: v1
kind: ConfigMap
metadata:
  name: nvidia-device-plugin-config
  namespace: kube-system
data:
  config.yaml: |
    version: v1
    flags:
      migStrategy: none
    sharing:
      timeSlicing:
        resources:
        - name: nvidia.com/gpu
          replicas: 4

Monitoring GPU Utilization and Health

Kubernetes doesn't natively expose GPU metrics. Use the NVIDIA DCGM exporter (part of the GPU Operator) to collect metrics like GPU utilization, memory usage, temperature, and power. Integrate with Prometheus and Grafana. Also, set up liveness probes for GPU pods to detect driver hangs. Common failure: GPU stuck in 'P2' state (high power) due to a stuck kernel—requires node reboot.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nvidia-dcgm-exporter
  namespace: gpu-operator
spec:
  selector:
    matchLabels:
      app: nvidia-dcgm-exporter
  template:
    metadata:
      labels:
        app: nvidia-dcgm-exporter
    spec:
      containers:
      - name: dcgm-exporter
        image: nvidia/dcgm-exporter:3.1.7-1.0.0-ubuntu20.04
        ports:
        - containerPort: 9400
          name: metrics

GPU Operator: The All-in-One Solution

The NVIDIA GPU Operator automates the deployment of all GPU-related components: drivers, device plugin, runtime, monitoring, and MIG configuration. It uses a DaemonSet to manage driver upgrades without node reboots. Recommended for production clusters to reduce manual steps. However, it adds complexity—debugging issues requires understanding the operator's reconciliation logic. Weigh against manual setup for small clusters.

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update
helm install gpu-operator nvidia/gpu-operator --namespace gpu-operator --create-namespace
# Wait for all pods to be Running
kubectl wait --for=condition=ready pod -l app.kubernetes.io/component=nvidia-driver -n gpu-operator --timeout=300s

Node Affinity and Taints for GPU Workloads

To ensure GPU workloads land on GPU-equipped nodes, use node affinity or taints. Common pattern: taint GPU nodes with nvidia.com/gpu=present:NoSchedule and add toleration to GPU pods. Also, use node affinity to prefer nodes with specific GPU models (e.g., A100 vs T4). Without these, CPU-only pods might occupy GPU nodes, wasting resources.

apiVersion: v1
kind: Pod
metadata:
  name: gpu-pod
spec:
  tolerations:
  - key: "nvidia.com/gpu"
    operator: "Exists"
    effect: "NoSchedule"
  containers:
  - name: cuda
    image: nvidia/cuda:12.2.0-base-ubuntu22.04
    command: ["nvidia-smi"]
    resources:
      limits:
        nvidia.com/gpu: 1

Handling GPU Memory and OOM

Kubernetes does not track GPU memory usage. A pod can request 1 GPU but use all its VRAM, starving other pods on the same GPU (if time-slicing) or causing OOM. Mitigate by using CUDA_VISIBLE_DEVICES to limit which GPUs a pod sees, and by setting memory limits on the container (CPU memory, not GPU). For MIG, each slice has fixed memory, so OOM is contained. Monitor with DCGM and set resource quotas.

import pynvml
pynvml.nvmlInit()
handle = pynvml.nvmlDeviceGetHandleByIndex(0)
info = pynvml.nvmlDeviceGetMemoryInfo(handle)
print(f'Total: {info.total / 1024**2} MB')
print(f'Used: {info.used / 1024**2} MB')
print(f'Free: {info.free / 1024**2} MB')

Advanced: GPU Sharing with MPS and CUDA MPS

CUDA Multi-Process Service (MPS) allows multiple processes to share a single GPU context, improving utilization for small kernels. Unlike time-slicing, MPS provides better performance isolation. Enable it via the device plugin's mps sharing strategy. However, MPS has limitations: it requires a control daemon and can be unstable with certain CUDA versions. Use for inference serving with many small requests.

apiVersion: v1
kind: ConfigMap
metadata:
  name: nvidia-device-plugin-config
  namespace: kube-system
data:
  config.yaml: |
    version: v1
    sharing:
      mps:
        resources:
        - name: nvidia.com/gpu
          replicas: 4

Troubleshooting Common GPU Issues

Common issues: (1) Pod stuck in Pending: no GPU node available or resource name wrong. Check node capacity and taints. (2) Pod starts but nvidia-smi fails: driver not installed or runtime not configured. Check pod logs for 'cannot find CUDA driver'. (3) GPU OOM: monitor with DCGM. (4) XID errors: hardware failure, drain node. (5) Device plugin not registering: check kubelet logs. Always start with kubectl describe pod and node conditions.

# Check node GPU capacity
kubectl get node gpu-node -o json | jq '.status.capacity'
# Check device plugin logs
kubectl logs -n kube-system -l name=nvidia-device-plugin-ds
# Check pod events
kubectl describe pod gpu-pod | grep -A5 Events