Kubernetes DaemonSets

You've just deployed a new logging agent across your 50-node Kubernetes cluster. Three hours later, you're paged because half the nodes have no logs. The rollout succeeded, but the DaemonSet didn't schedule on nodes with taints you forgot existed. This is the reality of DaemonSets: they look simple, but production will punish you for assumptions. DaemonSets are Kubernetes' answer to running a pod on every node — think kube-proxy, Calico, or Datadog agents. They're not for stateless apps; they're for infrastructure that must be everywhere. But 'everywhere' is a loaded word when you consider node taints, resource constraints, and rolling updates. In this guide, we'll cover DaemonSet mechanics, scheduling nuances, production patterns, and failure modes you'll encounter at scale.

What is a DaemonSet?

A DaemonSet is a Kubernetes workload resource that ensures a copy of a pod runs on every node in the cluster (or a subset, if node selectors are used). When nodes are added or removed, the DaemonSet controller automatically creates or deletes pods accordingly. This makes DaemonSets ideal for cluster-wide services that need to be present on each node, such as log collectors (Fluentd, Filebeat), monitoring agents (Prometheus Node Exporter, Datadog Agent), storage daemons (GlusterFS, Ceph), and network plugins (Calico, Flannel). Unlike Deployments, which aim for a desired number of replicas spread across nodes, DaemonSets guarantee one pod per node by default. This fundamental difference means DaemonSets are not scaled horizontally; they scale with the cluster size. The DaemonSet controller watches the Kubernetes API for node changes and ensures the desired state is maintained. It respects node taints and tolerations, node selectors, and affinity rules, giving you fine-grained control over which nodes run the pod.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd-elasticsearch
  namespace: kube-system
  labels:
    k8s-app: fluentd-logging
spec:
  selector:
    matchLabels:
      name: fluentd-elasticsearch
  template:
    metadata:
      labels:
        name: fluentd-elasticsearch
    spec:
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      containers:
      - name: fluentd-elasticsearch
        image: quay.io/fluentd_elasticsearch/fluentd:v2.5.2
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: dockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: dockercontainers
        hostPath:
          path: /var/lib/docker/containers

DaemonSet Scheduling: How Pods Land on Nodes

The DaemonSet controller uses the Kubernetes scheduler to place pods on nodes, but with a twist: it bypasses the default scheduler for initial placement. Instead, the controller directly creates pods on each node that matches the DaemonSet's node selector and tolerations. This means DaemonSet pods are scheduled even if the cluster has a custom scheduler or if the default scheduler is misconfigured. The controller watches for new nodes and creates pods immediately, without going through the scheduling queue. However, if a node becomes unschedulable (e.g., due to taints), the DaemonSet will not create a pod there unless the pod template includes matching tolerations. This is a common pitfall: forgetting to add tolerations for master nodes or nodes with custom taints. Additionally, DaemonSets respect node affinity and anti-affinity rules, but anti-affinity is rarely needed since there's only one pod per node by design. The scheduling behavior can be further controlled using the .spec.template.spec.nodeSelector field or by setting .spec.template.spec.affinity.nodeAffinity. For advanced use cases, you can use .spec.updateStrategy.rollingUpdate.maxSurge to control how many extra pods are created during rolling updates.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-exporter
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: node-exporter
  template:
    metadata:
      labels:
        app: node-exporter
    spec:
      nodeSelector:
        kubernetes.io/os: linux
      tolerations:
      - operator: Exists
      containers:
      - name: node-exporter
        image: prom/node-exporter:v1.3.1
        ports:
        - containerPort: 9100
          hostPort: 9100
        resources:
          requests:
            cpu: 100m
            memory: 50Mi
          limits:
            cpu: 200m
            memory: 100Mi

Rolling Updates and Rollbacks

DaemonSets support rolling updates to update the pod template (e.g., image version, resource limits) without downtime. The update strategy is configured via .spec.updateStrategy.type, which can be RollingUpdate (default) or OnDelete. With RollingUpdate, the DaemonSet controller updates pods one by one, respecting maxUnavailable and maxSurge parameters. maxUnavailable (default 1) specifies how many pods can be unavailable during the update. maxSurge (default 0) specifies how many extra pods can be created above the desired per-node count. Setting maxSurge to 1 allows the controller to create a new pod on a node before deleting the old one, reducing downtime for critical daemons. However, this consumes extra resources. The OnDelete strategy only updates pods when they are manually deleted, giving you full control. Rollbacks are performed by reverting the DaemonSet's pod template to a previous version. Kubernetes does not automatically track revisions for DaemonSets like it does for Deployments, so you must manually apply an older YAML or use kubectl rollout undo. To enable revision history, you can annotate the DaemonSet, but it's not built-in. For production, consider using a GitOps workflow to manage DaemonSet versions.

# Update the DaemonSet image
kubectl set image daemonset/fluentd-elasticsearch fluentd-elasticsearch=quay.io/fluentd_elasticsearch/fluentd:v2.5.3 -n kube-system

# Check rollout status
kubectl rollout status daemonset/fluentd-elasticsearch -n kube-system

# Rollback if needed
kubectl rollout undo daemonset/fluentd-elasticsearch -n kube-system

Resource Management and Node Pressure

DaemonSet pods share node resources with other workloads. If a DaemonSet pod consumes too much CPU or memory, it can cause resource starvation for application pods on that node. This is especially dangerous for daemons like log collectors or monitoring agents that might have unbounded resource usage. Always set resource requests and limits on DaemonSet containers. Requests guarantee the pod gets that amount of resources; limits prevent it from consuming more. For example, a Fluentd DaemonSet might request 100m CPU and 200Mi memory, with limits of 200m CPU and 400Mi memory. If the node runs out of resources, the kubelet may evict pods, starting with those exceeding their limits. DaemonSet pods are typically not critical enough to warrant priority classes, but you can set them if needed. Additionally, consider using resourceQuota at the namespace level to cap total DaemonSet resource usage. Another common issue is disk pressure: DaemonSets that write to hostPath volumes (e.g., log collectors) can fill up the node's disk. Monitor disk usage and set log rotation policies. Finally, DaemonSets with high network I/O (e.g., network plugins) can saturate the node's bandwidth. Use resource limits and quality of service (QoS) classes to mitigate.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: logging
spec:
  selector:
    matchLabels:
      app: filebeat
  template:
    metadata:
      labels:
        app: filebeat
    spec:
      containers:
      - name: filebeat
        image: docker.elastic.co/beats/filebeat:7.17.0
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
          limits:
            cpu: 200m
            memory: 200Mi
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: data
          mountPath: /usr/share/filebeat/data
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: data
        hostPath:
          path: /var/lib/filebeat-data

DaemonSets and Cluster Autoscaling

Cluster autoscalers (e.g., Karpenter, Cluster Autoscaler) add or remove nodes based on resource demands. DaemonSets interact with autoscaling in two ways: they consume resources on every node, and they must be considered when calculating node capacity. When a new node is added, the DaemonSet controller creates pods on it, consuming resources that could otherwise be used by pending pods. This can cause the autoscaler to add more nodes than necessary if DaemonSet resource requests are high. To avoid this, you can set .spec.template.spec.priorityClassName to a lower priority for DaemonSet pods, allowing application pods to preempt them. However, this is risky for critical daemons. Alternatively, configure the autoscaler to ignore DaemonSet resource consumption by setting --skip-nodes-with-system-pods=false (Cluster Autoscaler) or using karpenter.k8s.aws/do-not-disrupt annotations. Another approach is to use node affinity to exclude DaemonSets from certain node pools, but this defeats the purpose. Best practice: keep DaemonSet resource requests minimal and monitor their impact on autoscaling decisions. For example, a monitoring DaemonSet requesting 100m CPU per node on a 100-node cluster consumes 10 CPU cores total—non-trivial.

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: daemonset-priority
value: 1000
globalDefault: false
description: "Priority for DaemonSet pods"
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: low-priority-daemon
spec:
  template:
    spec:
      priorityClassName: daemonset-priority
      containers:
      - name: main
        image: busybox
        command: ["sleep", "infinity"]
        resources:
          requests:
            cpu: 10m
            memory: 10Mi

DaemonSets with Host Networking and Ports

DaemonSets often need to access host-level resources, such as network interfaces or ports. You can enable host networking by setting hostNetwork: true in the pod spec, which makes the pod use the node's network stack directly. This is common for network plugins (e.g., Calico, Flannel) and node monitoring tools. When using host networking, the pod can bind to host ports, but you must ensure port conflicts don't occur. Use hostPort to expose a container port on the host IP. For example, a Node Exporter DaemonSet might use hostPort: 9100 to make metrics available on the node's IP. However, hostPort is not recommended for production because it can cause port conflicts if multiple DaemonSets try to use the same port. Instead, use a hostNetwork with a unique port, or use a NodePort service to expose the daemon. Another consideration: when using host networking, the pod's network performance is identical to the host's, which is beneficial for latency-sensitive daemons. However, it also means the pod is not isolated from the host network, which is a security concern. For security, avoid host networking unless necessary.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-exporter
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: node-exporter
  template:
    metadata:
      labels:
        app: node-exporter
    spec:
      hostNetwork: true
      hostPID: true
      containers:
      - name: node-exporter
        image: prom/node-exporter:v1.3.1
        args:
        - --path.procfs=/host/proc
        - --path.sysfs=/host/sys
        ports:
        - containerPort: 9100
          hostPort: 9100
        volumeMounts:
        - name: proc
          mountPath: /host/proc
          readOnly: true
        - name: sys
          mountPath: /host/sys
          readOnly: true
      volumes:
      - name: proc
        hostPath:
          path: /proc
      - name: sys
        hostPath:
          path: /sys

DaemonSet and Persistent Volumes

DaemonSets can use persistent volumes, but with caveats. Since each pod runs on a different node, you typically need a volume that is node-local (e.g., hostPath) or a shared network filesystem (e.g., NFS, EFS). hostPath volumes are the most common, as they allow the pod to access the node's filesystem. However, hostPath volumes are not portable and can cause issues if the pod is rescheduled to a different node (which shouldn't happen with DaemonSets, but can during updates). For stateful daemons that need persistent storage across restarts, consider using a local persistent volume with node affinity, or a network volume like NFS. Be aware that network volumes introduce latency and potential single points of failure. Another option is to use emptyDir volumes for temporary data, but data is lost on pod restart. For log collectors, hostPath is standard. For databases running as DaemonSets (rare), you'd need a distributed storage solution. In production, avoid using DaemonSets for stateful workloads that require durable storage; use StatefulSets instead.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: log-collector
  namespace: logging
spec:
  selector:
    matchLabels:
      app: log-collector
  template:
    metadata:
      labels:
        app: log-collector
    spec:
      containers:
      - name: log-collector
        image: busybox
        command: ["sh", "-c", "tail -f /var/log/syslog"]
        volumeMounts:
        - name: varlog
          mountPath: /var/log
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
          type: Directory

DaemonSet Health Checks and Readiness Probes

DaemonSet pods should have liveness and readiness probes to ensure they are healthy and ready to serve traffic. Liveness probes restart the pod if it becomes unhealthy; readiness probes control whether the pod receives traffic from services. For DaemonSets that expose a service (e.g., monitoring agents), readiness probes are crucial to avoid routing traffic to a broken pod. However, DaemonSets often don't have a service fronting them; they are accessed via hostIP or DNS. In that case, readiness probes are less critical but still useful for rolling updates. A common pattern is to use an HTTP probe on a health endpoint, or a TCP probe on a known port. For daemons that take time to initialize (e.g., loading rules), set initialDelaySeconds appropriately. Avoid using exec probes that run heavy commands, as they can consume resources. Also, consider using startupProbe for slow-starting containers to prevent premature restarts. In production, always define at least a liveness probe to detect deadlocks or OOM kills.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: logging
spec:
  selector:
    matchLabels:
      app: fluentd
  template:
    metadata:
      labels:
        app: fluentd
    spec:
      containers:
      - name: fluentd
        image: fluent/fluentd:v1.14-1
        ports:
        - containerPort: 24224
        livenessProbe:
          tcpSocket:
            port: 24224
          initialDelaySeconds: 10
          periodSeconds: 10
        readinessProbe:
          tcpSocket:
            port: 24224
          initialDelaySeconds: 5
          periodSeconds: 10

DaemonSet Security: ServiceAccounts and RBAC

DaemonSet pods often need to interact with the Kubernetes API to discover nodes, watch resources, or report metrics. This requires a ServiceAccount with appropriate RBAC permissions. For example, a network plugin DaemonSet might need permissions to create routes or modify nodes. Always follow the principle of least privilege: grant only the permissions the daemon needs. Create a dedicated ServiceAccount for the DaemonSet and bind it to a Role or ClusterRole. Avoid using the default ServiceAccount, which may have excessive permissions. Also, consider using Pod Security Standards (PSS) or OPA/Gatekeeper to enforce security policies on DaemonSet pods. For hostNetwork DaemonSets, restrict capabilities and enable seccomp profiles. Another security concern: DaemonSet pods running as privileged containers can escape to the host. If your daemon requires privileged mode (e.g., for network configuration), ensure it's absolutely necessary and audit the image. In production, scan DaemonSet images for vulnerabilities and use image signing.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluentd
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluentd
rules:
- apiGroups: [""]
  resources: ["pods", "namespaces"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: fluentd
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: fluentd
subjects:
- kind: ServiceAccount
  name: fluentd
  namespace: kube-system
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: kube-system
spec:
  template:
    spec:
      serviceAccountName: fluentd
      containers:
      - name: fluentd
        image: fluent/fluentd:v1.14-1

DaemonSet Monitoring and Observability

Monitoring DaemonSets is critical because they run on every node and can impact cluster health. Key metrics to track: number of pods scheduled vs. desired (should be equal), pod restart count, resource usage (CPU, memory, disk), and rollout status. Use Prometheus to scrape metrics from DaemonSet pods, or use the Kubernetes API to monitor DaemonSet status. Set up alerts for when a DaemonSet pod is not running on a node (e.g., due to taints or resource constraints). Also monitor for crash loops: a DaemonSet in a crash loop can cause cascading failures. For log collectors, ensure logs are being shipped correctly; missing logs from a node indicate a problem. Use tools like kubectl describe daemonset to see events, or use a dashboard like Grafana with the Kubernetes mixin. In production, implement a health check that verifies DaemonSet coverage: every node should have a running pod. This can be done with a custom script or using the kube-state-metrics exporter.

# Check DaemonSet status
kubectl get daemonsets -A

# Get detailed status
kubectl describe daemonset fluentd -n kube-system

# Check pod distribution
kubectl get pods -n kube-system -l name=fluentd -o wide

# Alert if number of pods != number of ready nodes
NODES=$(kubectl get nodes --no-headers | wc -l)
PODS=$(kubectl get pods -n kube-system -l name=fluentd --field-selector=status.phase=Running --no-headers | wc -l)
if [ $PODS -ne $NODES ]; then
  echo "DaemonSet not covering all nodes: $PODS pods on $NODES nodes"
fi

Common DaemonSet Pitfalls and How to Avoid Them

1. Forgetting tolerations: DaemonSet pods won't schedule on tainted nodes. Always add tolerations for known taints (e.g., master node taint). 2. Resource limits too high: DaemonSet resource requests can consume significant cluster capacity. Keep them minimal. 3. Host port conflicts: Multiple DaemonSets using the same hostPort will fail. Use unique ports or avoid hostPort. 4. Rolling update misconfiguration: Setting maxUnavailable too high can cause mass pod deletion. Use maxSurge for critical daemons. 5. Ignoring node selectors: If you use nodeSelector, ensure it matches all intended nodes. Otherwise, some nodes will be missed. 6. Not setting terminationGracePeriodSeconds: DaemonSet pods may need time to flush buffers before shutdown. Set an appropriate grace period. 7. Using DaemonSets for stateful workloads: DaemonSets are not designed for stateful applications; use StatefulSets. 8. Lack of monitoring: Without coverage alerts, you may not notice missing pods. 9. Privileged containers without need: Avoid privileged mode unless absolutely necessary. 10. Not testing updates: Always test DaemonSet updates in a staging environment first.

# Example of a DaemonSet with common fixes
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: good-daemon
spec:
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  template:
    spec:
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      terminationGracePeriodSeconds: 60
      containers:
      - name: main
        image: busybox
        command: ["sleep", "infinity"]
        resources:
          requests:
            cpu: 10m
            memory: 10Mi
          limits:
            cpu: 50m
            memory: 50Mi
        livenessProbe:
          exec:
            command: ["true"]
          initialDelaySeconds: 5
          periodSeconds: 10

DaemonSet Alternatives and When Not to Use Them

DaemonSets are not always the right choice. For workloads that don't need to run on every node, use Deployments or StatefulSets. For example, a web server should be a Deployment with replicas spread across nodes. If you need to run a task on each node once (e.g., node initialization), consider a Job or CronJob. For workloads that require ordered deployment and stable network identities, use StatefulSets. Another alternative is to use node affinity with a Deployment to ensure pods are scheduled on specific nodes, but this is less reliable than DaemonSets. For cluster-wide services that can tolerate missing a node (e.g., caching), a Deployment with anti-affinity might suffice. Also, consider using Kuberenetes Operators that manage node-level agents. In production, evaluate whether you truly need a pod on every node. If the service can function with fewer replicas, a Deployment is more flexible and easier to manage. DaemonSets add operational overhead: they consume resources on every node, and updates affect the entire cluster. Use them only for infrastructure that must be omnipresent.

# Alternative: Deployment with node anti-affinity
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - my-app
            topologyKey: kubernetes.io/hostname
      containers:
      - name: app
        image: nginx