Kubernetes Troubleshooting — Complete Guide

Welcome to Kubernetes Troubleshooting — Complete Guide. We'll break this down from first principles.

1. Understanding the Kubernetes Troubleshooting Hierarchy

Kubernetes troubleshooting follows a layered approach: start from the bottom (pods) and work up to the cluster level. Most issues manifest as pod failures, but the root cause often lies in the control plane, networking, or resource constraints. The hierarchy includes: Pods → Services/Ingress → Nodes → Control Plane → Storage/Networking. Always check pod logs and events first, then examine node health, and finally inspect cluster-level components like kube-apiserver or etcd. This prevents wasted effort on symptoms rather than causes. For example, a CrashLoopBackOff might be due to a missing ConfigMap, which is a pod-level config issue, but if all pods on a node fail, suspect a node problem. Use kubectl get events --all-namespaces to see cluster-wide events quickly.

kubectl get events --all-namespaces --sort-by='.lastTimestamp' | tail -20

2. Diagnosing Pod Failures: CrashLoopBackOff and ImagePullBackOff

Pod failures are the most common Kubernetes issues. CrashLoopBackOff means the container starts but crashes repeatedly. ImagePullBackOff means the container image cannot be pulled. For CrashLoopBackOff, check logs with kubectl logs <pod> --previous to see the last crash's output. Common causes: missing environment variables, misconfigured secrets, or application bugs. For ImagePullBackOff, verify the image name, tag, and registry credentials. Use kubectl describe pod <pod> to see the exact error. If the image is private, ensure the imagePullSecret is correctly configured. Also check if the node has disk space issues preventing image extraction. A systematic approach: describe the pod, check logs, then inspect the node's disk and network.

# Get pod details
kubectl describe pod my-app-7d4f8b6c9-abcde

# View logs from previous crash
kubectl logs my-app-7d4f8b6c9-abcde --previous

# Check image pull secret
kubectl get secret my-registry-key -o yaml

3. Debugging Services and Endpoints

When a service is unreachable, the problem often lies in endpoint configuration or network policies. First, verify the service exists and has endpoints: kubectl get svc and kubectl get endpoints. If endpoints are empty, the selector doesn't match any pods. Check pod labels: kubectl get pods --show-labels. Also ensure the service port matches the container port. For headless services, endpoints should list pod IPs. If endpoints exist but the service is unreachable, test connectivity from a pod: kubectl exec -it <pod> -- curl <service-name>:<port>. Network policies might block traffic; check with kubectl describe networkpolicy. Also verify kube-proxy is running on nodes and iptables rules are correct.

# Check service and endpoints
kubectl get svc my-service
kubectl get endpoints my-service

# Test connectivity from a pod
kubectl run test-pod --image=busybox --rm -it --restart=Never -- sh -c "wget -qO- http://my-service:8080"

# Check network policies
kubectl get networkpolicy -n default

4. Node-Level Troubleshooting: Disk, Memory, and Network

Node issues can cause widespread pod failures. Common problems: disk pressure, memory pressure, or network unavailability. Check node status with kubectl get nodes and kubectl describe node <node>. Look for conditions like DiskPressure, MemoryPressure, or PIDPressure. For disk pressure, check disk usage on the node via SSH or use kubectl debug node/<node> -it --image=busybox to run a debug pod. Memory pressure may require adjusting resource limits or adding nodes. Network issues can be diagnosed with kubectl get nodes -o wide to see internal IPs and then ping from another node. Also check kubelet logs: journalctl -u kubelet -n 100 on the node. If the node is NotReady, restart kubelet or check the container runtime.

# Check node status
kubectl get nodes
kubectl describe node worker-1

# Debug node with ephemeral container
kubectl debug node/worker-1 -it --image=busybox -- chroot /host

# Check disk usage inside node debug pod
df -h

# Check kubelet logs (on the node)
journalctl -u kubelet -n 50 --no-pager

5. Control Plane Troubleshooting: API Server, Scheduler, and Controller Manager

Control plane failures affect the entire cluster. Symptoms: kubectl commands hang or return errors, pods not being scheduled, or resources not being reconciled. Check control plane pod status: kubectl get pods -n kube-system. Common issues: etcd leader election failures, API server certificate expiration, or scheduler misconfiguration. For API server issues, check its logs: kubectl logs -n kube-system kube-apiserver-master. If the API server is down, you can't use kubectl; SSH into the master node and check the kube-apiserver process. For scheduler issues, check if pending pods exist: kubectl get pods --field-selector=status.phase=Pending. If pods are pending but nodes are available, the scheduler might be stuck. Restarting the scheduler pod often helps.

# Check control plane pods
kubectl get pods -n kube-system | grep -E 'apiserver|scheduler|controller|etcd'

# Check API server logs
kubectl logs -n kube-system kube-apiserver-master --tail=50

# Check pending pods
kubectl get pods --all-namespaces --field-selector=status.phase=Pending

# Check etcd cluster health (from master node)
etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt --key=/etc/kubernetes/pki/etcd/healthcheck-client.key endpoint health

6. Networking Troubleshooting: DNS, CNI, and Ingress

Networking issues are notoriously tricky. Common problems: DNS resolution failures, CNI plugin misconfiguration, or Ingress controller not routing traffic. For DNS, check CoreDNS pods: kubectl get pods -n kube-system -l k8s-app=kube-dns. If they are running, test DNS from a pod: kubectl exec -it <pod> -- nslookup kubernetes.default.svc.cluster.local. If DNS fails, check CoreDNS logs and ConfigMap. For CNI issues, check if pods get IPs: kubectl get pods -o wide and see if IPs are assigned. If pods have no IP, the CNI plugin (Calico, Flannel, etc.) might be misconfigured. Check CNI daemonset logs. For Ingress, verify the Ingress resource and controller logs: kubectl logs -n ingress-nginx ingress-nginx-controller-xxxxx.

# Test DNS resolution
kubectl run dns-test --image=busybox --rm -it --restart=Never -- nslookup kubernetes.default.svc.cluster.local

# Check CoreDNS logs
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=20

# Check pod IPs
kubectl get pods -o wide

# Check Ingress controller logs
kubectl logs -n ingress-nginx deployment/ingress-nginx-controller --tail=50

7. Storage Troubleshooting: PersistentVolumeClaims and Mount Issues

Storage issues cause pods to fail to start or become unresponsive. Common problems: PVC stuck in Pending, pod unable to mount volume, or volume permissions errors. Check PVC status: kubectl get pvc. If Pending, the StorageClass might not exist or the provisioner is unavailable. Verify StorageClass: kubectl get storageclass. If the PVC is Bound but pod can't mount, check the pod events: kubectl describe pod <pod>. Common mount errors include wrong filesystem type or permission denied. For NFS or cloud volumes, ensure the node has the necessary drivers. Also check if the volume is already mounted by another pod (ReadWriteOnce). Use kubectl get pv to see the underlying volume.

# Check PVC status
kubectl get pvc
kubectl describe pvc my-pvc

# Check StorageClass
kubectl get storageclass

# Check pod events for mount errors
kubectl describe pod my-app-7d4f8b6c9-abcde | grep -A5 Mount

# Check PV details
kubectl get pv
kubectl describe pv pvc-xxxxx

8. Using kubectl Debug and Ephemeral Containers

Kubernetes 1.18+ introduced ephemeral containers for debugging running pods without restarting them. This is invaluable for troubleshooting production issues where you can't modify the pod spec. Use kubectl debug to add an ephemeral container to a running pod. For example, to debug a pod with a network issue: kubectl debug -it <pod> --image=nicolaka/netshoot --target=<container>. The ephemeral container shares the same network namespace, so you can run curl, tcpdump, etc. You can also debug nodes: kubectl debug node/<node> -it --image=busybox. This creates a pod with host access. Ephemeral containers are temporary and removed when the pod is deleted.

# Debug a running pod with netshoot
kubectl debug -it my-app-7d4f8b6c9-abcde --image=nicolaka/netshoot --target=my-app -- sh

# Inside the ephemeral container
curl http://my-service:8080

# Debug a node
kubectl debug node/worker-1 -it --image=busybox -- chroot /host

# Check processes on the node
ps aux | grep kubelet

9. Logging and Monitoring: Centralized Logs and Metrics

Centralized logging and monitoring are essential for proactive troubleshooting. Use tools like Prometheus for metrics and Loki/Elasticsearch for logs. In Kubernetes, common patterns: Fluentd or Filebeat to ship logs, and Prometheus Operator for metrics. For quick ad-hoc monitoring, use kubectl top to see resource usage: kubectl top pods and kubectl top nodes. If metrics aren't available, ensure metrics-server is installed. For logs, use kubectl logs --tail=100 --follow for real-time streaming. In production, set up alerts for common failure modes: pod restarts, node pressure, and API server latency. Use kubectl get events --watch to monitor cluster events in real-time.

# Check resource usage
kubectl top pods
kubectl top nodes

# Stream logs from multiple pods
kubectl logs -l app=my-app --tail=10 --follow

# Watch cluster events
kubectl get events --watch --all-namespaces

# Check metrics-server
kubectl get deployment metrics-server -n kube-system

10. Common Pitfalls and Anti-Patterns

Avoid these common mistakes: 1) Ignoring resource limits — pods without limits can starve other pods. 2) Using latest tag — leads to unpredictable behavior. 3) Not setting pod disruption budgets — causes downtime during node maintenance. 4) Overlooking network policies — default allow-all can be a security risk. 5) Skipping health checks — liveness and readiness probes prevent routing traffic to unhealthy pods. 6) Using hostNetwork without understanding the implications. 7) Not backing up etcd — without backups, cluster recovery is impossible. 8) Manually editing pods — always use deployments. 9) Ignoring pod anti-affinity — can cause all replicas to land on the same node. 10) Not using namespaces — leads to resource conflicts.

apiVersion: v1
kind: Pod
metadata:
  name: my-app
spec:
  containers:
  - name: my-app
    image: my-app:1.0.0
    resources:
      requests:
        memory: "256Mi"
        cpu: "250m"
      limits:
        memory: "512Mi"
        cpu: "500m"
    livenessProbe:
      httpGet:
        path: /healthz
        port: 8080
      initialDelaySeconds: 5
      periodSeconds: 10
    readinessProbe:
      httpGet:
        path: /ready
        port: 8080
      initialDelaySeconds: 5
      periodSeconds: 10

11. Advanced Troubleshooting: Using kubectl plugin and krew

Kubectl plugins extend its functionality. Install krew, the plugin manager: kubectl krew install <plugin>. Essential plugins: kubectl-neat to clean up YAML output, kubectl-tree to show resource ownership hierarchy, kubectl-oom to find pods killed by OOM, kubectl-whisper for secret management, and kubectl-sniff to capture network traffic. For example, kubectl sniff <pod> -p <port> starts a remote tcpdump. kubectl oom lists pods that have been OOMKilled. These plugins save time and provide deeper insights. To install krew: kubectl krew install sniff oom tree neat.

# Install krew (if not installed)
(
  set -x; cd "$(mktemp -d)" &&
  OS="$(uname | tr '[:upper:]' '[:lower:]')" &&
  ARCH="$(uname -m | sed -e 's/x86_64/amd64/' -e 's/arm64/aarch64/')" &&
  KREW="krew-${OS}_${ARCH}" &&
  curl -fsSLO "https://github.com/kubernetes-sigs/krew/releases/latest/download/${KREW}.tar.gz" &&
  tar zxvf "${KREW}.tar.gz" &&
  ./"${KREW}" install krew
)

# Add krew to PATH
export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH"

# Install plugins
kubectl krew install sniff oom tree neat

# Use plugins
kubectl sniff my-pod -p 8080
kubectl oom
kubectl tree deployment my-app

12. Building a Troubleshooting Runbook

A runbook standardizes troubleshooting steps, reducing MTTR. Start with a decision tree: Is the issue pod-level? Node-level? Cluster-level? For each, list commands and expected outputs. Include sections for common scenarios: CrashLoopBackOff, service unreachable, node NotReady, etc. Document escalation paths and contact information. Use tools like Backstage or GitBook to host the runbook. Keep it version-controlled and update it after every incident. Example structure: 1) Initial checks (events, top, describe), 2) Pod issues (logs, previous, debug), 3) Service issues (endpoints, connectivity), 4) Node issues (conditions, disk, kubelet), 5) Control plane (apiserver, etcd), 6) Escalation. Automate common checks with scripts or chatbots.

# Runbook: Pod CrashLoopBackOff
# Step 1: Check events
kubectl get events --field-selector involvedObject.kind=Pod --sort-by='.lastTimestamp'
# Step 2: Describe pod
kubectl describe pod <pod-name>
# Step 3: Check logs
kubectl logs <pod-name> --previous
# Step 4: Check resource usage
kubectl top pod <pod-name>
# Step 5: Check node resources
kubectl top node <node-name>
# Step 6: If image pull issue, check secret
kubectl get secret <image-pull-secret>
# Step 7: If all else fails, escalate to SRE team
# Contact: #sre-oncall