DevOps Advanced

Kubernetes Pods and Deployments

Q: How do I perform a canary deployment without a service mesh?

Use `kubectl rollout pause` after the Deployment creates one new Pod. The new Pod will be added to the Service endpoints automatically. Monitor error rates and latency on the canary Pod. If healthy, run `kubectl rollout resume` to complete the rollout. If unhealthy, run `kubectl rollout undo` to revert. Combine with `maxSurge: 1` to ensure only one canary Pod is created during the pause.

📅 March 06, 2026 ⏱ 3 min read 🎯 Advanced

Where developers are forged. · Structured learning · Free forever.

📍 Part of: Kubernetes → Topic 2 of 12

Master Kubernetes Pods and Deployments: architectural differences, zero-downtime rolling updates, production-grade health check strategies, and debugging patterns for senior engineers.

🔥 Advanced — solid DevOps foundation required

In this tutorial, you'll learn

Master Kubernetes Pods and Deployments: architectural differences, zero-downtime rolling updates, production-grade health check strategies, and debugging patterns for senior engineers.

Pods are ephemeral and non-self-healing — never deploy individual Pods for production workloads.
Deployments are controllers that maintain the desired number of Pod replicas and handle lifecycle management.
Use 'RollingUpdate' with 'maxUnavailable: 0' to achieve zero-downtime application updates.

✦ Plain-English analogy ✦ Real code with output ✦ Interview questions

⚡Quick Answer

Pod: Ephemeral, non-self-healing. Shares a network namespace (localhost) and volumes across containers.
Deployment: Manages ReplicaSets which manage Pods. Handles rolling updates, rollbacks, and scaling.
ReplicaSet: The intermediate controller that ensures N replicas exist. You rarely interact with it directly.
Probes: Liveness (restart if deadlocked) vs Readiness (remove from service endpoints if not ready).
maxSurge: More surge = faster rollout but higher peak resource usage.
maxUnavailable: 0 = zero-downtime but slower rollout. Higher = faster but riskier.
Creating Pods directly instead of Deployments. Direct Pods are not self-healing. If the node dies, the Pod stays dead forever.

🚨 START HERE

Pod and Deployment Triage Commands

Rapid commands to isolate Pod lifecycle and Deployment issues.

🟡Pod in CrashLoopBackOff.

Immediate ActionCheck previous container logs and exit code.

Commands

kubectl logs <pod> --previous --tail=50

kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.exitCode}'

Fix NowExit code 137 = OOMKill (increase memory limit). Exit code 1 = application error (check logs). Exit code 143 = SIGTERM (check graceful shutdown).

🟡Pod stuck in Pending.

Immediate ActionCheck Pod events and node resources.

Commands

kubectl describe pod <pod> | grep -A 20 Events

kubectl describe nodes | grep -A 5 'Allocated resources'

Fix NowIf 'Insufficient cpu/memory', scale cluster or reduce requests. If 'node(s) had taint', add tolerations. If 'waiting for volume', check PVC status.

🟡Deployment rollout stuck.

Immediate ActionCheck rollout status and new Pod readiness.

Commands

kubectl rollout status deployment/<name> --timeout=30s

kubectl get pods -l app=<label> -o wide | grep -v Running

Fix NowIf new Pods are not Ready, check readiness probe. If rollout is stuck, rollback: `kubectl rollout undo deployment/<name>`.

🟡502 errors during deploy.

Immediate ActionCheck Service endpoints during rollout.

Commands

kubectl get endpoints <service> -w

kubectl get pod -l app=<label> -o jsonpath='{range .items[*]}{.metadata.name} {.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}'

Fix NowIf endpoints are empty during rollout, Pods are not passing readiness probes. Add startup probe and increase terminationGracePeriodSeconds.

🟡Pod evicted from node.

Immediate ActionCheck node conditions and Pod QoS class.

Commands

kubectl describe node <node> | grep -A 10 Conditions

kubectl get pod <pod> -o jsonpath='{.status.qosClass}'

Fix NowIf QoS is BestEffort, set resource requests and limits. If node has DiskPressure, clean up unused images. If MemoryPressure, reduce Pod memory usage.

Production IncidentRolling Update Caused 502 Errors for 8 Minutes During Production DeployA routine deployment of a payment API caused sustained 502 errors for 8 minutes. The rollout appeared successful in kubectl, but users experienced connection refused errors as new Pods received traffic before they were ready.

SymptomHTTP 502 and connection refused errors spiked immediately after kubectl apply. The rollout completed in 3 minutes but 502 errors persisted for 5 additional minutes. kubectl get pods showed all Pods in Running state. No CrashLoopBackOff. No OOMKill events.

AssumptionThe new application version had a bug that caused it to reject requests.

Root causeThe Deployment had maxUnavailable: 0 (correct for zero-downtime) but the readiness probe was misconfigured: initialDelaySeconds: 5 and periodSeconds: 10 with failureThreshold: 1. The application took 45 seconds to warm its cache and connect to the database. During this 45-second window, the readiness probe failed once after 5 seconds, but Kubernetes had already added the Pod to the Service endpoints because the first probe check had not yet run. The kube-proxy rules were updated to route traffic to the new Pod before it was actually ready. Additionally, the old Pods were terminated immediately when the new Pods passed their first readiness check, creating a window where neither old nor new Pods were fully serving traffic.

Fix1. Added a startup probe with failureThreshold: 60 and periodSeconds: 5 (300 seconds max startup time) to gate liveness and readiness probes until the app is fully booted. 2. Changed readiness probe initialDelaySeconds to 0 (startup probe handles the delay) and set failureThreshold: 3 with periodSeconds: 5. 3. Added terminationGracePeriodSeconds: 60 with a pre-stop hook that sleeps 15 seconds to drain in-flight connections before the container is killed. 4. Set maxSurge: 1 to ensure at least one new Pod is fully ready before old Pods are terminated. 5. Added a PodDisruptionBudget with minAvailable: 2 to prevent simultaneous termination of multiple Pods.

Key Lesson

Readiness probes must reflect actual readiness, not just process liveness. A Pod that is 'running' is not the same as a Pod that is 'ready'.Startup probes are mandatory for applications with warm-up times greater than 30 seconds. Without them, liveness probes kill the container during boot.Always combine terminationGracePeriodSeconds with a pre-stop hook to drain in-flight connections before container shutdown.Test rolling updates in staging with real traffic patterns. The first time you see a rollout failure should not be in production.PodDisruptionBudgets prevent the Deployment from terminating too many Pods simultaneously during updates or node drains.

Production Debug GuideSymptom-first investigation path for Pod lifecycle and Deployment rollout failures.

Pod stuck in CrashLoopBackOff.→Check container logs with kubectl logs <pod> --previous to see why it crashed. Check for OOMKill (exit code 137), missing environment variables, or failed startup dependencies. If the crash happens during startup, add a startup probe.

Pod stuck in Pending with no events.→Check node resources with kubectl describe nodes | grep -A 5 Allocatable. Check for insufficient CPU/memory, PVC binding failures, or taints/tolerations mismatches. If no nodes can schedule the Pod, it stays Pending indefinitely.

Deployment rollout hangs at 'Waiting for rollout'.→Check kubectl rollout status deployment/<name>. If maxUnavailable is 0 and a new Pod cannot become ready, the rollout blocks. Check readiness probe failures. Check for PDB conflicts that prevent Pod termination.

Pods restarting frequently but not in CrashLoopBackOff.→Check liveness probe configuration. If failureThreshold is too low or periodSeconds is too aggressive, healthy-but-slow Pods are killed. Check liveness probe endpoint latency. Consider using a startup probe to gate liveness.

502/503 errors during Deployment rollout.→Check readiness probe timing. Ensure new Pods are ready before old Pods are terminated. Check terminationGracePeriodSeconds and pre-stop hooks. Verify maxSurge and maxUnavailable settings. Check Service endpoints during rollout: kubectl get endpoints <service> -w.

Pod evicted with reason 'Evicted' and no OOMKill.→The kubelet evicted the Pod due to node pressure (disk, memory, PID). Check kubectl describe node <node> | grep Conditions. Check if the Pod's QoS class is BestEffort (first to be evicted). Set resource requests and limits to achieve Guaranteed QoS.

Every production Kubernetes workload ultimately runs as Pods managed by Deployments. The Pod is the atomic scheduling unit — a group of containers sharing a network namespace and volumes. The Deployment is the declarative controller that ensures the right number of Pods exist, handles rolling updates, and rolls back on failure.

Misconfiguring either object causes production incidents: Pods without resource limits cause noisy-neighbor OOMKills, Deployments without proper probes cause 502 errors during rollouts, and direct Pod creation bypasses self-healing entirely. Understanding the reconciliation loop — how the Deployment controller continuously drives current state toward desired state — is the foundation for debugging every higher-level Kubernetes object.

Pod Basics: The Atomic Unit

In the world of Kubernetes, the Pod is the atomic unit of scheduling. While you might be used to thinking in terms of 'containers,' Kubernetes thinks in 'Pods.' A Pod can host a single container, or a tightly coupled group of containers (like an app container and a 'sidecar' logging agent) that need to share the same local network (localhost) and storage volumes.

Crucially, Pods are ephemeral. They are born, they live, and they die. They are never 'repaired'; they are replaced.

io/thecodeforge/k8s/pod.yaml · YAML

123456789101112131415161718192021222324252627282930

# Package: io.thecodeforge.k8s.manifests
# pod.yaml — used primarily for local debugging, rarely in production
apiVersion: v1
kind: Pod
metadata:
  name: forge-api-pod
  labels:
    app: forge-api
    env: production
spec:
  containers:
    - name: api-container
      image: io.thecodeforge/api:1.2.0
      ports:
        - containerPort: 8080
      resources:
        requests:
          memory: "128Mi"
          cpu: "100m"
        limits:
          memory: "256Mi"
          cpu: "500m"
      env:
        - name: APP_COLOR
          value: "blue"
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: cloud-secrets
              key: db-password

▶ Output

# Note: Direct Pods are not self-healing. If the node dies, the pod stays dead.

Mental Model

Pods Are Cattle, Not Pets

If your application breaks when a Pod restarts, your application is not cloud-native. It must be stateless, discover services by DNS, and store data externally.

Pod IP is assigned at creation and changes on restart. Use Service DNS for discovery.
Container filesystem is ephemeral. Use PersistentVolumes for data that must survive restarts.
Pods in the same Pod share localhost networking. Containers in different Pods do not.
Sidecar containers (logging, proxy) share the Pod's lifecycle. If one crashes, the Pod is restarted.
Init containers run before the main container. They block Pod startup until they complete successfully.

📊 Production Insight

The most dangerous Pod anti-pattern is creating Pods directly (via kubectl run or a Pod manifest) for production workloads. Direct Pods have no self-healing: if the node crashes, the Pod is gone forever with no replacement. The Deployment controller is what provides self-healing, rolling updates, and rollback. Every production workload must use a Deployment, StatefulSet, or DaemonSet — never a raw Pod.

🎯 Key Takeaway

Pods are the atomic scheduling unit but they are ephemeral and non-self-healing. They share a network namespace and volumes across containers. Never create Pods directly for production workloads — always use a Deployment.

Deployments: Orchestrating the Desired State

A Deployment is a high-level object that manages a ReplicaSet, which in turn manages Pods. Its job is to ensure the 'Desired State' matches the 'Current State.' If you tell a Deployment you want 3 replicas, and a node crashes taking one Pod with it, the Deployment controller notices the discrepancy and immediately schedules a new Pod on a healthy node.

Deployments are also the primary vehicle for Rolling Updates. By manipulating the maxSurge and maxUnavailable parameters, you can swap out version 1.0 for 2.0 without dropping a single user request.

io/thecodeforge/k8s/deployment.yaml · YAML

1234567891011121314151617181920212223242526272829303132333435363738

# deployment.yaml — The production standard
apiVersion: apps/v1
kind: Deployment
metadata:
  name: forge-api-deployment
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: forge-api
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: forge-api
    spec:
      containers:
        - name: api
          image: io.thecodeforge/api:2.0.1
          ports:
            - containerPort: 8080
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 15
            periodSeconds: 20
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 10

▶ Output

# kubectl apply -f deployment.yaml
# deployment.apps/forge-api-deployment created

Mental Model

The Reconciliation Loop

When debugging, always ask: 'What does the controller think the current state is, and what does it think the desired state is?' The gap between them is the action it will take.

Deployment creates ReplicaSets. ReplicaSets create Pods. You interact with Deployments.
Each rollout creates a new ReplicaSet. Old ReplicaSets are kept for rollback (default% of desired pods are always up template: : 10 revisions).
The Deployment controller manages the transition: scale up new ReplicaSet, scale down old ReplicaSet.
maxSurge: How many extra Pods can exist above the desired count during rollout.
maxUnavailable: How many Pods can be missing below the desired count during rollout.

📊 Production Insight

The maxSurge and maxUnavailable parameters directly control rollout speed vs resource usage. maxSurge: 25%, maxUnavailable: 0 means: during rollout, create up to 25% extra new Pods before terminating old Pods. This ensures zero-downtime but requires 125% of normal resource capacity. For resource-constrained clusters, set maxSurge: 0, maxUnavailable: 25% to terminate old Pods first (faster, less resource usage, but brief capacity reduction). The worst configuration is maxSurge: 0, maxUnavailable: 0 — the rollout blocks forever because it cannot create new Pods (no surge) and cannot terminate old Pods (no unavailability allowed).

🎯 Key Takeaway

Deployments manage ReplicaSets which manage Pods. The reconciliation loop continuously drives current state toward desired state. maxSurge and maxUnavailable control rollout speed vs resource usage. Never set both to 0 — the rollout will block forever.

Production Operations: Essential kubectl Patterns

Managing Deployments in production requires more than just apply. You need to be able to inspect the rollout history, trigger instant rollbacks, and scale on demand.

io/thecodeforge/k8s/deployment-ops.sh · BASH

123456789101112131415161718

# 1. Trigger a manual rolling update by changing the image
kubectl set image deployment/forge-api-deployment api=io.thecodeforge/api:2.1.0

# 2. Watch the rollout in real-time
kubectl rollout status deployment/forge-api-deployment

# 3. Emergency! Roll back to the previous stable version
kubectl rollout undo deployment/forge-api-deployment

# 4. View the rollout history to find a specific revision
kubectl rollout history deployment/forge-api-deployment

# 5. Scale the workload horizontally (manual scaling)
kubectl scale deployment/forge-api-deployment --replicas=10

# 6. Deep inspection of pod failures
kubectl describe pod <pod-name>
kubectl logs -f deployment/forge-api-deployment --all-containers

▶ Output

# rollout "forge-api-deployment" successfully rolled back

Mental Model

Rollback Is Not Free

The default revisionHistoryLimit is 10. After 10 rollouts, the oldest ReplicaSet is deleted and you cannot rollback to it. Increase this limit for critical services.

Rollback = scale up old ReplicaSet + scale down new ReplicaSet. No image pull needed if cached.
revisionHistoryLimit (default 10): How many old ReplicaSets to keep. Increase to 20-50 for critical services.
kubectl rollout pause/resume: Pause a rollout mid-way to test a subset of new Pods before completing.
kubectl rollout undo --to-revision=N: Rollback to a specific revision, not just the previous one.
kubectl rollout restart: Triggers a rolling restart without changing the image. Useful for picking up ConfigMap changes.

📊 Production Insight

The kubectl rollout pause command is the foundation of canary deployments without a service mesh. Pause the rollout after one new Pod is created, send a small percentage of traffic to it via Service endpoint selection, monitor error rates, then either resume or undo. This gives you canary behavior with standard Kubernetes objects. Combine with maxSurge: 1 to ensure only one canary Pod is created during the pause.

🎯 Key Takeaway

Production operations require more than kubectl apply. Use rollout undo for instant rollbacks, rollout pause for canary deployments, and rollout restart for ConfigMap-driven restarts. Keep old images available — rollback fails if the image is garbage-collected.

🗂 Pod Management Objects: Deployment vs StatefulSet vs DaemonSet vs Job

Understanding when to use each controller for different workload types.

Aspect	Deployment	StatefulSet	DaemonSet	Job
Use case	Stateless applications (APIs, web servers)	Stateful applications (databases, queues)	Node-level agents (log collectors, monitors)	One-off or batch tasks (migrations, reports)
Pod identity	Interchangeable. No stable identity.	Stable identity (ordinal index: pod-0, pod-1).	One Pod per node (or subset of nodes).	Single Pod (or N completions). Runs to completion.
Scaling	Horizontal (add/remove replicas freely).	Ordered: scale up 0->1->2. Scale down in reverse.	Automatic: one Pod per matching node.	parallelism controls concurrent Pods.
Storage	Ephemeral or shared PersistentVolume.	PersistentVolumeClaim per Pod (stable storage).	HostPath or shared storage.	Ephemeral. Output to external storage.
Rolling update	Supported (RollingUpdate strategy).	Supported (OrderedReady or Parallel).	Supported (RollingUpdate or OnDelete).	Not applicable (Pods run to completion).
Networking	Pod IP changes on restart. Use Service.	Stable network identity (pod-0.svc.cluster.local).	Pod IP tied to node IP. Use hostNetwork.	Pod IP is ephemeral.
Self-healing	Yes. Replaces failed Pods automatically.	Yes. Replaces failed Pods with same identity.	Yes. Replaces failed Pods on the same node.	Yes. Retries failed Pods up to backoffLimit.
Production example	Web API, microservice, frontend	PostgreSQL, Kafka, Redis cluster	Fluentd, node-exporter, Cilium agent	Database migration, backup job, report generation

🎯 Key Takeaways

Pods are ephemeral and non-self-healing — never deploy individual Pods for production workloads.
Deployments are controllers that maintain the desired number of Pod replicas and handle lifecycle management.
Use 'RollingUpdate' with 'maxUnavailable: 0' to achieve zero-downtime application updates.
Liveness probes manage the container lifecycle (restarts), while Readiness probes manage the traffic flow (service discovery).
Resource 'requests' determine scheduling (which node); 'limits' determine runtime enforcement (CPU throttling/OOM killing).
Startup probes are mandatory for applications with warm-up times greater than 30 seconds. Without them, liveness probes kill containers during boot.
PodDisruptionBudgets prevent simultaneous Pod termination during node drains and rolling updates.
Rollback is not free — it fails if the old image has been garbage-collected. Keep old images available and increase revisionHistoryLimit for critical services.

⚠ Common Mistakes to Avoid

✕Creating Pods directly for production workloads. Direct Pods have no self-healing — if the node crashes, the Pod is gone forever with no replacement. Always use a Deployment.

✕Setting maxSurge: 0 and maxUnavailable: 0 simultaneously. The rollout blocks forever because it cannot create new Pods (no surge) and cannot terminate old Pods (no unavailability allowed).

✕Using liveness probes that check downstream dependencies (database, cache). If the dependency is slow, the liveness probe fails, the container restarts, and the restart increases load on the dependency, causing a cascading failure loop.

✕Not using startup probes for applications with warm-up times greater than 30 seconds. The liveness probe kills the container during boot, creating a CrashLoopBackOff loop.

✕Setting readiness probe initialDelaySeconds too low. The Pod is added to Service endpoints before it is ready to serve traffic, causing 502 errors during rollout.

✕Not setting terminationGracePeriodSeconds for applications that need time to drain in-flight connections. The default 30 seconds may not be enough. Add a pre-stop hook that sleeps 10-15 seconds to allow load balancers to de-register the Pod.

✕Not using PodDisruptionBudgets. During node drains or rolling updates, Kubernetes may terminate multiple Pods simultaneously, causing capacity loss. PDBs enforce minimum availability.

✕Hardcoding Pod IPs in application configuration. Pod IPs change on every restart. Use Service DNS names (service-name.namespace.svc.cluster.local) for discovery.

✕Storing state in the container's local filesystem without a PersistentVolume. Data is lost on every Pod restart. Use PVs for data that must survive restarts.

✕Not setting resource requests and limits. Without requests, the scheduler cannot make intelligent placement decisions. Without limits, a single Pod can consume all node resources and cause OOMKill on other Pods.

✕Using the default revisionHistoryLimit (10) for critical services. After 10 rollouts, old ReplicaSets are deleted and rollback is impossible. Increase to 20-50 for critical services.

✕Not monitoring Deployment rollout status. A stuck rollout (maxUnavailable: 0 with failing readiness probes) blocks indefinitely. Set up alerts on `kubectl rollout status` timeouts.

Interview Questions on This Topic

QDescribe the 'reconciliation loop' in a Deployment controller. How does it maintain the desired state?
QYour application takes 60 seconds to warm up its cache. If you don't configure probes correctly, users see 502 errors during a rollout. Which probe do you use and how do you configure it?
QExplain the 'Sidecar Pattern' in a Pod. Give a real-world example of why two containers would need to share a Pod instead of being in separate Pods.
QWhat is the difference between 'maxSurge' and 'maxUnavailable' in a RollingUpdate strategy, and how do they impact cluster capacity during a deployment?
QHow would you perform a 'Canary Deployment' using only standard Kubernetes Deployment objects and Services?
QA Pod is stuck in CrashLoopBackOff. Walk me through your debugging process from the first command you would run to identifying the root cause.
QWhat is the difference between a liveness probe and a readiness probe? What happens if you use a liveness probe to check database connectivity?
QExplain the relationship between a Deployment, a ReplicaSet, and a Pod. Why does Kubernetes have this three-layer hierarchy instead of Deployments managing Pods directly?
QWhat happens when you run kubectl rollout undo? Does it redeploy the old image? What are the failure modes?
QHow do you design a zero-downtime deployment strategy in Kubernetes? What configuration parameters matter and why?

Frequently Asked Questions

What is the difference between a liveness probe and a readiness probe?

A Liveness probe tells K8s if your container is 'alive'. If it fails, the kubelet kills the container and restarts it according to the restartPolicy. A Readiness probe tells K8s if the container is 'ready' to handle traffic. If it fails, the Pod's IP is removed from the Endpoints of any associated Service, so no traffic reaches it, but the container is NOT restarted. You use readiness for apps that need to load large cache files or perform migrations before serving requests.

Why should I set both requests and limits for my containers?

Requests ensure your Pod is guaranteed a certain amount of resources on a node. Limits prevent a 'noisy neighbor' scenario where one buggy container consumes all the CPU or RAM on a node, causing other Pods to fail. In production, setting requests equal to limits (Guaranteed QoS) provides the most predictable performance.

How does a Deployment know which Pods it owns?

Deployments use Label Selectors. The spec.selector.matchLabels in the Deployment must match the spec.template.metadata.labels of the Pod. This loose coupling allows you to 'adopt' or 'orphan' Pods simply by changing labels, though this is rarely done manually.

What happens to a Deployment if a Node fails?

The Control Plane detects that the Node is 'NotReady'. After a timeout (pod-eviction-timeout), the Deployment controller notices the missing replicas and schedules new Pods on the remaining healthy nodes to restore the desired state.

What is a startup probe and when should I use it?

A startup probe runs before liveness and readiness probes. While the startup probe has not yet succeeded, liveness and readiness probes are disabled. Use it for applications with startup times greater than 30 seconds. Set failureThreshold high enough to cover the maximum expected startup time (e.g., failureThreshold: 60 with periodSeconds: 5 = 300 seconds max). Without a startup probe, the liveness probe kills the container during boot, causing CrashLoopBackOff.

How do I perform a canary deployment without a service mesh?

Use kubectl rollout pause after the Deployment creates one new Pod. The new Pod will be added to the Service endpoints automatically. Monitor error rates and latency on the canary Pod. If healthy, run kubectl rollout resume to complete the rollout. If unhealthy, run kubectl rollout undo to revert. Combine with maxSurge: 1 to ensure only one canary Pod is created during the pause.

🔥

Naren Founder & Author

Developer and founder of TheCodeForge. I built this site because I was tired of tutorials that explain what to type without explaining why it works. Every article here is written to make concepts actually click.

About Naren Get in touch

Forged with 🔥 at TheCodeForge.io — Where Developers Are Forged