etcd Disk Latency — How 800ms Killed the Kubernetes Cluster

Kubernetes is not a deployment tool. It is a distributed state reconciliation engine. Every component — from the scheduler to the kubelet — operates on the same principle: watch the desired state in etcd, compare it with observed state, and act to close the gap. This is the mental model that unlocks real debugging capability.

The control plane is the brain. etcd is the memory. The kubelet is the muscle on each node. The scheduler decides placement. When any of these components degrades, the symptoms are often misleading — a Pod stuck in Pending looks like a scheduling problem but is frequently an etcd latency issue or a resource quota misconfiguration.

The common misconception is that Kubernetes 'runs containers.' It does not. Kubernetes manages the desired state of workloads. The container runtime (containerd, CRI-O) runs containers. Kubernetes tells the runtime what to run, monitors whether it is running, and corrects deviations. This distinction matters when debugging crashes, image pull failures, and networking issues.

What etcd Latency Actually Does to Kubernetes

etcd is the distributed key-value store that backs Kubernetes, holding all cluster state — pods, services, configmaps, secrets. The core mechanic: every write to etcd must be committed to a majority of nodes (quorum) before it's considered durable. This means a single slow disk on one node can stall the entire cluster. In practice, etcd's performance is measured by fsync latency: the time to flush a write to disk. Kubernetes control-plane components — kube-apiserver, scheduler, controller-manager — all depend on etcd's linearizable reads and writes. When fsync latency exceeds 100ms, watch timeouts and leader elections cascade. At 800ms, the cluster enters a death spiral: heartbeats fail, leaders step down, and no new writes succeed. You use etcd in every Kubernetes cluster, but its sensitivity to disk I/O is often underestimated. Understanding this matters because a single slow disk — not CPU, not memory — is the most common cause of control-plane outages in production.

The 4 Essential Objects: Pod, Service, Deployment, Namespace

Before diving into the architecture, you need a concrete mental model of the four objects you'll use every day. Kubernetes exposes hundreds of resource types, but 80% of your interactions will involve these four.

Pod is the smallest deployable unit. A Pod wraps one or more containers, gives them a shared network namespace (one IP per Pod), and optionally shared storage volumes. Containers in the same Pod can communicate via localhost. Pods are ephemeral — they can be killed and rescheduled at any time. Never run a single Pod without a controller (Deployment, StatefulSet, DaemonSet).

Service provides a stable network endpoint for a set of Pods. Because Pods can die and be replaced with new IPs, a Service gives a fixed IP (ClusterIP) and DNS name that load-balances across the healthy Pods. The Service uses label selectors to determine which Pods belong to it.

Deployment is the most common controller. It declares the desired state for your stateless applications: how many replicas, which container image, resource limits, health checks, update strategy. The Deployment controller creates a ReplicaSet, which creates the Pods. When you update the Pod template, the Deployment creates a new ReplicaSet and gradually scales it up and the old one down (rolling update).

Namespace is a virtual cluster boundary. It isolates resources, RBAC, and network policies. Every resource lives in a namespace — except cluster-scoped resources like Nodes and PersistentVolumes. Use namespaces to separate environments (dev, staging, prod) or teams.

Together, these objects form the foundation: you define a Deployment that creates Pods, expose them via a Service, and organize everything in a Namespace.

# 1. Namespace
apiVersion: v1
kind: Namespace
metadata:
  name: production
---
# 2. Deployment creating 3 replicas of a web app
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deploy
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.27
        ports:
        - containerPort: 80
---
# 3. Service exposing the Deployment internally
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
  namespace: production
spec:
  selector:
    app: nginx
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
  type: ClusterIP

Control Plane Architecture: The Brain of the Cluster

The Kubernetes control plane consists of four components that work together to maintain cluster state. Understanding each component's role — and its failure modes — is essential for production operations.

kube-apiserver is the front door. Every kubectl command, every controller reconciliation, every kubelet status report goes through the API server. It validates requests, persists state to etcd, and serves as the watch endpoint for all controllers. It is stateless — you can run multiple replicas behind a load balancer for HA.

etcd is the single source of truth. It is a distributed, consistent key-value store built on the Raft consensus protocol. All cluster state — Pod definitions, ConfigMaps, Secrets, node registrations — lives in etcd. If etcd loses quorum, the cluster cannot make any state changes. etcd is the most critical component and the most commonly under-provisioned.

kube-scheduler watches for unscheduled Pods and assigns them to nodes. It does not run Pods — it only writes the nodeName field. The kubelet on the assigned node then pulls the image and starts the container. The scheduler uses a two-phase process: filtering (eliminate infeasible nodes) and scoring (rank feasible nodes, pick the highest score).

kube-controller-manager runs the control loops. Each controller watches a specific resource type and reconciles actual state with desired state. The Deployment controller ensures the right number of replicas exist. The Node controller detects when nodes go unhealthy. The Endpoint controller updates Service endpoints as Pods come and go.

# Control Plane Health Check — Run this to verify all components are healthy
# Save as check-control-plane.sh

# 1. API Server health (returns 200 if healthy)
curl -k https://localhost:6443/healthz
# Expected: "ok"

# 2. etcd cluster health
ETCDCTL_API=3 etcdctl endpoint health \n  --endpoints=https://127.0.0.1:2379 \n  --cacert=/etc/kubernetes/pki/etcd/ca.crt \n  --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \n  --key=/etc/kubernetes/pki/etcd/healthcheck-client.key
# Expected: "is healthy"

# 3. etcd cluster member status
ETCDCTL_API=3 etcdctl endpoint status \n  --endpoints=https://127.0.0.1:2379 \n  --cacert=/etc/kubernetes/pki/etcd/ca.crt \n  --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \n  --key=/etc/kubernetes/pki/etcd/healthcheck-client.key \n  --write-out=table
# Shows: ID, Status, Version, DB Size, Raft Term, Raft Index

# 4. Scheduler and Controller-Manager leader election
kubectl get endpoints kube-scheduler -n kube-system -o jsonpath='{.metadata.annotations.control-plane\.alpha\.kubernetes\.io/leader}'
kubectl get endpoints kube-controller-manager -n kube-system -o jsonpath='{.metadata.annotations.control-plane\.alpha\.kubernetes\.io/leader}'

# 5. All control plane components running
kubectl get pods -n kube-system -o wide

Control Plane vs Data Plane Architecture

Kubernetes is divided into two logical planes: the control plane (brain) and the data plane (muscle). The control plane makes decisions about the cluster state — what should run, where it should run, and whether the current state matches the desired state. The data plane executes those decisions — it runs the actual containers, provides the network connectivity, and reports back the observed state.

The control plane components (kube-apiserver, etcd, scheduler, controller-manager) typically run on dedicated master nodes, though in smaller clusters they may be colocated. The data plane consists of the worker nodes, each running kubelet, kube-proxy, the container runtime, and the CNI plugin.

The key architectural insight: control plane components communicate with each other and with etcd, but they never directly interact with the user containers. All interactions go through the API server. The kubelet on each worker node polls the API server for Pods assigned to its node, then instructs the container runtime to pull images and start containers. kube-proxy watches the API server for Service changes and programs iptables/IPVS rules accordingly.

This separation means that if the control plane fails, existing containers continue running (the kubelet is autonomous for running workloads) but you cannot make any changes. Conversely, if a worker node fails, the control plane detects it (via the Node controller) and reschedules the Pods on healthy nodes after a timeout.

# Check kubelet and kube-proxy status on a worker node
# Run this on the worker node itself

# 1. Check kubelet is running
systemctl status kubelet

# 2. Check kube-proxy (runs as a DaemonSet, check from master)
kubectl get pods -n kube-system | grep kube-proxy

# 3. Check container runtime is responsive
crictl pods

# 4. Verify that the node can reach the API server
curl -k https://<api-server-ip>:6443/healthz

# 5. Test that local kubelet can actually start a Pod
kubectl run test --image=busybox --restart=Never -- sleep 30
kubectl get pods -n default
kubectl delete pod test

The Scheduler: How Kubernetes Decides Where Pods Run

The kube-scheduler is the component that assigns Pods to nodes. It does not run Pods — it only writes the spec.nodeName field on the Pod object. The kubelet on the assigned node then pulls the image and starts the container.

The scheduler uses a two-phase process:

Filtering (Feasibility): Eliminate nodes that cannot run the Pod. Filter reasons include: insufficient CPU/memory, node taints the Pod cannot tolerate, node affinity mismatches, volume zone constraints, and Pod topology spread constraints. After filtering, if zero nodes remain, the Pod stays in Pending.

Scoring (Ranking): Rank the feasible nodes by a set of scoring plugins. Default scoring includes: NodeResourcesBalancedAllocation (prefer nodes with balanced CPU/memory usage), ImageLocality (prefer nodes that already have the container image), InterPodAffinity (prefer nodes where affinity rules are satisfied), and TaintToleration (prefer nodes with fewer taints). The node with the highest weighted score wins.

The scheduler makes decisions based on the state of the cluster at scheduling time. It does not predict future load. It does not rebalance existing Pods. Once a Pod is scheduled, only explicit actions (eviction, deletion, preemption) can move it.

# Example: Pod with scheduling constraints
# This Pod will ONLY be scheduled on nodes with the label 'disktype=ssd'
# and will prefer nodes in zone 'us-east-1a'
apiVersion: v1
kind: Pod
metadata:
  name: io-thecodeforge-payment-service
  namespace: production
spec:
  # Hard requirement: node MUST have this label
  nodeSelector:
    disktype: ssd

  # Soft preference: scheduler tries to place here, but can choose elsewhere
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 80
          preference:
            matchExpressions:
              - key: topology.kubernetes.io/zone
                operator: In
                values:
                  - us-east-1a
    # Pod affinity: prefer to run near other payment-service Pods
    podAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 50
          podAffinityTerm:
            labelSelector:
              matchLabels:
                app: payment-service
            topologyKey: kubernetes.io/hostname

  # Tolerations: allow scheduling on nodes with the 'dedicated=high-cpu' taint
  tolerations:
    - key: dedicated
      operator: Equal
      value: high-cpu
      effect: NoSchedule

  # Topology spread: distribute replicas evenly across zones
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: topology.kubernetes.io/zone
      whenUnsatisfiable: DoNotSchedule
      labelSelector:
        matchLabels:
          app: payment-service

  containers:
    - name: payment-service
      image: registry.thecodeforge.io/payment-service:v2.4.1
      resources:
        requests:
          cpu: "500m"
          memory: "512Mi"
        limits:
          cpu: "1000m"
          memory: "1Gi"

Pod Networking: How Containers Talk to Each Other

Kubernetes networking has three fundamental requirements, enforced by the CNI (Container Network Interface) plugin:

1. Every Pod gets its own IP address, unique across the cluster.
2. Pods on any node can communicate with Pods on any other node without NAT.
3. Agents on a node (kubelet, system daemons) can communicate with all Pods on that node.

These requirements are simple to state but complex to implement. The CNI plugin (Calico, Cilium, Flannel, AWS VPC CNI) is responsible for wiring this up. It allocates IP addresses from the node's Pod CIDR range, sets up network interfaces inside the Pod's network namespace, and configures routing rules so Pods can reach each other across nodes.

kube-proxy handles Service networking. It watches the API server for Service and Endpoint objects, then programs iptables rules (or IPVS rules) on each node. When a Pod connects to a Service's ClusterIP, the kernel's iptables rules intercept the connection and DNAT it to one of the backend Pod IPs. This is why Service IPs are virtual — they do not exist on any network interface.

# Debugging Pod networking step by step

# 1. Verify Pod has an IP address
kubectl get pods -n production -o wide
# If Pod IP is <none>, the CNI plugin failed to assign an address

# 2. Check if the CNI plugin is healthy
kubectl get pods -n kube-system | grep -E 'calico|cilium|flannel|aws-node'

# 3. Verify Pod CIDR allocation per node
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.podCIDR}{"\n"}{end}'
# Each node must have a unique, non-overlapping CIDR

# 4. Test Pod-to-Pod connectivity across nodes
kubectl exec -it pod-on-node-a -- ping <pod-ip-on-node-b>
# If this fails but intra-node works, the CNI cross-node routing is broken

# 5. Check Service endpoints
kubectl get endpoints payment-service -n production
# If endpoints are empty, no Pods match the Service's selector

# 6. Test Service DNS resolution
kubectl exec -it <pod> -- nslookup payment-service.production.svc.cluster.local
# If DNS fails, check CoreDNS pods: kubectl get pods -n kube-system | grep coredns

# 7. Inspect iptables rules for a Service
# (run on the node where your Pod is running)
iptables-save | grep <service-cluster-ip>

Kubernetes Storage: PersistentVolumes, Claims, and StorageClasses

Kubernetes storage decouples Pod lifecycle from data life. A Pod can be deleted and recreated, but its data persists if it uses a PersistentVolume (PV) and PersistentVolumeClaim (PVC). This is critical for stateful workloads like databases.

PersistentVolume (PV) is a piece of storage in the cluster that has been provisioned by an administrator or dynamically by a StorageClass. It is a cluster resource, like a node. PVs have a capacity and access mode (ReadWriteOnce, ReadOnlyMany, ReadWriteMany).

PersistentVolumeClaim (PVC) is a request for storage by a user. It specifies size and access mode. Kubernetes binds a PVC to a PV that meets the requirements. If no matching PV exists, the PVC remains Pending — unless a StorageClass with a dynamic provisioner is referenced.

StorageClass defines a class of storage. It specifies the provisioner (e.g., kubernetes.io/aws-ebs), parameters (type, IOPS), and reclaim policy. When a PVC requests a StorageClass, the provisioner automatically creates a PV that satisfies the claim.

# StorageClass for AWS gp3 volumes with 3000 IOPS
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: io-thecodeforge-fast
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  iops: "3000"
  throughput: "125"
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
---
# PVC that uses the StorageClass above
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: io-thecodeforge-payment-db-pvc
  namespace: production
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi
  storageClassName: io-thecodeforge-fast
---
# Pod using the PVC
apiVersion: v1
kind: Pod
metadata:
  name: io-thecodeforge-payment-db
  namespace: production
spec:
  containers:
    - name: postgres
      image: postgres:16
      env:
        - name: PGDATA
          value: /var/lib/postgresql/data/pgdata
      volumeMounts:
        - name: data
          mountPath: /var/lib/postgresql/data
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: io-thecodeforge-payment-db-pvc

Namespaces, Resource Quotas, and Multi-Tenancy

Namespaces are virtual clusters within a physical cluster. They provide isolation boundaries for resources, RBAC, and network policies. Every resource lives in a namespace — except cluster-scoped resources like Nodes and PersistentVolumes.

ResourceQuota limits aggregate resource consumption within a namespace. You can set quotas on CPU, memory, Pod count, PVC storage, and even the number of Services. Without quotas, a single misconfigured application can consume all cluster resources and starve others.

LimitRange sets default requests/limits and min/max constraints for Pods in a namespace. This prevents a Pod from requesting an absurd amount of resources or running without any limits.

Multi-tenancy with Namespaces is common: each team gets its own namespace, with RBAC restricting cross-namespace access. But true multi-tenancy (running untrusted workloads) requires additional isolation — consider virtual clusters (vClusters) or sandbox containers (gVisor, Kata Containers).

# ResourceQuota for a namespace
apiVersion: v1
kind: ResourceQuota
metadata:
  name: io-thecodeforge-team-quota
  namespace: team-a
spec:
  hard:
    requests.cpu: "10"
    requests.memory: 20Gi
    limits.cpu: "20"
    limits.memory: 40Gi
    persistentvolumeclaims: "10"
    requests.storage: 500Gi
    pods: "50"
    services: "10"
---
# LimitRange to enforce default resource boundaries
apiVersion: v1
kind: LimitRange
metadata:
  name: io-thecodeforge-default-limits
  namespace: team-a
spec:
  limits:
    - default:
        cpu: "500m"
        memory: 512Mi
      defaultRequest:
        cpu: "100m"
        memory: 128Mi
      max:
        cpu: "2"
        memory: 4Gi
      min:
        cpu: "50m"
        memory: 64Mi
      type: Container

Kubernetes vs Docker Compose: When to Use Each

Docker Compose and Kubernetes both orchestrate containers, but they serve fundamentally different use cases. Docker Compose is a single-host orchestration tool designed for development environments and small deployments. Kubernetes is a multi-host, production-grade orchestration system with automated healing, scaling, and rolling updates.

Choose Docker Compose when: you are developing locally, running CI integrations tests, or deploying a simple application on a single VM where orchestration overhead is not justified.

Choose Kubernetes when: you need high availability, rolling updates, auto-scaling, multi-node deployment, or run multiple microservices that need advanced networking (service discovery, load balancing, network policies). Many teams start with Docker Compose for development and then write Kubernetes manifests for production — maintaining both can be an overhead, but tools like Kompose can convert Compose files to Kubernetes YAML.

# Docker Compose version (docker-compose.yml)
version: '3.8'
services:
  web:
    image: nginx:1.27
    ports:
      - "80:80"
  app:
    build: .
    environment:
      - DATABASE_URL=postgres://db:5432/mydb
    depends_on:
      - db
  db:
    image: postgres:16
    environment:
      POSTGRES_PASSWORD_FILE: /run/secrets/db_password
    secrets:
      - db_password
    volumes:
      - pgdata:/var/lib/postgresql/data

secrets:
  db_password:
    file: ./secrets/db_password.txt

volumes:
  pgdata:

---
# Equivalent Kubernetes manifest (simplified)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - name: nginx
        image: nginx:1.27
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector:
    app: web
  ports:
  - port: 80
  type: LoadBalancer
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  replicas: 5
  selector:
    matchLabels:
      app: app
  template:
    metadata:
      labels:
        app: app
    spec:
      containers:
      - name: app
        image: myapp:latest
        env:
        - name: DATABASE_URL
          value: postgres://db:5432/mydb
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-password
              key: password
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: db
spec:
  serviceName: db
  selector:
    matchLabels:
      app: db
  template:
    metadata:
      labels:
        app: db
    spec:
      containers:
      - name: postgres
        image: postgres:16
        env:
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-password
              key: password
        volumeMounts:
        - name: data
          mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 10Gi
---
apiVersion: v1
kind: Secret
metadata:
  name: db-password
type: Opaque
data:
  password: <base64-encoded-password>
---
apiVersion: v1
kind: Service
metadata:
  name: db
spec:
  selector:
    app: db
  ports:
  - port: 5432
  clusterIP: None  # Headless service for StatefulSet

The Evolution of Deployment — Why We Stopped Trusting Bare Metal

You don't understand Kubernetes until you understand the deployment hell it replaced. Before containers, you had two choices: dump a JAR on a physical server and pray nothing else touched the port, or waste 40% of your budget on VM overhead because each app needed its own OS instance.

Virtualization fixed the hardware waste but introduced its own cancer — golden images that rotted over time, configuration drift that turned production into a snowflake zoo, and boot times measured in coffee breaks. Then came containers. Docker gave you repeatable build artifacts and second-level startup. But now you had 50 containers on a single VM and no sane way to manage them.

That's the gap Kubernetes fills. Not as a container manager — as a control system. It takes your container images and applies a desired state loop. You say "I want 3 replicas of payment-api behind a stable DNS name." Kubernetes makes it true, then keeps it true. No SSH, no manual restart, no "it works on my machine." The whole industry pivoted from pet servers to cattle because manual operations don't scale past 5 microservices.

// io.thecodeforge — devops tutorial

# What deployment looked like in 2014
# One physical server. One app. Hours to reprovision.
server:
  hostname: prod-payments-01
  os: CentOS 7.2
  app: payments.jar v1.3.2
  ports:
    - 8080
  dependencies:
    - postgresql-9.6
  scaling: buy another server, wait 3 days

# What Kubernetes does instead
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments
spec:
  replicas: 3
  selector:
    matchLabels:
      app: payments
  template:
    spec:
      containers:
        - name: payments
          image: payments:v1.3.2
          ports:
            - containerPort: 8080

Why Kubernetes Stands Out — The Desired State Loop

Every other orchestrator tells you how to start processes. Kubernetes tells you how the system should look and makes reality match the spec. This is the single most important concept to internalize.

When you write a Deployment, you declare: "3 replicas, port 8080, liveness probe hitting /healthz." The control plane stores that intent in etcd. Then the kubelet on each node continuously checks: "Does my pod match what etcd says? No? Fix it." This isn't a one-time deploy. It's a running reconciliation loop that fires every second until you delete the resource.

Why does this matter? Because production never stays still. A node crashes — the controller sees 2 replicas instead of 3 and spawns a replacement. A pod runs out of memory — the restart policy kills and re-creates it. Traffic spikes — your HorizontalPodAutoscaler reads the metrics and tells the deployment to scale to 10. No human touching a terminal at 3 AM.

The magic isn't the containers. It's the control theory applied to distributed systems. You describe the steady state. Kubernetes enforces it. Period.

// io.thecodeforge — devops tutorial

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-gateway
spec:
  replicas: 3
  selector:
    matchLabels:
      app: gateway
  template:
    spec:
      containers:
        - name: gateway
          image: api-gateway:2.4.1
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 10
          resources:
            requests:
              cpu: 500m
              memory: 256Mi

Real-World Kubernetes: Where the Theory Dies

You've read the docs. You've deployed a pod. Now what? The real value of Kubernetes isn't container orchestration — it's the patterns that survive production. Three use cases define modern k8s: stateless web backends, event-driven batch jobs, and stateful data pipelines.

Stateless apps are the entry drug. Horizontal Pod Autoscaler + Deployment + Service = you can absorb traffic spikes without waking up at 3 AM. Batch jobs use Jobs and CronJobs to replace cron on bare metal — way easier to restart failed pods than re-ssh into a dead VM. Stateful stuff uses StatefulSets with PersistentVolumeClaims for databases like PostgreSQL or Kafka. You don't need to manage the database lifecycle in k8s — just give it stable storage and a stable network identity.

The trap? Thinking every app belongs in k8s. Latency-sensitive workloads, GPU training jobs that don't scale horizontally, or anything that needs raw hardware access — leave those on bare metal or spot instances. Kubernetes is not a Swiss Army knife. It's a hammer. Use it on nails.

// io.thecodeforge — devops tutorial

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: web-backend-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: user-service
  minReplicas: 3
  maxReplicas: 50
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-report
spec:
  schedule: "0 2 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: report
            image: reporting:2.1.0
            command: ["/bin/report"]
          restartPolicy: OnFailure

Kuberenetes Projects That Actually Teach You Something

Stop running nginx in a playground. Build something that breaks, then fix it. Three projects will teach you more than any certification: a multi-service web app with zero-downtime deploys, a CI/CD pipeline that runs entirely inside the cluster, and a GitOps setup that auto-remediates drift.

First project: Deploy a frontend + API + database. Use rolling updates with readiness probes. Simulate a bad deploy — watch the probe kill it and rollback automatically. You'll understand why livenessProbe and readinessProbe are not optional. Second: Run a Jenkins or Argo Workflows executor inside k8s with dynamic pod-per-build. Learn how PersistentVolumeClaims hold workspace data and how cluster autoscaling handles build spikes. Third: Set up Argo CD or Flux with a Git repo. Break the cluster state manually — watch it self-heal. That's the Desired State Loop in action, not theory.

These projects force you to hit real problems: pod eviction, OOMKilled containers, RBAC misconfigurations, and etcd latency when the control plane gets hammered. You'll stop treating k8s like magic and start treating it like a distributed system that demands respect.

// io.thecodeforge — devops tutorial

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-gateway
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  template:
    spec:
      containers:
      - name: gateway
        image: myorg/api-gateway:2.4.0
        ports:
        - containerPort: 8080
        readinessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 10
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 15
          periodSeconds: 20

What Is Kubernetes? The Bare Minimum You Need to Know

Kubernetes is a container orchestration platform that automates deployment, scaling, and management of containerized applications. At its core, you manage a cluster: a set of machines called nodes. One node is the master (control plane), the rest are workers (data plane). You define your app's desired state — how many replicas, which image, what ports — in a YAML manifest. Kubernetes then ensures the cluster matches that state, healing failures, scaling load, and rolling updates. A Pod is the smallest unit: one or more containers sharing networking and storage. Deployments manage replica sets. Services provide stable network endpoints. This declarative approach means you tell Kubernetes what you want, not how to achieve it. The system handles the rest, watching for drift and correcting it automatically.

// io.thecodeforge — devops tutorial
thecodeforge: basics/kubernetes
apiVersion: v1
kind: Pod
metadata:
  name: nginx-basic
  labels:
    app: webserver
spec:
  containers:
    - name: nginx
      image: nginx:1.25
      ports:
        - containerPort: 80
      resources:
        requests:
          memory: "128Mi"
          cpu: "250m"
        limits:
          memory: "256Mi"
          cpu: "500m"

Key Primitives You Must Understand

Beyond Pods, four objects form Kubernetes' backbone. A Deployment manages Pod lifecycles: rolling updates, rollbacks, replica scaling. It creates a ReplicaSet that watches pod counts. A Service provides stable networking — Pods get ephemeral IPs, but a Service gives a fixed ClusterIP or LoadBalancer. Ingress routes external HTTP/S traffic to Services. ConfigMaps and Secrets decouple configuration from images. Volumes (PersistentVolumeClaims) persist data beyond Pod restarts. Namespaces isolate resources within a cluster. RBAC (Role-Based Access Control) locks down who can do what. These primitives layer on each other: Deployment → ReplicaSet → Pod → Container. Knowing which to use and when separates beginners from pros. Start with a Deployment + Service pair; that covers 80% of use cases.

// io.thecodeforge — devops tutorial
thecodeforge: basics/primitives
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deploy
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:1.25
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-svc
spec:
  selector:
    app: nginx
  ports:
    - port: 80
      targetPort: 80
  type: ClusterIP