Kubernetes Internals Explained — Architecture, Scheduling, and Production Gotchas

Kubernetes is not a deployment tool. It is a distributed state reconciliation engine. Every component — from the scheduler to the kubelet — operates on the same principle: watch the desired state in etcd, compare it with observed state, and act to close the gap. This is the mental model that unlocks real debugging capability.

The control plane is the brain. etcd is the memory. The kubelet is the muscle on each node. The scheduler decides placement. When any of these components degrades, the symptoms are often misleading — a Pod stuck in Pending looks like a scheduling problem but is frequently an etcd latency issue or a resource quota misconfiguration.

The common misconception is that Kubernetes 'runs containers.' It does not. Kubernetes manages the desired state of workloads. The container runtime (containerd, CRI-O) runs containers. Kubernetes tells the runtime what to run, monitors whether it is running, and corrects deviations. This distinction matters when debugging crashes, image pull failures, and networking issues.

Control Plane Architecture: The Brain of the Cluster

The Kubernetes control plane consists of four components that work together to maintain cluster state. Understanding each component's role — and its failure modes — is essential for production operations.

kube-apiserver is the front door. Every kubectl command, every controller reconciliation, every kubelet status report goes through the API server. It validates requests, persists state to etcd, and serves as the watch endpoint for all controllers. It is stateless — you can run multiple replicas behind a load balancer for HA.

etcd is the single source of truth. It is a distributed, consistent key-value store built on the Raft consensus protocol. All cluster state — Pod definitions, ConfigMaps, Secrets, node registrations — lives in etcd. If etcd loses quorum, the cluster cannot make any state changes. etcd is the most critical component and the most commonly under-provisioned.

kube-scheduler watches for unscheduled Pods and assigns them to nodes. It does not run Pods — it only writes the nodeName field. The kubelet on the assigned node then pulls the image and starts the container. The scheduler uses a two-phase process: filtering (eliminate infeasible nodes) and scoring (rank feasible nodes, pick the highest score).

kube-controller-manager runs the control loops. Each controller watches a specific resource type and reconciles actual state with desired state. The Deployment controller ensures the right number of replicas exist. The Node controller detects when nodes go unhealthy. The Endpoint controller updates Service endpoints as Pods come and go.

# Control Plane Health Check — Run this to verify all components are healthy
# Save as check-control-plane.sh

# 1. API Server health (returns 200 if healthy)
curl -k https://localhost:6443/healthz
# Expected: "ok"

# 2. etcd cluster health
ETCDCTL_API=3 etcdctl endpoint health \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
  --key=/etc/kubernetes/pki/etcd/healthcheck-client.key
# Expected: "is healthy"

# 3. etcd cluster member status
ETCDCTL_API=3 etcdctl endpoint status \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
  --key=/etc/kubernetes/pki/etcd/healthcheck-client.key \
  --write-out=table
# Shows: ID, Status, Version, DB Size, Raft Term, Raft Index

# 4. Scheduler and Controller-Manager leader election
kubectl get endpoints kube-scheduler -n kube-system -o jsonpath='{.metadata.annotations.control-plane\.alpha\.kubernetes\.io/leader}'
kubectl get endpoints kube-controller-manager -n kube-system -o jsonpath='{.metadata.annotations.control-plane\.alpha\.kubernetes\.io/leader}'

# 5. All control plane components running
kubectl get pods -n kube-system -o wide

The Scheduler: How Kubernetes Decides Where Pods Run

The kube-scheduler is the component that assigns Pods to nodes. It does not run Pods — it only writes the spec.nodeName field on the Pod object. The kubelet on the assigned node then pulls the image and starts the container.

The scheduler uses a two-phase process:

Filtering (Feasibility): Eliminate nodes that cannot run the Pod. Filter reasons include: insufficient CPU/memory, node taints the Pod cannot tolerate, node affinity mismatches, volume zone constraints, and Pod topology spread constraints. After filtering, if zero nodes remain, the Pod stays in Pending.

Scoring (Ranking): Rank the feasible nodes by a set of scoring plugins. Default scoring includes: NodeResourcesBalancedAllocation (prefer nodes with balanced CPU/memory usage), ImageLocality (prefer nodes that already have the container image), InterPodAffinity (prefer nodes where affinity rules are satisfied), and TaintToleration (prefer nodes with fewer taints). The node with the highest weighted score wins.

The scheduler makes decisions based on the state of the cluster at scheduling time. It does not predict future load. It does not rebalance existing Pods. Once a Pod is scheduled, only explicit actions (eviction, deletion, preemption) can move it.

# Example: Pod with scheduling constraints
# This Pod will ONLY be scheduled on nodes with the label 'disktype=ssd'
# and will prefer nodes in zone 'us-east-1a'
apiVersion: v1
kind: Pod
metadata:
  name: io-thecodeforge-payment-service
  namespace: production
spec:
  # Hard requirement: node MUST have this label
  nodeSelector:
    disktype: ssd

  # Soft preference: scheduler tries to place here, but can choose elsewhere
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 80
          preference:
            matchExpressions:
              - key: topology.kubernetes.io/zone
                operator: In
                values:
                  - us-east-1a
    # Pod affinity: prefer to run near other payment-service Pods
    podAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 50
          podAffinityTerm:
            labelSelector:
              matchLabels:
                app: payment-service
            topologyKey: kubernetes.io/hostname

  # Tolerations: allow scheduling on nodes with the 'dedicated=high-cpu' taint
  tolerations:
    - key: dedicated
      operator: Equal
      value: high-cpu
      effect: NoSchedule

  # Topology spread: distribute replicas evenly across zones
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: topology.kubernetes.io/zone
      whenUnsatisfiable: DoNotSchedule
      labelSelector:
        matchLabels:
          app: payment-service

  containers:
    - name: payment-service
      image: registry.thecodeforge.io/payment-service:v2.4.1
      resources:
        requests:
          cpu: "500m"
          memory: "512Mi"
        limits:
          cpu: "1000m"
          memory: "1Gi"

Pod Networking: How Containers Talk to Each Other

Kubernetes networking has three fundamental requirements, enforced by the CNI (Container Network Interface) plugin:

1. Every Pod gets its own IP address, unique across the cluster.
2. Pods on any node can communicate with Pods on any other node without NAT.
3. Agents on a node (kubelet, system daemons) can communicate with all Pods on that node.

These requirements are simple to state but complex to implement. The CNI plugin (Calico, Cilium, Flannel, AWS VPC CNI) is responsible for wiring this up. It allocates IP addresses from the node's Pod CIDR range, sets up network interfaces inside the Pod's network namespace, and configures routing rules so Pods can reach each other across nodes.

kube-proxy handles Service networking. It watches the API server for Service and Endpoint objects, then programs iptables rules (or IPVS rules) on each node. When a Pod connects to a Service's ClusterIP, the kernel's iptables rules intercept the connection and DNAT it to one of the backend Pod IPs. This is why Service IPs are virtual — they do not exist on any network interface.

# Debugging Pod networking step by step

# 1. Verify Pod has an IP address
kubectl get pods -n production -o wide
# If Pod IP is <none>, the CNI plugin failed to assign an address

# 2. Check if the CNI plugin is healthy
kubectl get pods -n kube-system | grep -E 'calico|cilium|flannel|aws-node'

# 3. Verify Pod CIDR allocation per node
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.podCIDR}{"\n"}{end}'
# Each node must have a unique, non-overlapping CIDR

# 4. Test Pod-to-Pod connectivity across nodes
kubectl exec -it pod-on-node-a -- ping <pod-ip-on-node-b>
# If this fails but intra-node works, the CNI cross-node routing is broken

# 5. Check Service endpoints
kubectl get endpoints payment-service -n production
# If endpoints are empty, no Pods match the Service's selector

# 6. Test Service DNS resolution
kubectl exec -it <pod> -- nslookup payment-service.production.svc.cluster.local
# If DNS fails, check CoreDNS pods: kubectl get pods -n kube-system | grep coredns

# 7. Inspect iptables rules for a Service
# (run on the node where your Pod is running)
iptables-save | grep <service-cluster-ip>

Introduction to Kubernetes

Introduction to Kubernetes is a core concept in DevOps. Rather than starting with a dry definition, let's see it in action and understand why it exists.

// TheCodeForge — Introduction to Kubernetes example
// Always use meaningful names, not x or n
package io.thecodeforge.kubernetes;

public class ForgeExample {
    public static void main(String[] args) {
        String topic = "Introduction to Kubernetes";
        System.out.println("Learning: " + topic);
    }
}

Frequently Asked Questions