Kubernetes Network Policies: Zero-Trust Segmentation in Production

Most teams get Kubernetes running, deploy their apps, and move on — never realizing their payment service can freely dial their logging sidecar, which can freely dial their database, which can freely reach the internet. That's not paranoia; that's the default. Kubernetes was designed for rapid connectivity, not zero-trust isolation. The moment you run multiple tenants, compliance workloads, or anything that touches PII or financial data, that open-door model becomes a liability.

Network Policies solve this by letting you express intent in YAML: only Pods with this label may reach my database on port 5432, from this namespace only, and my database can reach nothing outbound except DNS. The CNI plugin — not the Kubernetes API server — enforces those rules in the kernel using iptables, eBPF, or nftables depending on your stack. That distinction matters enormously for debugging and performance.

This is not a syntax reference. It covers how policies are evaluated and merged, how to write airtight ingress and egress rules without accidentally blackholing DNS, how to verify enforcement at the network level rather than trusting your YAML applied cleanly, and the production mistakes that silently leave clusters wide open.

How Network Policy Enforcement Actually Works — The CNI Layer

Here's the thing most tutorials skip: the Kubernetes API server doesn't enforce Network Policies. It just stores them. The actual enforcement happens inside your CNI plugin — Calico, Cilium, Weave, Antrea — which watches the API server for NetworkPolicy objects and translates them into kernel-level firewall rules on each node.

With Calico on older kernels, that means iptables chains per endpoint. With Cilium, it's eBPF programs loaded into the kernel that intercept packets at the socket layer before they ever hit iptables — significantly lower latency and dramatically better observability. With Flannel, enforcement is zero because Flannel doesn't implement Network Policies at all. This is one of the most common production surprises: a team applies policies and believes they're enforced, but their CNI silently ignores them.

Policy evaluation works like a firewall whitelist. If no NetworkPolicy selects a Pod, all traffic is allowed. The moment any policy selects a Pod — via podSelector — that Pod enters an implicit 'default deny' for the traffic directions that policy governs. Multiple policies selecting the same Pod are unioned together: a packet is allowed if it matches any one of them. There's no precedence, no ordering, no 'deny' rule type in the core API. You get whitelisting only, which is both a simplicity win and a constraint you need to design around.

# STEP 1: Apply a default-deny baseline to a namespace.
# This selects ALL pods in the namespace (empty podSelector matches everything)
# and specifies BOTH policyTypes — so both ingress and egress are now default-deny.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all-traffic
  namespace: payments
spec:
  podSelector: {}              # Empty selector = matches every Pod in this namespace
  policyTypes:
    - Ingress                  # Explicitly govern inbound traffic
    - Egress                   # Explicitly govern outbound traffic
  # No ingress or egress rules defined here — that's intentional.
  # The absence of rules under a governed policyType means: deny everything.
  # This is your zero-trust baseline. Now you add back only what you need.

Writing Precise Ingress and Egress Rules — With the DNS Trap Explained

Once you've applied default-deny, you need to surgically re-open only the traffic paths your application legitimately needs. Ingress rules control what can reach your Pod. Egress rules control what your Pod can reach. Both use the same selector primitives: podSelector, namespaceSelector, and ipBlock, which you can combine with AND logic inside a single from/to entry, or use OR logic across multiple entries.

The subtlety that burns everyone: a from entry with both podSelector AND namespaceSelector means the source must match BOTH selectors simultaneously — it's an AND. Two separate from entries each with their own selector is an OR. The indentation in YAML is load-bearing here. Get it wrong and you either over-permit or under-permit with no error from the API server.

The DNS trap is equally nasty. When you lock down egress, your Pods immediately lose DNS resolution because they can no longer reach CoreDNS on port 53 UDP/TCP. Every connection attempt fails not with a 'connection refused' but with a timeout waiting for DNS — which takes 30 seconds to surface. Always add an explicit egress rule for CoreDNS as part of your default-deny rollout, or you'll wonder why your app is broken when your network policy looks correct.

# This policy governs the 'api-server' Pods in the 'payments' namespace.
# It allows:
#   INGRESS: Only from Pods labeled 'app: frontend' in the 'web' namespace
#   EGRESS:  Only to the PostgreSQL database pods on port 5432
#            AND to CoreDNS on port 53 (critical — without this, DNS breaks)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-server-traffic-rules
  namespace: payments
spec:
  podSelector:
    matchLabels:
      app: api-server
  policyTypes:
    - Ingress
    - Egress

  ingress:
    - from:
        # AND logic: source must be in namespace 'web' AND have label 'app: frontend'
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: web
          podSelector:
            matchLabels:
              app: frontend
      ports:
        - protocol: TCP
          port: 8080

  egress:
    # Rule 1: Allow outbound to PostgreSQL pods only
    - to:
        - podSelector:
            matchLabels:
              app: postgres-primary
      ports:
        - protocol: TCP
          port: 5432

    # Rule 2: Allow DNS resolution — NEVER omit this in an egress-restricted policy
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

Verifying Real Enforcement and Debugging Policy Failures in Production

Applying a NetworkPolicy and assuming it works is a mistake you only make once in production. The API server accepts any syntactically valid policy regardless of whether your CNI supports it. You need to verify enforcement at the traffic level, not the YAML level.

The gold-standard test is running a temporary Pod in the source namespace and attempting a connection directly — not through a Service mesh or load balancer that might bypass node-level rules. Use kubectl run with --rm -it to spin up a throwaway Pod, then use curl, nc, or wget to probe the target. A dropped connection times out; a policy-permitted connection either succeeds or returns an application-level error (which is actually what you want to see — it means the packet reached the target).

For Cilium clusters, cilium monitor and the Hubble UI are exceptionally powerful — they show you in real time which policies matched or dropped each flow, with source/destination Pod identity, namespace, and labels. For Calico clusters, calicoctl get networkpolicy and iptables -L -n --line-numbers on the node running your Pod reveal the actual enforced rules. Always test both directions — a policy that allows egress from Pod A to Pod B doesn't automatically allow ingress to Pod B from Pod A unless Pod B also has a matching ingress rule.

#!/usr/bin/env bash
# Verify Network Policy enforcement empirically.
# Tests both allowed and blocked paths.

set -euo pipefail

TARGET_NAMESPACE="payments"
TARGET_SERVICE="api-server"
TARGET_PORT="8080"
ALLOWED_SOURCE_NAMESPACE="web"
BLOCKED_SOURCE_NAMESPACE="monitoring"
CONNECT_TIMEOUT_SECONDS="3"

echo "=== Network Policy Enforcement Verification ==="
echo ""

# Test 1: Allowed path
echo "[TEST 1] Allowed ingress: frontend (web) -> api-server (payments)"
ALLOWED_RESULT=$(kubectl run policy-test-allowed \
  --namespace="$ALLOWED_SOURCE_NAMESPACE" \
  --image=curlimages/curl:8.5.0 \
  --restart=Never --rm --quiet -it \
  -- curl --silent --max-time "$CONNECT_TIMEOUT_SECONDS" \
       --output /dev/null --write-out "%{http_code}" \
       "http://${TARGET_SERVICE}.${TARGET_NAMESPACE}.svc.cluster.local:${TARGET_PORT}/health" \
  2>/dev/null || echo "FAILED")

if [ "$ALLOWED_RESULT" = "200" ]; then
  echo "  PASS: HTTP 200 received"
else
  echo "  FAIL: Expected HTTP 200, got '$ALLOWED_RESULT'"
fi

echo ""

# Test 2: Blocked path
echo "[TEST 2] Blocked ingress: prometheus (monitoring) -> api-server (payments)"
BLOCKED_RESULT=$(kubectl run policy-test-blocked \
  --namespace="$BLOCKED_SOURCE_NAMESPACE" \
  --image=curlimages/curl:8.5.0 \
  --restart=Never --rm --quiet -it \
  -- curl --silent --max-time "$CONNECT_TIMEOUT_SECONDS" \
       --output /dev/null --write-out "%{http_code}" \
       "http://${TARGET_SERVICE}.${TARGET_NAMESPACE}.svc.cluster.local:${TARGET_PORT}/health" \
  2>/dev/null || echo "TIMEOUT")

if [ "$BLOCKED_RESULT" = "TIMEOUT" ] || [ "$BLOCKED_RESULT" = "000" ]; then
  echo "  PASS: Connection timed out — blocked source is correctly dropped"
else
  echo "  FAIL: Expected timeout, got '$BLOCKED_RESULT' — policy NOT enforced!"
fi

echo ""
echo "=== Verification Complete ==="

Production Patterns: Namespace Isolation, Monitoring Carve-outs and Label Hygiene

In a real multi-tenant cluster, you can't write policies Pod-by-Pod. You need namespace-scoped baselines combined with additive per-workload rules. The pattern that works at scale is: one default-deny policy per namespace applied by your CD pipeline at namespace creation, then application-specific policies delivered alongside each Helm chart or Kustomize overlay.

Monitoring is the most common carve-out needed. Prometheus needs to scrape metrics from every namespace, but you don't want to globally allow all ingress. The clean solution is a namespace label like monitoring.io/allow-scrape: 'true' and a policy in each target namespace that allows ingress from the monitoring namespace on port 9090 or whatever your metrics port is. This keeps control local to the target namespace.

Label hygiene is non-negotiable. Network Policies inherit whatever labels your Pods have — if a developer changes a label during a refactor, the policy selector silently stops matching and the Pod falls back to default-deny behavior with no warning event. Use immutable labels like app: payment-api for security selectors and mutable labels like version: v2 only for routing. Audit your selectors in CI with kubectl get pods -l app=api-server -n payments and fail the pipeline if the expected count is zero.

# Prometheus scrape carve-out for the 'payments' namespace.
# Complements the default-deny-all policy already in place.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-prometheus-scrape
  namespace: payments
  labels:
    policy-type: monitoring-carveout
    managed-by: platform-team
spec:
  podSelector:
    matchLabels:
      monitoring.io/expose-metrics: "true"
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: monitoring
          podSelector:
            matchLabels:
              app: prometheus
      ports:
        - protocol: TCP
          port: 8080

Network Policy Performance: iptables vs eBPF at Scale

The CNI enforcement mechanism directly impacts network latency and control plane load. Understanding the performance characteristics of your CNI is critical for capacity planning and troubleshooting latency issues that appear only at scale.

#!/usr/bin/env bash
# Check NetworkPolicy rule count and CNI performance characteristics.
# Run on a node with calicoctl or cilium CLI installed.

set -euo pipefail

# For Calico: count iptables rules per endpoint
echo "=== Calico iptables Rule Count ==="
if command -v calicoctl &> /dev/null; then
  calicoctl get networkpolicy -A -o wide | wc -l
  echo "Total iptables chains on this node:"
  iptables -L -n | grep -c '^Chain'
  echo "Total iptables rules on this node:"
  iptables -L -n | grep -c '^[0-9]'
else
  echo "calicoctl not found — skipping Calico check"
fi

echo ""

# For Cilium: check eBPF policy programs
echo "=== Cilium eBPF Policy Status ==="
if command -v cilium &> /dev/null; then
  cilium status
  echo ""
  echo "Policy verdicts (last 100 flows):"
  cilium monitor --type policy-verdict -n payments | tail -20
else
  echo "cilium CLI not found — skipping Cilium check"
fi

echo ""

# General: check for NetworkPolicy count per namespace
echo "=== NetworkPolicy Distribution ==="
for ns in $(kubectl get namespaces -o jsonpath='{.items[*].metadata.name}'); do
  count=$(kubectl get networkpolicy -n "$ns" --no-headers 2>/dev/null | wc -l)
  if [ "$count" -gt 0 ]; then
    echo "  $ns: $count policies"
  fi
done

Frequently Asked Questions