Kubernetes Network Policies: Default-Deny Egress Blocks DNS

Most teams get Kubernetes running, deploy their apps, and move on — never realizing their payment service can freely dial their logging sidecar, which can freely dial their database, which can freely reach the internet. That's not paranoia; that's the default. Kubernetes was designed for rapid connectivity, not zero-trust isolation. The moment you run multiple tenants, compliance workloads, or anything that touches PII or financial data, that open-door model becomes a liability.

Network Policies solve this by letting you express intent in YAML: only Pods with this label may reach my database on port 5432, from this namespace only, and my database can reach nothing outbound except DNS. The CNI plugin — not the Kubernetes API server — enforces those rules in the kernel using iptables, eBPF, or nftables depending on your stack. That distinction matters enormously for debugging and performance.

This is not a syntax reference. It covers how policies are evaluated and merged, how to write airtight ingress and egress rules without accidentally blackholing DNS, how to verify enforcement at the network level rather than trusting your YAML applied cleanly, and the production mistakes that silently leave clusters wide open.

Why Default-Deny Egress Breaks DNS

Kubernetes Network Policies are firewall rules that control traffic between pods at the IP address or port level (OSI layer 3 or 4). They are implemented by the CNI plugin (e.g., Calico, Cilium, Weave) and are evaluated per-pod based on label selectors. The core mechanic: a policy selects a set of pods and defines ingress and/or egress rules — if no policy selects a pod, all traffic is allowed; once any policy selects it, all traffic not explicitly permitted is denied.

In practice, applying a default-deny egress policy (a policy that selects all pods with no egress rules) immediately blocks all outbound traffic, including DNS lookups to the cluster's CoreDNS service. This happens because DNS runs on UDP/TCP port 53, and unless your egress rule explicitly allows traffic to the kube-system namespace on port 53, the pod cannot resolve any hostnames. This is a common first-day surprise for teams adopting network policies.

Use default-deny egress when you need to enforce least-privilege networking — for example, in multi-tenant clusters or PCI/HIPAA environments. But always pair it with an explicit egress rule that permits DNS traffic to the CoreDNS service IP or namespace selector. Without that, your pods will fail to resolve service names, causing cascading failures in service discovery, health checks, and external API calls.

How Network Policy Enforcement Actually Works — The CNI Layer

Here's the thing most tutorials skip: the Kubernetes API server doesn't enforce Network Policies. It just stores them. The actual enforcement happens inside your CNI plugin — Calico, Cilium, Weave, Antrea — which watches the API server for NetworkPolicy objects and translates them into kernel-level firewall rules on each node.

With Calico on older kernels, that means iptables chains per endpoint. With Cilium, it's eBPF programs loaded into the kernel that intercept packets at the socket layer before they ever hit iptables — significantly lower latency and dramatically better observability. With Flannel, enforcement is zero because Flannel doesn't implement Network Policies at all. This is one of the most common production surprises: a team applies policies and believes they're enforced, but their CNI silently ignores them.

Policy evaluation works like a firewall whitelist. If no NetworkPolicy selects a Pod, all traffic is allowed. The moment any policy selects a Pod — via podSelector — that Pod enters an implicit 'default deny' for the traffic directions that policy governs. Multiple policies selecting the same Pod are unioned together: a packet is allowed if it matches any one of them. There's no precedence, no ordering, no 'deny' rule type in the core API. You get whitelisting only, which is both a simplicity win and a constraint you need to design around.

# STEP 1: Apply a default-deny baseline to a namespace.
# This selects ALL pods in the namespace (empty podSelector matches everything)
# and specifies BOTH policyTypes — so both ingress and egress are now default-deny.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all-traffic
  namespace: payments
spec:
  podSelector: {}              # Empty selector = matches every Pod in this namespace
  policyTypes:
    - Ingress                  # Explicitly govern inbound traffic
    - Egress                   # Explicitly govern outbound traffic
  # No ingress or egress rules defined here — that's intentional.
  # The absence of rules under a governed policyType means: deny everything.
  # This is your zero-trust baseline. Now you add back only what you need.

Writing Precise Ingress and Egress Rules — With the DNS Trap Explained

Once you've applied default-deny, you need to surgically re-open only the traffic paths your application legitimately needs. Ingress rules control what can reach your Pod. Egress rules control what your Pod can reach. Both use the same selector primitives: podSelector, namespaceSelector, and ipBlock, which you can combine with AND logic inside a single from/to entry, or use OR logic across multiple entries.

The subtlety that burns everyone: a from entry with both podSelector AND namespaceSelector means the source must match BOTH selectors simultaneously — it's an AND. Two separate from entries each with their own selector is an OR. The indentation in YAML is load-bearing here. Get it wrong and you either over-permit or under-permit with no error from the API server.

The DNS trap is equally nasty. When you lock down egress, your Pods immediately lose DNS resolution because they can no longer reach CoreDNS on port 53 UDP/TCP. Every connection attempt fails not with a 'connection refused' but with a timeout waiting for DNS — which takes 30 seconds to surface. Always add an explicit egress rule for CoreDNS as part of your default-deny rollout, or you'll wonder why your app is broken when your network policy looks correct.

# This policy governs the 'api-server' Pods in the 'payments' namespace.
# It allows:
#   INGRESS: Only from Pods labeled 'app: frontend' in the 'web' namespace
#   EGRESS:  Only to the PostgreSQL database pods on port 5432
#            AND to CoreDNS on port 53 (critical — without this, DNS breaks)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-server-traffic-rules
  namespace: payments
spec:
  podSelector:
    matchLabels:
      app: api-server
  policyTypes:
    - Ingress
    - Egress

  ingress:
    - from:
        # AND logic: source must be in namespace 'web' AND have label 'app: frontend'
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: web
          podSelector:
            matchLabels:
              app: frontend
      ports:
        - protocol: TCP
          port: 8080

  egress:
    # Rule 1: Allow outbound to PostgreSQL pods only
    - to:
        - podSelector:
            matchLabels:
              app: postgres-primary
      ports:
        - protocol: TCP
          port: 5432

    # Rule 2: Allow DNS resolution — NEVER omit this in an egress-restricted policy
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

Verifying Real Enforcement and Debugging Policy Failures in Production

Applying a NetworkPolicy and assuming it works is a mistake you only make once in production. The API server accepts any syntactically valid policy regardless of whether your CNI supports it. You need to verify enforcement at the traffic level, not the YAML level.

The gold-standard test is running a temporary Pod in the source namespace and attempting a connection directly — not through a Service mesh or load balancer that might bypass node-level rules. Use kubectl run with --rm -it to spin up a throwaway Pod, then use curl, nc, or wget to probe the target. A dropped connection times out; a policy-permitted connection either succeeds or returns an application-level error (which is actually what you want to see — it means the packet reached the target).

For Cilium clusters, cilium monitor and the Hubble UI are exceptionally powerful — they show you in real time which policies matched or dropped each flow, with source/destination Pod identity, namespace, and labels. For Calico clusters, calicoctl get networkpolicy and iptables -L -n --line-numbers on the node running your Pod reveal the actual enforced rules. Always test both directions — a policy that allows egress from Pod A to Pod B doesn't automatically allow ingress to Pod B from Pod A unless Pod B also has a matching ingress rule.

#!/usr/bin/env bash
# Verify Network Policy enforcement empirically.
# Tests both allowed and blocked paths.

set -euo pipefail

TARGET_NAMESPACE="payments"
TARGET_SERVICE="api-server"
TARGET_PORT="8080"
ALLOWED_SOURCE_NAMESPACE="web"
BLOCKED_SOURCE_NAMESPACE="monitoring"
CONNECT_TIMEOUT_SECONDS="3"

echo "=== Network Policy Enforcement Verification ==="
echo ""

# Test 1: Allowed path
echo "[TEST 1] Allowed ingress: frontend (web) -> api-server (payments)"
ALLOWED_RESULT=$(kubectl run policy-test-allowed \
  --namespace="$ALLOWED_SOURCE_NAMESPACE" \
  --image=curlimages/curl:8.5.0 \
  --restart=Never --rm --quiet -it \
  -- curl --silent --max-time "$CONNECT_TIMEOUT_SECONDS" \
       --output /dev/null --write-out "%{http_code}" \
       "http://${TARGET_SERVICE}.${TARGET_NAMESPACE}.svc.cluster.local:${TARGET_PORT}/health" \
  2>/dev/null || echo "FAILED")

if [ "$ALLOWED_RESULT" = "200" ]; then
  echo "  PASS: HTTP 200 received"
else
  echo "  FAIL: Expected HTTP 200, got '$ALLOWED_RESULT'"
fi

echo ""

# Test 2: Blocked path
echo "[TEST 2] Blocked ingress: prometheus (monitoring) -> api-server (payments)"
BLOCKED_RESULT=$(kubectl run policy-test-blocked \
  --namespace="$BLOCKED_SOURCE_NAMESPACE" \
  --image=curlimages/curl:8.5.0 \
  --restart=Never --rm --quiet -it \
  -- curl --silent --max-time "$CONNECT_TIMEOUT_SECONDS" \
       --output /dev/null --write-out "%{http_code}" \
       "http://${TARGET_SERVICE}.${TARGET_NAMESPACE}.svc.cluster.local:${TARGET_PORT}/health" \
  2>/dev/null || echo "TIMEOUT")

if [ "$BLOCKED_RESULT" = "TIMEOUT" ] || [ "$BLOCKED_RESULT" = "000" ]; then
  echo "  PASS: Connection timed out — blocked source is correctly dropped"
else
  echo "  FAIL: Expected timeout, got '$BLOCKED_RESULT' — policy NOT enforced!"
fi

echo ""
echo "=== Verification Complete ==="

Production Patterns: Namespace Isolation, Monitoring Carve-outs and Label Hygiene

In a real multi-tenant cluster, you can't write policies Pod-by-Pod. You need namespace-scoped baselines combined with additive per-workload rules. The pattern that works at scale is: one default-deny policy per namespace applied by your CD pipeline at namespace creation, then application-specific policies delivered alongside each Helm chart or Kustomize overlay.

Monitoring is the most common carve-out needed. Prometheus needs to scrape metrics from every namespace, but you don't want to globally allow all ingress. The clean solution is a namespace label like monitoring.io/allow-scrape: 'true' and a policy in each target namespace that allows ingress from the monitoring namespace on port 9090 or whatever your metrics port is. This keeps control local to the target namespace.

Label hygiene is non-negotiable. Network Policies inherit whatever labels your Pods have — if a developer changes a label during a refactor, the policy selector silently stops matching and the Pod falls back to default-deny behavior with no warning event. Use immutable labels like app: payment-api for security selectors and mutable labels like version: v2 only for routing. Audit your selectors in CI with kubectl get pods -l app=api-server -n payments and fail the pipeline if the expected count is zero.

# Prometheus scrape carve-out for the 'payments' namespace.
# Complements the default-deny-all policy already in place.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-prometheus-scrape
  namespace: payments
  labels:
    policy-type: monitoring-carveout
    managed-by: platform-team
spec:
  podSelector:
    matchLabels:
      monitoring.io/expose-metrics: "true"
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: monitoring
          podSelector:
            matchLabels:
              app: prometheus
      ports:
        - protocol: TCP
          port: 8080

Network Policy Performance: iptables vs eBPF at Scale

The CNI enforcement mechanism directly impacts network latency and control plane load. Understanding the performance characteristics of your CNI is critical for capacity planning and troubleshooting latency issues that appear only at scale.

#!/usr/bin/env bash
# Check NetworkPolicy rule count and CNI performance characteristics.
# Run on a node with calicoctl or cilium CLI installed.

set -euo pipefail

# For Calico: count iptables rules per endpoint
echo "=== Calico iptables Rule Count ==="
if command -v calicoctl &> /dev/null; then
  calicoctl get networkpolicy -A -o wide | wc -l
  echo "Total iptables chains on this node:"
  iptables -L -n | grep -c '^Chain'
  echo "Total iptables rules on this node:"
  iptables -L -n | grep -c '^[0-9]'
else
  echo "calicoctl not found — skipping Calico check"
fi

echo ""

# For Cilium: check eBPF policy programs
echo "=== Cilium eBPF Policy Status ==="
if command -v cilium &> /dev/null; then
  cilium status
  echo ""
  echo "Policy verdicts (last 100 flows):"
  cilium monitor --type policy-verdict -n payments | tail -20
else
  echo "cilium CLI not found — skipping Cilium check"
fi

echo ""

# General: check for NetworkPolicy count per namespace
echo "=== NetworkPolicy Distribution ==="
for ns in $(kubectl get namespaces -o jsonpath='{.items[*].metadata.name}'); do
  count=$(kubectl get networkpolicy -n "$ns" --no-headers 2>/dev/null | wc -l)
  if [ "$count" -gt 0 ]; then
    echo "  $ns: $count policies"
  fi
done

The Two Kinds of Pod Isolation (And Why Default-Deny Is a Lie)

Network policies don't add security—they remove connectivity. That sounds backwards until you understand how Kubernetes handles pod isolation. When you create a NetworkPolicy that selects a pod, three things happen: First, the CNI drops all traffic that doesn't match an explicit allow rule. Second, the pod becomes 'isolated' in Kubernetes terms. Third, and this is where teams get burned: isolation only applies to interfaces the policy explicitly selects. Any traffic that bypasses a selected interface—like hostNetwork pods or traffic routed through the node itself—sails right through your 'default deny' policy. The official docs call this selective isolation. I call it a footgun. If you have a pod running with hostNetwork: true, no amount of NetworkPolicy magic touches its traffic. You need node-level firewall rules (iptables, nftables) or a service mesh sidecar to intercept that path. The two sorts of isolation are: namespace-wide isolation (select all pods in the namespace) and per-pod isolation (select specific labels). You can use both to build layered security, but never assume a default-deny policy covers the entire attack surface. It covers the pod network only.

// io.thecodeforge — devops tutorial

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: payments-api
spec:
  podSelector: {}
  policyTypes:
  - Ingress
---
# This pod is NOT protected by the above policy
apiVersion: v1
kind: Pod
metadata:
  name: vault-agent
  namespace: payments-api
spec:
  hostNetwork: true  # <-- bypasses all network policies
  containers:
  - name: vault
    image: hashicorp/vault:1.15

Default Policies: The Three You Must Write Before Breakfast

Teams that skip default policies end up debugging why their payments namespace can talk to prod databases at 3 AM. Kubernetes applies no default isolation—zero. A pod can shout into the void or whisper to any other pod until you write a policy. There are three defaults you need in every production cluster: default-deny-all-ingress, default-deny-all-egress, and an allow-dns-egress for kube-dns. The first two are simple: an empty podSelector with empty rules blocks everything. The third one is where 90% of teams fail. They write a default-deny egress, deploy it, and their pods can't resolve service names. DNS runs on UDP port 53 to the kube-dns service IP (usually 10.96.0.10). If your egress default-deny doesn't include an allow rule for that IP on UDP 53, you get to learn what a CoreDNS timeout looks like in production. Not fun. The pattern is: deny-all-ingress, deny-all-egress, then allow-DNS-egress to the cluster DNS IP. Everything else graduates to explicit allow rules. This is zero trust for the pod network. It's not optional. It's the baseline that every security audit will ask about. And yes, you should apply these default policies to every namespace via a cluster-wide controller or Open Policy Agent rule—because humans forget to write them for new namespaces.

// io.thecodeforge — devops tutorial

# Default-deny-all ingress
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: payments-api
spec:
  podSelector: {}
  policyTypes:
  - Ingress
---
# Default-deny-all egress + DNS allow
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-egress-plus-dns
  namespace: payments-api
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - ipBlock:
        cidr: 10.96.0.10/32  # cluster DNS IP
    ports:
    - port: 53
      protocol: UDP

Stop Guessing: How NetworkPolicy Treats Already-Open Connections

You apply a deny-all NetworkPolicy and expect instant silence. What happens to TCP sockets that were already established before the policy landed? They stay open. Kubernetes NetworkPolicy is connection-unaware — it evaluates packets, not sessions. The CNI plugin (Calico, Cilium, etc.) tracks conntrack entries from iptables or eBPF maps. Existing connections in ESTABLISHED state bypass new ingress/egress rules because the conntrack entry was created before your policy existed.

This is a production trap. If you roll out a strict default-deny policy during business hours, pre-existing SSH sessions, database connections, or monitoring scrapes won't be cut off immediately. They linger until the connection times out or the application tears it down. The fix: drain traffic or restart pods after policy changes. Don't trust a policy audit that shows blocked packets while live connections still flow. Conntrack doesn't lie, but it has memory.

// io.thecodeforge — devops tutorial

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress
  # Existing ESTABLISHED connections survive this.
  # Check conntrack on nodes:
  # $ conntrack -L -p tcp --state ESTABLISHED
  # Or for Cilium:
  # $ cilium bpf conntrack list | grep ESTABLISHED

Port Range Targeting: Why Your Micro-Service Needs It and Your Devs Don't

The ports field in NetworkPolicy accepts a single port or a named port on a pod. But some applications — like gRPC servers on ephemeral high ports or monitoring agents scanning port ranges — require targeting a contiguous range. Kubernetes 1.25+ added endPort to the ports spec. Without it, you'd write five separate ingress rules for ports 30000-30004. With endPort, you write one line.

The WHY: Port ranges reduce policy bloat and misconfiguration risk. If your service listens on ports 8080-8090, a single rule with port: 8080, endPort: 8090 covers all. But be brutal about scope — don't open a range because you're lazy. Define named ports on pods and reference those instead. Named ports are self-documenting and survive port renumbering. Reserve endPort for cases where you truly cannot predict the exact port (e.g., sidecar injection, dynamic service mesh ports). If your devs ask for a port range 'just in case', push back. They're asking for a security hole.

// io.thecodeforge — devops tutorial

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-metrics-range
spec:
  podSelector:
    matchLabels:
      app: prometheus
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: monitoring
    ports:
    - protocol: TCP
      port: 9090
      endPort: 9100  # covers 9090, 9091, ..., 9100 (11 ports)

Explicitly Allow Necessary Pod-to-Pod Communications

Default-deny policies are the safe foundation, but they break your application unless you explicitly permit the exact pod-to-pod traffic your services depend on. The mistake is writing permissive catch-all rules like 'allow all from namespace X'. Instead, define precise policies that match pod labels, ports, and protocols your microservices actually use. For example, an API pod should only accept connections from a frontend pod on port 8080, not from a database pod. To discover these requirements, audit your application's connection map: list every pod that talks to every other pod, then encode each edge as a distinct network policy. This prevents accidental exposure and ensures that when a new pod is deployed, it cannot communicate unless a matching policy exists. The rule is simple: every allowed conversation must be intentional and visible in your YAML files, not implicitly inherited from namespace membership.

// io.thecodeforge — devops tutorial
// Explicit allow: frontend → API on port 8080
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend-to-api
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          tier: frontend
    ports:
    - protocol: TCP
      port: 8080

Summary: Network Policies Are Contracts, Not Filters

Kubernetes Network Policies enforce identity-based, least-privilege networking between pods. They are not firewall ACLs; they are declarative contracts enforced by the CNI plugin. The core pattern is: default deny all traffic, then explicitly allow only the minimal paths required by your application. You must handle the DNS trap — allow egress to CoreDNS or the cluster DNS service IP — and understand that already-established connections are not retroactively cut. For production, adopt namespace isolation, label hygiene, and eBPF-based CNIs for scale. Never rely on namespace-level allow rules; use fine-grained pod selectors. Port range targeting improves security for microservices with dynamic ports. Finally, always test enforcement with real probes: deploy a debug pod and confirm denied packets are actually dropped. Network policies are the single source of truth for pod communication; if your app breaks after adding them, your policy is incomplete, not the feature.

// io.thecodeforge — devops tutorial
// Minimal default-deny baseline
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress