Kubernetes Network Policies: Zero-Trust Segmentation in Production
Most teams get Kubernetes running, deploy their apps, and move on — never realizing their payment service can freely dial their logging sidecar, which can freely dial their database, which can freely reach the internet. That's not paranoia; that's the default. Kubernetes was designed for rapid connectivity, not zero-trust isolation. The moment you run multiple tenants, compliance workloads, or anything that touches PII or financial data, that open-door model becomes a liability. A single compromised Pod in a flat network is a foothold into everything.
Network Policies solve this by letting you express intent in YAML: 'only Pods with this label may reach my database on port 5432, from this namespace only, and my database can reach nothing outbound except DNS.' The CNI plugin — not the Kubernetes API server — enforces those rules in the kernel using iptables, eBPF, or nftables depending on your stack. That distinction matters enormously for debugging and performance.
By the end of this article you'll understand how policies are evaluated and merged, how to write airtight ingress and egress rules without accidentally blackholing DNS, how to verify enforcement at the network level rather than just trusting your YAML applied cleanly, and the three production mistakes that silently leave clusters wide open even when teams think they're locked down.
How Network Policy Enforcement Actually Works — The CNI Layer
Here's the thing most tutorials skip: the Kubernetes API server doesn't enforce Network Policies. It just stores them. The actual enforcement happens inside your CNI plugin — Calico, Cilium, Weave, Antrea — which watches the API server for NetworkPolicy objects and translates them into kernel-level firewall rules on each node.
With Calico on older kernels, that means iptables chains per endpoint. With Cilium, it's eBPF programs loaded into the kernel that intercept packets at the socket layer before they ever hit iptables — significantly lower latency and dramatically better observability. With Flannel, enforcement is zero because Flannel doesn't implement Network Policies at all. This is one of the most common production surprises: a team applies policies and believes they're enforced, but their CNI silently ignores them.
Policy evaluation works like a firewall whitelist. If no NetworkPolicy selects a Pod, all traffic is allowed. The moment any policy selects a Pod — via podSelector — that Pod enters an implicit 'default deny' for the traffic directions that policy governs. Multiple policies selecting the same Pod are unioned together: a packet is allowed if it matches any one of them. There's no precedence, no ordering, no 'deny' rule type in the core API. You get whitelisting only, which is both a simplicity win and a constraint you need to design around.
# STEP 1: Apply a default-deny baseline to a namespace. # This selects ALL pods in the namespace (empty podSelector matches everything) # and specifies BOTH policyTypes — so both ingress and egress are now default-deny. # Without specifying both policyTypes explicitly, only the directions you mention # are affected. A policy with only 'ingress' rules leaves egress wide open. apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny-all-traffic namespace: payments # Scope is always per-namespace spec: podSelector: {} # Empty selector = matches every Pod in this namespace policyTypes: - Ingress # Explicitly govern inbound traffic - Egress # Explicitly govern outbound traffic # No ingress or egress rules defined here — that's intentional. # The absence of rules under a governed policyType means: deny everything. # This is your zero-trust baseline. Now you add back only what you need.
# Verify the policy was stored (not yet verifying enforcement — see section 3 for that):
kubectl get networkpolicy -n payments
NAME POD-SELECTOR AGE
default-deny-all-traffic <none> 4s
Writing Precise Ingress and Egress Rules — With the DNS Trap Explained
Once you've applied default-deny, you need to surgically re-open only the traffic paths your application legitimately needs. Ingress rules control what can reach your Pod. Egress rules control what your Pod can reach. Both use the same selector primitives: podSelector, namespaceSelector, and ipBlock, which you can combine with AND logic inside a single from/to entry, or use OR logic across multiple entries.
The subtlety that burns everyone: a from entry with both podSelector AND namespaceSelector means the source must match BOTH selectors simultaneously — it's an AND. Two separate from entries each with their own selector is an OR. The indentation in YAML is load-bearing here. Get it wrong and you either over-permit or under-permit with no error from the API server.
The DNS trap is equally nasty. When you lock down egress, your Pods immediately lose DNS resolution because they can no longer reach CoreDNS on port 53 UDP/TCP. Every connection attempt fails not with a 'connection refused' but with a timeout waiting for DNS — which takes 30 seconds to surface. Always add an explicit egress rule for CoreDNS as part of your default-deny rollout, or you'll wonder why your app is broken when your network policy looks correct.
# This policy governs the 'api-server' Pods in the 'payments' namespace. # It allows: # INGRESS: Only from Pods labeled 'app: frontend' in the 'web' namespace # EGRESS: Only to the PostgreSQL database pods on port 5432 # AND to CoreDNS on port 53 (critical — without this, DNS breaks) apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: api-server-traffic-rules namespace: payments spec: podSelector: matchLabels: app: api-server # This policy governs Pods with this label policyTypes: - Ingress - Egress ingress: - from: # AND logic: source must be in namespace 'web' AND have label 'app: frontend' # This is ONE entry with two fields — that means AND. - namespaceSelector: matchLabels: kubernetes.io/metadata.name: web # Namespace label (auto-applied by k8s 1.21+) podSelector: # Note: same list item as namespaceSelector — AND! matchLabels: app: frontend ports: - protocol: TCP port: 8080 # Only accept traffic on the API's actual listen port egress: # Rule 1: Allow outbound to PostgreSQL pods only - to: - podSelector: matchLabels: app: postgres-primary # Only the primary DB — not replicas, not other services ports: - protocol: TCP port: 5432 # Rule 2: Allow DNS resolution — NEVER omit this in an egress-restricted policy # CoreDNS lives in kube-system namespace and listens on both UDP and TCP port 53 - to: - namespaceSelector: matchLabels: kubernetes.io/metadata.name: kube-system podSelector: matchLabels: k8s-app: kube-dns ports: - protocol: UDP port: 53 - protocol: TCP # TCP fallback for large DNS responses port: 53
# Test ingress is working from allowed source:
kubectl exec -n web deploy/frontend -- curl -s http://api-server.payments.svc.cluster.local:8080/health
{"status":"ok"}
# Test ingress is blocked from an unauthorized pod:
kubectl exec -n monitoring deploy/prometheus -- curl -s --max-time 3 http://api-server.payments.svc.cluster.local:8080/health
curl: (28) Connection timed out after 3001 milliseconds
# Timeout (not refused) is the expected CNI drop behavior — packets are silently dropped
Verifying Real Enforcement and Debugging Policy Failures in Production
Applying a NetworkPolicy and assuming it works is a mistake you only make once in production. The API server accepts any syntactically valid policy regardless of whether your CNI supports it. You need to verify enforcement at the traffic level, not the YAML level.
The gold-standard test is running a temporary Pod in the source namespace and attempting a connection directly — not through a Service mesh or load balancer that might bypass node-level rules. Use kubectl run with --rm -it to spin up a throwaway Pod, then use curl, nc, or wget to probe the target. A dropped connection times out; a policy-permitted connection either succeeds or returns an application-level error (which is actually what you want to see — it means the packet reached the target).
For Cilium clusters, cilium monitor and the Hubble UI are exceptionally powerful — they show you in real time which policies matched or dropped each flow, with source/destination Pod identity, namespace, and labels. For Calico clusters, calicoctl get networkpolicy and iptables -L -n --line-numbers on the node running your Pod reveal the actual enforced rules. Always test both directions — a policy that allows egress from Pod A to Pod B doesn't automatically allow ingress to Pod B from Pod A unless Pod B also has a matching ingress rule.
#!/usr/bin/env bash # Run this script to empirically verify your Network Policy enforcement. # It tests both allowed and blocked paths and reports results clearly. # Requires: kubectl configured with cluster access set -euo pipefail TARGET_NAMESPACE="payments" TARGET_SERVICE="api-server" TARGET_PORT="8080" ALLOWED_SOURCE_NAMESPACE="web" BLOCKED_SOURCE_NAMESPACE="monitoring" CONNECT_TIMEOUT_SECONDS="3" echo "=== Network Policy Enforcement Verification ===" echo "" # Test 1: Allowed path — frontend in 'web' namespace should reach api-server echo "[TEST 1] Allowed ingress: frontend (web) -> api-server (payments)" ALLOWED_RESULT=$(kubectl run policy-test-allowed \ --namespace="$ALLOWED_SOURCE_NAMESPACE" \ --image=curlimages/curl:8.5.0 \ --restart=Never \ --rm \ --quiet \ -it \ -- curl \ --silent \ --max-time "$CONNECT_TIMEOUT_SECONDS" \ --output /dev/null \ --write-out "%{http_code}" \ "http://${TARGET_SERVICE}.${TARGET_NAMESPACE}.svc.cluster.local:${TARGET_PORT}/health" \ 2>/dev/null || echo "FAILED") if [ "$ALLOWED_RESULT" = "200" ]; then echo " PASS: HTTP 200 received — ingress from allowed source is working" else echo " FAIL: Expected HTTP 200, got '$ALLOWED_RESULT' — check policy or service" fi echo "" # Test 2: Blocked path — prometheus in 'monitoring' namespace should be dropped echo "[TEST 2] Blocked ingress: prometheus (monitoring) -> api-server (payments)" BLOCKED_RESULT=$(kubectl run policy-test-blocked \ --namespace="$BLOCKED_SOURCE_NAMESPACE" \ --image=curlimages/curl:8.5.0 \ --restart=Never \ --rm \ --quiet \ -it \ -- curl \ --silent \ --max-time "$CONNECT_TIMEOUT_SECONDS" \ --output /dev/null \ --write-out "%{http_code}" \ "http://${TARGET_SERVICE}.${TARGET_NAMESPACE}.svc.cluster.local:${TARGET_PORT}/health" \ 2>/dev/null || echo "TIMEOUT") if [ "$BLOCKED_RESULT" = "TIMEOUT" ] || [ "$BLOCKED_RESULT" = "000" ]; then echo " PASS: Connection timed out — blocked source is correctly dropped" else echo " FAIL: Expected timeout/drop, got '$BLOCKED_RESULT' — policy NOT enforced!" echo " Check if your CNI supports NetworkPolicy (Flannel does not)" fi echo "" echo "=== Verification Complete ===" # For Cilium clusters: use Hubble to see policy decisions in real time # hubble observe --namespace payments --type drop # hubble observe --namespace payments --type policy-verdict
[TEST 1] Allowed ingress: frontend (web) -> api-server (payments)
PASS: HTTP 200 received — ingress from allowed source is working
[TEST 2] Blocked ingress: prometheus (monitoring) -> api-server (payments)
PASS: Connection timed out — blocked source is correctly dropped
=== Verification Complete ===
Production Patterns: Namespace Isolation, Monitoring Carve-outs and Label Hygiene
In a real multi-tenant cluster, you can't write policies Pod-by-Pod. You need namespace-scoped baselines combined with additive per-workload rules. The pattern that works at scale is: one default-deny policy per namespace applied by your CD pipeline at namespace creation, then application-specific policies delivered alongside each Helm chart or Kustomize overlay.
Monitoring is the most common carve-out needed. Prometheus needs to scrape metrics from every namespace, but you don't want to globally allow all ingress. The clean solution is a namespace label like monitoring.io/allow-scrape: 'true' and a policy in each target namespace that allows ingress from the monitoring namespace on port 9090 or whatever your metrics port is. This keeps control local to the target namespace.
Label hygiene is non-negotiable. Network Policies inherit whatever labels your Pods have — if a developer changes a label during a refactor, the policy selector silently stops matching and the Pod falls back to default-deny behavior with no warning event. Use immutable labels like app: payment-api for security selectors and mutable labels like version: v2 only for routing. Audit your selectors in CI with kubectl get pods -l app=api-server -n payments and fail the pipeline if the expected count is zero. Never let a NetworkPolicy reference a label that isn't actively verified to exist on running Pods.
# This policy lives in the 'payments' namespace and carves out access # for Prometheus to scrape metrics without opening broad ingress. # It complements the default-deny-all policy already in place. # # Prerequisite: the 'monitoring' namespace must have the label # kubernetes.io/metadata.name: monitoring # (Kubernetes 1.21+ applies this automatically) apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-prometheus-scrape namespace: payments labels: policy-type: monitoring-carveout # Label your policies so you can audit them managed-by: platform-team spec: podSelector: matchLabels: monitoring.io/expose-metrics: "true" # Only scrape pods that opt in # Add this label to any pod whose metrics endpoint should be scraped # Pods without this label remain invisible to Prometheus traffic policyTypes: - Ingress # We're only adding ingress here; egress governed elsewhere ingress: - from: - namespaceSelector: matchLabels: kubernetes.io/metadata.name: monitoring # Only from the monitoring namespace podSelector: matchLabels: app: prometheus # AND only from the Prometheus pod itself, not anything in monitoring ports: - protocol: TCP port: 8080 # Metrics endpoint — adjust to match your /metrics port --- # How to label a deployment so it gets scraped: # kubectl label pods -l app=api-server -n payments monitoring.io/expose-metrics=true # # Verification — from Prometheus pod, this should return metrics: # kubectl exec -n monitoring deploy/prometheus \ # -- curl -s http://api-server.payments.svc.cluster.local:8080/metrics | head -5
# List all policies in the payments namespace to confirm your layered setup:
kubectl get networkpolicy -n payments -o wide
NAME POD-SELECTOR AGE
default-deny-all-traffic <none> 2h
api-server-traffic-rules app=api-server 2h
allow-prometheus-scrape monitoring.io/expose-metrics=true 4s
# Confirm the label is on the right pods:
kubectl get pods -n payments -l monitoring.io/expose-metrics=true
NAME READY STATUS RESTARTS AGE
api-server-7d9f8b6c4-xr2mq 1/1 Running 0 2h
| Aspect | Calico (iptables mode) | Cilium (eBPF mode) |
|---|---|---|
| Policy enforcement layer | iptables chains per endpoint on each node | eBPF programs loaded at socket/TC layer |
| Observability | iptables rule counters, calicoctl | Hubble UI, per-flow policy verdict logging |
| Performance at scale | iptables rule count grows O(n) — degrades at 1000+ Pods | eBPF hash maps are O(1) lookup — scales linearly |
| Layer 7 policies | Not supported in core Kubernetes API | Supported natively (HTTP method, path, gRPC) |
| DNS-based egress policies | Requires Calico GlobalNetworkPolicy (proprietary) | Built-in with Cilium's DNS-aware egress |
| Installation complexity | Moderate — well-documented, mature | Higher — requires kernel 4.9+ (5.10+ for full features) |
| Network Policy API support | Full compliance | Full compliance + extended CRDs |
| Packet drop behavior | iptables DROP — silent timeout | eBPF DROP — silent timeout, but Hubble shows it |
| Production maturity | Battle-tested since 2016 | Rapidly maturing — preferred for new clusters 2022+ |
🎯 Key Takeaways
- The Kubernetes API server stores Network Policies but never enforces them — enforcement lives entirely in your CNI plugin. If your CNI doesn't support policies (Flannel), they are silently ignored with zero warnings.
- An empty podSelector in a NetworkPolicy matches ALL Pods in the namespace — not zero Pods. This is counterintuitive but is how you write a namespace-wide default-deny baseline.
- Multiple
from/toentries in a policy are ORed together; fields within a single entry are ANDed. This YAML indentation distinction determines your actual security posture and produces no errors when wrong. - Always include a DNS egress carve-out (UDP+TCP port 53 to kube-dns) before rolling out default-deny egress, or every service discovery call in your cluster will silently time out after 30 seconds.
⚠ Common Mistakes to Avoid
- ✕Mistake 1: Specifying only 'Ingress' in policyTypes when you meant to lock down both directions — Symptom: your Pods can't be reached but can freely dial any outbound destination, including the internet. The Pod is half-locked. Fix: Always list both 'Ingress' and 'Egress' in policyTypes when applying a default-deny baseline. If you only want to control ingress for a specific policy, that's fine, but your default-deny policy must explicitly enumerate both types.
- ✕Mistake 2: Forgetting the DNS egress carve-out when enabling default-deny egress — Symptom: app Pods appear healthy per readiness checks but fail on any real traffic with 30-second delays and 'could not resolve host' errors in logs. The CNI blocks UDP/53 to CoreDNS. Fix: Always include an egress rule to kube-system Pods labeled k8s-app=kube-dns on UDP port 53 AND TCP port 53 as part of your default-deny egress rollout. Template this into your namespace bootstrap automation.
- ✕Mistake 3: Using AND-logic when OR-logic is needed (or vice versa) in from/to selectors — Symptom: either traffic from an expected source is still blocked (over-restrictive AND) or an unintended namespace can reach your Pod (over-permissive OR). No error is produced — the policy applies cleanly but does the wrong thing. Fix: Remember that fields within a single '-' list entry are ANDed; separate '-' entries are ORed. Draw the intended access matrix on paper first, then translate each allowed source into either a single entry (AND) or multiple entries (OR). Verify empirically using the curl test pattern shown in section 3.
Interview Questions on This Topic
- QA NetworkPolicy is applied to a namespace with default-deny-all. A developer reports their Pod can reach the database but can't resolve any hostnames. What's wrong and how do you fix it without relaxing security?
- QExplain the difference between a podSelector and a namespaceSelector in a 'from' clause. What does it mean when both appear in the same list entry versus as separate entries? Give an example of when you'd need each.
- QYour team applies a NetworkPolicy that looks correct but packets are still flowing between Pods that should be blocked. Walk me through how you'd diagnose whether this is a CNI issue, a policy syntax issue, or a label mismatch — and what commands you'd run.
Frequently Asked Questions
Does a Kubernetes Network Policy affect traffic between Pods in the same namespace?
Yes, absolutely. Network Policies apply to all Pod-to-Pod traffic regardless of whether the source and destination are in the same namespace or different ones. By default, same-namespace traffic is also wide open. A default-deny policy with an empty podSelector will block same-namespace traffic too, and you'll need explicit ingress rules to re-permit it.
Can Kubernetes Network Policies block traffic from outside the cluster — like from a load balancer?
For traffic entering via a Service of type LoadBalancer or NodePort, the source IP seen by the Pod is typically the node's IP or the load balancer's IP due to SNAT — not the original client IP. This means ipBlock rules targeting external IPs may not work as expected unless you set externalTrafficPolicy: Local on the Service to preserve source IPs. Network Policies work best for Pod-to-Pod east-west traffic; perimeter security for north-south traffic usually belongs in a separate ingress controller or cloud firewall.
What happens if two Network Policies select the same Pod with conflicting rules?
There are no conflicts in the traditional sense because Network Policies are purely additive whitelists — there's no 'deny' rule in the standard API. If two policies both select the same Pod, their ingress and egress rules are unioned: a packet is allowed if it satisfies any matching rule from any policy. You can never use a second policy to override and deny something a first policy allows. If you need deny semantics, you need a CNI-specific extension like Calico's GlobalNetworkPolicy or Cilium's CiliumNetworkPolicy.
Written and reviewed by senior developers with real-world experience across enterprise, startup and open-source projects. Every article on TheCodeForge is written to be clear, accurate and genuinely useful — not just SEO filler.