Kubernetes Namespaces and ResourceQuotas

You're running a shared Kubernetes cluster for three teams. Everything's fine until one team deploys a memory-hungry batch job that OOM-kills your production database pod. No alerts, no warning—just a cascading failure because you didn't isolate resources. This is the reality of multi-tenant clusters without Namespaces and ResourceQuotas. Namespaces are your first line of defense: they create logical boundaries for resources, RBAC, and network policies. But boundaries alone don't prevent resource starvation—that's where ResourceQuotas come in. They act as hard caps, ensuring no single team can consume more than their fair share. In this guide, we'll build a production-grade namespace strategy from scratch, covering quotas, limits, and real-world failure modes. By the end, you'll know how to design a cluster that survives even the most aggressive deployments.

Why Namespaces Alone Are Not Enough

Namespaces provide logical isolation, but they don't enforce resource limits. Without ResourceQuotas, a single namespace can consume all cluster resources, starving others. In production, this leads to unpredictable failures: pods evicted, latency spikes, and cascading outages. Consider a scenario where a CI/CD pipeline in the 'ci' namespace spawns hundreds of pods during a build storm. Without quotas, those pods can consume all available CPU, causing production pods in the 'prod' namespace to throttle or crash. ResourceQuotas are the mechanism to prevent this. They set hard limits on aggregate resource consumption per namespace, ensuring that even if one namespace goes rogue, others remain unaffected. This is not optional for multi-tenant clusters—it's a requirement for any production deployment with more than one team or application.

apiVersion: v1
kind: Namespace
metadata:
  name: ci

ResourceQuota Fundamentals: What You Can Limit

ResourceQuotas can limit compute resources (CPU, memory) and object counts (pods, services, secrets, etc.). Compute quotas are specified as 'requests' and 'limits'—you can cap the total requested resources or the total limit across all pods in a namespace. Object count quotas prevent namespace bloat by capping the number of resources like ConfigMaps or PersistentVolumeClaims. A common pitfall is forgetting to set quotas for ephemeral storage or extended resources (e.g., GPUs). In production, always include at least CPU and memory quotas, plus pod count to prevent infinite scaling loops. Quotas are enforced at admission time: if a new pod would exceed the quota, the API server rejects it with a 403 Forbidden. This is a hard stop, not a soft limit—no overcommit is allowed beyond the quota.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-quota
  namespace: team-a
spec:
  hard:
    requests.cpu: "4"
    requests.memory: 8Gi
    limits.cpu: "8"
    limits.memory: 16Gi
    pods: "10"

Setting Up Your First ResourceQuota

To create a ResourceQuota, define a YAML manifest with 'spec.hard' listing the resources you want to limit. Apply it to a namespace using 'kubectl apply -f quota.yaml -n <namespace>'. Once applied, any pod creation that would exceed the quota is rejected. It's critical to test quotas in a non-production namespace first. A common mistake is setting quotas too low, causing legitimate workloads to fail. Start with generous limits based on historical usage, then tighten over time. Use 'kubectl describe resourcequota' to see current usage and hard limits. Quotas are dynamic—you can update them without restarting anything. However, reducing a quota below current usage will not evict running pods; it only prevents new ones. Plan for this by monitoring usage before tightening.

kubectl create namespace team-a
kubectl apply -f - <<EOF
apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-quota
  namespace: team-a
spec:
  hard:
    requests.cpu: "4"
    requests.memory: 8Gi
    limits.cpu: "8"
    limits.memory: 16Gi
    pods: "10"
EOF
kubectl describe resourcequota compute-quota -n team-a

LimitRanges: The Companion to ResourceQuotas

ResourceQuotas set aggregate limits, but they don't enforce per-pod constraints. A single pod could request all 8Gi of memory, starving other pods. LimitRanges fill this gap by setting min/max/default resource requests and limits for individual containers or pods. They work alongside quotas: quotas cap the total, LimitRanges cap each pod. For example, you can set a default memory request of 256Mi and a max of 2Gi per container. This prevents any single pod from hogging resources. LimitRanges are also enforced at admission time. If a pod doesn't specify resource requests/limits, the LimitRange injects defaults. This is crucial for ensuring all pods have predictable resource profiles. Without LimitRanges, a pod with no requests could be scheduled anywhere, causing resource contention.

apiVersion: v1
kind: LimitRange
metadata:
  name: mem-limit-range
  namespace: team-a
spec:
  limits:
  - max:
      memory: 2Gi
    min:
      memory: 128Mi
    default:
      memory: 512Mi
    defaultRequest:
      memory: 256Mi
    type: Container

Priority Classes and ResourceQuota Interaction

Not all workloads are equal. Critical system pods (e.g., DNS, ingress controllers) should be protected from eviction during resource pressure. PriorityClasses allow you to assign priority to pods, and the scheduler uses this to preempt lower-priority pods when resources are scarce. However, ResourceQuotas do not respect priority by default—a quota applies to all pods regardless of priority. This means a high-priority pod can still be blocked if the namespace quota is exhausted. To handle this, you can create separate namespaces for critical workloads with dedicated quotas. Alternatively, use ResourceQuota's 'scopes' to apply different quotas to different priority classes. For example, you can set a quota for 'BestEffort' pods (those without resource requests) separately from 'Guaranteed' pods. This gives you fine-grained control over resource allocation.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: best-effort-quota
  namespace: team-a
spec:
  hard:
    pods: "5"
  scopes:
  - BestEffort
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: guaranteed-quota
  namespace: team-a
spec:
  hard:
    pods: "10"
    requests.cpu: "4"
    requests.memory: 8Gi
  scopes:
  - NotBestEffort

Monitoring Quota Usage and Alerts

Quotas are only effective if you monitor them. Use 'kubectl describe resourcequota' to see current usage, but for production, set up Prometheus metrics. Kubernetes exposes quota metrics like 'kube_resourcequota' with labels for namespace, resource, and type (hard/used). Alert on usage exceeding 80% of hard limit to give teams time to request increases. Common failure modes: a quota is reached silently, causing deployments to fail with cryptic 'Forbidden' errors. Teams often don't know why their pods aren't starting. Implement a dashboard showing quota usage per namespace. Also, set up alerts for quota modifications—unauthorized changes to quotas can cause chaos. In one incident, a developer accidentally deleted a quota, allowing unlimited resource consumption. Monitor quota events via audit logs.

groups:
- name: quota-alerts
  rules:
  - alert: ResourceQuotaNearLimit
    expr: |
      sum by (namespace, resource) (
        kube_resourcequota{type="used"}
      ) / sum by (namespace, resource) (
        kube_resourcequota{type="hard"}
      ) > 0.8
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "ResourceQuota {{ $labels.resource }} in {{ $labels.namespace }} is >80% used"

Handling Quota Exhaustion in CI/CD Pipelines

CI/CD pipelines are notorious for triggering quota exhaustion. A merge storm can spawn dozens of parallel builds, each creating pods that consume quota. If the pipeline namespace hits its quota, builds fail with 'Forbidden' errors. To mitigate, use separate namespaces per branch or per PR, each with its own quota. This isolates failures and prevents one branch from blocking others. Alternatively, use dynamic quotas via operators that adjust limits based on time of day or load. Another approach: set a low pod count quota per namespace and use a queue to serialize builds. This ensures predictable resource usage. In production, we saw a team's CI namespace hit quota every afternoon during peak development hours. The fix: increase pod quota and add a LimitRange to cap per-pod resources.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: ci-quota
  namespace: ci
spec:
  hard:
    pods: "20"
    requests.cpu: "8"
    requests.memory: 16Gi
    limits.cpu: "16"
    limits.memory: 32Gi

Advanced: Dynamic Quota Management with Operators

Static quotas work for stable workloads, but dynamic environments (e.g., dev/test clusters) benefit from adjustable quotas. Operators like the 'Quota Operator' or custom controllers can adjust quotas based on cluster load, time of day, or team activity. For example, you can increase quotas during business hours and reduce them at night. This maximizes resource utilization without manual intervention. Another use case: automatically create namespaces with default quotas for new projects. This enforces governance from day one. However, dynamic quotas introduce complexity—ensure your operator handles race conditions and doesn't exceed cluster capacity. In production, we built a simple operator that reads quota definitions from a ConfigMap and applies them. This allowed teams to self-serve quota increases via GitOps.

package main

import (
	"context"
	"fmt"
	corev1 "k8s.io/api/core/v1"
	"k8s.io/apimachinery/pkg/api/resource"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/client-go/kubernetes"
	"k8s.io/client-go/rest"
)

func main() {
	config, _ := rest.InClusterConfig()
	clientset, _ := kubernetes.NewForConfig(config)
	quota := &corev1.ResourceQuota{
		ObjectMeta: metav1.ObjectMeta{
			Name:      "dynamic-quota",
			Namespace: "team-a",
		},
		Spec: corev1.ResourceQuotaSpec{
			Hard: corev1.ResourceList{
				corev1.ResourcePods: resource.MustParse("20"),
				corev1.ResourceRequestsCPU: resource.MustParse("8"),
				corev1.ResourceRequestsMemory: resource.MustParse("16Gi"),
			},
		},
	}
	clientset.CoreV1().ResourceQuotas("team-a").Create(context.TODO(), quota, metav1.CreateOptions{})
	fmt.Println("Quota created dynamically")
}

Common Pitfalls and How to Avoid Them

Pitfall 1: Forgetting to set quotas on new namespaces. Use admission controllers (e.g., OPA/Gatekeeper) to enforce that every namespace has a quota. Pitfall 2: Setting quotas too low for stateful workloads. StatefulSets often require persistent volumes and stable network identities—ensure your quota includes enough storage and pod count. Pitfall 3: Not accounting for overhead. System pods (kube-proxy, CNI) run in namespaces like kube-system; don't forget to set quotas there too. Pitfall 4: Confusing 'requests' and 'limits' in quotas. Quotas on 'limits' cap the total limit, not the actual usage. If pods burst above their requests, they can still be throttled. Pitfall 5: Ignoring ephemeral storage quotas. Pods writing logs or temporary data can fill node disks. Always set 'requests.ephemeral-storage' and 'limits.ephemeral-storage'.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: full-quota
  namespace: team-a
spec:
  hard:
    pods: "10"
    requests.cpu: "4"
    requests.memory: 8Gi
    limits.cpu: "8"
    limits.memory: 16Gi
    requests.ephemeral-storage: "50Gi"
    limits.ephemeral-storage: "100Gi"
    persistentvolumeclaims: "5"
    configmaps: "20"
    secrets: "20"
    services: "10"

Testing Quota Enforcement in a Staging Environment

Before rolling out quotas to production, test them thoroughly in staging. Create a namespace with the same quotas you plan to use, then deploy workloads that push against the limits. Verify that pods exceeding quota are rejected with the correct error message. Also test edge cases: what happens when a quota is exactly full? Can you update a quota while pods are running? (Yes, but it doesn't affect running pods.) Use 'kubectl auth can-i' to verify that only authorized users can modify quotas. Automate quota testing in your CI pipeline: create a namespace, apply quota, attempt to create too many pods, and assert the failure. This ensures quota changes don't break deployments. In production, we run a daily quota validation job that checks all namespaces have quotas and alerts on missing ones.

#!/bin/bash
NAMESPACE=test-quota
kubectl create namespace $NAMESPACE
kubectl apply -f - <<EOF
apiVersion: v1
kind: ResourceQuota
metadata:
  name: test-quota
  namespace: $NAMESPACE
spec:
  hard:
    pods: "1"
EOF
# Try to create two pods; second should fail
kubectl run pod1 --image=nginx -n $NAMESPACE
kubectl run pod2 --image=nginx -n $NAMESPACE 2>&1 || echo "Expected failure"
kubectl delete namespace $NAMESPACE

Integrating with Cluster Autoscaler and Quotas

Cluster Autoscaler adds nodes when pods are unschedulable due to resource constraints. However, if a pod is unschedulable because of a ResourceQuota (not node resources), the autoscaler won't help—it only scales based on node-level constraints. This is a common misunderstanding: teams see pending pods and expect the cluster to scale, but the real issue is quota exhaustion. To diagnose, check 'kubectl describe pod' for events like 'FailedCreateResourceQuota'. If the cluster is underutilized but pods are pending, quotas are likely the culprit. In production, we set up alerts that distinguish between 'unschedulable due to node resources' and 'unschedulable due to quota'. This prevents unnecessary cluster scaling and helps teams request quota increases promptly.

kubectl get pods --all-namespaces --field-selector=status.phase=Pending -o json | jq '.items[] | {name: .metadata.name, namespace: .metadata.namespace, reason: .status.conditions[0].reason}'

Production Strategy: Multi-Tenant Quota Design

Designing quotas for multi-tenant clusters requires balancing isolation with utilization. Start by defining tiers: critical (production), standard (staging), and best-effort (dev/CI). Assign larger quotas to critical namespaces and enforce strict limits on best-effort ones. Use namespaces per team or per application, not per environment—this prevents a single team's dev namespace from consuming resources needed for another team's production. Implement a quota review process: teams submit requests via a GitOps PR, and an operator applies the change. This creates an audit trail. Also, set a cluster-wide 'reserved' quota for system components (kube-system, monitoring). Monitor quota utilization trends and adjust quarterly. In production, we use a 'namespace template' that includes default quotas, LimitRanges, and NetworkPolicies, applied automatically via a controller.

apiVersion: v1
kind: Namespace
metadata:
  name: ${TEAM}-${ENV}
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: default-quota
  namespace: ${TEAM}-${ENV}
spec:
  hard:
    pods: "20"
    requests.cpu: "8"
    requests.memory: 16Gi
---
apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
  namespace: ${TEAM}-${ENV}
spec:
  limits:
  - default:
      cpu: 500m
      memory: 512Mi
    defaultRequest:
      cpu: 100m
      memory: 256Mi
    type: Container