Kubernetes Cost Optimization — Kubecost, Karpenter, Spot

Welcome to Kubernetes Cost Optimization — Kubecost, Karpenter, Spot. We'll break this down from first principles.

Why Kubernetes Costs Spiral Out of Control

Kubernetes abstracts infrastructure, making it easy to deploy more resources than needed. The default behavior of most operators and Helm charts is to request far more CPU and memory than necessary. Without cost visibility, teams accumulate orphaned resources, over-provisioned nodes, and idle workloads. The result: cloud bills that grow linearly with cluster size, not actual utilization. In production, we've seen teams double their AWS bill by simply enabling cluster autoscaler without limits. The root cause is a lack of feedback loops — developers don't see the cost impact of their resource requests.

kubectl top pods -A --sort-by=cpu | head -20
# Compare to resource requests:
kubectl get pods -A -o json | jq '.items[] | {name: .metadata.name, requests: .spec.containers[].resources.requests}'

Kubecost: Real-Time Cost Visibility and Allocation

Kubecost is an open-source tool that maps Kubernetes resource usage to cloud costs. It breaks down spend by namespace, deployment, label, or pod. Install it via Helm, and within minutes you get a dashboard showing cost per team, per service, and per cluster. The key metric is 'efficiency' — the ratio of actual usage to requested resources. Kubecost also provides recommendations for right-sizing requests and limits. In production, we use Kubecost to generate monthly chargebacks to engineering teams. Without it, cost allocation is guesswork. Kubecost integrates with AWS, GCP, and Azure to pull real-time pricing.

helm repo add kubecost https://kubecost.github.io/cost-analyzer/
helm upgrade -i kubecost kubecost/cost-analyzer \
  --namespace kubecost --create-namespace \
  --set kubecostToken="your-token" \
  --set prometheus.nodeExporter.enabled=false \
  --set global.prometheus.enabled=true
# Access via port-forward:
kubectl port-forward --namespace kubecost service/kubecost-cost-analyzer 9090:9090

Right-Sizing Workloads with Kubecost Recommendations

Kubecost's right-sizing recommendations analyze historical usage (default 7 days) and suggest optimal CPU/memory requests. Apply them via the UI or programmatically using the API. The recommendation engine uses the 95th percentile of usage to avoid under-provisioning. In production, we batch-apply recommendations during off-peak hours. A common pitfall: blindly applying recommendations without considering traffic spikes. Always set a buffer (e.g., 20% above the 95th percentile). Kubecost also supports 'request sizing' for DaemonSets and sidecars.

# Get recommendations via API
curl -s "http://localhost:9090/model/savings/requestSizing" | jq '.recommendations'
# Apply using kubecost-recommender (custom script)
# Example output:
{
  "namespace": "production",
  "container": "app",
  "currentRequest": {"cpu": "500m", "memory": "512Mi"},
  "recommendedRequest": {"cpu": "250m", "memory": "256Mi"}
}

Karpenter: Dynamic Node Provisioning for Cost Efficiency

Karpenter is an open-source node autoscaler for Kubernetes that launches just-in-time nodes based on pod resource requirements. Unlike the traditional Cluster Autoscaler, Karpenter works at the pod level, not the node group level. It can launch different instance types (including spot) and terminates nodes when they're no longer needed. Karpenter reduces waste by bin-packing pods tightly and avoiding over-provisioning. In production, Karpenter can cut node costs by 30-50% compared to static node groups. It supports AWS, and soon GCP and Azure.

apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: default
spec:
  template:
    spec:
      requirements:
        - key: "karpenter.sh/capacity-type"
          operator: In
          values: ["spot", "on-demand"]
        - key: "node.kubernetes.io/instance-type"
          operator: In
          values: ["m5.large", "m5.xlarge", "c5.large"]
      nodeClassRef:
        name: default
  limits:
    cpu: 1000
  disruption:
    consolidationPolicy: WhenUnderutilized
    expireAfter: 720h

Spot Instances: The Biggest Cost Saver with Karpenter

Spot instances are spare cloud compute capacity offered at up to 90% discount. Karpenter can automatically use spot instances for interruptible workloads. The risk: spot instances can be reclaimed with 2-minute notice. Mitigate by using pod disruption budgets, graceful shutdown hooks, and multi-az deployments. Karpenter's spot-to-spot consolidation ensures you always use the cheapest spot instance type available. In production, we run 80% of our batch jobs and stateless services on spot. The key is to never run stateful workloads (databases, Kafka) on spot without proper replication.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: stateless-app
spec:
  replicas: 3
  template:
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: "topology.kubernetes.io/zone"
          whenUnsatisfiable: DoNotSchedule
      containers:
        - name: app
          image: nginx
          resources:
            requests:
              cpu: 500m
              memory: 512Mi
      # Graceful shutdown for spot interruptions
      terminationGracePeriodSeconds: 30
      # PreStop hook to drain connections
      lifecycle:
        preStop:
          exec:
            command: ["/bin/sh", "-c", "sleep 20"]

Combining Karpenter and Spot for Maximum Savings

The real power is using Karpenter with spot instances as the default capacity type. Configure Karpenter to prefer spot but fall back to on-demand if spot is unavailable. This ensures availability while maximizing savings. Use node pools with different instance families to increase spot capacity pool diversity. In production, we set spot allocation strategy to 'lowest-price' but with a cap to avoid using extremely cheap but unreliable instances. Monitor spot interruption rates via Karpenter metrics. The combination of Karpenter + spot typically yields 50-70% savings over on-demand only.

apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: spot-preferred
spec:
  template:
    spec:
      requirements:
        - key: "karpenter.sh/capacity-type"
          operator: In
          values: ["spot"]
        - key: "node.kubernetes.io/instance-type"
          operator: In
          values: ["m5.large", "m5.xlarge", "c5.large", "c5.xlarge"]
      nodeClassRef:
        name: default
  disruption:
    consolidationPolicy: WhenUnderutilized
    expireAfter: 720h
  limits:
    cpu: 500
---
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: on-demand-fallback
spec:
  template:
    spec:
      requirements:
        - key: "karpenter.sh/capacity-type"
          operator: In
          values: ["on-demand"]
      nodeClassRef:
        name: default
  disruption:
    consolidationPolicy: WhenUnderutilized
  limits:
    cpu: 200

Pod Disruption Budgets and Graceful Shutdowns

Pod Disruption Budgets (PDBs) ensure a minimum number of pods remain available during voluntary disruptions (node drains, spot interruptions). For spot instances, PDBs are critical. Set minAvailable or maxUnavailable based on your application's redundancy. Combine with preStop hooks to gracefully drain connections. In production, we use PDBs with maxUnavailable: 1 for deployments with 3+ replicas. For stateful workloads, use pod anti-affinity to spread across nodes. Without PDBs, a spot interruption can take down all replicas.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: app-pdb
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: stateless-app
---
# PreStop hook in deployment
lifecycle:
  preStop:
    exec:
      command: ["/bin/sh", "-c", "sleep 30"]

Monitoring and Alerting for Cost Anomalies

Cost optimization is not a one-time effort. Set up monitoring and alerting for cost anomalies. Use Kubecost alerts for daily budget thresholds, spot interruption rates, and node utilization. Integrate with PagerDuty or Slack. In production, we have alerts for: namespace cost > $500/day, spot interruption rate > 5%, and node utilization < 20%. Also monitor Karpenter metrics like nodes created, terminated, and spot interruption count. Use Prometheus and Grafana for dashboards. Without monitoring, cost savings degrade over time as teams add new workloads.

apiVersion: v1
kind: ConfigMap
metadata:
  name: kubecost-alerts
  namespace: kubecost
data:
  alerts.json: |
    [
      {
        "type": "budget",
        "threshold": 500,
        "window": "daily",
        "aggregation": "namespace",
        "targets": ["slack"]
      }
    ]

Advanced: Node Consolidation and Bin Packing

Karpenter's consolidation feature automatically terminates underutilized nodes and reschedules pods onto cheaper or more efficient instances. This is called 'bin packing'. Enable consolidationPolicy: WhenUnderutilized to trigger consolidation when a node's utilization drops below a threshold. Karpenter also supports 'do not consolidate' for nodes with critical workloads. In production, we use consolidation aggressively for spot nodes but conservatively for on-demand. Monitor consolidation events to ensure they don't cause cascading restarts. Bin packing can reduce node count by 20-30%.

apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: aggressive-consolidation
spec:
  disruption:
    consolidationPolicy: WhenUnderutilized
    consolidateAfter: 5m
  limits:
    cpu: 1000

Cost Optimization for Stateful Workloads

Stateful workloads (databases, message queues) are harder to optimize because they require persistent storage and stable nodes. Use local SSDs with node affinity, or EBS volumes with volume binding mode WaitForFirstConsumer. For cost savings, consider using spot instances for read replicas or non-leader nodes. Use Karpenter's node templates to attach specific volumes. In production, we run Cassandra on spot for non-leader nodes, saving 60% on compute. The leader node runs on on-demand. Always test with chaos engineering before production rollout.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: cassandra
spec:
  serviceName: cassandra
  replicas: 3
  selector:
    matchLabels:
      app: cassandra
  template:
    metadata:
      labels:
        app: cassandra
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: "karpenter.sh/capacity-type"
                    operator: In
                    values: ["spot"]
      containers:
        - name: cassandra
          image: cassandra:4.0
          volumeMounts:
            - name: data
              mountPath: /var/lib/cassandra
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: gp3
        resources:
          requests:
            storage: 100Gi

Governance: Enforcing Cost Policies with OPA/Gatekeeper

To prevent cost waste from the start, enforce policies using Open Policy Agent (OPA) or Gatekeeper. For example, require all pods to have resource limits, enforce minimum node count, or block expensive instance types. In production, we have policies that: require CPU and memory limits, block instance types larger than m5.xlarge, and enforce spot-only for non-production namespaces. Gatekeeper can also validate Karpenter NodePool configurations. Without governance, cost optimization is reactive.

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredResources
metadata:
  name: require-resource-limits
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
  parameters:
    limits:
      - cpu: "500m"
        memory: "512Mi"
    requests:
      - cpu: "100m"
        memory: "128Mi"

Putting It All Together: A Production Cost Optimization Pipeline

The ultimate setup: Kubecost for visibility and recommendations, Karpenter with spot-first node pools, PDBs for resilience, Gatekeeper for governance, and Prometheus/Grafana for monitoring. Automate the feedback loop: Kubecost recommendations trigger a CI job that updates resource requests, which then get deployed. Use Karpenter's metrics to adjust spot diversification. In production, this pipeline reduced our monthly bill by 60% over 6 months. The key is to iterate: start with visibility, then right-size, then spot, then consolidate. Don't try everything at once.

# Example GitHub Actions workflow
name: Cost Optimization
on:
  schedule:
    - cron: '0 0 * * 0' # weekly
jobs:
  optimize:
    runs-on: ubuntu-latest
    steps:
      - name: Fetch Kubecost recommendations
        run: |
          curl -s http://kubecost:9090/model/savings/requestSizing | jq '.recommendations' > recs.json
      - name: Apply recommendations
        run: |
          # Custom script to patch deployments
          ./apply-recs.sh recs.json