Prometheus OOMKill: High-Cardinality Labels in Kubernetes

TSDB stores data in blocks. Each block covers a 2-hour time range and contains a chunks directory (compressed metric samples) and an index. The head block is the in-memory write-ahead log (WAL) that receives all new samples. Every 2 hours, the head block is compacted into a persistent block and flushed to disk. Old blocks are compacted into larger blocks (e.g., 2h blocks into 1-day blocks) to reduce the number of files.", "code": { "language": "yaml", "filename": "prometheus-retention-config.yaml", "code": "# Prometheus StatefulSet with retention and storage configuration. # For the kube-prometheus-stack Helm chart, these go in prometheus.prometheusSpec. apiVersion: monitoring.coreos.com/v1 kind: Prometheus metadata: name: kube-prometheus namespace: monitoring spec: # Retention: how long to keep data locally. # 15d is typical. Longer retention = more disk and memory. retention: 15d

# Retention by size: delete oldest blocks when storage exceeds this. # Use this as a safety net alongside time-based retention. retentionSize: 50GB

# Storage: PVC for persistent TSDB blocks. storage: volumeClaimTemplate: spec: storageClassName: fast-ssd accessModes: - ReadWriteOnce resources: requests: storage: 100Gi

# Resources: Prometheus is memory-hungry. # Rule of thumb: ~16KB per active time series. # 1M series = 16GB RAM. Plan accordingly. resources: requests: cpu: '1' memory: 16Gi limits: cpu: '4' memory: 32Gi

# External labels: applied to all metrics when using Thanos/Cortex. # Identifies which Prometheus instance scraped the data. externalLabels: cluster: production-us-east-1 environment: production

# Thanos sidecar: uploads blocks to object storage for long-term retention. thanos: objectStorageConfig: name: thanos-objstore-config key: objstore.yml

# Sample limit: max series per scrape target. # Safety net against high-cardinality targets. serviceMonitorSelectorNilUsesHelmValues: false podMonitorSelectorNilUsesHelmValues: false", "output": "Prometheus configured with 15-day retention, 50GB size limit, fast SSD storage, and Thanos sidecar for long-term block upload to object storage." }, "callout": { "type": "mental_model", "title": "When to Add Thanos or Cortex", "text": "Prometheus is designed for short-term, per-cluster monitoring. It cannot federate queries across clusters, cannot store data longer than a few weeks efficiently, and is a single point of failure. Thanos and Cortex solve these problems.", "hook": "Add Thanos when you need: cross-cluster query federation, retention beyond 30 days, or HA for Prometheus.", "bullets": [ "Prometheus: Single cluster, short-term (days to weeks). No native HA or federation.", "Thanos: Sidecar uploads blocks to S3/GCS. Querier federates across Prometheus instances. Compactor reduces storage costs. Best for multi-cluster with object storage.", "Cortex: Hor Cloud. Best for multi-tenant SaaS platforms.", "VictoriaMetrics: Drop-in Prometheus replacement with better compression and lower resource usage. Best for single-cluster with high cardinality.", "Decision: Use Thanos for multi-cluster with object storage. Use Cortex for multi-tenant SaaS. Use VictoriaMetrics for single-cluster resource optimization." ] }, "production_insight": "Prometheus's memory usage is directly proportional to the number of active time series in the TSDB head block. Each active series consumes approximately 16KB of memory. If you have 2 million active series, you need approximately 32GB of RAM for the head block alone, plus overhead for queries and compaction. Monitor prometheus_tsdb_head_series and process_resident_memory_bytes. Set retention based on disk capacity: 15 days at 1M series with 15s scrape interval equals approximately 50GB of disk. Use fast SSDs for TSDB — network-attached storage introduces latency that slows compaction and can cause WAL corruption during power loss.", "decision_tree": { "title": "Prometheus Storage Decision Tree", "items": [ { "condition": "Single cluster, retention under 15 days, under 5M active series", "result": "Standalone Prometheus with local TSDB and PVC on fast SSD. No additional components needed." }, { "condition": "Multiple clusters, need cross-cluster query federation", "result": "Deploy Thanos Sidecar on each Prometheus. Add Thanos Querier for global view. Add Thanos Store Gateway for historical queries." }, { "condition": "Retention beyond 30 days or more than 5M active series", "result": "Add Thanos Sidecar with object storage (S3/GCS). Add Thanos Compactor to reduce storage costs. Keep local retention short (7d) and rely on object storage for long-term." }, { "condition": "Multi-tenant SaaS platform with per-customer isolation", "result": "Use Cortex or Grafana Mimir for horizontal scalability and per-tenant resource limits." }, { "condition": "Single cluster with extreme cardinality (10M+ series)", "result": "Consider VictoriaMetrics as a drop-in replacement. It offers better compression (1izontally scalable multi-tenant Prometheus backend. Used by Grafana0x) and lower memory usage than Prometheus TSDB." } ] }, "key_takeaway": "Prometheus TSDB is a block-based storage engine with an in-memory head block. Memory is proportional to active series count. Use fast SSDs for TSDB. Add Thanos for cross-cluster federation and long-term retention beyond local disk capacity. Plan capacity based on series count: 1M series equals approximately 16GB RAM and approximately 50GB disk for 15 days." } ]

What Prometheus Actually Is (and Isn’t) for Kubernetes

Prometheus is not a full platform. It’s a pull-based metrics system with a time-series database and a query language. That’s it. No dashboards. No built-in long-term storage. No synthetic monitoring. What it does well: scrape HTTP endpoints, store labeled metrics, fire alerts on expressions.

Inside Kubernetes, this matters because pods die, reschedule, and scale. Prometheus discovers targets via the Kubernetes API — no static config files. Each service, pod, or endpoint gets labels. Those labels are your only join key between metrics. Mess them up and your alerts break silently.

Prometheus uses a single binary. No external dependencies. The operator wraps it in CRDs like ServiceMonitor so you don’t write raw scrape configs. But understand this: the operator is just a config generator. If you don’t know what a scrape interval or relabel config does, you’re cargo-culting YAML, not monitoring.

Senior take: Prometheus assumes you understand your architecture. If your metrics don’t map to your deployment topology, you’re blind.

// io.thecodeforge — devops tutorial

# What Prometheus says about itself in its own config
# This is the closest to a truth-of-record you'll get

global:
  scrape_interval: 15s      # How often to scrape targets
  evaluation_interval: 15s   # How often to evaluate rules

scrape_configs:
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod            # Discover targets from pod metadata
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        target_label: app

Why Prometheus Owns Kubernetes Monitoring (and Where It Bleeds)

Prometheus won the container monitoring war for one reason: service discovery. In a static VM world, you list IPs in a config file. In Kubernetes, pods come and go every deploy. Prometheus talks to the Kubernetes API, discovers new pods in seconds, and starts scraping. No manual sync. No config reload.

But that strength is also its weakness. Service discovery means Prometheus holds a live connection to the API server. If your cluster has 10,000 pods and each pod exposes 1000 metrics, Prometheus will OOM before lunch. Labels multiply cardinality exponentially. A label like request_id with 10 million unique values will crash the TSDB.

The big bleed: long-term retention. Prometheus isn’t built for storing months of data. The default retention is 15 days. You need Thanos or Cortex for that. And if you’re scraping 500+ targets, a single Prometheus will hit performance walls. Shard by namespace or use federation.

Real talk: Prometheus works great for 99% of clusters. The 1% are hyperscale edge cases. Don’t over-engineer until you hit those limits.

// io.thecodeforge — devops tutorial

# This metric will kill Prometheus in production
# Avoid high-cardinality labels at ALL COSTS

# BAD - do NOT deploy this
http_requests_total{endpoint="/api/v1/users", method="GET", status="200", request_id="abc-123-def"} 1
http_requests_total{endpoint="/api/v1/users", method="GET", status="200", request_id="abc-124-def"} 1
# ... 10 million unique request_ids

Prometheus Architecture: The Minimal Mental Model You Need

You don’t need to memorize every component in the architecture diagram. You need the three things Prometheus actually does: scrape, store, alert.

First, the scraper. Prometheus runs a HTTP client that hits /metrics endpoints on your targets. It uses service discovery to know where to hit. Each scrape produces a sample — a timestamp, a value, and a set of labels. The scraper doesn’t push; the target must expose an endpoint.

Second, the TSDB (time-series database). Samples get stored on local disk. Default retention is 15 days, 512MB block size. The TSDB is append-only with periodic compaction. It’s fast but not distributed. If the pod dies, you lose data. That’s why you add Thanos or Cortex for HA.

Third, the alerting engine. Prometheus evaluates recording and alerting rules every evaluation_interval. If a rule matches, it fires an alert to Alertmanager. Alertmanager handles deduplication, grouping, and routing to Slack, PagerDuty, email.

That’s it. Three components. Everything else — exporters, operators, dashboards — is auxiliary. Master this model and you can debug any Prometheus issue in production.

// io.thecodeforge — devops tutorial

# Absolute minimal Prometheus config to monitor itself
# This is the skeleton. Everything else is customization.

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - "alert.rules"

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

Key Terminologies: Stop Pretending You Know What These Mean

Before you touch Prometheus in Kubernetes, you need the vocabulary straight. Not the marketing fluff — the actual implementation details that break your cluster at 2am.

A Target is anything Prometheus scrapes: a pod, a node, a service endpoint. Metadata from the target becomes labels. A Metric is a time series — a name plus a set of labels with a timestamped float value. That's it. No hidden complexity.

Labels are key-value pairs that identify the metric's dimension. They're not tags — they ARE the identity. Change a label and you create a new time series. Cardinality is the number of unique label combinations. High cardinality kills Prometheus. Period.

An Alert fires when a PromQL expression is true. Silences are global mute buttons for specific labels. Recording rules precompute expensive queries so your dashboards don't fall over. ServiceMonitor is a CRD that tells the Operator which services to scrape. ScrapeConfig is the older, more manual way.

Know the difference between counter (only increases), gauge (goes up and down), and histogram (buckets duration). Mix them up and your queries lie.

// io.thecodeforge — devops tutorial

// A minimal scrape config showing key terms in the wild
scrape_configs:
  - job_name: 'node-exporter'
    kubernetes_sd_configs:
      - role: node
    relabel_configs:
      - source_labels: [__meta_kubernetes_node_name]
        target_label: node
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: 'node_load1'
        action: keep
    # node_load1 is a gauge, not a counter
    # Each node becomes a different time series (target)

Key Features: Why Prometheus Wins for Kubernetes Monitoring

Prometheus dominates Kubernetes monitoring not because it's easy, but because its architecture matches how Kubernetes works. Its key features: pull-based scraping means no agent configuration inside containers—Prometheus discovers targets via Kubernetes APIs, adapting to pod churn automatically. The multi-dimensional data model pairs metric names with arbitrary key-value labels, enabling slicing by deployment, namespace, or pod. PromQL's aggregation operators (sum, rate, histogram_quantile) allow real-time SLO calculations without stored aggregations. Alertmanager handles deduplication, silencing, and grouping before paging you. Service discovery happens natively through pod annotations or ServiceMonitors, eliminating static configs. The operator pattern lets you define scraping rules as CRDs, versioned and GitOps-friendly. Prometheus is stateless by design—scrape failures don't cascade, and data is ephemeral unless remote-stored. Where it bleeds is long-term storage and cardinality explosion; choose Thanos or Cortex for retention beyond a month. Key decision: every new metric label adds memory pressure—team-wide label conventions enforce discipline.

// io.thecodeforge — devops tutorial

// Key features: pull model, label-based data, PromQL
// This sample shows a ServiceMonitor using labels
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: app-monitor
  labels:
    release: prometheus
spec:
  selector:
    matchLabels:
      app: myapp
  endpoints:
  - port: http
    interval: 15s
  namespaceSelector:
    any: true
// Labels enable queries like: rate(http_requests_total{app="myapp"}[5m])

Demo: Monitor Any Third-Party App With Prometheus in Kubernetes

Third-party apps rarely expose Prometheus metrics natively. The fix: deploy a sidecar exporter or use the kube-state-metrics pattern. Steps: First, identify what metrics you need—CPU? request latency? HTTP status codes? Second, find or build a metrics exporter. For Redis, use the redis-exporter. For Nginx, the nginx-prometheus-exporter. For custom apps, write a lightweight exporter in Go or Python exposing :2112/metrics on the /metrics endpoint. Third, deploy the exporter as a sidecar container in the same pod as your third-party app—this guarantees they share the network namespace. Fourth, annotate the pod with prometheus.io/scrape: "true" and prometheus.io/port: "2112". Fifth, verify Prometheus discovers the target via its Targets page. Sixth, write recording rules for common aggregations (e.g., avg 5m CPU) to avoid expensive live PromQL. Seventh, set up Alertmanager for critical thresholds. This pattern works for legacy databases, message queues, or any black-box service. Don't modify the third-party app—wrap it.

// io.thecodeforge — devops tutorial

// Sidecar pattern: exporter + third-party app in one pod
apiVersion: v1
kind: Pod
metadata:
  name: redis-app
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "9121"
spec:
  containers:
  - name: redis
    image: redis:7
  - name: exporter
    image: oliver006/redis_exporter:latest
    args: ["--redis.addr=redis://localhost:6379"]
    ports:
    - containerPort: 9121

Prerequisites Before Installing Prometheus on Kubernetes

Before deploying the kube-prometheus-stack, you need a running Kubernetes cluster (v1.19+) with sufficient resources—at least 4 CPU cores and 8GB RAM across nodes for a basic setup. Install kubectl and Helm v3, and ensure your cluster has PersistentVolume support for Prometheus and Alertmanager data durability. You must also have cluster-admin RBAC permissions. For Loki integration, prepare an object storage backend like S3 or MinIO, as Loki stores logs externally. Understanding basic PromQL and Kubernetes namespaces is assumed. Without these, the stack will fail silently, leaving you blind to metrics gaps. This setup is not plug-and-play; it demands a solid infrastructure foundation to avoid data loss or performance degradation in production.

// io.thecodeforge — devops tutorial
# Verify cluster readiness before proceeding
apiVersion: v1
kind: Pod
metadata:
  name: readiness-check
spec:
  containers:
  - name: checker
    image: bitnami/kubectl:latest
    command: ["sh", "-c", "kubectl version --short && helm version"]
  restartPolicy: Never
---
# Expected output:
# Client Version: v1.28.0
# version.BuildInfo{Version:"v3.14.0"}

Installing kube-prometheus-stack and Loki for Unified Monitoring

Install the kube-prometheus-stack via Helm to get Prometheus, Alertmanager, and Grafana in one chart. Add the Prometheus community repo and run: helm install monitoring prometheus-community/kube-prometheus-stack --namespace monitoring --create-namespace. This deploys default scraping for Kubernetes components and node metrics. For log aggregation, install Loki using the Grafana Helm chart: helm install loki grafana/loki-stack --namespace monitoring --set grafana.enabled=false,prometheus.enabled=false. Loki pairs with Promtail (deployed automatically) to push container logs to the Loki backend. You can then query logs in Grafana alongside metrics. This combination gives you a single pane of glass for metrics and logs, reducing context switching during incidents. Adjust storage retention via values.yaml to avoid unbounded disk usage.

// io.thecodeforge — devops tutorial
# Install kube-prometheus-stack
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm upgrade --install monitoring prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace \
  --set prometheus.prometheusSpec.retention=7d

# Install Loki stack (logs)
helm repo add grafana https://grafana.github.io/helm-charts
helm upgrade --install loki grafana/loki-stack \
  --namespace monitoring \
  --set grafana.enabled=false,prometheus.enabled=false

Advantages of Prometheus and Loki for Kubernetes Monitoring

Prometheus offers a pull-based model that perfectly fits Kubernetes' dynamic pod lifecycle—auto-discovery via service monitors means no manual target configuration. Its powerful PromQL allows slicing metrics by labels, enabling precise alerting and ad-hoc queries. The kube-prometheus-stack bundles dashboards, recording rules, and alerting out of the box, slashing initial setup time. Loki complements this with log aggregation that reuses Kubernetes labels, letting you jump from a high-metric spike to relevant log lines instantly. Both are cloud-native, scale horizontally (via Thanos for Prometheus, and Loki's microservices mode), and integrate with Grafana for unified visualization. This stack reduces MTTR by correlating metrics and logs without leaving your dashboard. Plus, the ecosystem is vast—community exporters cover databases, caches, and HTTP services.

// io.thecodeforge — devops tutorial
# Example: auto-discovery via ServiceMonitor
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: example-app
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: my-app
  endpoints:
  - port: http
    interval: 15s
---
# Prometheus auto-targets pods matching label app: my-app

Limitations of Prometheus and Loki in Kubernetes Environments

Prometheus is not designed for long-term storage; its local TSDB retains data for weeks at most, requiring Thanos or Cortex for months of history. It struggles with high-cardinality metrics—labels like user IDs or request paths can balloon memory usage past available limits. The pull model fails if targets are ephemeral (e.g., serverless) or behind network segmentation. Loki, while efficient for logs, has slow query performance on large volumes without proper index configuration; log parsing is limited to simple label extraction. Both tools lack built-in multi-tenancy—separate instances or proxies are needed for team isolation. Finally, the operational overhead is non-trivial: upgrading Helm charts, tuning scrape intervals, and managing retention policies demand dedicated DevOps effort. For very small clusters, a lighter alternative like VictoriaMetrics might suffice.

// io.thecodeforge — devops tutorial
# Mitigate cardinality: limit labels in metrics
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  namespace: monitoring
data:
  prometheus.yml: |
    global:
      scrape_configs:
      - job_name: 'safe-app'
        metrics_path: /metrics
        relabel_configs:
        - action: labeldrop
          regex: 'user_id|request_id'  # Drop high-cardinality labels