Prometheus OOMKill: High-Cardinality Labels in Kubernetes

Running Kubernetes in production without monitoring is like flying a commercial aircraft with the instrument panel blacked out. Everything might feel fine until it catastrophically isn't. Prometheus is used by over 84% of Kubernetes production environments — not because it's the easiest tool, but because it's the most powerful pull-based metrics system that was purpose-built for dynamic, containerized infrastructure.

The real problem Prometheus solves is the ephemeral nature of Kubernetes workloads. Traditional monitoring tools expect your target IPs to stay fixed. In Kubernetes, a pod's IP changes every restart. Prometheus solves this with Kubernetes-native service discovery — it queries the Kubernetes API server directly to find what's alive right now, not what was alive when you wrote the config.

This is not a getting-started guide. It covers scrape configurations with relabeling, custom application metrics using client libraries, recording rules to avoid query-time explosions, Alertmanager integration, and the five most expensive mistakes teams make in production.

Prometheus Stack (Operator) Component Visual

The Prometheus Operator for Kubernetes introduces a set of Custom Resource Definitions (CRDs) that declaratively define the Prometheus monitoring stack. Understanding how these components fit together is essential before diving into scrape configuration.

Additionally, the ScrapeConfig CRD (introduced in Prometheus Operator v0.65+) provides a lower-level way to define scrape targets with full Prometheus scrape_config semantics, bypassing the semantic filters of ServiceMonitor/PodMonitor.

A typical production deployment creates a Prometheus CR, an Alertmanager CR, and one or more ServiceMonitor or ScrapeConfig CRs, plus PrometheusRule CRs for alerts. The operator watches these CRDs and reconciles the state of Prometheus and Alertmanager instances.

The diagram below shows the relationships: the Operator watches CRDs, creates Prometheus/Alertmanager pods, and translates ServiceMonitors into scrape configs.

# Minimal Prometheus CR — operator creates the deployment
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: kube-prometheus
  namespace: monitoring
spec:
  serviceAccountName: prometheus
  serviceMonitorSelector:
    matchLabels:
      app: myapp
  resources:
    requests:
      memory: 16Gi
---
# ServiceMonitor — operator translates to scrape targets
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: myapp-monitor
  labels:
    app: myapp
spec:
  selector:
    matchLabels:
      app: myapp
  endpoints:
  - port: metrics
    interval: 30s

How Prometheus Service Discovery Works Inside Kubernetes

Prometheus uses a pull model — it reaches out to targets and scrapes metrics endpoints, typically on path /metrics, at a configured interval. In a static world you'd list IPs. In Kubernetes, Prometheus uses kubernetes_sd_configs to query the Kubernetes API and discover pods, services, endpoints, nodes, and ingresses dynamically.

When Prometheus starts, it authenticates to the API server using a ServiceAccount token mounted in its pod. It then watches specific resource types. For the endpoints role, Prometheus discovers every Endpoints object across the cluster. For each endpoint address it finds, it creates a scrape target. The magic happens during relabeling — a pipeline that runs before the scrape and lets you filter, rename, and attach labels using values pulled directly from Kubernetes metadata (pod annotations, namespace labels, service names).

The annotation `prometheus.io/scrape: 'true' is a community convention that Prometheus relabeling configs check. If the annotation exists and is true, the pod is scraped. This means enabling monitoring for a new application is as simple as adding three lines to its pod spec — no Prometheus config reload needed. Prometheus reconciles new targets automatically every scrape_interval`.

Understanding the target lifecycle is critical for production. Targets move through states: up, down, and unknown. A target goes unknown when Prometheus can't reach the endpoint at all (network issue or pod not started). It goes down when the HTTP scrape returns a non-200 status or times out. Staleness markers are injected after a target disappears — this prevents old time series from polluting range queries.

# This is the core Prometheus scrape configuration for Kubernetes pod discovery.
# It lives inside your prometheus.yml (or a PrometheusRule CR if using the operator).

scrape_configs:
  - job_name: 'kubernetes-pods'
    # Prometheus will query the Kubernetes API server to find all pods.
    kubernetes_sd_configs:
      - role: pod
        # Restrict discovery to a specific namespace for security isolation.
        namespaces:
          names:
            - production
            - staging

    # relabel_configs runs BEFORE each scrape — it filters and transforms targets.
    relabel_configs:
      # STEP 1: Only scrape pods that explicitly opt in via annotation.
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: 'true'

      # STEP 2: Allow pods to declare a custom metrics path.
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)

      # STEP 3: Allow pods to declare a custom port for scraping.
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: '([^:]+)(?::\d+)?;(\d+)'
        replacement: '$1:$2'
        target_label: __address__

      # STEP 4: Carry namespace as a label.
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: kubernetes_namespace

      # STEP 5: Carry the pod name as a label.
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: kubernetes_pod_name

      # STEP 6: Carry the app label from the pod.
      - source_labels: [__meta_kubernetes_pod_label_app]
        action: replace
        target_label: app

      # STEP 7: Drop targets running on hostNetwork.
      - source_labels: [__meta_kubernetes_pod_host_ip, __address__]
        regex: '(\d+\.\d+\.\d+\.\d+);\1:\d+'
        action: drop

ServiceMonitor vs ScrapeConfig Decision Guide

When using the Prometheus Operator, you have two primary ways to define scrape targets: ServiceMonitor (and its cousin PodMonitor) and the newer ScrapeConfig CRD (available since Prometheus Operator v0.65). Understanding which to use is critical for production architecture.

ServiceMonitor is the original CRD. It is designed to scrape a Kubernetes Service. You specify a service selector (by labels), and the Operator automatically discovers all endpoints behind that service. It adds semantic constraints: the service must expose one or more ports, and you can optionally specify a path, interval, and metric relabeling. ServiceMonitor is high-level: the Operator handles converting service endpoints to individual pod IP addresses.

ScrapeConfig is a lower-level CRD. It directly mirrors the Prometheus scrape_config block. You define kubernetes_sd_configs, relabel_configs, metric_relabel_configs, etc., exactly as you would in a raw Prometheus YAML file. There are no implicit service-based semantics. This gives you full control but requires more expertise.

The decision tree below helps pick the right one.

A common production pattern: use ServiceMonitor for all user-facing services (HTTP APIs, gRPC), and ScrapeConfig for infrastructure components (node-exporter, kube-state-metrics, or custom exporters that expose metrics on non-standard ports).

# ServiceMonitor — simple, operator-managed
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: myapp-service-monitor
  labels:
    app: myapp
spec:
  selector:
    matchLabels:
      app: myapp
  endpoints:
  - port: metrics
    interval: 30s
    path: /metrics
    metricRelabelings:
    - sourceLabels: [__name__]
      regex: '.*_total'
      action: keep
---
# ScrapeConfig — full control, raw Prometheus config
apiVersion: monitoring.coreos.com/v1
kind: ScrapeConfig
metadata:
  name: myapp-scrape-config
  labels:
    app: myapp
spec:
  scrapeConfig:
    job_name: 'myapp-scrape'
    kubernetes_sd_configs:
    - role: pod
      namespaces:
        names: [production]
    relabel_configs:
    - sourceLabels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
      action: keep
      regex: 'true'
    - sourceLabels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
      action: replace
      regex: '([^:]+)(?::\d+)?;(\d+)'
      replacement: '$1:$2'
      targetLabel: __address__
    metric_relabel_configs:
    - sourceLabels: [__name__]
      regex: '.*_total'
      action: keep

Exposing Custom Application Metrics with the Prometheus Client Libraries

Kubernetes infrastructure metrics (CPU, memory, network) come from kube-state-metrics and node-exporter. But the metrics that make or break your SLOs are application-level: request latency, error rates, queue depth, cache hit ratio. These come from instrumenting your own code.

Prometheus has four core metric types you need to understand at the semantic level, not just the API level:

Counter — a value that only goes up (resets to zero on restart). Use it for total requests, total errors, total bytes sent. Never use a counter for something that can decrease. PromQL's rate() and increase() functions unwrap counters properly, handling resets.

Gauge — a value that can go up or down. Use it for current queue depth, active connections, temperature, memory usage. Don't use rate() on a gauge — it's meaningless.

Histogram — pre-aggregated bucketed observations. Use it for latency and request size. It exposes three time series: _bucket, _sum, and _count. The bucket boundaries you choose at instrumentation time are permanent — you can't change them without restarting the process.

Summary — client-side computed quantiles. Use it only when you need accurate quantiles and can't aggregate across instances (summaries can't be aggregated in PromQL). In Kubernetes with multiple replicas, histograms are almost always the right choice over summaries.

package main

import (
	"math/rand"
	"net/http"
	"strconv"
	"time"

	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promauto"
	"github.com/prometheus/client_golang/prometheus/promhttp"
)

var (
	httpRequestsTotal = promauto.NewCounterVec(
		prometheus.CounterOpts{
			Name: "payment_service_http_requests_total",
			Help: "Total number of HTTP requests processed by the payment service.",
		},
		[]string{"method", "endpoint", "status_code"},
	)

	httpRequestDuration = promauto.NewHistogramVec(
		prometheus.HistogramOpts{\n\t\t\tName: \"payment_service_http_request_duration_seconds\",\n\t\t\tHelp: \"HTTP request duration in seconds, bucketed by endpoint.\",\n\t\t\tBuckets: []float64{0.005, 0.01, 0.025, 0.05, 0.1, 0.2, 0.5, 1.0, 2.5},\n\t\t},\n\t\t[]string{\"method\", \"endpoint\"},\n\t)\n\n\tinFlightRequests = promauto.NewGauge(\n\t\tprometheus.GaugeOpts{\n\t\t\tName: \"payment_service_in_flight_requests\",\n\t\t\tHelp: \"Number of HTTP requests currently being processed.\",\n\t\t},\n\t)\n\n\tpaymentQueueDepth = promauto.NewGauge(\n\t\tprometheus.GaugeOpts{\n\t\t\tName: \"payment_service_queue_depth\",\n\t\t\tHelp: \"Current number of payments waiting in the processing queue.\",\n\t\t},\n\t)\n)\n\nfunc instrumentedHandler(endpoint string, handlerFunc http.HandlerFunc) http.HandlerFunc {\n\treturn func(responseWriter http.ResponseWriter, request *http.Request) {\n\t\tinFlightRequests.Inc()\n\t\tdefer inFlightRequests.Dec()\n\t\tstartTime := time.Now()\n\t\twrappedWriter := &statusCapturingWriter{ResponseWriter: responseWriter, statusCode: http.StatusOK}\n\t\thandlerFunc(wrappedWriter, request)\n\t\tdurationSeconds := time.Since(startTime).Seconds()\n\t\thttpRequestDuration.WithLabelValues(request.Method, endpoint).Observe(durationSeconds)\n\t\thttpRequestsTotal.WithLabelValues(\n\t\t\trequest.Method,\n\t\t\tendpoint,\n\t\t\tstrconv.Itoa(wrappedWriter.statusCode),\n\t\t).Inc()\n\t}\n}\n\ntype statusCapturingWriter struct {\n\thttp.ResponseWriter\n\tstatusCode int\n}\n\nfunc (scw *statusCapturingWriter) WriteHeader(code int) {\n\tscw.statusCode = code\n\tscw.ResponseWriter.WriteHeader(code)\n}\n\nfunc processPayment(responseWriter http.ResponseWriter, request *http.Request) {\n\tprocessingTime := time.Duration(5+rand.Intn(295)) * time.Millisecond\n\ttime.Sleep(processingTime)\n\tif rand.Float64() < 0.02 {\n\t\thttp.Error(responseWriter, \"upstream payment gateway timeout\", http.StatusGatewayTimeout)\n\t\treturn\n\t}\n\tresponseWriter.WriteHeader(http.StatusOK)\n\tresponseWriter.Write([]byte(`{\"status\":\"processed\"}`))\n}\n\nfunc main() {\n\tgo func() {\n\t\tfor {\n\t\t\tpaymentQueueDepth.Set(float64(rand.Intn(500)))\n\t\t\ttime.Sleep(5 * time.Second)\n\t\t}\n\t}()\n\thttp.HandleFunc(\"/api/v1/payments\", instrumentedHandler(\"/api/v1/payments\", processPayment))\n\thttp.Handle(\"/metrics\", promhttp.Handler())\n\tgo http.ListenAndServe(\":9091\", nil)\n\thttp.ListenAndServe(\":8080\", nil)\n}",
        "output": "# Prometheus scrapes http://payment-service-pod:9091/metrics and receives all four metric types."
      }

Production-Grade Recording Rules and Alerting That Won't Page You at 3am

Raw PromQL queries against high-cardinality data are expensive. A query like rate(http_requests_total[5m]) across 200 pods runs every time a dashboard loads. In large clusters, this causes Prometheus to churn through millions of samples per query, leading to query timeouts and the dreaded 'query timed out in expression evaluation' error.

Recording rules solve this by pre-computing expensive expressions and storing the result as a new time series. Prometheus evaluates recording rules on its evaluation interval (typically 1m), writes the result into its TSDB, and future queries read that cheap pre-computed series instead of re-scanning the raw data.

Naming matters. The Prometheus community convention for recording rule names is level:metric:operations. For example job:http_requests_total:rate5m means: aggregated at the job level, derived from http_requests_total, computed as a 5-minute rate. Sticking to this convention makes rules self-documenting and searchable.

Alerts in Prometheus are defined in the same YAML format as recording rules. The critical production insight is that alerts should express SLO burn rates, not raw thresholds. An alert that fires when error rate > 1% will fire constantly during minor blips. An alert based on a multi-window burn rate (Google's SRE model) only fires when you're burning through your error budget fast enough to exhaust it within a prediction window — dramatically reducing noise.

# PrometheusRule custom resource — picked up automatically by the Prometheus Operator.
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: payment-service-slo-rules
  namespace: production
  labels:
    prometheus: kube-prometheus
    role: alert-rules
spec:
  groups:
    - name: payment_service_recording_rules
      interval: 1m
      rules:
        - record: job_endpoint:payment_service_http_requests_total:rate5m
          expr: |
            rate(payment_service_http_requests_total[5m])

        - record: job_endpoint:payment_service_error_ratio:rate5m
          expr: |
            sum by (job, endpoint) (
              rate(payment_service_http_requests_total{status_code=~\"5..\"}[5m])\n            )\n            /\n            sum by (job, endpoint) (\n              rate(payment_service_http_requests_total[5m])\n            )\n\n        - record: job_endpoint:payment_service_latency_p99:rate5m\n          expr: |\n            histogram_quantile(\n              0.99,\n              sum by (job, endpoint, le) (\n                rate(payment_service_http_request_duration_seconds_bucket[5m])\n              )\n            )\n\n    - name: payment_service_alerts\n      rules:\n        - alert: PaymentServiceHighErrorBurnRate\n          expr: |\n            job_endpoint:payment_service_error_ratio:rate5m > (14.4 * 0.001)\n          for: 2m\n          labels:\n            severity: critical\n            team: payments\n            runbook_url: https://wiki.company.com/runbooks/payment-service-errors\n          annotations:\n            summary: \"Payment service burning error budget at critical rate\"\n            description: |\n              Endpoint {{ $labels.endpoint }} error ratio is {{ $value | humanizePercentage }}.\n\n        - alert: PaymentServiceHighLatency\n          expr: |\n            job_endpoint:payment_service_latency_p99:rate5m > 0.5\n          for: 5m\n          labels:\n            severity: warning\n            team: payments\n          annotations:\n            summary: \"Payment service p99 latency exceeds SLO threshold\"\n\n        - alert: PaymentQueueConsumerDead\n          expr: |\n            payment_service_queue_depth == 0\n            and\n            sum(job_endpoint:payment_service_http_requests_total:rate5m) > 10\n          for: 3m\n          labels:\n            severity: critical\n            team: payments\n          annotations:\n            summary: \"Payment queue depth is zero but traffic is flowing\"",
        "output": "# PrometheusRule applied. Operator picks it up via label matching."
      }

Alertmanager: Routing, Silencing, and Deduplication

Prometheus evaluates alert rules and sends firing alerts to Alertmanager. Alertmanager is a separate component responsible for deduplicating, grouping, routing, and silencing alerts before sending them to notification channels (PagerDuty, Slack, email, Opsgenie).

Alertmanager's routing tree is a hierarchical matching system. An alert's labels are matched against the route configuration. The first matching route determines the receiver. This means your alert labels (severity, team, service) must be carefully designed to match your routing tree.

# Alertmanager configuration — deployed as a Secret in the monitoring namespace.
# The Prometheus Operator picks this up from the alertmanager.yaml key.
global:
  resolve_timeout: 5m
  pagerduty_url: 'https://events.pagerduty.com/v2/enqueue'
  slack_api_url: 'https://hooks.slack.com/services/T00/B00/xxxx'

route:
  # Default receiver for alerts that don't match any sub-route.
  receiver: 'slack-catchall'
  group_by: ['alertname', 'namespace', 'endpoint']
  group_wait: 30s         # Wait 30s to group similar alerts.
  group_interval: 5m      # Send grouped updates every 5m.
  repeat_interval: 4h     # Re-send unresolved alerts every 4h.

  routes:
    # Critical alerts -> PagerDuty (page the on-call engineer).
    - match:
        severity: critical
      receiver: 'pagerduty-critical'
      group_wait: 10s       # Page faster for critical alerts.
      repeat_interval: 1h   # Re-page every hour if unresolved.

    # Warning alerts -> Slack channel (no page, just notification).
    - match:
        severity: warning
      receiver: 'slack-warnings'
      group_wait: 1m
      repeat_interval: 4h

    # Team-specific routing.
    - match:
        team: payments
      receiver: 'slack-payments-team'

receivers:
  - name: 'pagerduty-critical'
    pagerduty_configs:
      - service_key: '<pagerduty-integration-key>'
        severity: 'critical'
        description: '{{ .CommonAnnotations.summary }}'
        details:
          firing: '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
          num_firing: '{{ .Alerts.Firing | len }}'
          runbook: '{{ .CommonAnnotations.runbook_url }}'

  - name: 'slack-warnings'
    slack_configs:
      - channel: '#alerts-warnings'
        title: '{{ .GroupLabels.alertname }}'
        text: '{{ range .Alerts }}{{ .Annotations.summary }}{{ "\n" }}{{ end }}'
        send_resolved: true

  - name: 'slack-payments-team'
    slack_configs:
      - channel: '#payments-alerts'
        title: '[Payments] {{ .GroupLabels.alertname }}'
        text: '{{ range .Alerts }}{{ .Annotations.description }}{{ "\n" }}{{ end }}'

  - name: 'slack-catchall'
    slack_configs:
      - channel: '#alerts-catchall'
        title: 'Unrouted Alert: {{ .GroupLabels.alertname }}'

# Inhibition: suppress warning alerts when a critical alert is already firing
# for the same service. Prevents alert storms.
inhibit_rules:
  - source_match:
      severity: critical
    target_match:
      severity: warning
    equal: ['alertname', 'namespace']

PromQL Common Query Cheat Sheet

PromQL is the query language for Prometheus. Whether you're building Grafana dashboards, writing alerting rules, or debugging a production issue, having a mental library of common PromQL patterns saves hours. Below is a cheat sheet of the most useful queries for Kubernetes monitoring.

Infrastructure Metrics - CPU usage per pod: rate(container_cpu_usage_seconds_total{container!=\"POD\", image!=\"\"}[5m]) - Memory usage per pod: container_memory_working_set_bytes{container!=\"POD\", image!=\"\"} - Network receive bytes per pod: rate(container_network_receive_bytes_total[5m]) - Disk reads per pod: rate(container_fs_reads_bytes_total[5m])

Application Metrics (assuming custom metrics like payment_service_http_requests_total) - Request rate per second: rate(payment_service_http_requests_total[5m]) - Error rate per second: rate(payment_service_http_requests_total{status_code=~\"5..\"}[5m]) - Error ratio (percentage): sum(rate(payment_service_http_requests_total{status_code=~\"5..\"}[5m])) / sum(rate(payment_service_http_requests_total[5m])) - p99 latency: histogram_quantile(0.99, sum(rate(payment_service_http_request_duration_seconds_bucket[5m])) by (le, endpoint)) - Average latency: rate(payment_service_http_request_duration_seconds_sum[5m]) / rate(payment_service_http_request_duration_seconds_count[5m])

Prometheus Self-Monitoring - Active time series: prometheus_tsdb_head_series - Memory usage: process_resident_memory_bytes - Scrape duration per job: scrape_duration_seconds - Targets up per job: up (returns 1 if target is up, 0 if down)

Alerting Patterns - Pager-worthy: job_endpoint:payment_service_error_ratio:rate5m > (14.4 * 0.001) (14.4 x 0.1% = 1.44% error rate over 5m, burning through monthly SLO in hours) - No data: absent(prometheus_tsdb_head_series) — fires when Prometheus itself is down - Pod restart detection: changes(process_start_time_seconds[1h]) > 0

Cardinality Detection - Top 10 metric names by series count: topk(10, count by (__name__)({__name__=~\".+\"})) - Series count for a specific metric: count(payment_service_http_requests_total)", "code": { "language": "promql", "filename": "promql-cheat-sheet.promql", "code": "# ---- Infrastructure Metrics ----

# CPU usage rate (5m window) per pod rate(container_cpu_usage_seconds_total{container!=\"POD\", image!=\"\"}[5m])

# Memory working set per pod (Gauge, no rate) container_memory_working_set_bytes{container!=\"POD\", image!=\"\"}

# Network bytes received per second per pod rate(container_network_receive_bytes_total[5m])

# ---- Application Request Metrics ----

# Total request rate sum(rate(payment_service_http_requests_total[5m]))

# Error ratio (percentage of 5xx) sum(rate(payment_service_http_requests_total{status_code=~\"5..\"}[5m])) / sum(rate(payment_service_http_requests_total[5m]))

# p99 latency (uses histogram_quantile) histogram_quantile(0.99, sum(rate(payment_service_http_request_duration_seconds_bucket[5m])) by (le, endpoint))

# ---- Prometheus Self-Monitoring ----

# Active time series count prometheus_tsdb_head_series

# Prometheus process memory process_resident_memory_bytes

# ---- Cardinality Analysis ----

# Top 10 metric names by number of time series topk(10, count by (__name__)({__name__=~\".+\"}))

# ---- Alerting Patterns ----

# High error burn rate (SLO-based) job_endpoint:payment_service_error_ratio:rate5m > (14.4 * 0.001)

# Detect pod restart in last hour (counter reset) changes(process_start_time_seconds[1h]) > 0

# Absence of data (Prometheus itself down) absent(prometheus_tsdb_head_series)", "output": "These PromQL queries can be executed directly in the Prometheus UI Execute page, in Grafana panels, or in alert rules." }, "callout": { "type": "tip", "title": "Always Use Recording Rules for Repeated Queries", "text": "The error ratio query above is expensive: it scans all error series and total series, then divides. If you use this query in five dashboards and one alert, you're executing it six times every evaluation cycle. Create a recording rule job_endpoint:payment_service_error_ratio:rate5m and query that instead.", "hook": "Every PromQL query you write should be a candidate for a recording rule if it appears in more than one place.", "bullets": [ "Recording rules pre-compute expensive queries into cheap time series.", "Use rate() for counters, not raw values (counter resets break averages).", "Use histogram_quantile() with sum by (le, ...) for aggregated percentiles.", "Avoid * selectors in production — always filter by at least one label.", "Check query performance with the Prometheus UI's query analysis (explain button)." ] }, "production_insight": "The most common PromQL mistake is using rate() on a gauge — it produces meaningless results because gauges can go down. Another frequent error is forgetting to sum by the correct labels when using histogram_quantile(): you must include le (bucket upper bound) in the by clause, otherwise Prometheus returns an error. When debugging high memory, the topk(10, count by (__name__)({__name__=~\".+\"}))` query quickly identifies which metric name has the most time series — often the culprit is a high-cardinality label on a single metric.", "key_takeaway": "Master these core PromQL patterns: rate (counters), gauge (raw values), histogram_quantile (latency percentiles), topk (cardinality detection). Use recording rules to optimize expensive queries. Always filter by at least one label to avoid scanning the entire TSDB." }, { "heading": "Prometheus Storage: TSDB Internals, Retention, and Thanos/Cortex for Long-Term", "content": "Prometheus stores metrics in its own time-series database (TSDB). Understanding TSDB internals is critical for capacity planning, retention tuning, and deciding when to add long-term storage.