Prometheus Cardinality Explosion: Find and Fix It

The page comes at 3 AM: Prometheus OOMKilled, restarted, replaying WAL, OOMKilled again. Disk has terabytes free. Retention is a modest 15 days. The killer is invisible on every disk dashboard because it lives in RAM: active series in the TSDB head, each costing 2 to 3KB, multiplied by a label that seemed harmless in code review.

High-cardinality labels are how this happens. A user_id, request path with embedded IDs, or session token on a single counter multiplies one metric into hundreds of thousands of series. The math is brutal and fast: 250,000 users times a few label combinations crosses a 16 GB pod limit in hours, and the crash loop that follows gaps every dashboard at once.

This guide gives you the memory math, the queries that name the offending label in minutes, and the two production fixes — dropping labels at scrape time and aggregating with recording rules. You'll leave with sample_limit guardrails and alerts that catch the next explosion while it's still a graph wiggle, not a 3 AM page.

Cardinality Math: Why One Label Costs Gigabytes

Every unique combination of metric name and label values becomes one active series, and each one costs roughly 2 to 3KB of head memory for chunks, labels, and index entries. That sounds trivial until multiplication kicks in: a counter with user_id (250K values) times status (3 values) is 750,000 series and around 2 GB of RAM from a single metric.

Do the math before any label ships. Multiply the expected value counts of every label on the metric, multiply by 3KB, and compare against headroom. If one label's growth is unbounded — users, sessions, request IDs — the product grows forever and the pod limit is just a countdown.

Head memory is the constraint that matters. Disk retention, block size, and WAL replay all matter operationally, but OOMKill comes from the head. Dashboards for prometheus_tsdb_head_series and process_resident_memory_bytes belong next to every deploy pipeline, reviewed as carefully as latency graphs.

Break the 2 to 3KB into parts you can reason about. Label pairs cost bytes each plus index entries; the current head chunk holds up to 120 samples before sealing; the inverted index maps every label value to its series. Small labels on few series cost almost nothing, which is why method and status never hurt. The same bytes on 250,000 user values multiply into gigabytes because every part scales with series count. Run the multiplication for each new label in review: values times combinations times 3KB. Numbers on a napkin beat OOMKills at 3 AM every time.

# Active series in the head — the number that kills pods
prometheus_tsdb_head_series

# Memory tracking head growth (alert at 80% of pod limit)
process_resident_memory_bytes

# Top 10 metrics by series count — the offender is usually #1
sort_desc(topk(10, count by (__name__) ({__name__=~".+"})))

# Scrape size per job — the jumping job owns the bad label
scrape_samples_scraped{job="checkout"}

Which Labels Explode and Which Stay Safe

Labels split into two families. Bounded labels like method, status_code, and route templates have small fixed value sets — 4 methods, 5 statuses, 50 routes — and their products stay flat for years. They're the reason labels exist and are safe by construction.

Unbounded labels grow with traffic: user IDs, emails, session tokens, request UUIDs, and raw URL paths with embedded IDs. Each new user permanently adds series, so the head grows monotonically until the pod dies. These values belong in logs, traces, or warehouses — systems built for per-entity detail — never in metric labels.

The treacherous middle is labels that look bounded until errors strike: error messages with dynamic text, file paths with temp dirs, pod names in long-lived clusters. Normalize at instrumentation — lowercase, bucket, allowlist — so a new error string can't mint ten thousand series overnight.

Histograms multiply everything they touch. A histogram with 12 buckets and labels for method and status creates 12 times 4 times 5 series per metric name — plus _count and _sum. That is 242 series where a counter would create 20, before any unbounded label arrives. Native histograms compress this dramatically by storing bucket spans instead of indexed per-bucket series, which is why newer clients prefer them. Until you migrate, treat every histogram label as twelve labels in review. The bucket count is a hidden multiplier most teams forget to multiply.

Finding the Offending Label in Minutes

When head series spike, find the owner fast. Rank metrics by series count with a topk query — the offender is usually number one by a wide margin. Then count values of its labels to find the runaway: a payment_method with 4 values is innocent, a user_id with 250,000 values is guilty.

Correlate with scrape sizes. The scrape_samples_scraped series per job shows which job's exposition ballooned and exactly when, pointing at the deploy that added the label. Cross-check against deploy logs and the diff usually confesses within minutes.

Move quickly but measure first. Record the head series count, memory, and the offending label's value growth before changing anything. Those baselines prove the fix worked and size the guardrails you'll add next.

The TSDB status API gives machine-readable ground truth when PromQL feels indirect. GET /api/v1/status/tsdb returns head statistics including total series, chunk counts, and compaction numbers — scrape it into your capacity dashboard. For label-level forensics, /api/v1/label/<name>/values with match[] selectors counts values per label without writing queries. Compare value counts across a deploy boundary: the label whose cardinality jumps names the commit. When topk queries time out on overloaded servers, the status API answers in milliseconds because it reads head metadata, not series data. Add both endpoints to your incident runbook next to the PromQL versions.

# Value counts for the suspect label — runaway = tens of thousands+
count by (payment_method) (checkout_completed_total)

# Confirm growth started at a deploy (7-day window)
count(count by (__name__) ({__name__="checkout_completed_total"}))

Dropping and Aggregating: the Two Production Fixes

The production fix drops the explosive label at scrape time, before it ever reaches the head. metric_relabel_configs rules run after scraping against metric labels, and a labeldrop or regex drop on user_id deletes the dimension from every affected series. The metric survives aggregated; the per-user detail never enters memory.

The classic trap is placing the rule in relabel_configs, which runs before scraping against target labels only. The config reloads cleanly, the rule matches nothing, and cardinality climbs untouched. Always use metric_relabel_configs for metric labels — the name is the documentation.

Dashboards that genuinely need segments get recording rules instead. Sum the counter by plan tier or endpoint group into a new aggregated series, point dashboards at it, and drop the raw high-cardinality originals. You keep every business answer while deleting millions of stored series.

Choose the drop granularity deliberately. Dropping a whole metric with source_labels __name__ suits counters that should never have shipped; labeldrop suits metrics that are valuable without one explosive dimension. The keep action inverts the logic for allowlisting: keep only named metrics from a noisy exporter and drop the hundred you never query. Order matters — rules evaluate top to bottom, so place cheap exact matches before expensive regexes. Test every rule with promtool test rules or a staging reload plus series counts; a miswritten regex can drop production alerts silently. Review the rule file quarterly because exporters add metrics on upgrade.

# Drop the explosive label before it reaches the head
scrape_configs:
  - job_name: checkout
    sample_limit: 50000
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: 'checkout_completed_total'
        action: drop
        # Or keep the metric but drop just the label:
      - regex: 'user_id'
        action: labeldrop

# Recording rule preserving the business answer by segment
# rules/checkout.yml
- record: job:checkout_completed_total:by_tier
  expr: sum by (plan_tier) (checkout_completed_total)

Guardrails: sample_limit and Alerts That Page Early

sample_limit is the circuit breaker that converts silent growth into a loud, immediate scrape failure. Set per job near 150% of its measured normal scrape size: normal variance passes, a label explosion fails the whole scrape and pages someone before the head fills.

Pair it with two alerts. One fires when any job's scrape size crosses 80% of its limit — the early warning that buys weeks. The other watches head-series week-over-week growth above 20% and memory above 80% of the pod limit. Together they catch both the sudden deploy mistake and the slow unbounded creep.

Treat limit tuning as capacity work. Recompute normal sizes quarterly, review every new label in code review with the 3KB math, and require load-test evidence for metrics on hot paths. Guardrails decay as fleets grow; the review habit is what keeps them honest.

Know the two sibling limits. sample_limit caps samples per scrape for one target and fails that scrape loudly; target_limit caps the total across all targets of a job and spreads protection wider. Set sample_limit near 150 percent of normal per-target size and target_limit near 150 percent of the job total. When a scrape exceeds the limit the entire scrape fails — including healthy series — so pair limits with alerts at 80 percent that fire before the cliff. Document each limit beside the measurement that sized it; unexplained limits get deleted in cleanups and the protection vanishes. Limits are capacity decisions wearing config clothes.

# Early warning: job approaching its sample limit
scrape_samples_scraped / ignoring() group_left
  scrape_sample_limit > 0.8

# Slow creep: head series growing week over week
predict_linear(prometheus_tsdb_head_series[1d], 7 * 86400)
  > prometheus_tsdb_head_series * 1.2

Recovering Without Just Buying Bigger Pods

Recovery from an active explosion follows a fixed order. First drop the label at scrape to stop the bleeding — head series flatten within a couple of compactions. Then point dashboards at aggregated recording rules so nothing visible breaks. Then set sample_limit and growth alerts so it can't recur silently.

Resist the urge to fix retention or resize pods instead. Retention governs disk blocks, and bigger pods just reschedule the crash by the exact ratio of added memory to growth rate. Both feel productive while the series math keeps compounding underneath.

Finally, move the displaced detail somewhere it belongs. Per-user analytics live happily in warehouses fed by event streams; debugging detail lives in logs and traces. Prometheus keeps the aggregates that drive alerts and dashboards — that's the division of labor that survives scale.

Dropping labels affects new samples only — existing high-cardinality series age out through retention, not instantly. Expect head series to fall over hours as old chunks compact away, not seconds after reload. For immediate relief on a crashing server, the snapshot API plus a storage wipe and replay buys a clean head, but prefer patience unless OOMKills loop. Old series with dropped labels remain queryable under their original names until retention expires, which keeps history intact while stopping growth. Verify the decline with head-series graphs across two compaction cycles before declaring victory. The fix works on geological time; watch it like a geologist.