Prometheus rate() vs increase(): Fix Wrong Alerts

The alert looked airtight in review: error budget burn measured with increase() over 5 minutes against a threshold the team agreed on. Then it paged 14 times in one night during normal traffic, stayed silent through a real 20-minute outage the next week, and nobody trusts it anymore.

The bug is a unit mismatch hiding in plain sight. rate() answers per-second — how fast. increase() answers total — how many. An alert threshold written in one unit and evaluated in the other misfires by the exact ratio of the window length, and counter resets plus short windows add two more ways to be wrong.

This guide makes the semantics concrete with numbers you can verify in the expression browser. You'll learn which function pages and which reports, how resets are handled for you, why windows must respect scrape intervals, and how to fix the three alert patterns that mix them up. Eight minutes here ends the flapping-alert era for your team. Every example here runs in the expression browser, so you can prove each claim before touching alerts.

rate() Is Speed, increase() Is Distance

rate() and increase() read the same counter but answer different questions. rate() divides growth by window seconds, returning events per second — a speed. increase() returns the raw growth over the window, a total count. For steady traffic at 14 errors/sec, rate(m[5m]) reads 14 while increase(m[5m]) reads about 4,200.

The relationship is exact: increase over a window roughly equals rate times the window length in seconds. That identity is your debugging superpower — evaluate both side by side and the ratio should match the window. When it doesn't, something interesting happened inside the window, usually a reset.

Choose by consumer. Pagers need speeds with stable thresholds, so alerts use rate(). Humans need totals for standups and postmortems, so dashboards use increase(). Mixing them means thresholds silently scale with window edits, which is how teams end up paged by their own config.

The implementation detail behind rate explains its edge behavior. It takes the first and last samples in the window, adjusts for resets between them, then extrapolates the slope to the full window boundaries. Extrapolation assumes samples near both edges; a scrape gap at the window start inflates the result because the slope stretches over missing time. This is why wider windows beat narrow ones on gappy jobs — more real samples anchor both ends. When values look 10 percent high on scrape-edge cases, suspect extrapolation before suspecting traffic. The function is honest about what it saw; sparse windows just show it little.

# Per-second speed — for alerts with per-second thresholds
rate(http_errors_total[5m])
# > 25 means 25 errors/sec sustained

# Window total — for dashboards and burn accounting
increase(http_errors_total[5m])
# ~ rate * 300 during steady traffic (300s window)

# Wrong: threshold in one unit, function in the other
increase(http_errors_total[5m]) > 1000   # 1000 what? total, not per-second

Counter Resets Are Handled — Trust the Functions

Counters reset to zero on every exporter restart, and both functions expect it. When they see the value drop, they treat it as a reset: they add the growth before the drop to the growth after, instead of reporting a negative plunge. Your alert math survives deploys without special cases.

This is why alerting on raw counters is always wrong. A raw errors_total graph cliffs to zero at every restart, and any threshold on it pages on deploys and misses real growth between restarts. rate() and increase() exist precisely to absorb resets — let them.

Verify it yourself once and you'll trust it forever. Graph a raw counter across a restart, then overlay increase(m[5m]): the raw line cliffs while the increase line sails through unchanged. That ten-second experiment ends every debate about whether restarts corrupt alert math.

Resets interact with federation and recording rules in ways worth knowing. Federated counters reset when the scraping server restarts, independent of exporter restarts — two reset sources, both handled the same way. Recording rules that sum counters before rating change reset visibility: sum first and individual resets hide inside the total, which is usually what you want for availability math. Long-term-storage downsampling preserves the compensation because it keeps enough resolution. The practical rule stays simple: rate the rawest counter you have, as close to the source as possible, and let every layer absorb the resets that occur there.

# Raw counter cliffs to zero at every restart — never alert on this
http_errors_total

# increase() sails through the same restart unchanged
increase(http_errors_total[5m])

# Correct paging alert: per-second rate + sustained duration
- alert: CheckoutErrorsHigh
  expr: rate(http_errors_total[5m]) > 25
  for: 5m

Window Size vs Scrape Interval: Stop the Flapping

Range windows must respect the scrape interval. rate() needs at least two samples in the window to compute slope; with a 1m window on a 60s scrape job, most evaluations see one sample and extrapolate wildly. The result flaps between zero and huge on every evaluation cycle.

The rule is simple: windows at 4x the interval or wider. Fifteen-second scrapes pair with 5m windows; 60s scrapes need 5m minimum. Wider windows also smooth legitimate burstiness so alerts fire on sustained shifts rather than single bad scrapes.

Add for-durations as the second half of the fix. for: 5m requires the breach to persist across evaluations before paging, which filters deploy spikes and traffic bursts while catching real regressions within minutes. Window width and for-duration together are what make alerts trustworthy.

The for-duration is a state machine with three states: inactive, pending, firing. A breach starts pending; recovery before the duration ends returns to inactive with no page; sustained breach fires and stays firing until resolution plus a resolve timeout. This design filters spikes exactly because most spikes die in pending. Pair for with evaluation alignment: a 5m for on a 1m group needs five consecutive breaching evaluations, while the same for on a 5m group needs one. Shorter groups plus moderate for-durations catch real shifts fastest with fewest false pages. Tune the pair together, never alone.

Gauges Are Not Counters: Don't rate() Them

Gauges break rate() because the function assumes counter semantics: any decrease looks like a reset and gets added back as phantom growth. A queue-depth gauge dipping from 100 to 40 reads as a reset of 60, inflating the result with traffic that never existed.

Use the right tool per type. Counters (requests, errors, bytes sent) get rate() and increase(). Gauges (temperature, queue depth, memory) get raw-value alerts or delta() for change-over-window. The metric exposition or docs declare the type — check before wrapping.

Negative outputs are the tell. A rate() that returns negative values on a supposedly counter-like metric means it's actually a gauge, or the instrumentation resets it arbitrarily. Either way the function is wrong for the data, and the fix is at the query, not the threshold.

Two more gauge functions complete the toolkit. deriv() returns the per-second derivative of a gauge over the window — useful for disk-fill rates and temperature slopes where resets never occur. predict_linear() extends the derivative into the future, powering classic disk-full-in-N-hours alerts from filesystem free bytes. Both assume gauge semantics, so counter resets corrupt them exactly as rate() corrupts gauges. Match function family to metric type as a review checklist: counters get rate and increase, gauges get delta, deriv, and predict_linear, info metrics get no math at all. Type discipline eliminates the whole bug class.

# Wrong: rate() on a gauge invents phantom growth on every dip
rate(queue_depth[5m])

# Right: alert on the gauge value or its window change
delta(queue_depth[5m]) > 500
queue_depth > 10000

Rewriting Alerts You'll Actually Trust

Rewrite broken alerts in a fixed pattern. Paging rules use rate() with per-second thresholds and for-durations: rate(http_errors_total[5m]) > 25 for 5m. Info rules and dashboards use increase() totals: increase(http_errors_total[1h]) tracking errors per hour alongside business metrics.

Backtest before enabling paging. Replay the new expression against a week of history in the expression browser and compare pages against known incidents. Every false page in backtest is a free bug caught before it wakes someone; every missed real incident is a threshold to tune.

Name alerts with their units. CheckoutErrorsPerSecondHigh reads differently from CheckoutErrorsHigh, and reviewers catch mismatches when the unit is in the name. This naming habit is the cheapest prevention in the guide — one word that stops the whole bug class at review time.

Test alert expressions before they page anyone. promtool test rules evaluates rule files against scripted series and asserts firing behavior — write a test per alert with a normal-traffic case and an incident case. Store tests beside the rules file and run them in CI; a threshold edit that breaks the incident case fails the build instead of the on-call. Backtest on real history too: replay the expression over the last incident week in the browser and confirm it would have paged. Alerts with tests earn shorter for-durations because false pages stop being scary. Untested alerts deserve long durations and low trust.

# Page on speed, report on totals — one rule each
- alert: CheckoutErrorsHigh
  expr: rate(http_errors_total[5m]) > 25
  for: 5m
  labels:
    severity: page

# Dashboard panel tracks the business total beside it:
# increase(http_errors_total[1h])

Units Are the Whole Game

The deeper lesson is that PromQL functions carry units, and ignoring them breaks alerts as surely as ignoring units breaks physics. rate() yields per-second, increase() yields per-window, and thresholds must speak the same unit. Code review for alerts should check units first, logic second.

This discipline compounds. Teams that name units, backtest windows, and respect scrape intervals build alert suites that page rarely and correctly. Teams that don't accumulate muted alerts until they're blind during real outages — the exact failure mode of this guide's incident.

Start with your noisiest alert today. Check its function against its threshold unit, widen its window to 4x the interval, and add a for-duration. That single rewrite usually silences the worst pager while making the next real incident louder.

Burn-rate alerting applies these lessons at scale. Instead of one threshold, pair a fast-burn alert (high rate, short window, paging) with a slow-burn alert (lower rate, long window, ticket). The fast leg catches cliffs in minutes; the slow leg catches leaks in hours without paging for noise. Both legs use rate() with per-second thresholds, both carry for-durations, and both reset cleanly across deploys. Error budgets computed with increase() over 30 days sit on dashboards beside them, translating pages into budget language managers understand. This pattern ends the single-threshold flapping era permanently.