Alerting & On-Call — Why Silenced Services Hide Outages

At 3 AM, your phone screams. You scramble to your laptop, bleary-eyed, only to discover the alert fired because a CPU spike lasted four seconds and self-corrected before you even logged in. You've lost sleep over nothing — and this happens four nights a week.

I've seen this pattern at companies of every size. The specifics vary — sometimes it's CPU, sometimes it's memory, sometimes it's a disk utilization alert on a volume that autoscales — but the shape is always the same. Engineers get paged for things that didn't need human attention. They lose sleep, lose trust in the paging system, and eventually lose patience with the whole programme. The best engineers leave first, because they have options. The ones who stay start silencing things. And then the real fire happens.

This is not a monitoring problem. It's a culture problem that presents as a technical one. The technical symptoms are easy to diagnose: too many alerts, thresholds too low, no runbooks, no audit process. But the underlying culture problem is that most engineering teams treat adding alerts as free and removing alerts as risky. Every post-mortem ends with 'add monitoring.' Nobody's post-mortem ever ends with 'delete the alert that's been crying wolf for six months.' That asymmetry is how you get a paging system that cries wolf 40 times a week and then fails to wake anyone up when the real incident hits.

The root cause isn't that teams care too little about monitoring — it's that they add alerts reactively, after every incident, without ever pruning the ones that stop being useful. Over time, the alert system becomes a noise machine. Engineers stop trusting it, start silencing pages, and miss the signals that actually matter. The fix isn't more dashboards. It's disciplined, intentional alerting philosophy backed by concrete practices for thresholds, routing, escalation, and rotation design.

By the end of this article you'll know how to audit your existing alert stack and identify the ones that don't serve you, how to write alerts that fire on symptoms not causes, how to structure an on-call rotation that doesn't erode your team's wellbeing, and how to use real tooling — Prometheus alerting rules, PagerDuty routing logic, and runbook templates — to make all of it operational and repeatable. The goal isn't a perfect alerting system. It's a system your engineers trust enough to actually pay attention to.

Alert on Symptoms, Not Causes — The Golden Rule of Monitoring

The most common alerting mistake is alerting on what you think is wrong instead of what the user actually experiences. A high CPU alert fires and the engineer investigates — but CPU being high isn't inherently bad. Maybe a batch job is running. Maybe it's expected load from a traffic spike the autoscaler is still catching up to. Maybe the GC is doing a full collection. The user doesn't care about CPU. They care whether the checkout page loads and their payment goes through.

Symptomatic alerting means your alert fires on things users feel directly: high latency, elevated error rates, failed health checks, degraded availability. Google's SRE book formalised this as the Four Golden Signals — latency, traffic, errors, and saturation — and the framing has held up well. Alerts on these signals are almost always actionable because if error rate is 15%, something is broken for users right now, and the on-call engineer has a clear starting point.

Causal metrics like CPU, memory, and disk are better suited for dashboards and capacity planning, not paging. You investigate them after a symptom alert fires to understand why something is wrong — not to decide whether something is wrong. The distinction matters because it changes the engineer's mental model during an incident. A symptom alert says 'users are experiencing this.' A causal alert says 'something in your infrastructure crossed a threshold.' Only one of those tells you whether to act.

The practical test: before adding any alert, ask yourself 'If this fires at 3 AM, can the on-call engineer take a concrete action within five minutes?' Not 'investigate' — act. If the answer is 'investigate and check some dashboards and maybe escalate', it belongs on a dashboard, not a pager. The 3 AM test is deliberately adversarial: the human responding is sleep-deprived, potentially unfamiliar with the specific service, and operating under pressure. Write your alerts for that human, not for a well-rested engineer on a Tuesday morning.

Here's the pushback you'll hear, usually from engineers who've been on the wrong end of missed incidents: 'But what about catching problems early?' This is a reasonable concern wrapped around a false premise. Causal metrics do catch problems early — on dashboards, during business hours, reviewed by engineers who have context and aren't panicking. You don't need to wake someone up to look at a graph trending in the wrong direction. Set up a daily dashboard review as part of your team's operational rhythm. If a pattern in causal metrics consistently precedes a symptom, write a symptom-based alert for the user-facing impact, not the infrastructure metric that correlates with it. The correlation is interesting. The symptom is the truth.

The Four Golden Signals aren't a checklist to run through mechanically. They're a lens for asking the right question: is what I'm about to alert on something a user would feel? Latency tells you how long users wait. Traffic tells you how much demand you're handling and whether demand patterns are normal. Errors tell you how often you're actively failing users. Saturation tells you how close to the edge you are before one of the other three degrades. Any proposed alert that doesn't map cleanly to one of these four should face an explicit justification before it gets merged.

Structuring On-Call Rotations That Don't Destroy Your Team

An on-call rotation is a social contract as much as it is a technical system. Engineers who feel the rotation is fair, predictable, and actively supported stay in it. Engineers who feel it's a punishment — or worse, an invisible tax that everyone pretends doesn't cost anything — churn. And the engineers who leave first are always the ones who have other options: the senior engineers, the ones who built the systems, the ones whose absence creates the next generation of undocumented incidents.

The fundamentals of a healthy rotation start with team size. You need at least four engineers to build a weekly rotation that gives people genuine recovery time between shifts. With three engineers, one person is always either on-call or just came off on-call — cognitive load never fully resets. With two, you're alternating weeks between two people and calling it a rotation. With one, you're not running a rotation at all; you're running a hero, and heroes burn out or leave.

Secondary on-call — a second engineer ready to be escalated to if the primary doesn't acknowledge within ten minutes — is non-negotiable for any service with a real SLA. It serves two purposes. The obvious one is redundancy: if the primary engineer is genuinely unavailable, the incident still gets coverage. The less obvious one is diagnostic: if escalation to secondary happens more than once per week, your primary alert volume is too high. The secondary escalation rate is a canary metric for rotation health that most teams never look at.

On-call handoff meetings deserve more ceremony than they typically receive. The outgoing engineer should document what fired, what was investigated, what commands were run, and what follow-up work was created. Without this, the same incidents repeat because the institutional knowledge of 'oh, that alert fires every Tuesday when the ETL job runs — just acknowledge and wait four minutes' lives in one engineer's head and evaporates with every rotation change. The handoff document is where that knowledge becomes team property.

Compensation matters — and how you structure it signals what you believe on-call is worth. Explicit on-call pay, time-off-in-lieu, or reduced sprint commitments during on-call weeks all communicate that the organisation understands the cost. The specifics matter less than the consistency and transparency.

But here's what I've observed across many teams: the psychological contract matters more than the compensation number. An engineer who receives $500 per on-call week but has no control over alert volume, whose feedback from retrospectives never changes anything, and who watches the same false alarms fire week after week without being fixed — that engineer feels trapped regardless of the pay. An engineer who receives time-off-in-lieu and can see, in every sprint, that the team is actively working to reduce alert noise and improve runbooks — that engineer feels respected. The best on-call programs are characterised by visible investment in reducing the burden, not just by compensating for it.

In 2026, most teams run distributed rotations spanning multiple time zones. This is both an opportunity and a design challenge. Done well, a distributed rotation means nobody carries the full 24-hour burden of a weekly shift — you can hand off at a natural boundary between regions and keep business-hours coverage for each timezone. Done poorly, it creates coordination overhead, unclear escalation paths when the person on-call is 9 time zones away, and handoff meetings that nobody can attend at a reasonable hour. If your team spans more than two time zones, design your rotation explicitly for the timezone distribution — don't just apply a single-timezone rotation template and hope the scheduling works out.

Rotation schedule changes are the most underestimated source of trust erosion. Once you publish a rotation, treat changes with the same communication discipline you'd apply to a production deployment. Engineers plan childcare, travel, and personal commitments around the schedule. A last-minute swap that affects a weekend isn't just inconvenient — it damages the sense that the rotation is a fair and predictable system, which is the only thing that makes it sustainable long-term.

Runbooks, SLOs, and the Alert Audit Loop That Keeps You Sane

Every alert that fires should have a runbook. Not a wiki page describing the service architecture. Not a Confluence document that was last updated eighteen months ago. A runbook: a living, numbered document that tells the on-call engineer exactly what to check, what commands to run, and what decisions to make — optimised for a person who may be half-asleep, may not know this service deeply, and has about five minutes before stakeholders start asking questions.

The most common runbook failure is structure: engineers write runbooks like documentation. They lead with context — here's how this service works, here's its architecture, here's the dependency graph. That context belongs in the service's architecture docs. A runbook during an active incident needs to start with the thing the engineer does right now, not the context they'd need to understand the system from scratch. Lead with the first command they should run. Everything else is footnotes.

Runbooks don't need to be perfect on day one. The minimum viable runbook has three headings: 'Is this alert real?', 'Known mitigations', and 'When to escalate and who to call.' The first heading should have a specific command or dashboard link that lets the engineer confirm the alert reflects a real problem, not a metric collection glitch. The second should have the two or three mitigations that have worked historically, even if they're partial. The third should have an explicit escalation matrix — not 'contact the team' but 'if the database is unreachable, call the database team at this PagerDuty service; if the issue is the payment gateway, here's the vendor's emergency number.'

As incidents happen, the on-call engineer appends what they learned. Within three months of consistently following this pattern, you have battle-tested documentation that reflects reality rather than design intent. The gap between design intent and operational reality is where most runbooks fail — and closing that gap is what makes the difference between a runbook that helps and one that the engineer closes after 30 seconds because it doesn't match what they're seeing.

SLO-based alerting is a different philosophy entirely, and it's worth understanding the shift it requires. Instead of alerting when error rate exceeds 1% — an arbitrary threshold chosen by someone who had a reasonable gut feeling — you alert when you're consuming your monthly error budget faster than sustainable. This ties your paging directly to whether you're going to breach the reliability commitment you've made to your users. It dramatically reduces alert volume while ensuring that every page represents a genuine threat to that commitment.

The monthly alert audit is the most underrated practice in on-call operations. Pull a report of every alert that fired in the last 30 days. For each one: Was it actionable? Was there a documented response? If the same alert fired more than three times without a code fix being shipped, that's reliability debt — it belongs in the engineering backlog with a sprint assignment, not as a recurring 3 AM interruption that the team has collectively accepted as normal.

The audit meeting structure matters. Pull the data before the meeting — total alerts fired, percentage that resulted in an acknowledged incident with a documented response, mean time to acknowledge, and the ranked list of repeat offenders by firing frequency. Present it visually. The team's job during the meeting is to make three decisions about each alert on the list: keep it as is, fix the underlying condition that makes it fire too often, or delete it. Not discuss it — decide. Meetings that produce 'we should look into that' instead of 'deleted' or 'ticket created' waste their 30 minutes entirely.

The SLO frame changes the audit conversation in a valuable way. Instead of asking 'was this specific alert actionable?' you ask 'is our error budget on track this month?' If the budget is healthy and you have 80% remaining at the midpoint, many threshold alerts that fired are provably noise — they didn't threaten the SLO. If the budget is burning and you're at 40% at midpoint, you need more alerting sensitivity, not less. The budget number makes the decision about alert sensitivity a function of reliability risk rather than engineering anxiety.

Alert Routing, Deduplication, and Suppression — The Plumbing That Makes It Work

Routing is where most alerting systems silently break in ways that are invisible until an incident proves it. Prometheus fires correctly. Alertmanager receives the alert. The alert routes to the wrong team — or to nobody at all — and the incident goes undetected. This failure is particularly dangerous because everything looks healthy: the alert fired, Alertmanager processed it, PagerDuty shows the service as active. The failure is in the gap between those systems, specifically in a label that's missing or mismatched.

Every alert label you add is implicitly a routing decision. The severity label determines which receiver handles the alert — PagerDuty for critical, Slack for warning. The team label determines which team's escalation policy is invoked. The service label enables inhibition rules and deduplication. Get any of these wrong and the alert either goes to the wrong destination or routes to Alertmanager's default receiver, which most teams configure as a catch-all Slack channel that nobody monitors at 3 AM.

Deduplication is Alertmanager's mechanism for not paging you multiple times for the same underlying problem. Two Prometheus instances — for example, a primary and a replica, or two regional scrape targets — will both fire the same alert when a metric breaches. Alertmanager groups them by label identity into a single notification. This is elegant when it works. The failure mode: two alerts that describe the same problem but have different labels — perhaps because one alert uses service="checkout" and another uses service="checkout-api" — are treated as separate alerts and generate separate pages. Label consistency across your alert rules is a more important operational practice than most teams realise, and it becomes critical when you have more than a handful of services.

Suppression windows are your surgical tool for planned maintenance. Unlike blanket silences — which are blunt instruments that suppress all alerts from a service regardless of type — proper suppression targets specific alert names for specific time windows. The rule is simple and should be enforced at the tooling level: every silence requires a comment with a linked ticket URL, and every silence auto-expires within 4 hours maximum. If your maintenance window is longer than 4 hours, renew the silence manually. This creates intentional checkpoints where someone has to actively decide that the silence should continue.

Inhibition rules solve a specific and common problem: when a critical alert fires, you don't want five additional warning alerts for the same service all generating pages simultaneously. Alertmanager inhibition rules suppress lower-severity alerts when a higher-severity alert is already active for the same service. The result is one page for one incident instead of five pages for five symptoms of the same root cause. This is the difference between an on-call engineer who wakes up to a single clear incident and one who wakes up to a flood of notifications that obscures which one to start with.

One important operational practice that often gets skipped: test your routing after any label change. When a service is renamed, when an alert rule is refactored, when a team restructuring changes the team label values — routing breaks silently. amtool config routes test with a set of representative alert labels takes about 90 seconds and catches misroutes before an incident does. Add it to your CI pipeline as a validation step any time alertmanager.yml or alert rule files change.

Frequently Asked Questions