Alerting & On-Call — Why Silenced Services Hide Outages

At 3 AM, your phone screams. You scramble to your laptop, bleary-eyed, only to discover the alert fired because a CPU spike lasted four seconds and self-corrected before you even logged in. You've lost sleep over nothing — and this happens four nights a week.

I've seen this pattern at companies of every size. The specifics vary — sometimes it's CPU, sometimes it's memory, sometimes it's a disk utilization alert on a volume that autoscales — but the shape is always the same. Engineers get paged for things that didn't need human attention. They lose sleep, lose trust in the paging system, and eventually lose patience with the whole programme. The best engineers leave first, because they have options. The ones who stay start silencing things. And then the real fire happens.

This is not a monitoring problem. It's a culture problem that presents as a technical one. The technical symptoms are easy to diagnose: too many alerts, thresholds too low, no runbooks, no audit process. But the underlying culture problem is that most engineering teams treat adding alerts as free and removing alerts as risky. Every post-mortem ends with 'add monitoring.' Nobody's post-mortem ever ends with 'delete the alert that's been crying wolf for six months.' That asymmetry is how you get a paging system that cries wolf 40 times a week and then fails to wake anyone up when the real incident hits.

The root cause isn't that teams care too little about monitoring — it's that they add alerts reactively, after every incident, without ever pruning the ones that stop being useful. Over time, the alert system becomes a noise machine. Engineers stop trusting it, start silencing pages, and miss the signals that actually matter. The fix isn't more dashboards. It's disciplined, intentional alerting philosophy backed by concrete practices for thresholds, routing, escalation, and rotation design.

By the end of this article you'll know how to audit your existing alert stack and identify the ones that don't serve you, how to write alerts that fire on symptoms not causes, how to structure an on-call rotation that doesn't erode your team's wellbeing, and how to use real tooling — Prometheus alerting rules, PagerDuty routing logic, and runbook templates — to make all of it operational and repeatable. The goal isn't a perfect alerting system. It's a system your engineers trust enough to actually pay attention to.

Why Silenced Services Hide Outages

Alerting and on-call best practices define how a team detects, responds to, and resolves system anomalies with minimal human latency. The core mechanic is a tiered escalation pipeline: alerts route from automated detection (e.g., p99 latency > 500ms for 5 minutes) to a primary on-call engineer, then to a secondary, and finally to incident management if unacknowledged within a defined timeout (commonly 15 minutes).

In practice, effective alerting hinges on three properties: signal-to-noise ratio, actionable content, and escalation speed. A good alert fires only when human intervention is required — not for transient blips. Each alert must include the failing component, the observed vs. expected value, and a runbook link. Escalation must be automatic and time-bound; manual handoffs introduce minutes of delay.

Use these practices in any production system where downtime costs exceed the overhead of maintaining on-call rotations. They matter most for services with strict SLOs (e.g., 99.9% uptime) or where a single engineer cannot know every subsystem. Without them, teams experience alert fatigue, missed critical pages, and prolonged mean-time-to-acknowledge (MTTA).

Alert on Symptoms, Not Causes — The Golden Rule of Monitoring

The most common alerting mistake is alerting on what you think is wrong instead of what the user actually experiences. A high CPU alert fires and the engineer investigates — but CPU being high isn't inherently bad. Maybe a batch job is running. Maybe it's expected load from a traffic spike the autoscaler is still catching up to. Maybe the GC is doing a full collection. The user doesn't care about CPU. They care whether the checkout page loads and their payment goes through.

Symptomatic alerting means your alert fires on things users feel directly: high latency, elevated error rates, failed health checks, degraded availability. Google's SRE book formalised this as the Four Golden Signals — latency, traffic, errors, and saturation — and the framing has held up well. Alerts on these signals are almost always actionable because if error rate is 15%, something is broken for users right now, and the on-call engineer has a clear starting point.

Causal metrics like CPU, memory, and disk are better suited for dashboards and capacity planning, not paging. You investigate them after a symptom alert fires to understand why something is wrong — not to decide whether something is wrong. The distinction matters because it changes the engineer's mental model during an incident. A symptom alert says 'users are experiencing this.' A causal alert says 'something in your infrastructure crossed a threshold.' Only one of those tells you whether to act.

The practical test: before adding any alert, ask yourself 'If this fires at 3 AM, can the on-call engineer take a concrete action within five minutes?' Not 'investigate' — act. If the answer is 'investigate and check some dashboards and maybe escalate', it belongs on a dashboard, not a pager. The 3 AM test is deliberately adversarial: the human responding is sleep-deprived, potentially unfamiliar with the specific service, and operating under pressure. Write your alerts for that human, not for a well-rested engineer on a Tuesday morning.

Here's the pushback you'll hear, usually from engineers who've been on the wrong end of missed incidents: 'But what about catching problems early?' This is a reasonable concern wrapped around a false premise. Causal metrics do catch problems early — on dashboards, during business hours, reviewed by engineers who have context and aren't panicking. You don't need to wake someone up to look at a graph trending in the wrong direction. Set up a daily dashboard review as part of your team's operational rhythm. If a pattern in causal metrics consistently precedes a symptom, write a symptom-based alert for the user-facing impact, not the infrastructure metric that correlates with it. The correlation is interesting. The symptom is the truth.

The Four Golden Signals aren't a checklist to run through mechanically. They're a lens for asking the right question: is what I'm about to alert on something a user would feel? Latency tells you how long users wait. Traffic tells you how much demand you're handling and whether demand patterns are normal. Errors tell you how often you're actively failing users. Saturation tells you how close to the edge you are before one of the other three degrades. Any proposed alert that doesn't map cleanly to one of these four should face an explicit justification before it gets merged.

# prometheus_symptom_alerts.yml
# These rules live in your Prometheus alerting rules directory.
# Reference them via the 'rule_files' block in prometheus.yml:
#   rule_files:
#     - /etc/prometheus/rules/*.yml
#
# Validate before applying:
#   promtool check rules /etc/prometheus/rules/prometheus_symptom_alerts.yml

groups:
  - name: user_facing_symptoms
    # Prometheus evaluates these rules every 60 seconds.
    # Shorter intervals increase Prometheus load without meaningfully
    # improving detection time for real incidents.
    interval: 60s
    rules:

      # ──────────────────────────────────────────────────────────
      # GOOD ALERT: Fires on what users actually experience
      # This is a SYMPTOM. Users are receiving 5xx responses.
      # The on-call engineer's action is unambiguous:
      # check error logs, look at recent deploys, investigate downstream.
      # ──────────────────────────────────────────────────────────
      - alert: HighErrorRateShopping
        expr: |
          (
            sum(rate(http_requests_total{
              service="shopping-cart",
              status=~"5.."
            }[5m]))
            /
            sum(rate(http_requests_total{
              service="shopping-cart"
            }[5m]))
          ) > 0.05
        # 'for: 2m' means Prometheus waits 2 minutes of sustained breach
        # before transitioning from 'pending' to 'firing'.
        # This single parameter eliminates the majority of false pages
        # from transient spikes and single bad scrape cycles.
        # Cost: you detect 2 minutes later. Benefit: far fewer 3 AM false alarms.
        for: 2m
        labels:
          severity: critical
          team: checkout
          # 'service' label is critical for Alertmanager routing AND
          # for inhibition rules — both depend on consistent service labels.
          service: shopping-cart
        annotations:
          summary: "Shopping cart error rate above 5% for 2 minutes"
          # The runbook_url annotation is non-negotiable.
          # Engineers responding to this alert should never have to search
          # for documentation during an active incident.
          runbook_url: "https://runbooks.internal/shopping-cart-errors"
          description: "Current error rate: {{ $value | humanizePercentage }}. Check recent deployments and downstream dependencies first."
          # dashboard_url lets the engineer jump directly to the relevant
          # Grafana panel without navigating the dashboard hierarchy.
          dashboard_url: "https://grafana.internal/d/checkout-overview?var-service=shopping-cart"

      # ──────────────────────────────────────────────────────────
      # GOOD ALERT: p99 latency degradation users feel
      # 5-minute 'for' duration is appropriate for latency —
      # a 30-second latency spike is often a cold cache or single
      # slow request. 5 minutes of sustained high p99 means
      # something structural has changed.
      # ──────────────────────────────────────────────────────────
      - alert: CheckoutLatencyHigh
        expr: |
          histogram_quantile(
            0.99,
            sum(rate(http_request_duration_seconds_bucket{
              service="checkout",
              handler="/api/purchase"
            }[10m])) by (le)
          ) > 2.0
        for: 5m
        labels:
          severity: warning
          team: checkout
          service: checkout
        annotations:
          summary: "p99 checkout latency exceeded 2s SLO threshold"
          runbook_url: "https://runbooks.internal/checkout-latency"
          description: "p99 latency is {{ $value | humanizeDuration }}. SLO threshold is 2.0s. Check database query times and downstream payment API latency."
          dashboard_url: "https://grafana.internal/d/checkout-latency"

      # ──────────────────────────────────────────────────────────
      # BAD ALERT — kept here intentionally as a contrast.
      # Do NOT uncomment this. CPU is a cause, not a symptom.
      #
      # Problems with this alert:
      # 1. 'What should the engineer DO?' — there is no clear action.
      #    Check CPU... then what? If users aren't affected, go back to sleep?
      # 2. It fires on batch jobs, GC pauses, autoscaling warmup,
      #    and any other expected load pattern.
      # 3. The threshold (80%) is arbitrary — why not 75%? Why not 90%?
      #    There is no principled answer because CPU% alone isn't the thing
      #    you actually care about.
      # ──────────────────────────────────────────────────────────
      # - alert: HighCPU
      #   expr: node_cpu_utilization > 0.80
      #   for: 1m
      #   annotations:
      #     summary: "CPU is high"
      #
      # If you inherited this alert, delete it today.
      # Add CPU to your Grafana dashboard instead.

Structuring On-Call Rotations That Don't Destroy Your Team

An on-call rotation is a social contract as much as it is a technical system. Engineers who feel the rotation is fair, predictable, and actively supported stay in it. Engineers who feel it's a punishment — or worse, an invisible tax that everyone pretends doesn't cost anything — churn. And the engineers who leave first are always the ones who have other options: the senior engineers, the ones who built the systems, the ones whose absence creates the next generation of undocumented incidents.

The fundamentals of a healthy rotation start with team size. You need at least four engineers to build a weekly rotation that gives people genuine recovery time between shifts. With three engineers, one person is always either on-call or just came off on-call — cognitive load never fully resets. With two, you're alternating weeks between two people and calling it a rotation. With one, you're not running a rotation at all; you're running a hero, and heroes burn out or leave.

Secondary on-call — a second engineer ready to be escalated to if the primary doesn't acknowledge within ten minutes — is non-negotiable for any service with a real SLA. It serves two purposes. The obvious one is redundancy: if the primary engineer is genuinely unavailable, the incident still gets coverage. The less obvious one is diagnostic: if escalation to secondary happens more than once per week, your primary alert volume is too high. The secondary escalation rate is a canary metric for rotation health that most teams never look at.

On-call handoff meetings deserve more ceremony than they typically receive. The outgoing engineer should document what fired, what was investigated, what commands were run, and what follow-up work was created. Without this, the same incidents repeat because the institutional knowledge of 'oh, that alert fires every Tuesday when the ETL job runs — just acknowledge and wait four minutes' lives in one engineer's head and evaporates with every rotation change. The handoff document is where that knowledge becomes team property.

Compensation matters — and how you structure it signals what you believe on-call is worth. Explicit on-call pay, time-off-in-lieu, or reduced sprint commitments during on-call weeks all communicate that the organisation understands the cost. The specifics matter less than the consistency and transparency.

But here's what I've observed across many teams: the psychological contract matters more than the compensation number. An engineer who receives $500 per on-call week but has no control over alert volume, whose feedback from retrospectives never changes anything, and who watches the same false alarms fire week after week without being fixed — that engineer feels trapped regardless of the pay. An engineer who receives time-off-in-lieu and can see, in every sprint, that the team is actively working to reduce alert noise and improve runbooks — that engineer feels respected. The best on-call programs are characterised by visible investment in reducing the burden, not just by compensating for it.

In 2026, most teams run distributed rotations spanning multiple time zones. This is both an opportunity and a design challenge. Done well, a distributed rotation means nobody carries the full 24-hour burden of a weekly shift — you can hand off at a natural boundary between regions and keep business-hours coverage for each timezone. Done poorly, it creates coordination overhead, unclear escalation paths when the person on-call is 9 time zones away, and handoff meetings that nobody can attend at a reasonable hour. If your team spans more than two time zones, design your rotation explicitly for the timezone distribution — don't just apply a single-timezone rotation template and hope the scheduling works out.

Rotation schedule changes are the most underestimated source of trust erosion. Once you publish a rotation, treat changes with the same communication discipline you'd apply to a production deployment. Engineers plan childcare, travel, and personal commitments around the schedule. A last-minute swap that affects a weekend isn't just inconvenient — it damages the sense that the rotation is a fair and predictable system, which is the only thing that makes it sustainable long-term.

# pagerduty_escalation_policy.tf
# Terraform resource definitions for a PagerDuty escalation policy
# and primary on-call rotation schedule.
#
# Prerequisites:
#   - PagerDuty Terraform provider configured:
#     terraform {
#       required_providers {
#         pagerduty = {
#           source  = "PagerDuty/pagerduty"
#           version = "~> 3.0"
#         }
#       }
#     }
#   - PAGERDUTY_TOKEN environment variable set
#
# Apply: terraform init && terraform apply

# ──────────────────────────────────────────────
# ESCALATION POLICY
# Three levels: Primary → Secondary → EM
# This is the standard for any SLA-backed production service.
# ──────────────────────────────────────────────
resource "pagerduty_escalation_policy" "checkout_team" {
  name      = "Checkout Team — Production Escalation"
  # num_loops: after completing all escalation levels with no acknowledgment,
  # restart from Level 1 this many times before giving up.
  # 2 loops = 2 full escalation cycles before the incident goes unacknowledged.
  num_loops = 2

  # LEVEL 1 — Primary on-call engineer
  # Gets paged first via push notification + SMS simultaneously.
  # 10 minutes is the standard industry acknowledgment window —
  # long enough to wake up, short enough to limit outage duration.
  rule {
    escalation_delay_in_minutes = 10

    target {
      type = "schedule_reference"
      id   = pagerduty_schedule.checkout_primary_rotation.id
    }
  }

  # LEVEL 2 — Secondary on-call engineer
  # Triggered when primary doesn't acknowledge within 10 minutes.
  # This should be a RARE event — if it happens weekly, audit alert volume.
  # Secondary rotation uses different engineers than primary
  # so the same person isn't primary + secondary simultaneously.
  rule {
    escalation_delay_in_minutes = 10

    target {
      type = "schedule_reference"
      id   = pagerduty_schedule.checkout_secondary_rotation.id
    }
  }

  # LEVEL 3 — Engineering Manager
  # This level should fire fewer than once per quarter in a healthy team.
  # If it fires weekly, the EM needs to be in the alert audit conversation,
  # not just at the end of the escalation chain.
  rule {
    escalation_delay_in_minutes = 15

    target {
      type = "user_reference"
      id   = data.pagerduty_user.checkout_engineering_manager.id
    }
  }
}

# ──────────────────────────────────────────────
# PRIMARY ON-CALL SCHEDULE
# Weekly rotation across 4 engineers minimum.
# Fewer than 4 engineers = rotation gaps and burnout risk.
# ──────────────────────────────────────────────
resource "pagerduty_schedule" "checkout_primary_rotation" {
  name      = "Checkout — Primary On-Call"
  time_zone = "America/New_York"

  layer {
    name = "Weekly Primary Rotation"
    # Rotation starts Monday 9 AM — handoff during business hours
    # means the incoming engineer can get context before night coverage.
    start                        = "2026-01-06T09:00:00-05:00"
    rotation_turn_length_seconds = 604800  # 7 days exactly
    rotation_virtual_start       = "2026-01-06T09:00:00-05:00"

    # 4 engineers minimum for a healthy weekly rotation.
    # Each engineer is primary once every 4 weeks.
    # With 4 people: 1 week on, 3 weeks off.
    # With 6 people: 1 week on, 5 weeks off — meaningful recovery time.
    users = [
      data.pagerduty_user.alice.id,
      data.pagerduty_user.bob.id,
      data.pagerduty_user.carol.id,
      data.pagerduty_user.david.id,
    ]
  }

  # For distributed teams spanning multiple time zones,
  # use multiple layers with restrictions to implement follow-the-sun:
  # layer 1: AMER engineers, active 9 AM - 6 PM EST
  # layer 2: EMEA engineers, active 9 AM - 6 PM GMT
  # layer 3: APAC engineers, active 9 AM - 6 PM SGT
  # Each layer has a 'restriction' block defining their active hours.
  # This keeps on-call within business hours for each region.
}

Runbooks, SLOs, and the Alert Audit Loop That Keeps You Sane

Every alert that fires should have a runbook. Not a wiki page describing the service architecture. Not a Confluence document that was last updated eighteen months ago. A runbook: a living, numbered document that tells the on-call engineer exactly what to check, what commands to run, and what decisions to make — optimised for a person who may be half-asleep, may not know this service deeply, and has about five minutes before stakeholders start asking questions.

The most common runbook failure is structure: engineers write runbooks like documentation. They lead with context — here's how this service works, here's its architecture, here's the dependency graph. That context belongs in the service's architecture docs. A runbook during an active incident needs to start with the thing the engineer does right now, not the context they'd need to understand the system from scratch. Lead with the first command they should run. Everything else is footnotes.

Runbooks don't need to be perfect on day one. The minimum viable runbook has three headings: 'Is this alert real?', 'Known mitigations', and 'When to escalate and who to call.' The first heading should have a specific command or dashboard link that lets the engineer confirm the alert reflects a real problem, not a metric collection glitch. The second should have the two or three mitigations that have worked historically, even if they're partial. The third should have an explicit escalation matrix — not 'contact the team' but 'if the database is unreachable, call the database team at this PagerDuty service; if the issue is the payment gateway, here's the vendor's emergency number.'

As incidents happen, the on-call engineer appends what they learned. Within three months of consistently following this pattern, you have battle-tested documentation that reflects reality rather than design intent. The gap between design intent and operational reality is where most runbooks fail — and closing that gap is what makes the difference between a runbook that helps and one that the engineer closes after 30 seconds because it doesn't match what they're seeing.

SLO-based alerting is a different philosophy entirely, and it's worth understanding the shift it requires. Instead of alerting when error rate exceeds 1% — an arbitrary threshold chosen by someone who had a reasonable gut feeling — you alert when you're consuming your monthly error budget faster than sustainable. This ties your paging directly to whether you're going to breach the reliability commitment you've made to your users. It dramatically reduces alert volume while ensuring that every page represents a genuine threat to that commitment.

The monthly alert audit is the most underrated practice in on-call operations. Pull a report of every alert that fired in the last 30 days. For each one: Was it actionable? Was there a documented response? If the same alert fired more than three times without a code fix being shipped, that's reliability debt — it belongs in the engineering backlog with a sprint assignment, not as a recurring 3 AM interruption that the team has collectively accepted as normal.

The audit meeting structure matters. Pull the data before the meeting — total alerts fired, percentage that resulted in an acknowledged incident with a documented response, mean time to acknowledge, and the ranked list of repeat offenders by firing frequency. Present it visually. The team's job during the meeting is to make three decisions about each alert on the list: keep it as is, fix the underlying condition that makes it fire too often, or delete it. Not discuss it — decide. Meetings that produce 'we should look into that' instead of 'deleted' or 'ticket created' waste their 30 minutes entirely.

The SLO frame changes the audit conversation in a valuable way. Instead of asking 'was this specific alert actionable?' you ask 'is our error budget on track this month?' If the budget is healthy and you have 80% remaining at the midpoint, many threshold alerts that fired are provably noise — they didn't threaten the SLO. If the budget is burning and you're at 40% at midpoint, you need more alerting sensitivity, not less. The budget number makes the decision about alert sensitivity a function of reliability risk rather than engineering anxiety.

# slo_burn_rate_alert.yml
# Multi-window burn rate alerting — the approach from Google SRE Workbook Chapter 5.
#
# PREREQUISITES:
# - A defined SLO: for this example, 99.9% availability (0.1% error budget)
# - The service must emit http_requests_total with a 'status' label
# - A 30-day SLO measurement window (most common in production)
#
# HOW BURN RATE WORKS:
# If your SLO is 99.9%, your monthly error budget is 0.1% of all requests.
# A burn rate of 1x means you're consuming budget at exactly the rate that
# exhausts it in 30 days. A burn rate of 14x means you'll exhaust 30 days
# of budget in ~52 hours. That's when you wake someone up immediately.
# A burn rate of 3x means you'll exhaust budget in ~10 days — serious,
# but you have time to investigate during business hours.
#
# WHY MULTI-WINDOW:
# Single-window: 1h burn rate > 14x fires on a 5-minute spike, then resolves.
# Multi-window: BOTH the 1h window AND the 5m window must exceed the threshold.
# A 5-minute spike that self-corrects won't sustain the 1h window above 14x.
# This eliminates transient false positives without meaningful detection delay.

groups:
  - name: slo_burn_rates
    rules:

      # ──────────────────────────────────────────────────────────
      # FAST BURN — 14x rate (budget exhausted in ~52 hours)
      # This is a drop-everything incident. Page immediately.
      # Multi-window: 1h (detects sustained fast burn) AND
      #               5m (confirms it's still happening right now).
      # ──────────────────────────────────────────────────────────
      - alert: CheckoutSLOFastBurn
        expr: |
          (
            (
              1 - (
                sum(rate(http_requests_total{
                  service="checkout",
                  status!~"5.."
                }[1h]))
                /
                sum(rate(http_requests_total{
                  service="checkout"
                }[1h]))
              )
            ) / 0.001
          ) > 14
          and
          (
            (
              1 - (
                sum(rate(http_requests_total{
                  service="checkout",
                  status!~"5.."
                }[5m]))
                /
                sum(rate(http_requests_total{
                  service="checkout"
                }[5m]))
              )
            ) / 0.001
          ) > 14
        # 'for' duration is deliberately short here — fast burn needs fast detection.
        # The multi-window expression does the noise filtering,
        # so a short 'for' doesn't create false positives.
        for: 2m
        labels:
          severity: critical
          alert_type: slo_burn
          service: checkout
        annotations:
          summary: "Checkout SLO: fast burn — 30-day error budget exhausts in ~52 hours at current rate"
          runbook_url: "https://runbooks.internal/checkout-slo-burn"
          description: |
            Burn rate is {{ $value | printf "%.1f" }}x sustainable.
            At this rate, your monthly error budget exhausts in approximately
            {{ printf "%.0f" (div 720.0 $value) }} hours.
            Immediate investigation required — this is SLA-threatening."
          dashboard_url: "https://grafana.internal/d/slo-error-budget?var-service=checkout"

      # ──────────────────────────────────────────────────────────
      # SLOW BURN — 3x rate (budget exhausted in ~10 days)
      # Serious but not drop-everything. Investigate today.
      # Multi-window: 6h (confirms sustained trend, not a spike) AND
      #               30m (confirms it's still happening, not historical).
      # ──────────────────────────────────────────────────────────
      - alert: CheckoutSLOSlowBurn
        expr: |
          (
            (
              1 - (
                sum(rate(http_requests_total{
                  service="checkout",
                  status!~"5.."
                }[6h]))
                /
                sum(rate(http_requests_total{
                  service="checkout"
                }[6h]))
              )
            ) / 0.001
          ) > 3
          and
          (
            (
              1 - (
                sum(rate(http_requests_total{
                  service="checkout",
                  status!~"5.."
                }[30m]))
                /
                sum(rate(http_requests_total{
                  service="checkout"
                }[30m]))
              )
            ) / 0.001
          ) > 3
        # Longer 'for' here — slow burn is a trend, not an event.
        # We want to be certain before alerting at warning severity.
        for: 15m
        labels:
          severity: warning
          alert_type: slo_burn
          service: checkout
        annotations:
          summary: "Checkout SLO: slow burn — 30-day error budget exhausts in ~10 days at current rate"
          runbook_url: "https://runbooks.internal/checkout-slo-burn"
          description: |
            Burn rate is {{ $value | printf "%.1f" }}x sustainable.
            Current trajectory exhausts monthly budget in approximately
            {{ printf "%.0f" (div 720.0 $value) }} hours.
            Investigate during business hours — this does not require waking anyone up."
          dashboard_url: "https://grafana.internal/d/slo-error-budget?var-service=checkout"

Alert Routing, Deduplication, and Suppression — The Plumbing That Makes It Work

Routing is where most alerting systems silently break in ways that are invisible until an incident proves it. Prometheus fires correctly. Alertmanager receives the alert. The alert routes to the wrong team — or to nobody at all — and the incident goes undetected. This failure is particularly dangerous because everything looks healthy: the alert fired, Alertmanager processed it, PagerDuty shows the service as active. The failure is in the gap between those systems, specifically in a label that's missing or mismatched.

Every alert label you add is implicitly a routing decision. The severity label determines which receiver handles the alert — PagerDuty for critical, Slack for warning. The team label determines which team's escalation policy is invoked. The service label enables inhibition rules and deduplication. Get any of these wrong and the alert either goes to the wrong destination or routes to Alertmanager's default receiver, which most teams configure as a catch-all Slack channel that nobody monitors at 3 AM.

Deduplication is Alertmanager's mechanism for not paging you multiple times for the same underlying problem. Two Prometheus instances — for example, a primary and a replica, or two regional scrape targets — will both fire the same alert when a metric breaches. Alertmanager groups them by label identity into a single notification. This is elegant when it works. The failure mode: two alerts that describe the same problem but have different labels — perhaps because one alert uses service="checkout" and another uses service="checkout-api" — are treated as separate alerts and generate separate pages. Label consistency across your alert rules is a more important operational practice than most teams realise, and it becomes critical when you have more than a handful of services.

Suppression windows are your surgical tool for planned maintenance. Unlike blanket silences — which are blunt instruments that suppress all alerts from a service regardless of type — proper suppression targets specific alert names for specific time windows. The rule is simple and should be enforced at the tooling level: every silence requires a comment with a linked ticket URL, and every silence auto-expires within 4 hours maximum. If your maintenance window is longer than 4 hours, renew the silence manually. This creates intentional checkpoints where someone has to actively decide that the silence should continue.

Inhibition rules solve a specific and common problem: when a critical alert fires, you don't want five additional warning alerts for the same service all generating pages simultaneously. Alertmanager inhibition rules suppress lower-severity alerts when a higher-severity alert is already active for the same service. The result is one page for one incident instead of five pages for five symptoms of the same root cause. This is the difference between an on-call engineer who wakes up to a single clear incident and one who wakes up to a flood of notifications that obscures which one to start with.

One important operational practice that often gets skipped: test your routing after any label change. When a service is renamed, when an alert rule is refactored, when a team restructuring changes the team label values — routing breaks silently. amtool config routes test with a set of representative alert labels takes about 90 seconds and catches misroutes before an incident does. Add it to your CI pipeline as a validation step any time alertmanager.yml or alert rule files change.

# alertmanager_routing.yml
# Load via: alertmanager --config.file=alertmanager.yml
#
# Validate before reloading:
#   amtool check-config alertmanager.yml
#
# Reload without restart (Alertmanager supports hot reload):
#   curl -X POST localhost:9093/-/reload

global:
  resolve_timeout: 5m
  pagerduty_url: 'https://events.pagerduty.com/v2/enqueue'

# ──────────────────────────────────────────────────────────────
# INHIBITION RULES
# When a critical alert is firing for a service,
# suppress all warning-severity alerts for the same service.
#
# Without this: a database outage causes 8 separate alerts
# (latency warning, error rate warning, availability warning, etc.)
# and the on-call engineer gets 8 pages for one incident.
# With this: they get 1 page — the critical alert that matters.
#
# 'equal' defines which labels must match for inhibition to apply.
# 'service' AND 'team' must both match to avoid cross-team suppression.
# ──────────────────────────────────────────────────────────────
inhibit_rules:
  - source_match:
      severity: critical
    target_match:
      severity: warning
    # Both labels must match — inhibiting checkout critical
    # should not suppress payments warning.
    equal: ['service', 'team']

# ──────────────────────────────────────────────────────────────
# ROUTE TREE
# Routes are evaluated top-to-bottom, first match wins.
# The 'default' route at the root catches anything that
# doesn't match a specific child route.
#
# Key timing parameters:
# group_wait:      How long to buffer alerts before sending the first notification.
#                  30s for default (batch similar alerts). 10s for critical (act fast).
# group_interval:  How long to wait before sending updates to an existing alert group.
# repeat_interval: How long to wait before re-notifying about an unresolved alert.
# ──────────────────────────────────────────────────────────────
route:
  receiver: default-slack
  group_by: ['alertname', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

  routes:
    # Critical severity → immediate PagerDuty page
    # Short group_wait (10s) minimises detection-to-page delay
    - match:
        severity: critical
      receiver: pagerduty-critical
      group_wait: 10s
      repeat_interval: 1h

    # SLO burn rate alerts → always page regardless of time of day
    # Separate route ensures SLO alerts can't be accidentally
    # caught by a warning-level route if someone misconfigures severity
    - match:
        alert_type: slo_burn
      receiver: pagerduty-critical
      group_wait: 10s
      repeat_interval: 1h

    # Warning severity → Slack notification, no page
    # Engineers review these during business hours
    - match:
        severity: warning
      receiver: slack-warnings
      group_wait: 5m
      repeat_interval: 4h

# ──────────────────────────────────────────────────────────────
# RECEIVERS
# Each receiver defines a notification integration.
# The PagerDuty receiver requires a service integration key
# unique to each service — do not share keys across services,
# it makes incident routing in PagerDuty impossible to untangle.
# ──────────────────────────────────────────────────────────────
receivers:
  - name: default-slack
    slack_configs:
      - channel: '#alerts-default'
        send_resolved: true
        title: '[{{ .Status | toUpper }}] {{ .GroupLabels.alertname }}'
        text: '{{ range .Alerts }}{{ .Annotations.summary }}{{ end }}'

  - name: pagerduty-critical
    pagerduty_configs:
      - service_key: '<YOUR_PAGERDUTY_INTEGRATION_KEY>'
        # Pass severity through to PagerDuty for UI triage
        severity: '{{ .GroupLabels.severity }}'
        # Include runbook and dashboard links in PagerDuty incident details
        details:
          runbook: '{{ (index .Alerts 0).Annotations.runbook_url }}'
          dashboard: '{{ (index .Alerts 0).Annotations.dashboard_url }}'

  - name: slack-warnings
    slack_configs:
      - channel: '#alerts-warnings'
        send_resolved: true
        title: '[WARNING] {{ .GroupLabels.alertname }}'
        text: '{{ range .Alerts }}{{ .Annotations.summary }} | {{ .Annotations.runbook_url }}{{ end }}'

# ──────────────────────────────────────────────────────────────
# SILENCE MANAGEMENT — use amtool, never the UI for production
#
# Good: time-bounded, ticket-linked, specific alert name
# amtool silence add \
#   alertname="CheckoutLatencyHigh" \
#   service="checkout" \
#   --duration=4h \
#   --comment="TICKET-1234: Planned database migration window"
#
# Bad: blanket silence, no ticket, long duration
# amtool silence add \
#   service="checkout" \
#   --duration=168h \
#   --comment="noisy"
# ↑ This is the pattern from the production incident above.
#   It hid a 4-hour outage. Never do this.
# ──────────────────────────────────────────────────────────────

The Noise Budget — Why Your Pager Should Be Quiet 90% of the Time

If your on-call engineer's phone buzzes more than once a shift, you've already lost. Every alert is a tax on focus. A noisy pager trains people to ignore it. That's how real outages slip through.

Here's the hard truth: Most alerts are noise. CPU at 80% isn't an emergency. A five-minute latency spike isn't an outage. Your monitoring tools are cheap; your engineer's attention isn't.

Set a noise budget. Calculate your team's tolerance — maybe 10 alerts per week per rotation. Anything above that gets triaged. Either automate the response (self-healing) or kill the alert. If it fires but never requires human action for 30 days, it's not an alert — it's a metric. Graph it, don't page on it.

Your goal: when the pager goes off, it's a goddamn emergency. Every time. If your on-call engineer can't remember the last time they got paged for a real issue, you've built trust. That's the only metric that matters.

// io.thecodeforge — devops tutorial

alerting_policy:
  name: production-noise-budget
  rotation: weekly
  budget:
    max_alerts_per_rotation: 10
    alert_types:
      - critical
      - warning
    exceptions:
      - incident_id: INC-2025-039  # sev-1 only
  auto_remediation:
    cpu_high:
      threshold: 85%
      action: scale_up_replicas
  dead_alert_cleanup:
    if_fires_without_ack: 7 days
    action: disable

The Handoff Protocol That Prevents the 3 AM Drop

The worst call you'll ever get is the one where the previous shift 'forgot' to mention the database connection pool is leaking. I've seen it. The on-call engineer who just got paged has no context, no runbook, and zero clue why the cluster is melting.

Stop treating handoffs like a handshake at a party. Make them surgical. Every shift change MUST include: a written summary of any ongoing issues, the current state of all active alerts, and a quick sync. No exceptions. If your team is distributed across time zones, write it in a shared doc — not Slack. Slack scrolls away. Docs don't.

Standardize. A handoff checklist — timestamp, alerts fired, actions taken, next steps. If you're not doing this, you're gambling. One missed detail can turn a 10-minute incident into a 2-hour post-mortem about process failure.

Here's the rule: The incoming engineer should be able to start debugging within 30 seconds of reading the handoff. If they can't, you failed. Automate the status dump — pull alert history, open incidents, and runbook state into a single handoff report. No excuses.

// io.thecodeforge — devops tutorial

handoff_protocol:
  required:
    - current_alert_count: 3
    - active_incidents:
        - INC-045: database_conn_pool_leak
    - actions_taken: "restarted primary replica, monitor for 30 mins"
    - next_steps: "check slow queries at 0600 UTC"
  automation:
    generate_report:
      trigger: shift_end
      sources:
        - pagerduty_incidents
        - runbook_current_state
      output: shared_drive/handoffs/2025-04-14.md
    sync_required: true  # 5-min Slack huddle or async doc