GCP Alerting Policies: Notification Channels, Conditions, and Alert Fatigue

A single misconfigured alerting policy once caused my team to page 47 engineers at 3 AM for a 0.1% CPU spike that lasted 30 seconds. The noise buried a real database failover that went unnoticed for 12 minutes. Alerting in GCP isn't just about setting thresholds—it's about designing a system that respects human attention. Most teams either over-alert (fatigue) or under-alert (blind spots). This article cuts through the noise: you'll learn how to build notification channels that actually reach the right people, craft conditions that minimize false positives, and implement strategies like deduplication and escalation to keep your on-call sane. By the end, you'll have a production-ready alerting architecture that pages only when it matters.

Anatomy of a GCP Alerting Policy

A GCP Alerting Policy consists of three core components: conditions, notification channels, and documentation. Conditions define what triggers the alert—metric thresholds, log-based events, or absence of data. Notification channels specify where the alert goes: email, SMS, PagerDuty, Slack, or webhooks. Documentation is optional but critical for runbooks. Policies are evaluated every 60 seconds by default (configurable via evaluation frequency). Each policy can have multiple conditions (OR logic) and multiple notification channels. The key insight: conditions are evaluated independently, but notifications are deduplicated per policy. This means if two conditions fire simultaneously, you get one notification—not two. Understanding this lifecycle is step one to avoiding alert storms.

resource "google_monitoring_alert_policy" "high_cpu" {
  display_name = "High CPU Utilization"
  combiner      = "OR"
  conditions {
    display_name = "CPU > 80% for 5m"
    condition_threshold {
      filter     = "metric.type=\"compute.googleapis.com/instance/cpu/utilization\" AND resource.type=\"gce_instance\""
      duration   = "300s"
      comparison = "COMPARISON_GT"
      threshold_value = 0.8
      trigger {
        count = 1
      }
    }
  }
  notification_channels = [google_monitoring_notification_channel.email.name]
  alert_strategy {
    auto_close = "1800s"
  }
}

Notification Channels: Beyond Email

GCP supports email, SMS, PagerDuty, Slack, webhooks, Pub/Sub, and mobile push. Email is the default but should never be your primary channel for critical alerts—it's unreliable and slow. For production, use PagerDuty or Opsgenie for escalation and on-call scheduling. Slack is great for non-critical or informational alerts. Webhooks allow integration with custom tools (e.g., Jira, ServiceNow). Pub/Sub is ideal for event-driven architectures where alerts trigger automated remediation. Each channel has a rate limit: email is 10 per minute per policy, PagerDuty is unlimited but costs per incident. Pro tip: always configure at least two channels per policy—one primary (PagerDuty) and one fallback (email). This prevents missed alerts if the primary channel is down.

resource "google_monitoring_notification_channel" "pagerduty" {
  display_name = "PagerDuty Production"
  type         = "pagerduty"
  labels = {
    service_key = var.pagerduty_service_key
  }
}

resource "google_monitoring_notification_channel" "slack" {
  display_name = "Slack Alerts"
  type         = "slack"
  labels = {
    channel_name = "#alerts"
  }
  sensitive_labels {
    auth_token = var.slack_token
  }
}

Crafting Conditions That Don't Lie

Conditions are the heart of your alerting policy. GCP offers three types: metric threshold, metric absence, and log match. Metric threshold is the most common—fire when a metric crosses a value for a duration. The trap: using short durations (e.g., 60s) causes flapping. Always use at least 300s for production. Metric absence fires when data stops reporting—critical for dead instances. Log match fires when a log entry matches a filter—great for error rates. Advanced: use MQL (Monitoring Query Language) for complex conditions like ratio alerts (e.g., error rate / total requests). MQL allows sliding windows and custom aggregations. Avoid single-threshold alerts; use multiple thresholds with different severities (e.g., warning at 70%, critical at 90%).

fetch gce_instance
| metric 'logging.googleapis.com/user/error_count'
| align rate(1m)
| every 1m
| group_by [resource.instance_id], .sum()
| condition gt 0.05

Alert Fatigue: The Silent Killer

Alert fatigue occurs when too many alerts desensitize the on-call engineer. Symptoms: ignored pages, missed critical alerts, burnout. Root causes: overly sensitive thresholds, lack of deduplication, no severity levels. GCP provides tools to combat this: auto-close (automatically resolve alerts after a duration), notification rate limits (max notifications per minute), and alert grouping (combine related alerts). Best practice: implement a tiered severity system—P1 (critical, pages immediately), P2 (warning, pages during business hours), P3 (info, no page). Use auto-close for transient conditions. Also, use alerting policies with multiple conditions but a single notification channel to deduplicate. Finally, regularly review alert history and prune policies that never fire or always fire.

resource "google_monitoring_alert_policy" "cpu_warning" {
  display_name = "CPU Warning"
  combiner     = "OR"
  conditions {
    display_name = "CPU > 70% for 10m"
    condition_threshold {
      filter     = "metric.type=\"compute.googleapis.com/instance/cpu/utilization\" AND resource.type=\"gce_instance\""
      duration   = "600s"
      comparison = "COMPARISON_GT"
      threshold_value = 0.7
    }
  }
  alert_strategy {
    auto_close = "3600s"
    notification_rate_limit {
      period = "300s"
    }
  }
}

Log-Based Alerts: Catching the Unseen

Not all failures surface as metrics. Log-based alerts trigger when a log entry matches a filter—perfect for catching application errors, security events, or custom business logic failures. To create one, you first need a log-based metric (user-defined metric extracted from logs). Then create an alert policy on that metric. Example: alert on 500 errors exceeding 1% of total requests. Log-based alerts are powerful but can be noisy if logs are verbose. Always use a sliding window and aggregation to smooth out spikes. Also, beware of log ingestion delays—alerts may fire minutes after the event. For latency-sensitive cases, use metric-based alerts instead.

resource "google_logging_metric" "error_500" {
  name        = "error_500_count"
  filter      = "resource.type=\"gce_instance\" AND severity>=ERROR AND httpRequest.status=500"
  metric_descriptor {
    metric_kind = "DELTA"
    value_type  = "INT64"
    unit        = "1"
  }
}

resource "google_monitoring_alert_policy" "error_500_alert" {
  display_name = "500 Error Rate"
  combiner     = "OR"
  conditions {
    display_name = "500 errors > 10 in 5m"
    condition_threshold {
      filter     = "metric.type=\"logging.googleapis.com/user/error_500_count\""
      duration   = "300s"
      comparison = "COMPARISON_GT"
      threshold_value = 10
    }
  }
}

Escalation Policies: When the First Responder Doesn't Respond

GCP doesn't natively support escalation within alerting policies—you need an external tool like PagerDuty or Opsgenie. These tools integrate via webhook or direct channel. Configure escalation tiers: primary on-call, then secondary, then manager. Set timeouts (e.g., 5 minutes for P1, 15 for P2). If no acknowledgment, escalate. Also, implement 'heartbeat' alerts—if a service stops reporting, escalate immediately. Pro tip: use PagerDuty's 'auto-acknowledge' for low-severity alerts to reduce noise. For teams without PagerDuty, you can build custom escalation using Cloud Functions and Pub/Sub—but it's complex. Keep it simple: use a dedicated incident management tool.

{
  "routing_key": "YOUR_PD_SERVICE_KEY",
  "event_action": "trigger",
  "dedup_key": "alert-12345",
  "payload": {
    "summary": "CPU > 90% on instance-1",
    "severity": "critical",
    "source": "GCP Monitoring",
    "custom_details": {
      "instance": "instance-1",
      "metric": "cpu/utilization",
      "value": "0.95"
    }
  }
}

Documentation: The Unsung Hero

Every alert policy should include documentation—a runbook that tells the on-call engineer what to do. GCP allows attaching documentation in markdown. Include: what the alert means, possible causes, immediate steps, escalation contacts, and links to dashboards. Without documentation, even a perfect alert is useless—engineers waste time figuring out what to do. Pro tip: use variables in documentation like ${metric} and ${instance} to make it dynamic. Also, link to your incident response playbook. Documentation is stored in the policy and accessible via the GCP console or API. Make it a mandatory field in your Terraform templates.

resource "google_monitoring_alert_policy" "high_cpu_doc" {
  display_name = "High CPU with Runbook"
  documentation {
    content   = <<EOF
# High CPU Alert
- **Metric**: CPU utilization > 80% for 5m
- **Possible causes**: Traffic spike, memory leak, bad deployment
- **Steps**:
  1. Check dashboard: https://console.cloud.google.com/monitoring/dashboards/...
  2. SSH into instance: gcloud compute ssh ${resource.labels.instance_id}
  3. Run `top` and `htop` to identify process
  4. If deployment-related, rollback
- **Escalation**: Contact SRE team #sre-oncall
EOF
    mime_type = "text/markdown"
  }
  conditions { ... }
}

Testing Alerting Policies: Don't Wait for Production

Never deploy an alerting policy without testing it. GCP provides a 'Test' button in the console that simulates the condition. For metric thresholds, you can use the Metrics Explorer to see historical data and verify the threshold. For log-based alerts, use the Logs Explorer to check if your filter matches expected logs. Also, create a test policy that fires on a known metric (e.g., a custom metric you control) to validate the entire pipeline—from condition to notification. Automate testing with Terraform and a CI/CD pipeline: deploy to a test project, trigger the condition, and verify the alert arrives. This catches misconfigurations before they cause missed incidents.

#!/bin/bash
# Create a test metric and alert
PROJECT_ID="my-test-project"
METRIC_NAME="test_alert_metric"

# Write a test metric value
gcloud logging write $METRIC_NAME '{"message":"test","severity":"ERROR"}' --project=$PROJECT_ID

# Wait for metric to appear
sleep 60

# Check if alert fired (via monitoring API)
gcloud alpha monitoring incidents list \
  --filter="metric.type=logging.googleapis.com/user/$METRIC_NAME" \
  --project=$PROJECT_ID

Multi-Project Alerting: Centralized vs. Decentralized

In large organizations, you have multiple GCP projects. You can manage alerting per project (decentralized) or centrally from a single project using aggregated metrics. Decentralized is simpler but leads to duplication and inconsistency. Centralized uses a 'host project' that collects metrics from all projects via cross-project aggregation. To set up centralized alerting, enable 'global' metrics in the host project and create policies that filter by project. This reduces policy count and ensures consistent thresholds. However, centralized policies can't access project-specific labels easily. Compromise: use centralized for global alerts (e.g., overall error budget) and decentralized for project-specific (e.g., per-service CPU).

resource "google_monitoring_alert_policy" "global_cpu" {
  display_name = "Global CPU > 80%"
  combiner     = "OR"
  conditions {
    display_name = "CPU > 80% any project"
    condition_threshold {
      filter     = "metric.type=\"compute.googleapis.com/instance/cpu/utilization\" AND resource.type=\"gce_instance\" AND project_id=monitored_resource(\"project\")"
      duration   = "300s"
      comparison = "COMPARISON_GT"
      threshold_value = 0.8
      aggregations {
        alignment_period     = "60s"
        per_series_aligner   = "ALIGN_MEAN"
        cross_series_reducer = "REDUCE_MAX"
      }
    }
  }
}

Automating Alert Remediation with Cloud Functions

Alerts should trigger automated responses, not just pages. Use Cloud Functions or Cloud Run to listen to Pub/Sub notifications from alerting policies. For example, if a disk fills up, the function can run a cleanup script. Or if an instance is unhealthy, it can restart it. This reduces manual toil. To set up, create a Pub/Sub notification channel, then subscribe a Cloud Function to that topic. The function receives the alert payload and executes logic. Important: implement idempotency and rate limiting to avoid infinite loops. Also, log all actions for audit. Start with simple automations (e.g., restart VM) and gradually add complexity.

import base64
import json
from google.cloud import compute_v1

def restart_instance(event, context):
    """Restart a GCE instance when alerted."""
    pubsub_message = base64.b64decode(event['data']).decode('utf-8')
    alert = json.loads(pubsub_message)
    
    # Extract instance details from alert
    instance_name = alert['resource']['labels']['instance_id']
    zone = alert['resource']['labels']['zone']
    project = alert['resource']['labels']['project_id']
    
    client = compute_v1.InstancesClient()
    request = compute_v1.ResetInstanceRequest(
        project=project,
        zone=zone,
        instance=instance_name
    )
    client.reset(request)
    print(f"Restarted instance {instance_name} in {zone}")

Monitoring Alerting Health: Alert on Your Alerts

Your alerting system itself needs monitoring. What if a policy stops firing due to a configuration change? Or if notification channels fail? Use GCP's 'Alerting Policy Health' metrics: 'alert_policy/incident_count', 'alert_policy/notification_failure_count', and 'alert_policy/condition_evaluation_failures'. Create a meta-alert that pages the team if any policy has zero incidents for a week (possible misconfiguration) or if notification failures spike. Also, monitor the latency of alert delivery. This is your safety net. Without it, you might not know your alerts are broken until a real incident occurs.

fetch monitoring.googleapis.com/alert_policy
| metric 'alert_policy/notification_failure_count'
| align rate(1m)
| every 1m
| condition gt 0

Recommended Alerting Policies: Pre-Built Packages

GCP offers pre-built alerting policy packages for many services: Compute Engine, GKE, Cloud SQL, Cloud Load Balancing, and third-party integrations like MongoDB, Kafka, Elasticsearch. These packages include recommended alerting policies, sample dashboards, and key metrics. When you install a package, you can enable its recommended policies with one click, configure notification channels, and optionally modify thresholds. This is the fastest way to get baseline monitoring for a new service. Production insight: use recommended policies as a starting point, not the final configuration. The defaults are intentionally conservative (e.g., CPU > 80% for 5 minutes) and may need tuning for your workload. We enabled the GKE package's recommended alerts and immediately got paged for a transient memory spike. We adjusted the duration from 1 minute to 5 minutes—problem solved.

# Install recommended alerting policies via gcloud alpha
# (This feature is available in the GCP Console under Monitoring > Integrations)

# List available integrations
gcloud alpha monitoring integrations list

# Describe a specific integration (e.g., GKE)
gcloud alpha monitoring integrations describe gke

# Install GKE integration with recommended alerts (UI-based)
# Console: Monitoring > Integrations > GKE > Enable recommended alerts
# No direct gcloud command for this yet

Forecasted Metric-Value Alerting: Predict Problems Before They Happen

Cloud Monitoring supports forecasted metric-value alerting, which uses machine learning to predict when a metric will cross a threshold and alerts you before it happens. This is useful for metrics that follow predictable patterns, like disk usage growth or memory leak trends. Instead of waiting for the threshold to be breached, you get an alert when the forecast predicts a breach within a specified time window (e.g., within 24 hours). This gives you time to remediate proactively. Production insight: we use forecasted alerts for disk usage. A disk filling up at 2GB/day would breach the 80% threshold in 10 days. The forecasted alert pages us when the predicted breach is within 7 days—giving us time to clean up or resize. This reduced disk-related incidents by 90%.

fetch gce_instance
| metric 'compute.googleapis.com/instance/disk/percent_used'
| align next_older(1h)
| every 1h
| condition forecast(gt, 0.85, '24h')

Uptime Checks: External Monitoring for Your Services

Uptime checks verify that your service is reachable from outside GCP. They simulate user requests from multiple locations worldwide, checking HTTP, HTTPS, or TCP endpoints. You can configure uptime checks for load balancers, VM instances, or any public endpoint. When a check fails, it triggers an alerting policy. Uptime checks are critical for detecting region-level outages or DNS issues that internal metrics wouldn't catch. Production insight: we had a service that was healthy internally (all metrics green) but unreachable from certain regions due to a BGP issue. Only uptime checks from multiple regions caught this. Always configure uptime checks from at least 3 locations. Also, be aware of false positives from transient network issues—set the failure threshold to 3 consecutive failures before alerting.

resource "google_monitoring_uptime_check_config" "api_uptime" {
  display_name = "API Uptime Check"
  timeout      = "10s"
  period       = "60s"
  
  http_check {
    path         = "/healthz"
    port         = 443
    use_ssl      = true
    validate_ssl = true
  }
  
  monitored_resource {
    type = "uptime_url"
    labels = {
      host = "api.example.com"
    }
  }
  
  selected_regions = ["USA", "EUROPE", "ASIA_PACIFIC"]
}

resource "google_monitoring_alert_policy" "api_down" {
  display_name = "API Down"
  combiner     = "OR"
  conditions {
    display_name = "Uptime check failure"
    condition_threshold {
      filter     = "metric.type=\"monitoring.googleapis.com/uptime_check/check_passed\" AND resource.type=\"uptime_url\""
      duration   = "180s"
      comparison = "COMPARISON_LT"
      threshold_value = 1
    }
  }
}

Dynamic Severity Levels Using MQL

Instead of static severity labels (P1, P2, P3), you can use MQL to dynamically set severity based on the metric value. For example: severity is 'critical' if error rate > 10%, 'warning' if > 5%, 'info' otherwise. This reduces alert fatigue by routing less severe conditions to lower-priority channels without creating separate policies. Dynamic severity is configured in the MQL query using a case expression. Production insight: we use dynamic severity for latency alerts. p99 > 1s = P1 (page), p99 > 500ms = P2 (Slack), p99 > 200ms = P3 (dashboard). This gives us a single policy that scales severity with impact. Before dynamic severity, we had 3 separate policies with different thresholds and notification channels—maintenance was a nightmare.

fetch cloud_run_revision
| metric 'run.googleapis.com/request_latencies'
| filter resource.service_name == 'my-service'
| align percentile(99)
| every 1m
| condition
    case
      val() > 1.0 : 'critical'
      val() > 0.5 : 'warning'
      true        : 'info'
    end