Cloud Monitoring: Metrics Explorer, Dashboards, and Uptime Checks

You've just pushed a new deployment. Five minutes later, your pager goes off—but it's not your app; it's your cloud bill. A misconfigured auto-scaling group spun up 50 instances, and you're burning $200 an hour. This is the reality of monitoring without guardrails. Cloud monitoring isn't about pretty charts; it's about survival. Metrics Explorer lets you query raw data, Dashboards give you at-a-glance health, and Uptime Checks catch failures before customers do. In this guide, I'll show you how to build a monitoring stack that actually prevents incidents, not just reports them. No fluff, just production-tested patterns.

The Three Pillars of Cloud Monitoring

Cloud monitoring breaks down into three core capabilities: Metrics Explorer, Dashboards, and Uptime Checks. Metrics Explorer is your raw data query interface—think of it as SQL for time-series data. Dashboards aggregate those metrics into visual snapshots for different stakeholders. Uptime Checks are synthetic probes that verify your endpoints are reachable and responding correctly. Each serves a distinct purpose, but they're most powerful when used together. For example, an Uptime Check triggers an alert, which leads you to Metrics Explorer to diagnose, and you pin the relevant chart to a Dashboard for postmortem. In production, you'll spend 80% of your time in Metrics Explorer, 15% in Dashboards, and 5% configuring Uptime Checks—but that 5% saves your bacon.

gcloud logging metrics list
# Or for AWS:
aws cloudwatch list-metrics --namespace AWS/EC2

Metrics Explorer: Querying Like a Pro

Metrics Explorer is your Swiss Army knife for time-series data. In Google Cloud, it's part of Cloud Monitoring; in AWS, it's CloudWatch Metrics Insights. The key is knowing how to write efficient queries. Start with the metric name, then filter by resource labels (e.g., instance_id, region), and finally apply aggregations (avg, sum, percentile). Avoid querying raw data for every instance—use group-by to reduce cardinality. For example, to find the average CPU across all instances in a zone: fetch gce_instance | metric 'compute.googleapis.com/instance/cpu/utilization' | group_by [zone], .mean(). This returns one time series per zone, not per instance. In production, you'll often combine metrics: join CPU with memory to detect memory pressure causing CPU spikes.

fetch gce_instance
| metric 'compute.googleapis.com/instance/cpu/utilization'
| filter resource.zone =~ 'us-central1-.*'
| group_by [resource.zone], .mean()
| every 1m

Building Dashboards That Tell a Story

Dashboards are not just eye candy—they're your operational command center. A good dashboard answers three questions: Is the system healthy? What's degrading? What changed? Structure your dashboard with a top-level health row (latency, error rate, throughput), then drill-down sections for each service. Use consistent time ranges and color coding: green for healthy, yellow for warning, red for critical. Avoid clutter—limit to 6-8 charts per dashboard. In production, you'll have separate dashboards for different audiences: SREs get raw metrics, managers get SLI summaries, and developers get per-service breakdowns. Use templating to make dashboards reusable across environments (dev, staging, prod).

{
  "displayName": "Production Service Health",
  "gridLayout": {
    "columns": "2",
    "widgets": [
      {
        "title": "Latency P99",
        "xyChart": {
          "dataSets": [{
            "timeSeriesQuery": {
              "timeSeriesFilter": "fetch http_latency | metric 'custom.googleapis.com/latency' | filter metric.environment = 'prod' | align percentile(99)"
            }
          }]
        }
      }
    ]
  }
}

Uptime Checks: Synthetic Monitoring That Works

Uptime Checks are HTTP(S), TCP, or UDP probes that verify your service is reachable from outside. They're your first line of defense against DNS failures, certificate expiry, and regional outages. Configure checks from multiple locations (at least 3) to avoid false positives from a single probe's network issue. Set reasonable timeouts (e.g., 10 seconds) and retry logic (e.g., 2 failures before alerting). For APIs, validate response content with regex or status codes. In production, use uptime checks for critical endpoints like /healthz, /readyz, and login flows. Don't check every endpoint—focus on user-facing paths.

name: my-service-uptime
httpCheck:
  path: /healthz
  port: 443
  useSsl: true
  validateSsl: true
  acceptedResponseStatusCodes:
    - statusClass: STATUS_CLASS_2XX
period: 60s
timeout: 10s
selectedRegions:
  - USA
  - EUROPE
  - ASIA_PACIFIC

Alerting: Turning Data into Action

Metrics and dashboards are passive; alerting is active. The goal is to notify the right person at the right time with enough context to act. Use threshold-based alerts for known failure modes (e.g., CPU > 90% for 5 minutes) and anomaly detection for unknown ones. Avoid alert fatigue by setting appropriate durations and using severity levels (critical, warning, info). In production, every alert must have a runbook. If an alert fires and no one knows what to do, it's noise. Use alerting policies that suppress duplicates and group related incidents. For example, if 10 instances fail, send one alert with a list, not 10 separate pages.

displayName: High CPU Alert
conditions:
  - conditionThreshold:
      filter: metric.type="compute.googleapis.com/instance/cpu/utilization"
      aggregations:
        - alignmentPeriod: 60s
          perSeriesAligner: ALIGN_MEAN
      duration: 300s
      comparison: COMPARISON_GT
      thresholdValue: 0.9
alertStrategy:
  notificationRateLimit:
    period: 3600s
documentation:
  content: "Runbook: https://runbooks.internal/cpu-spike"

Logs-Based Metrics: When Metrics Aren't Enough

Not everything is captured by infrastructure metrics. Application-level errors, slow SQL queries, and user behavior require logs-based metrics. These are custom metrics derived from log entries using filters. For example, count all log lines containing 'ERROR' in a service and create a metric for error rate. This gives you the same querying power as Metrics Explorer but for application data. In production, use logs-based metrics for business KPIs (e.g., signup rate, checkout failures) and for debugging transient issues that don't appear in standard metrics. Be careful with cardinality—don't create metrics with high-cardinality labels like user_id.

fetch global
| metric 'logging.googleapis.com/user/error_count'
| filter resource.labels.service = 'checkout'
| group_by [], .count()
| every 1m

Cost Management: Monitoring Your Monitoring

Cloud monitoring isn't free. Metrics ingestion, log storage, and API calls all cost money. A common pitfall is over-collecting: every custom metric, every log line, every trace. In production, you need to balance visibility with cost. Use sampling for logs (e.g., keep 10% of debug logs, 100% of errors). Set retention policies: 30 days for metrics, 7 days for logs, 90 days for audit logs. Use budget alerts to notify you if monitoring costs exceed a threshold. Also, avoid high-cardinality metrics—they're expensive to store and query. For example, don't create a metric with a label for every user; aggregate by region or tier instead.

# Estimate Cloud Monitoring costs
gcloud monitoring metrics list --filter="metric.type=~custom.*" | wc -l
# Check log storage size
aws logs describe-log-groups --query 'logGroups[*].[logGroupName,storedBytes]'

Incident Response: From Alert to Resolution

When an alert fires, the clock starts. Your monitoring setup should guide you from notification to root cause. First, acknowledge the alert. Then, open the relevant dashboard—it should show the affected service's health. Use Metrics Explorer to correlate metrics: did latency spike before errors? Did CPU increase? Check recent deployments or config changes. Use logs to find specific error messages. In production, have a runbook for each alert type. For example, 'High CPU' runbook: check for recent code deploy, check for traffic spike, check for memory leak. After resolution, update the runbook with new findings. This turns incidents into learning opportunities.

# Step 1: Acknowledge
echo "Acknowledged alert: High CPU on checkout service"
# Step 2: Open dashboard
echo "Open: https://monitoring.cloud.google.com/dashboard/checkout"
# Step 3: Query metrics
gcloud logging metrics list --filter="metric.type=~checkout.*"

Multi-Cloud and Hybrid Monitoring

If you run workloads across AWS, GCP, and on-prem, you need a unified monitoring strategy. Use a tool like Prometheus with Thanos or Grafana Cloud to aggregate metrics from all sources. For uptime checks, use a third-party provider (e.g., Pingdom, Checkly) that can probe from multiple locations regardless of cloud provider. The challenge is label consistency: ensure all resources have the same tags (e.g., service, environment, team). In production, we use a sidecar agent on each VM that exports metrics to a central Prometheus server. This avoids vendor lock-in and gives a single pane of glass.

scrape_configs:
  - job_name: 'aws_ec2'
    ec2_sd_configs:
      - region: us-east-1
        access_key: ${AWS_ACCESS_KEY}
        secret_key: ${AWS_SECRET_KEY}
    relabel_configs:
      - source_labels: [__meta_ec2_tag_Service]
        target_label: service

Advanced: SLOs, SLIs, and Error Budgets

Service Level Objectives (SLOs) are the target reliability you promise your users. Service Level Indicators (SLIs) are the metrics that measure it (e.g., latency, error rate). Error budgets are the allowable failure within a time window (e.g., 99.9% uptime allows 8.76 hours of downtime per year). Monitoring these requires precise metric definitions. For example, an SLI for latency might be 'proportion of requests under 200ms over a 5-minute window'. Use Metrics Explorer to compute SLIs, then track them on a dedicated dashboard. Alert when error budget burn rate exceeds a threshold (e.g., 10% per day). In production, SLOs drive engineering priorities: if error budget is low, freeze new features and focus on reliability.

fetch http_latency
| metric 'custom.googleapis.com/latency'
| filter metric.latency < 200
| align rate(5m)
| group_by [], .mean()

Automating Monitoring as Code

Manual monitoring setup is error-prone and unscalable. Use Infrastructure as Code (IaC) to define dashboards, alerts, and uptime checks. Tools like Terraform, Pulumi, or Google Cloud Deployment Manager can manage monitoring resources. For example, define an alert policy in Terraform and apply it across environments. This ensures consistency and version control. In production, we store monitoring configs in the same repo as application code and review changes via pull requests. This also enables automated testing: a CI job can validate that alerts have runbooks and dashboards don't have broken queries.

resource "google_monitoring_alert_policy" "cpu_alert" {
  display_name = "High CPU Alert"
  combiner     = "OR"
  conditions {
    display_name = "CPU > 90%"
    condition_threshold {
      filter     = "metric.type=\"compute.googleapis.com/instance/cpu/utilization\""
      duration   = "300s"
      comparison = "COMPARISON_GT"
      threshold_value = 0.9
    }
  }
  alert_strategy {
    notification_rate_limit {
      period = "3600s"
    }
  }
}

Testing Your Monitoring

Your monitoring is only as good as its last test. Regularly verify that alerts fire correctly, dashboards render, and uptime checks detect failures. Use chaos engineering to inject failures (e.g., kill a pod, block traffic) and confirm your monitoring catches it. Schedule monthly 'fire drills' where the on-call engineer responds to a simulated incident. Also, test that runbooks are up-to-date and accessible. In production, we have a CI job that deploys a test metric and checks that the alert triggers within 5 minutes. If it doesn't, the build fails.

# Simulate high CPU by running a stress test
stress --cpu 4 --timeout 60 &
# Wait for alert to fire
sleep 120
# Check alert history
gcloud alpha monitoring alerts list --filter="displayName=High CPU Alert"