CloudWatch TreatMissingData — Why Silence Won't Page

Every production system breaks. The difference between a 5-minute outage and a 5-hour disaster? Almost always the same thing: how fast you knew something was wrong.

AWS CloudWatch is the native observability service at the heart of every serious AWS deployment. It's not glamorous, but skipping it is like flying a plane with no instruments. Fine until it isn't.

This article covers metrics, alarms, logs, and dashboards. How they connect. How to set them up with CLI and CloudFormation. And the three mistakes engineers make with CloudWatch that page them at 3 AM for no reason.

Why CloudWatch TreatMissingData Is the Difference Between a Pager and a Silent Night

CloudWatch TreatMissingData is a per-alarm configuration that defines how an alarm behaves when a metric stops reporting. The default behavior — 'missing' — treats the missing datapoint as a breach, but only if the alarm's evaluation period has already been in ALARM state. This means a metric that goes silent while the alarm is OK stays OK, which is almost never what you want. The core mechanic is simple: you choose one of four policies — missing, notBreaching, breaching, or ignore — and CloudWatch applies that policy to each missing datapoint during the alarm evaluation.

In practice, TreatMissingData matters because it decouples metric absence from alarm state. If you set it to 'breaching', any gap in data immediately triggers the alarm, which is useful for heartbeat metrics. If you set it to 'notBreaching', the alarm stays OK even if data stops, which is dangerous for critical metrics like CPU utilization or request latency. The 'ignore' option is rarely used because it excludes missing datapoints from the evaluation entirely, effectively shortening the evaluation period. The key property: TreatMissingData only applies when the metric is missing — not when it's NaN, null, or out of range.

Use TreatMissingData when you need to guarantee that a silent metric pages. The canonical example is a dead EC2 instance: if CloudWatch agent stops sending CPU metrics, you want the alarm to fire, not stay OK. Without explicit configuration, your alarm will remain OK until the next evaluation period finds a breach, which could be minutes or never. In production, always set TreatMissingData to 'breaching' for any alarm that monitors a critical health signal — otherwise, silence is treated as health.

CloudWatch Metrics: The Heartbeat of Your Infrastructure

A metric is just a time-stamped number with a name and a namespace. That's it. EC2 sends CloudWatch a CPUUtilization number every minute. RDS sends DatabaseConnections. Lambda sends Duration and Errors. These numbers stream in automatically — you don't write a single line of code to get them.

What makes metrics powerful is the dimension system. A dimension is a key-value pair that narrows down which resource a metric belongs to. For example, CPUUtilization by itself tells you nothing. CPUUtilization where InstanceId=i-0abc123 tells you exactly which server is melting. You can also publish your own custom metrics — think order count per minute, active WebSocket connections, or queue depth in your own application.

Metrics are stored in CloudWatch for 15 months, but the resolution degrades over time: data points are kept at 1-second resolution for 3 hours, then aggregated to 1-minute for 15 days, then 5-minute for 63 days, and finally 1-hour for 15 months. This matters when you're debugging an incident from three weeks ago — you'll only have 5-minute averages, not second-by-second data.

Knowing the retention and resolution schedule helps you set the right alarm evaluation periods and avoid false conclusions from aggregated data.

Custom metric cost trap: AWS charges $0.30 per custom metric per month. If you publish put-metric-data --dimensions Name=userId,Value=u-12345, each userId creates a separate billable metric. With 10,000 active users, that's $3,000 per month. Always use low-cardinality dimensions: Environment (prod/staging), Region, ServiceName, InstanceType. Never put unique identifiers in dimensions.

#!/bin/bash
# publish_custom_metric.sh
# Publishes a custom business metric to CloudWatch.
# Run this from an EC2 instance, a container, or your CI/CD pipeline.
# Prerequisite: AWS CLI installed and IAM role with cloudwatch:PutMetricData permission.

APP_NAME="checkout-service"
ENVIRONMENT="production"

# Simulate reading the number of orders processed in the last minute.
# In a real app you'd query your database or read from an in-memory counter.
ORDERS_PROCESSED=142

# Publish the custom metric to a namespace we own.
# Namespace acts like a folder — use a consistent naming convention.
aws cloudwatch put-metric-data \
  --namespace "TheCodeForge/${APP_NAME}" \
  --metric-name "OrdersProcessedPerMinute" \
  --value "${ORDERS_PROCESSED}" \
  --unit "Count" \
  --dimensions \
    "Name=Environment,Value=${ENVIRONMENT}" \
    "Name=AppName,Value=${APP_NAME}" \
  --region us-east-1

# Verify the metric was accepted (exit code 0 means success)
if [ $? -eq 0 ]; then
  echo "[OK] Metric published: ${ORDERS_PROCESSED} orders for ${APP_NAME} in ${ENVIRONMENT}"
else
  echo "[ERROR] Failed to publish metric. Check IAM permissions and region."
  exit 1
fi

CloudWatch Alarms: Turning Numbers Into Actions

A metric sitting in CloudWatch does nothing on its own. An alarm is what connects a metric to a response. You define a threshold, a comparison operator, and an evaluation period — and CloudWatch will flip the alarm state from OK to ALARM the moment that condition is met.

An alarm has exactly three states: OK (everything is fine), ALARM (threshold breached), and INSUFFICIENT_DATA (not enough data points have arrived yet, which happens right after you create an alarm or if the metric stops publishing). Understanding INSUFFICIENT_DATA is important — it's not the same as OK, and treating it that way is a common mistake.

Alarms can trigger three types of actions: SNS notifications (email, SMS, PagerDuty webhook), EC2 actions (stop, terminate, reboot, or recover an instance), and Auto Scaling actions (scale in or scale out). This is where CloudWatch becomes genuinely powerful — you can build self-healing infrastructure where an alarm automatically replaces a failing instance without any human involvement.

For composite alarms, you can combine multiple alarms with AND/OR logic. This lets you avoid alert fatigue by only paging someone when CPU is high AND error rate is also high — not when just one or the other spikes briefly.

The DatapointsToAlarm setting is your best defense against alert fatigue. If you set EvaluationPeriods=3 and DatapointsToAlarm=2, the alarm only fires if 2 out of 3 consecutive evaluation windows breach the threshold. A single 1-minute spike won't wake you at 3 AM. A sustained 3-minute problem will.

# cloudwatch_alarm.yaml
# CloudFormation template that creates a CloudWatch alarm on a Lambda function.
# Deploy with: aws cloudformation deploy --template-file cloudwatch_alarm.yaml \
#              --stack-name checkout-lambda-alarms --capabilities CAPABILITY_IAM

AWSTemplateFormatVersion: '2010-09-09'
Description: >-
  Alarm that fires when the checkout Lambda error rate exceeds 1% over 5 minutes.
  Sends an alert to the on-call SNS topic when triggered.

Parameters:
  LambdaFunctionName:
    Type: String
    Default: checkout-processor
    Description: The name of the Lambda function to monitor.

  OnCallSnsTopicArn:
    Type: String
    Description: ARN of the SNS topic that routes to PagerDuty or email.

Resources:

  # Alarm: triggers if Lambda errors exceed 5 in any 5-minute window.
  # EvaluationPeriods: how many periods must breach before alarm fires.
  # DatapointsToAlarm: of those periods, how many must actually breach.
  # Using 2-of-3 prevents a single noisy data point from waking someone at 3am.
  CheckoutLambdaErrorAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmName: !Sub '${LambdaFunctionName}-high-error-rate'
      AlarmDescription: >-
        Fires when checkout Lambda has more than 5 errors in a 5-minute period.
        Check Lambda logs in CloudWatch for stack traces.
      Namespace: AWS/Lambda                 # Built-in Lambda namespace — no setup needed
      MetricName: Errors
      Dimensions:
        - Name: FunctionName
          Value: !Ref LambdaFunctionName    # Scoped to our specific function
      Statistic: Sum                        # Add up all error counts in the period
      Period: 300                           # Each evaluation window is 5 minutes (300 seconds)
      EvaluationPeriods: 3                  # Look at the last 3 windows (15 minutes total)
      DatapointsToAlarm: 2                  # Alarm only if 2 out of 3 windows breach threshold
      Threshold: 5                          # More than 5 errors triggers the alarm
      ComparisonOperator: GreaterThanThreshold
      TreatMissingData: notBreaching        # No data = function not invoked = not an error
      AlarmActions:
        - !Ref OnCallSnsTopicArn            # Page on-call engineer via SNS
      OKActions:
        - !Ref OnCallSnsTopicArn            # Also notify when alarm recovers

Outputs:
  AlarmName:
    Description: Name of the created CloudWatch alarm
    Value: !Ref CheckoutLambdaErrorAlarm

CloudWatch Logs: Centralising and Querying Application Output

CloudWatch Logs is where your application output lives. The hierarchy works like this: a Log Group is the top-level container (one per application or service), Log Streams are individual sources within that group (one per Lambda invocation, one per EC2 instance, one per container), and Log Events are the individual timestamped lines inside each stream.

Logs arrive in CloudWatch either automatically (Lambda, ECS, and CloudTrail do this natively) or via the CloudWatch Logs Agent or the newer CloudWatch Agent installed on EC2 instances.

The real power is CloudWatch Logs Insights, a query language that lets you search and aggregate across gigabytes of logs in seconds. It's not SQL, but it's close enough that you'll feel at home immediately. You can filter for errors, extract fields from structured JSON logs, calculate percentiles, and visualise results as time-series charts.

Metric filters are another killer feature: they scan incoming log lines for a pattern and convert matches into CloudWatch metrics. This means you can turn a log line like ERROR: payment gateway timeout into an incrementing metric — and then alarm on that metric. No log aggregation pipeline, no third-party tool, no extra cost beyond the metric itself.

Logs Insights cost warning: You're charged per GB of data scanned ($0.005 per GB). A poorly written query that scans 100 GB costs $0.50. That's cheap for incident debugging. But a query that runs every minute as a dashboard widget will cost $720/month. Use Logs Insights for ad-hoc debugging only. For continuous metrics, use Metric Filters.

#!/bin/bash
# logs_insights_query.sh
# Runs a CloudWatch Logs Insights query to find the top 10 slowest
# API endpoints in the last hour, using structured JSON logs.
#
# Assumes your app logs JSON lines like:
# {"level":"INFO","endpoint":"/api/checkout","duration_ms":342,"status":200}

LOG_GROUP_NAME="/production/checkout-service"
QUERY_LOOKBACK_SECONDS=3600  # Last 1 hour

START_TIME=$(date -u -d "-${QUERY_LOOKBACK_SECONDS} seconds" +%s)  # Linux
# On macOS use: date -u -v -${QUERY_LOOKBACK_SECONDS}S +%s
END_TIME=$(date -u +%s)

echo "Starting Logs Insights query on: ${LOG_GROUP_NAME}"
echo "Time range: last ${QUERY_LOOKBACK_SECONDS} seconds"

# Start the query — Logs Insights runs asynchronously, so we get a query ID back.
QUERY_ID=$(aws logs start-query \
  --log-group-name "${LOG_GROUP_NAME}" \
  --start-time "${START_TIME}" \
  --end-time "${END_TIME}" \
  --query-string '
    fields @timestamp, endpoint, duration_ms, status
    | filter ispresent(duration_ms)          # Only include log lines that have this field
    | stats avg(duration_ms) as avg_duration,
            max(duration_ms) as max_duration,
            count() as request_count
      by endpoint
    | sort avg_duration desc                 # Slowest endpoints first
    | limit 10
  ' \
  --query 'queryId' \
  --output text \
  --region us-east-1)

echo "Query submitted. ID: ${QUERY_ID}"
echo "Waiting for results..."

# Poll until the query finishes (usually 2-10 seconds for an hour of logs)
while true; do
  STATUS=$(aws logs get-query-results \
    --query-id "${QUERY_ID}" \
    --query 'status' \
    --output text \
    --region us-east-1)

  if [ "${STATUS}" == "Complete" ]; then
    echo "Query complete. Results:"
    # Fetch and pretty-print the results
    aws logs get-query-results \
      --query-id "${QUERY_ID}" \
      --query 'results[*].[?field==`endpoint`].value | [0] | join(`,`, @)' \
      --output json \
      --region us-east-1
    break
  elif [ "${STATUS}" == "Failed" ]; then
    echo "[ERROR] Query failed. Check query syntax and log group name."
    exit 1
  fi

  sleep 2
done

Putting It Together: Dashboards and a Real Monitoring Architecture

A CloudWatch Dashboard is a customisable canvas where you pin metrics graphs, alarm states, and log query results side by side. The point isn't just pretty charts — it's reducing the time-to-understanding during an incident. When your on-call engineer gets paged at 2am, the first thing they open should be a dashboard that answers: is this an app problem, a database problem, or a network problem?

Good dashboard design follows the RED method: Rate (requests per second), Errors (error rate), and Duration (latency percentiles). Put those three graphs at the top. Below them, add the saturation metrics — CPU, memory, DB connections. At the bottom, link to Logs Insights queries for the most common failure modes.

Here's the architecture pattern that works in production: CloudWatch receives metrics and logs from all your services automatically. Alarms on the most critical thresholds fire to an SNS topic. That SNS topic routes to PagerDuty (or OpsGenie, or just email if you're early-stage). The on-call engineer opens the service dashboard, runs a Logs Insights query to get the stack trace, fixes the issue, and the OKAction on the alarm auto-resolves the incident. Everything is connected, traceable, and auditable.

This closed loop — metric to alarm to notification to dashboard to logs — is the entire CloudWatch mental model. Once you've internalised it, everything else is just configuration.

{
  "Comment": "CloudWatch Dashboard definition for the checkout service.",
  "Comment2": "Deploy with: aws cloudwatch put-dashboard --dashboard-name checkout-production --dashboard-body file://service_dashboard.json",
  "widgets": [
    {
      "type": "metric",
      "x": 0, "y": 0, "width": 8, "height": 6,
      "properties": {
        "title": "Request Rate — Checkout Lambda (invocations/min)",
        "view": "timeSeries",
        "stat": "Sum",
        "period": 60,
        "metrics": [
          [
            "AWS/Lambda",
            "Invocations",
            "FunctionName", "checkout-processor"
          ]
        ],
        "region": "us-east-1"
      }
    },
    {
      "type": "metric",
      "x": 8, "y": 0, "width": 8, "height": 6,
      "properties": {
        "title": "Error Rate — Checkout Lambda (errors/min)",
        "view": "timeSeries",
        "stat": "Sum",
        "period": 60,
        "metrics": [
          [
            "AWS/Lambda",
            "Errors",
            "FunctionName", "checkout-processor",
            { "color": "#d62728" }
          ]
        ],
        "region": "us-east-1",
        "annotations": {
          "horizontal": [
            {
              "label": "Alarm threshold",
              "value": 5,
              "color": "#ff7f0e"
            }
          ]
        }
      }
    },
    {
      "type": "metric",
      "x": 16, "y": 0, "width": 8, "height": 6,
      "properties": {
        "title": "P99 Latency — Checkout Lambda (ms)",
        "view": "timeSeries",
        "stat": "p99",
        "period": 60,
        "metrics": [
          [
            "AWS/Lambda",
            "Duration",
            "FunctionName", "checkout-processor"
          ]
        ],
        "region": "us-east-1"
      }
    },
    {
      "type": "alarm",
      "x": 0, "y": 6, "width": 24, "height": 2,
      "properties": {
        "title": "Active Alarms — Checkout Service",
        "alarms": [
          "arn:aws:cloudwatch:us-east-1:123456789012:alarm:checkout-processor-high-error-rate"
        ]
      }
    }
  ]
}

CloudWatch Agent: The Bridge Between Your Servers and Observability

Metrics and logs don't magically appear. You need the CloudWatch Agent. It's a daemon you install on EC2 or on-prem servers. It collects system-level metrics like memory, disk, and swap—stuff EC2 doesn't give you by default. It also pushes application logs. The old SSM Agent? That's for parameter store and commands. The CloudWatch Agent is for data. Install it. Configure it. Stop guessing why your CPU is pegged but memory looks fine. The WHY: Without the agent, you're blind to OS-level metrics. Your alarms fire late or never. Your logs sit on disks you can't query. The HOW: Drop the agent configuration JSON in Parameter Store, bootstrap it via user data. The agent picks up permissions from IAM. Use the unified CloudWatch agent—it replaced the old metrics-only version. Test your config with sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a status. If it's not running, neither are your alerts.

// io.thecodeforge
// CloudWatch Agent config: collects memory, disk, and app logs
{
  "agent": {
    "metrics_collection_interval": 60,
    "logfile": "/var/log/amazon-cloudwatch-agent.log"
  },
  "metrics": {
    "namespace": "Prod/WebServer",
    "metrics_collected": {
      "mem": {
        "measurement": ["mem_used_percent", "mem_available_percent"]
      },
      "disk": {
        "measurement": ["disk_used_percent"],
        "resources": ["/", "/data"]
      }
    }
  },
  "logs": {
    "logs_collected": {
      "files": {
        "collect_list": [
          {
            "file_path": "/var/log/application/error.log",
            "log_group_name": "/prod/web/error",
            "log_stream_name": "{instance_id}"
          }
        ]
      }
    }
  }
}

Logs Insights: Stop Grepping, Start Querying Your Way to Root Cause

You've got logs in CloudWatch. Now what? Grepping a thousand streams is for amateurs. Logs Insights is your SQL-for-logs. It queries across all log groups in a region. Syntax looks like SQL but isn't—it's purpose-built for log parsing. You fields @timestamp to get timestamps. You filter by error codes. You stats by bin(5m) to see time-based spikes. The WHY: When an alarm fires at 3 AM, you need to find the error pattern in seconds, not hours. Logs Insights paginates results—sort by @timestamp desc. Common pattern: filter @message like /(?i)(error|exception|timeout)/. Then stats count() by @timestamp, bin(1m). You'll spot the surge. Pro tip: Save queries as CloudWatch Logs Insights queries. Name them something searchable like "5xx errors last hour". Share them with your team. Your future self, paged at 2 AM, will thank you. Also: use limit 10000 to avoid timeouts on large log groups. Set a time range before running. Never run across all time—that's how you burn money and patience.

// io.thecodeforge
// Find HTTP 5xx errors in last 15 minutes, group by 1-minute buckets
fields @timestamp, @message
| filter @message like /HTTP\/[0-9\.]+ [45][0-9]{2}/
| filter @message like /5[0-9]{2}/
| stats count() as error_count by bin(1m)
| sort @timestamp desc
| limit 100