Microsoft Azure — Azure Alerts & Action Groups

Azure is Microsoft's cloud computing platform offering over 200 services. This article covers azure alerts & action groups with production-ready configurations, best practices, and hands-on examples.

Why Azure Alerts and Action Groups Matter in Production

In production, monitoring is not optional—it's the difference between a 5-minute outage and a 5-hour firefight. Azure Alerts paired with Action Groups form the backbone of incident response. Alerts detect anomalies (metric thresholds, log queries, activity events) and trigger notifications or automated remediation. Action Groups define who gets paged, which webhook fires, or which ITSM ticket gets created. Without them, you're flying blind. I've seen teams lose SLAs because they relied on manual dashboard watching. Automate your alerting from day one. The cost of a missed alert in production far exceeds the effort to set up proper Action Groups.

{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "resources": [
    {
      "type": "Microsoft.Insights/metricAlerts",
      "apiVersion": "2018-03-01",
      "name": "CPU-Over-90-Percent",
      "location": "global",
      "properties": {
        "description": "Alert when CPU > 90% for 5 minutes",
        "severity": 1,
        "enabled": true,
        "scopes": [
          "[resourceId('Microsoft.Compute/virtualMachines', 'prod-vm-01')]"
        ],
        "evaluationFrequency": "PT1M",
        "windowSize": "PT5M",
        "criteria": {
          "allOf": [
            {
              "metricName": "Percentage CPU",
              "metricNamespace": "Microsoft.Compute/virtualMachines",
              "operator": "GreaterThan",
              "threshold": 90,
              "timeAggregation": "Average"
            }
          ]
        },
        "actions": [
          {
            "actionGroupId": "[resourceId('Microsoft.Insights/actionGroups', 'On-Call-Engineers')]"
          }
        ]
      }
    }
  ]
}

Anatomy of an Azure Alert Rule

An alert rule consists of a resource scope, condition (signal logic), severity, and action group. The signal can be a metric (e.g., CPU > 90%), a log query (e.g., errors > 100 in 5 min), or an activity log event (e.g., VM deletion). Severity ranges from 0 (Critical) to 4 (Verbose). Use severity 0-2 for actionable alerts; 3-4 for informational. The condition includes operator, threshold, aggregation granularity, and frequency of evaluation. For metric alerts, you can also use dynamic thresholds that adapt to historical patterns—useful for seasonal workloads. Always set a proper window size to avoid flapping. For example, a 5-minute window with 1-minute frequency means the alert fires only if the condition persists for 5 consecutive minutes.

# Create a metric alert for VM CPU
$resourceId = "/subscriptions/.../resourceGroups/prod-rg/providers/Microsoft.Compute/virtualMachines/prod-vm-01"
$actionGroupId = "/subscriptions/.../resourceGroups/prod-rg/providers/Microsoft.Insights/actionGroups/On-Call-Engineers"

$condition = New-AzMetricAlertRuleV2Criteria `
  -MetricName "Percentage CPU" `
  -TimeAggregation Average `
  -Operator GreaterThan `
  -Threshold 90

Add-AzMetricAlertRuleV2 `
  -Name "CPU-Over-90-Percent" `
  -ResourceGroupName "prod-rg" `
  -WindowSize (New-TimeSpan -Minutes 5) `
  -Frequency (New-TimeSpan -Minutes 1) `
  -TargetResourceId $resourceId `
  -Condition $condition `
  -ActionGroupId $actionGroupId `
  -Severity 1

Action Groups: The Nerve Center of Incident Response

Action Groups are collections of notification channels and actions triggered by an alert. They can include email, SMS, voice call, webhook, Azure Function, Logic App, ITSM connector, or automation runbook. Each action type has its own use case: email for non-critical, SMS/voice for on-call, webhook for integrating with PagerDuty or Slack. You can also define multiple actions in one group—e.g., email the team and trigger a webhook to create a Jira ticket. Action Groups support role-based access control (RBAC) so only authorized users can modify them. In production, always have at least two notification channels (e.g., email + SMS) to avoid single points of failure. Test your Action Group by sending a test alert from the Azure portal.

{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "resources": [
    {
      "type": "Microsoft.Insights/actionGroups",
      "apiVersion": "2021-09-01",
      "name": "On-Call-Engineers",
      "location": "Global",
      "properties": {
        "groupShortName": "oncall",
        "enabled": true,
        "emailReceivers": [
          {
            "name": "Team Email",
            "emailAddress": "oncall@contoso.com",
            "useCommonAlertSchema": true
          }
        ],
        "smsReceivers": [
          {
            "name": "Primary On-Call",
            "countryCode": "1",
            "phoneNumber": "555-1234"
          }
        ],
        "webhookReceivers": [
          {
            "name": "PagerDuty",
            "serviceUri": "https://events.pagerduty.com/integration/.../enqueue",
            "useCommonAlertSchema": true
          }
        ]
      }
    }
  ]
}

Log Alert Rules: Querying Your Way to Insight

Log alerts run Log Analytics queries at a defined frequency and fire when results meet a threshold. They're powerful for detecting patterns like error spikes, slow queries, or security events. The query can be over a fixed time range (e.g., last 5 minutes) and you can aggregate results (e.g., count > 100). You can also use dimensions to split alerts by resource or property. For example, alert when any VM has > 50 failed logins. Log alerts are charged per query execution, so optimize your queries to be efficient. Use the 'Number of results' or 'Metric measurement' alert logic. 'Metric measurement' allows grouping and alerting on each group independently—ideal for per-resource thresholds.

// Alert when any VM has > 50 failed logins in 5 minutes
let threshold = 50;
SigninLogs
| where TimeGenerated > ago(5m)
| where ResultType == "50074" // Failed login
| summarize FailedCount = count() by Identity, DeviceDetail.deviceId
| where FailedCount > threshold

Activity Log Alerts: Reacting to Azure Resource Changes

Activity log alerts fire when specific operations occur on Azure resources, such as VM creation, deletion, or security policy changes. They are essential for compliance and security monitoring. For example, alert when someone deletes a resource group or modifies a network security group. You can filter by resource type, operation name, status, and caller. Activity log alerts are free (no additional cost) and have near-real-time latency. They are ideal for detecting unauthorized changes or tracking critical operations. In production, I recommend creating activity log alerts for: resource deletion, role assignment changes, and security policy modifications. Combine with Action Groups to notify the security team immediately.

{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "resources": [
    {
      "type": "microsoft.insights/activityLogAlerts",
      "apiVersion": "2017-04-01",
      "name": "Resource-Group-Deleted",
      "location": "Global",
      "properties": {
        "enabled": true,
        "scopes": [
          "/subscriptions/..."
        ],
        "condition": {
          "allOf": [
            {
              "field": "category",
              "equals": "Administrative"
            },
            {
              "field": "operationName",
              "equals": "Microsoft.Resources/subscriptions/resourcegroups/delete"
            },
            {
              "field": "status",
              "equals": "Succeeded"
            }
          ]
        },
        "actions": {
          "actionGroups": [
            {
              "actionGroupId": "[resourceId('Microsoft.Insights/actionGroups', 'Security-Team')]"
            }
          ]
        }
      }
    }
  ]
}

Smart Groups and Alert Processing Rules

Smart Groups automatically correlate related alerts into a single incident, reducing noise. They group alerts that share the same resource, alert rule, or time window. For example, if a VM goes down, you might get alerts for CPU, memory, and disk—Smart Groups combine them. Alert Processing Rules allow you to modify alert behavior at scale. You can suppress alerts during maintenance windows, apply action groups to all alerts in a subscription, or filter alerts by severity. For example, suppress all non-critical alerts during a planned deployment. Use these to manage alert fatigue and ensure only meaningful alerts reach on-call engineers.

{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "resources": [
    {
      "type": "Microsoft.AlertsManagement/actionRules",
      "apiVersion": "2021-08-08",
      "name": "Maintenance-Window-Suppression",
      "location": "Global",
      "properties": {
        "scopes": [
          "/subscriptions/..."
        ],
        "conditions": {
          "severity": {
            "operator": "Equals",
            "values": ["Sev3", "Sev4"]
          }
        },
        "actions": [
          {
            "actionType": "Suppression",
            "suppressionConfig": {
              "recurrenceType": "Once",
              "schedule": {
                "startDate": "2026-07-12",
                "endDate": "2026-07-13",
                "startTime": "02:00:00",
                "endTime": "06:00:00"
              }
            }
          }
        ],
        "description": "Suppress non-critical alerts during maintenance window"
      }
    }
  ]
}

Automated Remediation with Action Groups

Action Groups can trigger automated responses via Azure Automation runbooks, Azure Functions, or Logic Apps. For example, if a VM's CPU is high, you can scale up the VM or restart a service. This reduces mean time to resolution (MTTR). To implement, create an Automation account with a runbook, then add it as an action in your Action Group. The alert payload includes context (resource, condition) that the runbook can use. Be cautious: automated actions can cause cascading failures. Always include a check (e.g., verify the VM is still running) and add a manual approval step for critical actions. Test in a non-production environment first.

param(
    [object]$WebhookData
)

$alertContext = $WebhookData.RequestBody | ConvertFrom-Json
$vmName = $alertContext.data.context.resourceName
$resourceGroup = $alertContext.data.context.resourceGroupName

# Restart the VM
Stop-AzVM -Name $vmName -ResourceGroupName $resourceGroup -Force
Start-AzVM -Name $vmName -ResourceGroupName $resourceGroup

# Log the action
Write-Output "Restarted VM $vmName due to alert"

Testing and Validating Your Alert Pipeline

Never assume your alerts work. Test them regularly. Azure provides a 'Test action group' feature that sends a sample alert to all configured actions. For metric alerts, you can manually trigger a condition (e.g., run a CPU stress test). For log alerts, run the query manually and verify it returns expected results. Also test the end-to-end pipeline: alert fires → Action Group triggers → notification received → automated remediation runs. Schedule quarterly 'fire drills' where you simulate an outage and verify the alerting chain. Document the test results and fix any gaps. In production, a broken alert is worse than no alert—it gives false confidence.

#!/bin/bash
# Stress CPU to trigger alert
apt-get update && apt-get install -y stress
stress --cpu 4 --timeout 120

Cost Management and Best Practices

Azure Alerts have a pricing model: metric alerts are free up to a limit, log alerts are charged per query execution, and action group notifications (SMS, voice) have per-message costs. To control costs: use metric alerts where possible (free), optimize log queries to reduce execution frequency, and limit SMS/voice to critical alerts only. Use action groups with email for non-critical alerts. Monitor your alert costs in Azure Cost Management. Also, avoid creating duplicate alerts on the same condition. Use alert processing rules to suppress noisy alerts. Finally, document your alert rules and action groups in a runbook so the team understands the monitoring strategy.

{
  "type": "Microsoft.Insights/metricAlerts",
  "apiVersion": "2018-03-01",
  "name": "Disk-Space-Low",
  "properties": {
    "description": "Alert when disk space < 10% for 15 minutes",
    "severity": 2,
    "enabled": true,
    "scopes": ["..."],
    "evaluationFrequency": "PT5M",
    "windowSize": "PT15M",
    "criteria": {
      "allOf": [{
        "metricName": "Percentage Free Space",
        "operator": "LessThan",
        "threshold": 10,
        "timeAggregation": "Minimum"
      }]
    },
    "actions": [{
      "actionGroupId": "..."
    }]
  }
}

Integrating with ITSM and Incident Management Tools

Action Groups can integrate with IT Service Management (ITSM) tools like ServiceNow, Jira, or PagerDuty via webhooks or the ITSM connector. The ITSM connector supports bidirectional sync: Azure creates a ticket in the ITSM tool, and updates from the tool (e.g., resolution) can close the alert in Azure. This is critical for organizations that follow ITIL processes. To set up, configure the ITSM connection in Azure Monitor, then select it as an action in your Action Group. For webhook-based integrations, use the common alert schema for consistent payloads. Test the integration by sending a test alert. In production, ensure your ITSM tool can handle the alert volume—otherwise, it becomes a bottleneck.

{
  "type": "Microsoft.Insights/actionGroups",
  "apiVersion": "2021-09-01",
  "name": "ServiceNow-Integration",
  "properties": {
    "groupShortName": "snow",
    "enabled": true,
    "itsmReceivers": [
      {
        "name": "ServiceNow",
        "workspaceId": "...",
        "connectionId": "...",
        "ticketConfiguration": "{\"AssignmentGroup\":\"CloudOps\"}",
        "region": "EastUS"
      }
    ]
  }
}

Monitoring Alert Health and Metrics

Alerts themselves need monitoring. Azure provides metrics for alert rules: fired, resolved, and action group success/failure rates. Use these to detect if an alert is not firing or if notifications are failing. Set up a 'canary' alert—a simple metric alert that fires periodically (e.g., every hour) to verify the pipeline works. Monitor the alert rule's health via Azure Monitor metrics or log queries. For example, query the AzureActivity table for alert rule failures. If an action group consistently fails (e.g., email bounces), you need to fix it. In production, I've seen silent failures where an alert fired but the SMS provider rejected the message. Always have a dashboard showing alert health.

// Check for action group failures in last 24 hours
AzureActivity
| where TimeGenerated > ago(1d)
| where OperationName == "Microsoft.Insights/actionGroups/write"
| where ActivityStatus == "Failure"
| project TimeGenerated, Caller, Resource, FailureReason

Putting It All Together: A Production Alerting Strategy

A robust alerting strategy combines all the pieces: define clear severity levels, use metric alerts for infrastructure, log alerts for application errors, activity log alerts for security, and action groups with multiple channels. Implement automated remediation for common issues (e.g., restart a service) but with safeguards. Use alert processing rules to suppress noise during maintenance. Test everything regularly. Document your alert rules and action groups in a runbook. Finally, review and tune alerts quarterly—remove stale ones, adjust thresholds, and add new ones as your system evolves. The goal is to have a system where every alert is actionable, and every action is reliable.

{
  "severityLevels": {
    "Sev0": "Critical - immediate human action required",
    "Sev1": "Error - needs attention within 15 minutes",
    "Sev2": "Warning - investigate within 1 hour",
    "Sev3": "Informational - no action needed"
  },
  "alertTypes": [
    {
      "type": "Metric",
      "examples": ["CPU > 90%", "Disk < 10% free"],
      "actionGroup": "On-Call-Engineers"
    },
    {
      "type": "Log",
      "examples": ["Error count > 100 in 5 min"],
      "actionGroup": "App-Team"
    },
    {
      "type": "Activity Log",
      "examples": ["Resource group deleted"],
      "actionGroup": "Security-Team"
    }
  ],
  "remediation": {
    "automated": ["Restart VM on high CPU"],
    "manual": ["Database failover requires approval"]
  }
}

Simple Log Search Alerts: Near Real-Time Log Alerting

Simple log search alerts are a newer alert type that evaluates each log row individually for near real-time detection. Unlike traditional log search alerts that run aggregations on a schedule, simple log alerts fire immediately when a log entry matches the defined criteria. This is ideal for security events, authentication failures, or critical error patterns that require immediate action. The evaluation frequency is near real-time (under 1 minute latency from ingestion). In production, use simple log alerts for scenarios like: a specific user fails login 5 times, a SQL injection pattern is detected, or a critical service returns HTTP 500. The trade-off is that you can't use aggregations like count() or summarize — each row is evaluated independently. For aggregated thresholds (e.g., more than 100 errors in 5 minutes), use traditional log search alerts instead.

// Simple log alert: fire on each failed admin login
SigninLogs
| where ResultType == "50074"
| where Identity contains "admin"
| project TimeGenerated, Identity, IPAddress, AppDisplayName, ResultDescription

Metric Alerts with Dynamic Thresholds: ML-Powered Anomaly Detection

Dynamic thresholds use machine learning to model historical metric patterns and alert only when behavior deviates from the expected baseline. Unlike static thresholds (e.g., CPU > 90%), dynamic thresholds adapt to seasonal patterns, growth trends, and day-of-week variations. This eliminates the tedious process of tuning thresholds for each resource. Configure sensitivity (high, medium, low) to control how much deviation triggers an alert. In production, dynamic thresholds are invaluable for metrics with variable baselines — like CPU on a website that gets 10x traffic on Mondays. A static threshold would either miss spikes or cause false alarms. Combine dynamic thresholds with multi-resource alerts to monitor an entire VM scale set or AKS cluster with a single rule. In production, we saw false positive reduction of 70% after switching from static to dynamic thresholds for our API gateway metrics.

# Create metric alert with dynamic threshold
$condition = New-AzMetricAlertRuleV2Criteria `
  -MetricName "Percentage CPU" `
  -TimeAggregation Average `
  -Operator GreaterThan `
  -DynamicThresholdSensitivity Medium

Add-AzMetricAlertRuleV2 `
  -Name "CPU-Anomaly-Detection" `
  -ResourceGroupName "prod-rg" `
  -WindowSize (New-TimeSpan -Minutes 15) `
  -Frequency (New-TimeSpan -Minutes 5) `
  -TargetResourceId $vmssId `
  -Condition $condition `
  -ActionGroupId $actionGroupId `
  -Severity 2

Managing Alert Rules as Code with Bicep and Terraform

Treating alert rules and action groups as infrastructure code ensures consistency, version control, and repeatability across environments. Use Bicep, ARM templates, or Terraform to define alert rules alongside your application infrastructure. Store alert definitions in Git and deploy via CI/CD pipelines. In production, we use a modular Bicep approach: each team owns a module that defines their alert rules, and a central platform team manages shared action groups and alert processing rules. This prevents duplicate alerts (multiple teams alerting on the same condition) and ensures critical alerts always reach the right people. Also use Azure Policy to audit that every critical resource has at least one alert rule configured, preventing coverage gaps. Deploy using deployment stacks or Azure DevOps multi-stage pipelines with validation gates.

// Bicep module for alert rule with action group
param vmName string
param actionGroupId string

resource vmAlert 'Microsoft.Insights/metricAlerts@2018-03-01' = {
  name: 'CPU-High-${vmName}'
  location: 'global'
  properties: {
    description: 'Alert when CPU > 80% for 10 minutes'
    severity: 2
    enabled: true
    scopes: [resourceId('Microsoft.Compute/virtualMachines', vmName)]
    evaluationFrequency: 'PT5M'
    windowSize: 'PT10M'
    criteria: {
      allOf: [{
        metricName: 'Percentage CPU'
        metricNamespace: 'Microsoft.Compute/virtualMachines'
        operator: 'GreaterThan'
        threshold: 80
        timeAggregation: 'Average'
      }]
    }
    actions: [{ actionGroupId: actionGroupId }]
  }
}