Microsoft Azure — Azure Monitor

Azure is Microsoft's cloud computing platform offering over 200 services. This article covers azure monitor with production-ready configurations, best practices, and hands-on examples.

Azure Monitor: The Observability Backbone

Azure Monitor is the single pane of glass for observability across Azure, on-premises, and other clouds. It collects, analyzes, and acts on telemetry from your applications and infrastructure. Unlike piecemeal logging solutions, Azure Monitor provides a unified platform for metrics, logs, traces, and alerts. You configure data sources once and route them to Log Analytics workspaces, Application Insights, or both. The key architectural decision is whether to use a single workspace for simplicity or multiple workspaces for isolation. In production, we've seen teams hit data ingestion limits because they didn't plan for scale. Always estimate your daily data volume and set up budgets and retention policies upfront. Azure Monitor's pricing is consumption-based, so uncontrolled ingestion can surprise you with a hefty bill. Start with a Log Analytics workspace in the same region as your resources to minimize egress costs.

az monitor log-analytics workspace create \
  --resource-group rg-monitoring \
  --workspace-name law-prod-001 \
  --location eastus \
  --sku PerGB2018 \
  --retention-time 30

Data Collection: Agents, SDKs, and Diagnostics

Azure Monitor collects data through three primary channels: agents (Azure Monitor Agent, Dependency Agent), SDKs (Application Insights SDK), and diagnostics settings (Azure Diagnostics Extension). The Azure Monitor Agent (AMA) is the modern replacement for the Log Analytics Agent and the Diagnostics Extension. It uses data collection rules (DCRs) to define what to collect and where to send it. For VMs, install AMA via Azure Policy to enforce consistent collection across your fleet. For applications, use the Application Insights SDK to capture requests, dependencies, exceptions, and custom metrics. The SDK supports auto-instrumentation for .NET, Java, Node.js, and Python. In production, we've seen teams miss critical telemetry because they forgot to enable dependency tracking. Always enable distributed tracing to correlate frontend requests with backend calls. For containers, use Container Insights, which collects stdout/stderr and performance counters from AKS clusters.

# Install Azure Monitor Agent on Windows VM using DSC
Configuration InstallAMA {
    Import-DscResource -ModuleName 'PSDscResources'
    Node localhost {
        Script InstallAzureMonitorAgent {
            GetScript  = { @{ Result = (Get-Service -Name 'AzureMonitorAgent' -ErrorAction SilentlyContinue).Status } }
            TestScript = { $false }
            SetScript  = {
                Invoke-WebRequest -Uri 'https://go.microsoft.com/fwlink/?linkid=2192883' -OutFile 'AzureMonitorAgent.msi'
                Start-Process msiexec.exe -Wait -ArgumentList '/i AzureMonitorAgent.msi /quiet'
            }
        }
    }
}
InstallAMA -OutputPath 'C:\DSC'

Log Analytics: Querying with KQL

Log Analytics is the query engine behind Azure Monitor. It uses Kusto Query Language (KQL) to search, filter, aggregate, and join data from multiple tables. Common tables include Perf (performance counters), Heartbeat (agent health), and AppRequests (Application Insights). KQL is case-sensitive and pipe-based. Start with simple filters: Perf | where CounterName == '% Processor Time' | where Computer == 'web-01'. Then aggregate: summarize avg(CounterValue) by bin(TimeGenerated, 5m). For production troubleshooting, learn to use let statements to define variables and functions. Use the union operator to combine data from multiple tables. The render operator creates timecharts. Always limit results with take or where TimeGenerated > ago(1h) to avoid scanning too much data. In production, we've seen queries that scan months of data and time out. Use time filters aggressively. Save common queries as functions or Azure Workbooks for reuse.

let threshold = 90;
Perf
| where TimeGenerated > ago(1h)
| where CounterName == "% Processor Time"
| where InstanceName == "_Total"
| summarize avgCpu = avg(CounterValue) by Computer, bin(TimeGenerated, 5m)
| where avgCpu > threshold
| project Computer, TimeGenerated, avgCpu
| order by TimeGenerated asc

Alerts: Actionable Notifications

Azure Monitor Alerts proactively notify you when conditions are met. There are three types: metric alerts (near-real-time on numeric values), log alerts (based on KQL queries), and activity log alerts (on Azure resource changes). For metric alerts, use dynamic thresholds to automatically adjust baselines based on historical patterns — this reduces noise from predictable spikes. For log alerts, set the frequency and time window carefully. A common mistake is using a 5-minute frequency with a 5-minute window, which can cause duplicate alerts. Use a 1-minute frequency with a 5-minute window for faster detection. Always configure action groups with multiple channels: email, SMS, and webhook to trigger automation (e.g., Azure Functions or Logic Apps). In production, we've seen alerts that fire but no one acts on them because the action group is misconfigured. Test your alerts by manually triggering them during a maintenance window. Use alert processing rules to suppress alerts during planned maintenance.

az monitor metrics alert create \
  --name "High CPU Alert" \
  --resource-group rg-monitoring \
  --scopes /subscriptions/.../resourceGroups/rg-prod/providers/Microsoft.Compute/virtualMachines/web-01 \
  --condition "avg Percentage CPU > 90" \
  --window-size 5m \
  --evaluation-frequency 1m \
  --action /subscriptions/.../resourceGroups/rg-monitoring/providers/microsoft.insights/actionGroups/ag-prod \
  --description "Alert when CPU exceeds 90% for 5 minutes"

Application Insights: End-to-End Tracing

Application Insights (part of Azure Monitor) provides application performance monitoring (APM) with distributed tracing, dependency mapping, and user analytics. Instrument your app with the Application Insights SDK or enable auto-instrumentation via the Azure portal for supported runtimes. The SDK captures requests, dependencies (HTTP, SQL, Azure services), exceptions, and custom events. Use the built-in sampling to reduce data volume while preserving trace integrity. In production, set adaptive sampling to automatically adjust based on traffic. For high-throughput apps, use fixed-rate sampling (e.g., 10%). The end-to-end transaction view shows the full call chain from the browser to the database. Use the Application Map to visualize dependencies and identify bottlenecks. In production, we've seen a single slow SQL query cause cascading timeouts across microservices. The Application Map made it obvious.

using Microsoft.ApplicationInsights;
using Microsoft.ApplicationInsights.Extensibility;

var builder = WebApplication.CreateBuilder(args);

// Add Application Insights
builder.Services.AddApplicationInsightsTelemetry(builder.Configuration["APPLICATIONINSIGHTS_CONNECTION_STRING"]);

var app = builder.Build();

app.MapGet("/api/orders/{id}", async (int id, TelemetryClient telemetry) =>
{
    using var operation = telemetry.StartOperation<RequestTelemetry>($"GET /api/orders/{id}");
    try
    {
        // Simulate database call
        await Task.Delay(100);
        operation.Telemetry.Success = true;
        return Results.Ok(new { Id = id, Status = "Shipped" });
    }
    catch (Exception ex)
    {
        operation.Telemetry.Success = false;
        telemetry.TrackException(ex);
        return Results.Problem("Internal server error");
    }
});

app.Run();

Workbooks: Custom Dashboards and Reports

Azure Monitor Workbooks provide a flexible canvas for creating interactive dashboards and reports. They combine text, queries, metrics, and parameters into a single document. Workbooks are built with KQL queries, and you can add parameters like time range, resource picker, or dropdowns to make them interactive. Use workbooks for operational dashboards (e.g., 'VM Performance Overview'), incident response playbooks (e.g., 'SQL Database Troubleshooter'), or compliance reports. Workbooks support multiple visualization types: time charts, grids, tiles, and maps. You can export workbooks as JSON and version-control them in Git. In production, we created a 'Deployment Health' workbook that shows error rates, latency, and deployment markers. It helped us correlate a spike in 500 errors with a recent deployment. Workbooks can also be pinned to Azure Dashboards for at-a-glance monitoring.

{
  "version": "Notebook/1.0",
  "items": [
    {
      "type": 1,
      "content": {
        "json": "## VM Performance Dashboard"
      },
      "name": "text - 0"
    },
    {
      "type": 3,
      "content": {
        "version": "KqlItem/1.0",
        "query": "Perf\n| where TimeGenerated > ago(1h)\n| where CounterName == \"% Processor Time\"\n| summarize avgCpu = avg(CounterValue) by Computer, bin(TimeGenerated, 5m)\n| render timechart",
        "size": 0,
        "timeContext": {
          "durationMs": 3600000
        }
      },
      "name": "query - 1"
    }
  ]
}

Autoscale: Dynamic Resource Management

Azure Monitor integrates with Azure Autoscale to automatically adjust the number of instances based on metrics. You can scale out (add instances) when CPU exceeds a threshold and scale in (remove instances) when it drops. Autoscale works with Virtual Machine Scale Sets, App Service Plans, and Azure Spring Apps. Configure scale rules with a cool-down period (default 5 minutes) to avoid flapping. Use multiple rules for different metrics (e.g., CPU and queue length). In production, we've seen autoscale fail because the scale-out rule was too aggressive, causing a thundering herd of new VMs that overwhelmed the database. Always set a maximum instance limit to prevent runaway costs. Use predictive autoscale (preview) to anticipate demand based on historical patterns. Test autoscale with a load test before going live. Monitor autoscale events in the Activity Log to detect failures.

az monitor autoscale create \
  --resource-group rg-prod \
  --resource webapp-orders \
  --resource-type Microsoft.Web/sites \
  --count 2 \
  --min-count 2 \
  --max-count 10 \
  --name "WebApp Autoscale"

az monitor autoscale rule create \
  --resource-group rg-prod \
  --autoscale-name "WebApp Autoscale" \
  --scale out 1 \
  --condition "Percentage CPU > 75 avg 5m" \
  --cooldown 5

az monitor autoscale rule create \
  --resource-group rg-prod \
  --autoscale-name "WebApp Autoscale" \
  --scale in 1 \
  --condition "Percentage CPU < 25 avg 5m" \
  --cooldown 10

Cost Management: Monitoring and Budgets

Azure Monitor itself generates costs based on data ingestion, retention, and alert evaluations. Use the Azure Monitor pricing calculator to estimate costs. Set a daily ingestion cap on your Log Analytics workspace to prevent unexpected bills. When the cap is reached, data collection stops until the next day. Monitor your usage with the Usage and Estimated Costs workbook. Create budgets in Azure Cost Management to alert when spending exceeds thresholds. In production, we've seen a team accidentally enable collection of all performance counters (hundreds) on thousands of VMs, leading to a $50,000 monthly bill. Limit collection to essential counters only. Use basic logs (low-cost, no analytics) for verbose debug logs that you rarely query. Archive old logs to Azure Storage or export to Event Hubs for long-term retention. Regularly review your data collection rules and remove unused ones.

Usage
| where TimeGenerated > ago(30d)
| where IsBillable == true
| summarize BillableDataGB = sum(Quantity) / 1000. by DataType, bin(TimeGenerated, 1d)
| project DataType, Date = TimeGenerated, BillableDataGB
| order by Date asc, DataType asc

Integration with ITSM and Automation

Azure Monitor integrates with IT Service Management (ITSM) tools like ServiceNow, Jira, and PagerDuty via ITSM Connector. When an alert fires, it can automatically create an incident in your ITSM tool. Configure the connector with your tool's API endpoint and authentication. For automation, use Azure Automation runbooks or Logic Apps triggered by alerts. For example, when a VM is unresponsive, a runbook can restart it. Or when disk space is low, a Logic App can run a cleanup script. In production, we've seen automated remediation cause more harm than good if not carefully scoped. Always add a manual approval step for destructive actions (e.g., restarting a production database). Use Azure Policy to enforce that all alerts have an action group configured. Test your automation in a non-production environment first. Monitor the automation itself for failures.

param(
    [string]$VMName,
    [string]$ResourceGroupName
)

$connection = Get-AutomationConnection -Name AzureRunAsConnection
Connect-AzAccount -ServicePrincipal -Tenant $connection.TenantID -ApplicationId $connection.ApplicationID -CertificateThumbprint $connection.CertificateThumbprint

Write-Output "Restarting VM $VMName in resource group $ResourceGroupName"
Restart-AzVM -Name $VMName -ResourceGroupName $ResourceGroupName -Force
Write-Output "VM restart initiated."

Advanced: Custom Metrics and Logs

Beyond built-in metrics, you can send custom metrics and logs to Azure Monitor using the REST API or SDKs. Custom metrics are useful for business KPIs (e.g., number of orders per minute) or application-specific performance indicators. They are stored in the same metrics database and can trigger alerts. Use the Azure Monitor Metrics REST API to push custom metrics. For custom logs, use the Log Analytics HTTP Data Collector API. Define a custom table with a schema that matches your data. In production, we've seen teams push too many custom metrics (thousands of time series) causing performance degradation and high costs. Limit custom metrics to a few hundred. Use dimensions to add context (e.g., region, customer tier) instead of creating separate metrics. For logs, use a structured format (JSON) and include a timestamp. Set a retention policy on custom tables to avoid indefinite storage.

import requests
import json
from datetime import datetime

# Azure Monitor custom metric endpoint
url = "https://<region>.monitoring.azure.com/<subscription-id>/resourceGroups/<rg>/providers/Microsoft.Insights/metrics"

headers = {
    "Authorization": "Bearer <access-token>",
    "Content-Type": "application/json"
}

body = {
    "time": datetime.utcnow().isoformat() + "Z",
    "data": {
        "baseData": {
            "metric": "OrdersProcessed",
            "namespace": "ecommerce",
            "dimNames": ["region"],
            "series": [
                {
                    "dimValues": ["us-east"],
                    "min": 100,
                    "max": 200,
                    "sum": 1500,
                    "count": 10
                }
            ]
        }
    }
}

response = requests.post(url, headers=headers, data=json.dumps(body))
print(response.status_code)

Security and Compliance Monitoring

Azure Monitor plays a key role in security monitoring by collecting security logs from Azure resources. Enable diagnostic settings to send Azure Activity Logs, NSG flow logs, and Azure AD logs to Log Analytics. Use Azure Sentinel (built on Log Analytics) for SIEM capabilities. Create alerts for suspicious activities like multiple failed logins, unusual geo-locations, or changes to critical resources. In production, we've seen security alerts that were ignored because they were too noisy. Tune your security alerts by using KQL to filter out known benign patterns. For compliance, use Azure Policy to enforce that diagnostic settings are enabled on all resources. Export logs to a separate workspace for security team access. Monitor for configuration changes that weaken security posture (e.g., opening a port to the internet). Use the Azure Security Center integration to get recommendations based on collected data.

SigninLogs
| where TimeGenerated > ago(1h)
| where ResultType != 0
| summarize FailedAttempts = count() by UserPrincipalName, IPAddress, bin(TimeGenerated, 5m)
| where FailedAttempts > 10
| project UserPrincipalName, IPAddress, TimeGenerated, FailedAttempts

Troubleshooting Common Production Issues

Even with proper monitoring, issues arise. Common problems include: missing data (agent not sending), high latency (query performance), and alert storms (misconfigured thresholds). For missing data, check the Heartbeat table: if a VM hasn't sent a heartbeat in 5 minutes, the agent may be down. Use the Azure Monitor Agent Health workbook to see agent status. For slow queries, use the 'Query Performance' blade in Log Analytics to identify expensive queries. Add indexes on frequently filtered columns (e.g., TimeGenerated). For alert storms, review alert history and adjust thresholds or use alert processing rules. In production, we've seen a single misconfigured alert generate 10,000 emails in an hour. Always set a cooldown on alerts. Use 'smart groups' to group related alerts. For Application Insights, check sampling rates: if sampling is too aggressive, you may miss rare errors. Use fixed-rate sampling for critical services.

Heartbeat
| where TimeGenerated > ago(10m)
| summarize LastHeartbeat = max(TimeGenerated) by Computer
| where LastHeartbeat < ago(5m)
| project Computer, LastHeartbeat, Status = "Missing"

Managed Prometheus and Azure Managed Grafana: Cloud-Native Metrics

Azure Monitor managed service for Prometheus provides a fully managed, Prometheus-compatible monitoring solution for Kubernetes and virtual machines. It collects metrics via the Azure Monitor Agent with a Prometheus scraper and stores them in an Azure Monitor workspace. Use PromQL queries and Prometheus alert rules for real-time monitoring. Azure Managed Grafana provides a fully managed Grafana instance with prebuilt dashboards for Kubernetes, VMs, and Azure services. Connect Grafana to your Azure Monitor workspace to visualize Prometheus metrics alongside Azure Monitor metrics and logs. For AKS clusters, enable managed Prometheus during cluster creation (default in AKS Automatic) or via Azure Policy. Prebuilt Grafana dashboards include: Kubernetes Compute Resources (Cluster, Namespace, Pod, Node), Kubernetes Networking, Kubelet, Node Exporter, and USE Method dashboards. For Azure VMs, use the OpenTelemetry-based metrics experience with prebuilt dashboards for platform metrics, detailed metrics, and process monitoring. To reduce costs, use the cost optimization settings in Container Insights to control which tables are collected — disable verbose logs and focus on performance counters and Prometheus metrics. For cross-cloud visibility, connect self-hosted Grafana or Grafana Cloud to Azure Monitor workspaces as a data source.

# Create an Azure Monitor workspace
az monitor account create \
  --name amw-prod-001 \
  --resource-group rg-monitoring \
  --location eastus

# Link Azure Managed Grafana workspace (create first)
az grafana create \
  --name grafana-prod \
  --resource-group rg-monitoring \
  --location eastus \
  --zone-redundancy Enabled

# Add Azure Monitor workspace as data source to Grafana
az grafana data-source create \
  --name grafana-prod \
  --definition '{"name":"Azure Managed Prometheus","type":"prometheus","url":"https://amw-prod-001.eastus.prometheus.monitor.azure.com","access":"proxy"}'

Container Insights: Deep Monitoring for AKS Clusters

Container Insights provides comprehensive monitoring for AKS and Arc-enabled Kubernetes clusters. It collects stdout/stderr logs, Kubernetes events, performance metrics, and inventory data using a containerized Azure Monitor Agent. Data is stored in a Log Analytics workspace for KQL queries and in Azure Monitor metrics for near-real-time alerting. For AKS Automatic, Container Insights is enabled by default. For AKS Standard, enable it during cluster creation or via the Monitoring tab. Key capabilities: monitor node and pod health, view container logs in context, analyze resource usage (CPU, memory, disk, network), and correlate with Prometheus metrics from Managed Prometheus. Use the cost optimization settings to control data collection — choose between 'All (Default)', 'Performance', 'Logs and events', or 'Inventory' table groupings. For advanced troubleshooting, use live logs (Stream Azure Diagnostics) to view container logs in real time without waiting for ingestion. Set up alerts for common scenarios: pod restarts, OOM kills, node disk pressure, and pending pods. Use Container Insights workbooks for capacity planning, resource optimization, and deployment tracking. Integrate with Azure Policy to enforce Container Insights on all new clusters.

ContainerLogV2
| where TimeGenerated > ago(1h)
| where LogMessage contains "ERROR" or LogMessage contains "Exception"
| project TimeGenerated, ContainerName, PodName, LogMessage
| order by TimeGenerated desc
| take 50

// Pod restart analysis
KubePodInventory
| where TimeGenerated > ago(1h)
| where PodRestartCount > 0
| project TimeGenerated, Namespace, Name = PodName, NodeName, PodRestartCount
| order by PodRestartCount desc

VM Insights and Network Monitoring: End-to-End Visibility for VMs

VM Insights monitors the health and performance of Azure VMs and Arc-enabled servers using the Azure Monitor Agent. It collects guest OS metrics (CPU, memory, disk, network), running processes, and dependencies (connections to other services). Enable VM Insights per VM or at scale via Azure Policy. The prebuilt VM performance dashboards show top resource consumers, process-level metrics, and connection maps. Use Azure Monitor dashboards with Grafana for rich VM monitoring visualizations — prebuilt dashboards for platform metrics, OpenTelemetry metrics, process monitoring, and Log Analytics data. For network monitoring, use Azure Network Watcher: Connection Monitor for end-to-end connectivity checks between VMs and external endpoints, Network Performance Monitor for latency and packet loss, and NSG flow logs for traffic analysis. Set up alerts for common VM issues: high CPU, disk space < 10%, memory pressure, and failed network connections. For Azure Virtual Desktop and Citrix workloads, use specialized monitoring with VM Insights and Azure Workbooks. Combine VM Insights with Change Tracking to correlate performance degradation with configuration changes. Use Azure Policy to enforce VM Insights and diagnostic settings on all VMs.

// Top CPU-consuming processes (VM Insights)
InsightsMetrics
| where TimeGenerated > ago(1h)
| where Origin == "vm.azm.ms" and Namespace == "Process" and Name == "ProcessCpu"
| summarize avgCpu = avg(Val) by Computer, Tags
| where Tags contains "processName"
| extend ProcessName = tostring(parse_json(Tags).processName)
| project Computer, ProcessName, avgCpu
| order by avgCpu desc
| take 20

// Failed network connections (Connection Monitor)
NetworkMonitoring
| where TimeGenerated > ago(1h)
| where ConnectionStatus == "Fail"
| project TimeGenerated, Source, Destination, FailureReason
| order by TimeGenerated desc