Microsoft Azure — Virtual Machine Scale Sets

Azure is Microsoft's cloud computing platform offering over 200 services. This article covers virtual machine scale sets with production-ready configurations, best practices, and hands-on examples.

Why Virtual Machine Scale Sets?

Virtual Machine Scale Sets (VMSS) are Azure's answer to elastic, auto-scaling compute. If you're running stateless workloads—web servers, microservices, batch processing—VMSS lets you scale out and in based on demand, without manual intervention. Unlike individual VMs, a scale set gives you a single API to manage hundreds of identical instances, with built-in load balancing and health monitoring. The key insight: VMSS is not a VM; it's a management layer for a fleet of VMs. You define a VM configuration (image, size, network), and Azure handles the rest. This is critical for DevOps pipelines where you need to deploy updates across all instances without downtime. In production, we've seen teams waste hours manually scaling VMs during traffic spikes—VMSS eliminates that. But it's not magic: you must design for statelessness, use managed disks for consistency, and plan for instance failures. The real power comes from combining VMSS with Azure Load Balancer or Application Gateway for traffic distribution, and Azure Monitor for autoscale rules. If you're still using individual VMs for scalable workloads, you're doing it wrong.

az vmss create \
  --resource-group myResourceGroup \
  --name myScaleSet \
  --image UbuntuLTS \
  --admin-username azureuser \
  --generate-ssh-keys \
  --instance-count 2 \
  --vm-sku Standard_DS1_v2 \
  --load-balancer myLoadBalancer \
  --backend-pool-name myBackendPool

VMSS vs. Availability Sets: When to Use What

A common confusion: should I use a VMSS or an Availability Set? Both provide high availability, but they serve different purposes. Availability Sets distribute VMs across fault domains and update domains to protect against hardware failures and planned maintenance. They're ideal for stateful workloads where you need to control each VM individually—like a database cluster. VMSS, on the other hand, is designed for stateless, identical instances that can be scaled automatically. The key difference: Availability Sets are about availability; VMSS is about scalability. In production, we often combine both: use an Availability Set for the database tier, and VMSS for the web tier. But never use VMSS for stateful workloads unless you have a robust external state layer. Also, note that VMSS supports automatic OS updates and instance repair, which Availability Sets don't. If you need to roll out updates across hundreds of VMs, VMSS is the only sane choice. My rule of thumb: if you can replace an instance without data loss, use VMSS. Otherwise, stick with Availability Sets or a managed service like Azure SQL.

az vm availability-set create \
  --resource-group myResourceGroup \
  --name myAvailabilitySet \
  --platform-fault-domain-count 2 \
  --platform-update-domain-count 5

Designing the VM Configuration: Images, Extensions, and Custom Scripts

The VM configuration is the heart of your scale set. You define the OS image, VM size, storage, network, and extensions. For production, use custom images from Azure Compute Gallery (formerly Shared Image Gallery) to ensure consistency and faster deployment. Avoid using marketplace images directly because they may change or have different configurations. Extensions are crucial: they run post-deployment scripts to install software, configure monitoring, or join a domain. The Custom Script Extension is the most common—it executes a script (from Azure Storage or GitHub) on every instance. But beware: extensions run on every boot or reimage, so make them idempotent. Also, consider using the Desired State Configuration (DSC) extension for declarative configuration management. For production, we always include the Azure Monitor Agent extension for metrics and logs. One gotcha: extension failures can cause the VM to be marked unhealthy. Test your extensions thoroughly. And never hardcode secrets in scripts—use Azure Key Vault with Managed Identity.

{
  "name": "customScript",
  "properties": {
    "publisher": "Microsoft.Azure.Extensions",
    "type": "CustomScript",
    "typeHandlerVersion": "2.1",
    "autoUpgradeMinorVersion": true,
    "settings": {
      "fileUris": ["https://raw.githubusercontent.com/example/install-app.sh"],
      "commandToExecute": "./install-app.sh"
    }
  }
}

Autoscaling: Rules, Metrics, and Best Practices

Autoscaling is the killer feature of VMSS. You define rules based on metrics like CPU, memory, or custom metrics from Application Insights. Azure Monitor collects these metrics and triggers scale-out or scale-in actions. But autoscaling is not magic—it requires careful tuning. Common mistakes: scaling too aggressively (causing thrash), using only CPU (which may not reflect actual load), or ignoring cool-down periods. For production, use a combination of metrics: CPU for compute-bound workloads, request count per instance for web apps, and queue length for batch processing. Always set minimum and maximum instance counts to avoid runaway costs or insufficient capacity. Also, implement predictive autoscaling if your workload has predictable patterns. One critical setting: the scale-in policy. By default, Azure scales in the newest instances, but you can choose to scale in the oldest or use a custom policy. For stateful workloads (if you must), use the 'NewestVM' policy to avoid disrupting long-running processes. And always test autoscaling under load before going live.

{
  "autoscale": {
    "enabled": true,
    "profiles": [
      {
        "name": "default",
        "capacity": {
          "minimum": "2",
          "maximum": "10",
          "default": "2"
        },
        "rules": [
          {
            "metricTrigger": {
              "metricName": "Percentage CPU",
              "metricResourceId": "/subscriptions/.../vmss/myScaleSet",
              "timeGrain": "PT1M",
              "statistic": "Average",
              "timeWindow": "PT5M",
              "timeAggregation": "Average",
              "operator": "GreaterThan",
              "threshold": 75
            },
            "scaleAction": {
              "direction": "Increase",
              "type": "ChangeCount",
              "value": "1",
              "cooldown": "PT5M"
            }
          }
        ]
      }
    ]
  }
}

Updating VMSS Instances: Rolling Upgrades and Reimaging

Updating a scale set is where many teams stumble. You can't just SSH into each instance; you need a strategy. VMSS supports two upgrade modes: Automatic and Manual. Automatic applies updates to all instances immediately—risky for production. Manual gives you control: you can update instances one by one or in batches. For production, use Manual mode with rolling upgrades via Azure DevOps or PowerShell. The typical workflow: update the VMSS model (new image, extension, or configuration), then perform a rolling upgrade that reimages instances in batches, respecting health probes. Always define a health probe (e.g., HTTP endpoint) so Azure knows when an instance is healthy before moving to the next batch. Also, consider using 'max unhealthy instances' settings to control the blast radius. Another approach: use instance repair to automatically replace unhealthy instances. But beware: reimaging an instance loses all local data—again, statelessness is key. For zero-downtime updates, use a deployment slot pattern: deploy a new scale set alongside the old one, shift traffic, then tear down the old set.

az vmss rolling-upgrade start \
  --resource-group myResourceGroup \
  --name myScaleSet \
  --max-batch-instance-percent 20 \
  --max-unhealthy-instance-percent 10 \
  --max-unhealthy-upgraded-instance-percent 10 \
  --pause-time-between-batches PT30S

Networking and Load Balancing: Front-End Traffic Distribution

A scale set without a load balancer is just a group of VMs. You need to distribute traffic across instances. Azure provides Azure Load Balancer (Layer 4) and Application Gateway (Layer 7). For most web workloads, use Application Gateway with SSL termination and path-based routing. The load balancer must be in the same virtual network as the scale set. When you create a VMSS with a load balancer, Azure automatically adds the instances to the backend pool. But there are nuances: you must configure health probes on the load balancer to detect unhealthy instances. Also, consider using NAT rules for RDP/SSH access to individual instances—but in production, use Azure Bastion instead. One common issue: source network address translation (SNAT) port exhaustion. Each outbound connection from a VM uses a SNAT port. If your app makes many outbound calls, you may run out. Mitigate by using Azure NAT Gateway or assigning a public IP to each instance (not recommended for scale). Also, ensure your load balancer's backend pool is configured for the correct protocol (TCP or UDP). For high throughput, use Floating IP (Direct Server Return) for scenarios like DNS or NTP.

az network application-gateway create \
  --resource-group myResourceGroup \
  --name myAppGateway \
  --sku Standard_v2 \
  --capacity 2 \
  --vnet-name myVNet \
  --subnet appGatewaySubnet \
  --http-settings-cookie-based-affinity Disabled \
  --frontend-port 80 \
  --http-settings-protocol Http \
  --routing-rule-type Basic \
  --servers 10.0.1.4 10.0.1.5

Monitoring and Diagnostics: Keeping Your Scale Set Healthy

You can't manage what you don't measure. VMSS integrates with Azure Monitor for metrics, logs, and alerts. Essential metrics: CPU, memory, disk I/O, and network. But also track scale set-specific metrics like 'Total VM Count' and 'Scale Actions'. Enable boot diagnostics to capture serial console output and screenshots—invaluable for troubleshooting boot failures. For application-level monitoring, install the Azure Monitor Application Insights agent. Set up alerts for critical conditions: high CPU, low disk space, or scale set capacity hitting limits. One often-overlooked feature: instance health. VMSS can report an instance as unhealthy if its health probe fails. Use this to trigger automatic repairs or alerts. Also, configure diagnostic settings to stream logs to Log Analytics or Azure Storage for long-term analysis. In production, we set up dashboards that show real-time instance counts, average CPU, and error rates. And we have runbooks that automatically scale out or restart instances based on alerts. Remember: monitoring is not just for reactive troubleshooting; use historical data to right-size your scale set and optimize costs.

{
  "properties": {
    "workspaceId": "/subscriptions/.../workspaces/myWorkspace",
    "logs": [
      {
        "category": "ScaleSetLogs",
        "enabled": true,
        "retentionPolicy": {
          "days": 30,
          "enabled": true
        }
      }
    ],
    "metrics": [
      {
        "category": "AllMetrics",
        "enabled": true,
        "retentionPolicy": {
          "days": 30,
          "enabled": true
        }
      }
    ]
  }
}

Cost Optimization: Rightsizing and Reserved Instances

VMSS can quickly rack up costs if not managed. The biggest lever is instance size: choose the right VM SKU for your workload. Use Azure Advisor recommendations to identify underutilized instances. For predictable workloads, use Reserved Instances (RI) or Azure Savings Plans to save up to 72% compared to pay-as-you-go. But RIs require a 1- or 3-year commitment—only buy for baseline capacity. For burstable workloads, consider using B-series VMs that accumulate credits. Also, enable auto-scaling to scale in during low traffic—but set a minimum to avoid cold starts. Another cost-saving tactic: use Spot VMs for fault-tolerant batch jobs. Spot VMs can be evicted, so they're not for production web servers. But for CI/CD agents or data processing, they can slash costs by 90%. Monitor your costs with Azure Cost Management and set budgets with alerts. One hidden cost: data egress. If your scale set sends data to the internet, you pay per GB. Use Azure CDN or put the scale set in the same region as your users. Finally, regularly review and delete unused resources like old scale sets or load balancers.

az reservation purchase \
  --reserved-resource-type VirtualMachines \
  --sku Standard_DS2_v2 \
  --term P1Y \
  --quantity 5 \
  --scope Single \
  --resource-group myResourceGroup

Disaster Recovery and Business Continuity

VMSS is regional—if the region goes down, your scale set goes down. For disaster recovery, you need a multi-region strategy. The simplest approach: deploy identical scale sets in two regions with Azure Traffic Manager or Azure Front Door in front. Use active-passive (one region handles traffic, the other is standby) or active-active (both regions serve traffic). For active-passive, keep the standby scale set at minimum capacity to reduce costs. Use Azure Site Recovery to replicate VMSS configuration and data? Actually, Site Recovery doesn't support VMSS directly. Instead, use Azure DevOps pipelines to redeploy the scale set in the secondary region from your infrastructure-as-code templates. For state, use geo-replicated Azure SQL or Cosmos DB. Test your failover regularly—we do it quarterly. One common failure: DNS propagation delays. Use Traffic Manager with fast failover settings (monitoring interval 10 seconds). Also, consider using Availability Zones within a region to protect against datacenter failures. VMSS supports zonal deployment: you can pin instances to specific zones or let Azure balance them. For critical workloads, deploy across three zones.

{
  "properties": {
    "dnsConfig": {
      "relativeName": "myapp",
      "ttl": 30
    },
    "monitorConfig": {
      "protocol": "HTTP",
      "port": 80,
      "path": "/health",
      "intervalInSeconds": 10,
      "timeoutInSeconds": 5,
      "toleratedNumberOfFailures": 2
    },
    "endpoints": [
      {
        "name": "primary",
        "type": "azureEndpoints",
        "targetResourceId": "/subscriptions/.../vmss/primaryScaleSet",
        "endpointLocation": "eastus"
      },
      {
        "name": "secondary",
        "type": "azureEndpoints",
        "targetResourceId": "/subscriptions/.../vmss/secondaryScaleSet",
        "endpointLocation": "westus"
      }
    ]
  }
}

Security: Managed Identity, Network Security, and Secrets Management

Security in VMSS starts with identity. Never use static credentials. Assign a Managed Identity to the scale set so instances can authenticate to Azure services without storing keys. Use Azure RBAC to grant the identity only the permissions it needs. For network security, use Network Security Groups (NSGs) to restrict inbound and outbound traffic. Apply NSGs at the subnet level for the scale set. For web workloads, only allow ports 80/443 from the load balancer. Also, enable Azure DDoS Protection on the virtual network. For secrets, use Azure Key Vault with the virtual machine extension to inject certificates or connection strings at boot time. Never bake secrets into images. Another critical aspect: OS security. Use Azure Security Center to assess vulnerabilities and apply patches. Enable automatic OS upgrades for VMSS to keep instances patched. For compliance, use Azure Policy to enforce rules like 'VMSS must use managed disks' or 'VMSS must have autoscale enabled'. Finally, monitor security events with Azure Sentinel. One production gotcha: if you use custom scripts, ensure they don't expose secrets in logs. Redirect output to a secure location.

{
  "name": "KeyVaultForWindows",
  "properties": {
    "publisher": "Microsoft.Azure.KeyVault",
    "type": "KeyVaultForWindows",
    "typeHandlerVersion": "1.0",
    "settings": {
      "secretsManagementSettings": {
        "pollingIntervalInS": "3600",
        "certificateStoreLocation": "LocalMachine",
        "certificateStoreName": "My",
        "observedCertificates": ["https://myvault.vault.azure.net/secrets/mycert"]
      }
    }
  }
}

Troubleshooting Common VMSS Failures

Even with best practices, things go wrong. Here are the most common VMSS failures and how to fix them. 1) Instances stuck in 'Creating' state: usually a quota issue. Check your vCPU quota in the region and request an increase. 2) Health probe failures: the application isn't responding on the expected port. Check the app logs and ensure the health endpoint returns 200. 3) Extension failures: the custom script failed. Look at the extension status in the portal or run 'az vmss extension list'. Common causes: script errors, network timeouts, or missing dependencies. 4) Autoscale not triggering: metrics may not be flowing. Verify the diagnostic settings and that the metric is available. 5) Scale-in evicting the wrong instances: configure the scale-in policy. 6) Rolling upgrade hangs: an instance may be unhealthy. Set max unhealthy instance percent higher or fix the health probe. 7) Outbound connectivity issues: SNAT port exhaustion or NSG blocking outbound traffic. Use Azure NAT Gateway. 8) Disk space filling up: logs or temp files. Use custom script to clean up or attach a data disk. For each failure, start with Azure Monitor and instance view. The serial console is your friend for boot issues. And always have a rollback plan: keep the previous VMSS model or use deployment slots.

# Check instance view
az vmss get-instance-view --resource-group myResourceGroup --name myScaleSet --instance-id 0

# Check extension status
az vmss extension list --resource-group myResourceGroup --vmss-name myScaleSet

# View serial console (requires boot diagnostics)
az vmss boot-diagnostics get-serial-console --resource-group myResourceGroup --name myScaleSet --instance-id 0

Infrastructure as Code: Deploying VMSS with ARM, Bicep, or Terraform

Manual creation of VMSS is fine for testing, but for production you need Infrastructure as Code (IaC). Azure Resource Manager (ARM) templates are the native option, but they're verbose. Bicep is a cleaner DSL that compiles to ARM. Terraform is cross-platform and popular in multi-cloud shops. Whichever you choose, the principles are the same: define the VMSS, load balancer, autoscale rules, and extensions in code. Store the templates in Git and use CI/CD to deploy. For production, we use Bicep because it's simpler and has native Azure support. Key considerations: parameterize everything (instance count, VM size, admin password). Use modules to reuse common patterns. Always deploy to a staging environment first. One common mistake: hardcoding the VMSS name or resource group. Use unique names with deployment stamps. Also, manage state carefully with Terraform—use remote state storage. For secrets, reference Key Vault in your templates. Finally, use deployment scripts to run post-deployment validation. IaC turns your infrastructure into a repeatable, version-controlled artifact. If you're not using IaC for VMSS, you're one click away from disaster.

param location string = resourceGroup().location
param vmssName string = 'myScaleSet'
param instanceCount int = 2

resource vmss 'Microsoft.Compute/virtualMachineScaleSets@2023-03-01' = {
  name: vmssName
  location: location
  sku: {
    name: 'Standard_DS1_v2'
    capacity: instanceCount
  }
  properties: {
    upgradePolicy: {
      mode: 'Manual'
    }
    virtualMachineProfile: {
      storageProfile: {
        imageReference: {
          publisher: 'Canonical'
          offer: 'UbuntuServer'
          sku: '18.04-LTS'
          version: 'latest'
        }
      }
      osProfile: {
        computerNamePrefix: 'vmss'
        adminUsername: 'azureuser'
        linuxConfiguration: {
          disablePasswordAuthentication: true
          ssh: {
            publicKeys: [
              {
                path: '/home/azureuser/.ssh/authorized_keys'
                keyData: 'ssh-rsa AAAAB3NzaC1yc2E...'
              }
            ]
          }
        }
      }
      networkProfile: {
        networkInterfaceConfigurations: [
          {
            name: 'nic'
            properties: {
              primary: true
              ipConfigurations: [
                {
                  name: 'ipconfig'
                  properties: {
                    subnet: {
                      id: '/subscriptions/.../subnets/default'
                    }
                  }
                }
              ]
            }
          }
        ]
      }
    }
  }
}