Microsoft Azure — ExpressRoute & Hybrid Connectivity

Azure is Microsoft's cloud computing platform offering over 200 services. This article covers expressroute & hybrid connectivity with production-ready configurations, best practices, and hands-on examples.

Why ExpressRoute? The Case for Private Connectivity

Public internet is fine for dev/test, but production workloads demand predictable latency, bandwidth, and security. ExpressRoute bypasses the internet, providing a private connection from your on-premises network to Azure. This eliminates packet loss from ISP congestion, reduces attack surface, and enables SLA-backed uptime (99.95% for redundant circuits). For hybrid architectures—think SAP HANA, SQL Server Always On, or real-time analytics—ExpressRoute is non-negotiable. Without it, you're gambling on best-effort routing. I've seen outages caused by BGP flapping over VPN tunnels that simply wouldn't happen with a dedicated circuit. If your CFO asks why you're spending $Xk/month on a circuit, show them the cost of a 30-minute production outage.

#!/bin/bash
# Check ExpressRoute circuit and peering status
az network express-route list --query "[].{Name:name, ProvisioningState:provisioningState, CircuitProvisioningState:circuitProvisioningState, ServiceProviderProvisioningState:serviceProviderProvisioningState}" -o table

ExpressRoute Topologies: Choosing the Right Model

ExpressRoute supports three connectivity models: CloudExchange Co-location, Point-to-Point Ethernet, and Any-to-Any (IPVPN). Co-location is ideal when your data center is in the same facility as an MSEE (Microsoft Enterprise Edge) – you get low latency and high bandwidth. Point-to-point is for direct fiber connections between your router and the MSEE. IPVPN uses an MPLS provider to connect multiple sites to Azure. For most enterprises, a combination works: IPVPN for branch offices, point-to-point for core data centers. Avoid the trap of a single circuit – always deploy redundant circuits across different peering locations. I've seen a backhoe cut through a fiber bundle taking down an entire region. Use two providers or two diverse paths.

resource "azurerm_express_route_circuit" "primary" {
  name                  = "er-circuit-primary"
  location              = azurerm_resource_group.main.location
  resource_group_name   = azurerm_resource_group.main.name
  service_provider_name = "Equinix"
  peering_location      = "Silicon Valley"
  bandwidth_in_mbps     = 1000
  sku {
    tier   = "Premium"
    family = "MeteredData"
  }
  allow_global_reach = true
}

resource "azurerm_express_route_circuit" "secondary" {
  name                  = "er-circuit-secondary"
  location              = azurerm_resource_group.main.location
  resource_group_name   = azurerm_resource_group.main.name
  service_provider_name = "Level 3"
  peering_location      = "Los Angeles"
  bandwidth_in_mbps     = 1000
  sku {
    tier   = "Premium"
    family = "MeteredData"
  }
  allow_global_reach = true
}

BGP Routing: The Backbone of ExpressRoute

ExpressRoute uses BGP to exchange routes between your on-premises network and Azure. You establish two BGP sessions per circuit: one for private peering (RFC 1918 addresses) and one for Microsoft peering (public services like Office 365). For private peering, advertise your on-premises prefixes to Azure and receive Azure virtual network prefixes. Use BGP communities to tag routes for traffic engineering. Always filter routes on both sides – never accept default routes from Azure unless you want to blackhole traffic. I've seen outages caused by misconfigured AS numbers or missing route filters. Use a dedicated public ASN (65515-65520 for private use) and enable BFD for fast convergence. Set MED values to prefer one circuit over another.

import subprocess
import json

def check_bgp_status():
    result = subprocess.run(
        ["az", "network", "express-route", "peering", "list",
         "--circuit-name", "my-circuit", "--resource-group", "my-rg"],
        capture_output=True, text=True
    )
    peerings = json.loads(result.stdout)
    for peering in peerings:
        print(f"Peering Type: {peering['peeringType']}")
        print(f"Primary Peer IP: {peering['primaryPeerAddressPrefix']}")
        print(f"Secondary Peer IP: {peering['secondaryPeerAddressPrefix']}")
        print(f"ASN: {peering['peerASN']}")
        print(f"State: {peering['state']}")
        print("---")

if __name__ == "__main__":
    check_bgp_status()

Configuring ExpressRoute Gateway: High Availability and Performance

The ExpressRoute gateway sits in your Azure virtual network and routes traffic to/from the circuit. For production, use the UltraPerformance SKU (10 Gbps) or ErGw3AZ (zone-redundant). Always deploy the gateway in an active-active configuration – this requires two gateway instances and two connections to the circuit. The gateway uses BGP to learn routes from on-premises. Set the gateway subnet to /27 or larger to accommodate scaling. I've seen gateways fail because the subnet was too small. Enable FastPath for low-latency traffic (bypasses the gateway for certain flows). Monitor gateway metrics like CPU, packets per second, and routes advertised. If you exceed 1000 routes, consider route aggregation.

resource "azurerm_subnet" "gateway" {
  name                 = "GatewaySubnet"
  resource_group_name  = azurerm_resource_group.main.name
  virtual_network_name = azurerm_virtual_network.main.name
  address_prefixes     = ["10.0.1.0/27"]
}

resource "azurerm_express_route_gateway" "main" {
  name                = "er-gateway"
  location            = azurerm_resource_group.main.location
  resource_group_name = azurerm_resource_group.main.name
  virtual_network_id  = azurerm_virtual_network.main.id
  scale_units         = 2
}

resource "azurerm_express_route_connection" "primary" {
  name                             = "primary-connection"
  express_route_gateway_id         = azurerm_express_route_gateway.main.id
  express_route_circuit_peering_id = azurerm_express_route_circuit_peering.primary.id
  routing_weight                   = 10
}

Redundancy and Failover: Designing for Resilience

ExpressRoute offers two levels of redundancy: circuit-level and connection-level. At circuit level, deploy two circuits from different providers to different peering locations. At connection level, each circuit has two BGP sessions (primary and secondary) to two MSEE routers. For failover, use BGP attributes: local preference, AS path prepend, and MED. Typically, set local preference higher on the primary circuit. Use BFD (Bidirectional Forwarding Detection) for sub-second failure detection. Test failover regularly – I've seen configurations that looked correct but failed during actual outages because of asymmetric routing. Implement Azure Monitor alerts for BGP session state and circuit health. Use ExpressRoute Monitor to visualize path health.

#!/bin/bash
# Simulate circuit failure by disabling BGP peering on primary
az network express-route peering update \
  --circuit-name er-circuit-primary \
  --resource-group my-rg \
  --peering-type AzurePrivatePeering \
  --state Disabled

# Wait for failover
sleep 30

# Check connectivity
ping -c 4 10.0.0.4  # Azure gateway IP

# Re-enable
az network express-route peering update \
  --circuit-name er-circuit-primary \
  --resource-group my-rg \
  --peering-type AzurePrivatePeering \
  --state Enabled

Monitoring and Troubleshooting ExpressRoute

Azure provides several tools for monitoring ExpressRoute: Azure Monitor metrics (ARP availability, BGP availability, bits in/out), ExpressRoute Monitor (end-to-end network performance), and Network Performance Monitor (NPM). Set alerts on BGP session down, circuit provisioning state changes, and high latency. For troubleshooting, start with the Azure portal's ExpressRoute diagnostics – it runs connectivity checks. Common issues: BGP peering not established (check ASN, VLAN ID, IP addresses), route limits exceeded (max 1000 routes per peering), and ARP resolution failures. Use the az network express-route list-route-tables command to verify learned routes. I've seen cases where on-premises firewalls blocked BGP (TCP 179) or ICMP, causing false positives.

#!/bin/bash
# Check BGP session state
az network express-route peering show \
  --circuit-name my-circuit \
  --resource-group my-rg \
  --peering-type AzurePrivatePeering \
  --query "{PrimaryState:primaryAzurePort, SecondaryState:secondaryAzurePort, State:state}"

# List routes learned from on-premises
az network express-route list-route-tables \
  --circuit-name my-circuit \
  --resource-group my-rg \
  --peering-type AzurePrivatePeering \
  --device-path primary

# Check ARP table
az network express-route list-arp-tables \
  --circuit-name my-circuit \
  --resource-group my-rg \
  --peering-type AzurePrivatePeering \
  --device-path primary

Security Considerations: Segmentation and Encryption

ExpressRoute traffic is private but not encrypted by default. For sensitive data, use IPsec over ExpressRoute or Azure VPN Gateway in parallel. Segment traffic using VLANs and BGP communities. For example, tag production traffic with community 12076:5000 and dev with 12076:5100, then apply routing policies. Use Azure Firewall or NVA to inspect traffic between on-premises and Azure. Implement network security groups (NSGs) on subnets connected via ExpressRoute. I've seen breaches where on-premises networks had direct access to Azure VMs without any firewall. Also, consider using ExpressRoute with Azure Private Link to expose services privately. Never advertise your entire on-premises network to Azure – use specific /24 or smaller prefixes.

resource "azurerm_network_security_group" "expressroute_nsg" {
  name                = "expressroute-nsg"
  location            = azurerm_resource_group.main.location
  resource_group_name = azurerm_resource_group.main.name

  security_rule {
    name                       = "Allow-OnPrem-Web"
    priority                   = 100
    direction                  = "Inbound"
    access                     = "Allow"
    protocol                   = "Tcp"
    source_port_range          = "*"
    destination_port_range     = "443"
    source_address_prefixes    = ["10.0.0.0/8"]
    destination_address_prefix = "10.1.0.0/16"
  }

  security_rule {
    name                       = "Deny-All-Other-OnPrem"
    priority                   = 200
    direction                  = "Inbound"
    access                     = "Deny"
    protocol                   = "*"
    source_port_range          = "*"
    destination_port_range     = "*"
    source_address_prefixes    = ["10.0.0.0/8"]
    destination_address_prefix = "*"
  }
}

Global Reach: Connecting ExpressRoute Circuits Across Regions

ExpressRoute Global Reach allows you to connect two ExpressRoute circuits from different regions, enabling private connectivity between on-premises sites through the Microsoft backbone. This replaces expensive MPLS links for inter-site communication. Enable Global Reach on each circuit (requires Premium add-on). Traffic between sites stays on Microsoft's network, reducing latency and improving security. Use cases: disaster recovery between regions, connecting branch offices to a central data center. I've seen companies save 40% on WAN costs by replacing MPLS with Global Reach. However, be aware of bandwidth limits – each circuit supports up to 10 Gbps. Also, Global Reach does not support IPv6 yet.

resource "azurerm_express_route_circuit" "circuit1" {
  name                  = "er-circuit-silicon-valley"
  location              = "westus"
  resource_group_name   = azurerm_resource_group.main.name
  service_provider_name = "Equinix"
  peering_location      = "Silicon Valley"
  bandwidth_in_mbps     = 1000
  sku {
    tier   = "Premium"
    family = "MeteredData"
  }
  allow_global_reach = true
}

resource "azurerm_express_route_circuit" "circuit2" {
  name                  = "er-circuit-new-york"
  location              = "eastus"
  resource_group_name   = azurerm_resource_group.main.name
  service_provider_name = "Zayo"
  peering_location      = "New York"
  bandwidth_in_mbps     = 1000
  sku {
    tier   = "Premium"
    family = "MeteredData"
  }
  allow_global_reach = true
}

resource "azurerm_express_route_circuit_peering" "peering1" {
  circuit_name       = azurerm_express_route_circuit.circuit1.name
  resource_group_name = azurerm_resource_group.main.name
  peering_type       = "AzurePrivatePeering"
  peer_asn           = 65001
  vlan_id            = 100
  primary_peer_address_prefix = "10.0.0.0/30"
  secondary_peer_address_prefix = "10.0.0.4/30"
  shared_key         = "A1b2C3d4"
}

resource "azurerm_express_route_circuit_peering" "peering2" {
  circuit_name       = azurerm_express_route_circuit.circuit2.name
  resource_group_name = azurerm_resource_group.main.name
  peering_type       = "AzurePrivatePeering"
  peer_asn           = 65002
  vlan_id            = 200
  primary_peer_address_prefix = "10.0.1.0/30"
  secondary_peer_address_prefix = "10.0.1.4/30"
  shared_key         = "E5f6G7h8"
}

Cost Optimization: Right-Sizing Your ExpressRoute

ExpressRoute pricing includes circuit bandwidth (metered or unlimited data) and data transfer out. For predictable traffic, unlimited data is cheaper. For bursty traffic, metered is better. Use Azure Cost Management to analyze traffic patterns. Consider using ExpressRoute Direct for dedicated 10/100 Gbps ports – cheaper at high bandwidths. For dev/test, use a smaller circuit (50 Mbps) and scale up. I've seen teams over-provision 10 Gbps circuits for workloads that peak at 1 Gbps. Monitor utilization and right-size quarterly. Also, use ExpressRoute with Azure Private Link to reduce data transfer costs by keeping traffic within Azure. Remember: Premium add-on adds cost but enables Global Reach and higher route limits.

#!/bin/bash
# Get ExpressRoute circuit metrics for cost analysis
az monitor metrics list \
  --resource /subscriptions/.../resourceGroups/my-rg/providers/Microsoft.Network/expressRouteCircuits/my-circuit \
  --metric "BitsInPerSecond" "BitsOutPerSecond" \
  --interval PT1H \
  --aggregation Average \
  --start-time 2026-07-01T00:00:00Z \
  --end-time 2026-07-12T00:00:00Z \
  --output table

Migration from VPN to ExpressRoute: A Phased Approach

Migrating from site-to-site VPN to ExpressRoute requires careful planning to avoid downtime. Start by provisioning the ExpressRoute circuit and configuring BGP peering alongside the existing VPN. Use BGP to prefer ExpressRoute over VPN (e.g., lower MED on ExpressRoute). Gradually shift traffic by adjusting route advertisements. Test each phase: first, move non-critical traffic, then production. Monitor for asymmetric routing – ensure both paths have consistent routes. I've seen migrations fail because the VPN was torn down before ExpressRoute was fully validated. Keep the VPN as a backup for at least 30 days. Use Azure Traffic Manager or a load balancer to distribute traffic during migration. Document rollback procedures.

#!/bin/bash
# Phase 1: Add ExpressRoute routes with lower MED
az network express-route peering update \
  --circuit-name my-circuit \
  --resource-group my-rg \
  --peering-type AzurePrivatePeering \
  --set "peerings[0].microsoftPeeringConfig.routeFilter.rules[0].action=Permit"

# Phase 2: Remove VPN routes after verification
# (Manual step after monitoring)

# Phase 3: Tear down VPN gateway
az network vpn-gateway delete \
  --name vpn-gateway \
  --resource-group my-rg

ExpressRoute Direct: Dedicated Bandwidth for High-Scale Workloads

ExpressRoute Direct provides dedicated 10 Gbps or 100 Gbps ports directly from your on-premises router to Microsoft's edge. This is ideal for high-throughput workloads like media streaming, large-scale data replication, or HPC. You manage the physical layer – no service provider involvement. Direct offers lower latency and higher bandwidth than standard circuits. However, you must have physical access to an MSEE location. Provisioning takes longer (weeks vs days). Use cases: real-time financial trading, video transcoding, or big data analytics. I've seen Direct used for Azure NetApp Files replication. Be aware of the minimum commitment: 10 Gbps per port. For redundancy, order two ports from different MSEE locations.

resource "azurerm_express_route_port" "direct" {
  name                = "er-direct-port"
  location            = "westus"
  resource_group_name = azurerm_resource_group.main.name
  peering_location    = "Silicon Valley"
  bandwidth_in_gbps   = 10
  encapsulation       = "Dot1Q"
}

resource "azurerm_express_route_circuit" "direct_circuit" {
  name                  = "er-direct-circuit"
  location              = azurerm_express_route_port.direct.location
  resource_group_name   = azurerm_resource_group.main.name
  express_route_port_id = azurerm_express_route_port.direct.id
  bandwidth_in_gbps     = 10
  sku {
    tier   = "Premium"
    family = "MeteredData"
  }
}

ExpressRoute with Azure VMware Solution (AVS)

Azure VMware Solution (AVS) runs VMware SDDC natively in Azure. ExpressRoute is the primary connectivity method for AVS, providing low-latency access to on-premises vCenter and workloads. You connect AVS to your on-premises network via ExpressRoute Global Reach or a circuit. AVS requires a dedicated ExpressRoute circuit with at least 1 Gbps bandwidth. Use BGP to advertise AVS management networks (e.g., /22 for vCenter, NSX-T). I've seen performance issues when using VPN with AVS due to jitter. For vMotion or HCX, ExpressRoute is mandatory. Plan for IP address overlap – use network segmentation. Monitor AVS ExpressRoute metrics for bandwidth saturation. AVS also supports ExpressRoute FastPath for reduced latency.

#!/bin/bash
# Create AVS private cloud
az vmware private-cloud create \
  --name avs-cloud \
  --resource-group my-rg \
  --location westus \
  --sku av36 \
  --network-block 10.0.0.0/22

# Authorize ExpressRoute circuit
az vmware authorization create \
  --name er-auth \
  --private-cloud avs-cloud \
  --resource-group my-rg

# Get authorization key
az vmware authorization list \
  --private-cloud avs-cloud \
  --resource-group my-rg \
  --query "[0].expressRouteAuthorizationKey" -o tsv