Microsoft Azure — AKS Cluster Design & Security

Azure is Microsoft's cloud computing platform offering over 200 services. This article covers aks cluster design & security with production-ready configurations, best practices, and hands-on examples.

1. Why AKS Cluster Design Matters

AKS simplifies Kubernetes management, but default settings are not production-ready. A poorly designed cluster leads to cost overruns, security breaches, and performance degradation. This article walks through a production-grade AKS design, covering network topology, identity, RBAC, pod security, monitoring, and cost optimization. Each section builds on the previous, culminating in a secure, observable, and efficient cluster.

az aks show --name myCluster --resource-group myRG --query "{networkProfile:networkProfile, identity:identity, addonProfiles:addonProfiles}" -o json

2. Network Topology: Azure CNI with Calico

Azure CNI assigns each pod an IP from the VNet subnet, enabling direct pod-to-pod communication and integration with Azure networking. Calico provides network policies for micro-segmentation. Choose a subnet size that accommodates node scaling: each node reserves up to 30 IPs for pods (configurable via --max-pods). For production, use a /16 subnet (65k IPs) for flexibility. Enable Azure Network Policy or Calico; Calico offers richer policy capabilities like egress filtering and namespace isolation.

az aks create \
  --resource-group prod-rg \
  --name prod-aks \
  --node-count 3 \
  --enable-cluster-autoscaler \
  --min-count 3 \
  --max-count 10 \
  --network-plugin azure \
  --network-policy calico \
  --vnet-subnet-id /subscriptions/.../subnets/aks-subnet \
  --pod-cidr 10.244.0.0/16 \
  --service-cidr 10.0.0.0/16 \
  --dns-service-ip 10.0.0.10 \
  --docker-bridge-address 172.17.0.1/16

3. Identity and Access Management: Managed Identity and RBAC

AKS supports system-assigned or user-assigned managed identities for the cluster itself. Use user-assigned for better control and separation of concerns. For node pools, use a separate identity per pool. RBAC at the cluster level controls who can access the Kubernetes API. Integrate with Azure AD for authentication: enable Azure AD integration with AKS managed Azure AD. This allows conditional access policies and MFA. For service-to-service auth, use Azure AD pod identity or workload identity (preferred).

az aks create \
  --enable-aad \
  --aad-admin-group-object-ids <admin-group-id> \
  --aad-tenant-id <tenant-id> \
  --assign-identity /subscriptions/.../resourcegroups/prod-rg/providers/Microsoft.ManagedIdentity/userAssignedIdentities/aks-identity \
  --enable-azure-rbac \
  --name prod-aks \
  --resource-group prod-rg

4. Pod Security: Azure Policy and Pod Identity

Azure Policy for AKS enforces security standards at scale. Use built-in policies like 'Kubernetes cluster containers should not share host process ID or host IPC namespace' and 'Kubernetes cluster containers should run as non-privileged'. For pod identity, use Azure AD Workload Identity (preview) instead of the deprecated aad-pod-identity. Workload identity uses federated identity credentials, eliminating the need for custom resource definitions and reducing complexity.

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sPSPPrivilegedContainer
metadata:
  name: psp-privileged-container
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
  parameters:
    allowedPrivileged: false

5. Secrets Management: Azure Key Vault with CSI Driver

Store secrets in Azure Key Vault and mount them into pods using the Secrets Store CSI Driver. This avoids storing secrets in etcd and enables rotation without pod restarts. Enable the driver via addon: az aks enable-addons --addons azure-keyvault-secrets-provider. Use a user-assigned managed identity for the driver to access Key Vault. For production, enable auto-rotation and set the rotation poll interval to 2h.

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: my-secrets
spec:
  provider: azure
  parameters:
    usePodIdentity: false
    useVMManagedIdentity: true
    userAssignedIdentityID: <identity-client-id>
    keyvaultName: my-keyvault
    objects: |
      array:
        - |
          objectName: db-password
          objectType: secret
    tenantId: <tenant-id>

6. Monitoring and Logging: Container Insights and Prometheus

Enable Container Insights (Azure Monitor for containers) for out-of-the-box metrics, logs, and alerts. It collects stdout/stderr, performance counters, and Kubernetes events. For deeper observability, deploy Prometheus and Grafana using the Azure Monitor managed service for Prometheus (preview). This integrates with Container Insights and provides long-term storage. Configure alerts for node CPU > 80%, pod restarts > 3 in 5m, and OOMKilled events.

az aks enable-addons \
  --addons monitoring \
  --name prod-aks \
  --resource-group prod-rg \
  --workspace-resource-id /subscriptions/.../resourcegroups/prod-rg/providers/Microsoft.OperationalInsights/workspaces/prod-law

7. Cost Optimization: Node Pools, Spot Instances, and Cluster Autoscaler

Use multiple node pools for different workloads: system pool (critical system pods) and user pools (application workloads). System pool should have at least 2 nodes with a fixed size (e.g., Standard_D2s_v3). User pools can use spot instances for non-critical, fault-tolerant workloads. Enable cluster autoscaler with min/max counts per pool. Use Azure Reservations for baseline capacity and spot for burst. Monitor node utilization with kubectl top nodes and right-size instance types.

az aks nodepool add \
  --resource-group prod-rg \
  --cluster-name prod-aks \
  --name spotpool \
  --priority Spot \
  --eviction-policy Delete \
  --spot-max-price -1 \
  --enable-cluster-autoscaler \
  --min-count 1 \
  --max-count 5 \
  --node-vm-size Standard_D4s_v3

8. Disaster Recovery: Multi-Region and Backup

For critical workloads, deploy AKS clusters in paired regions (e.g., East US 2 and Central US). Use Azure Traffic Manager or Front Door to route traffic. For stateful workloads, use Azure Disks with zone-redundant storage (ZRS) or Azure Files with geo-redundant storage (GRS). Backup etcd and persistent volumes using Velero with Azure Blob storage. Test recovery procedures regularly. Configure pod anti-affinity to spread replicas across nodes and zones.

apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: daily-backup
  namespace: velero
spec:
  schedule: "0 2 * * *"
  template:
    includedNamespaces:
      - '*'
    excludedResources:
      - nodes
      - events
      - events.events.k8s.io
    ttl: 720h
    storageLocation: default

9. Upgrade Strategy: Blue-Green and Canary Deployments

AKS upgrades (Kubernetes version, node image) can cause downtime. Use blue-green deployment: create a new node pool with the target version, migrate workloads, then delete the old pool. For application upgrades, use canary deployments with Flagger or Argo Rollouts. These tools gradually shift traffic and roll back on failures. Always test upgrades in a non-production environment first. Monitor upgrade progress with kubectl get events --watch.

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: app-canary
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: app
  service:
    port: 80
  analysis:
    interval: 1m
    threshold: 5
    maxWeight: 50
    stepWeight: 10
    metrics:
      - name: request-success-rate
        thresholdRange:
          min: 99
        interval: 1m

10. Compliance and Governance: Azure Policy and Guardrails

Azure Policy for AKS enforces organizational standards. Assign built-in policies like 'Kubernetes clusters should not use the default namespace' and 'Kubernetes clusters should have network policies enabled'. For custom policies, use Azure Policy's Rego-based constraints (Gatekeeper v3). Audit mode first, then enforce. Use Azure Blueprints to deploy a compliant AKS environment with policies, RBAC, and monitoring pre-configured.

az policy assignment create \
  --name 'enforce-network-policies' \
  --policy /providers/Microsoft.Authorization/policyDefinitions/... \
  --scope /subscriptions/.../resourceGroups/prod-rg \
  --params '{"effect":{"value":"Deny"}}'

11. Networking: Ingress Controllers and Service Mesh

For production traffic, use an ingress controller like NGINX or Application Gateway Ingress Controller (AGIC). AGIC integrates with Azure Application Gateway for L7 load balancing and WAF. For service-to-service communication, consider a service mesh like Istio or Linkerd. Istio provides mTLS, traffic splitting, and observability. However, service mesh adds complexity; only use if you need fine-grained traffic management. For most cases, Azure CNI + Calico + NGINX ingress suffices.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx
  rules:
  - host: app.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: app-service
            port:
              number: 80

12. Putting It All Together: Production AKS Checklist

Before going live, verify: (1) Azure CNI with Calico network policies, (2) Azure AD integration with RBAC, (3) Azure Policy enforcing pod security, (4) Key Vault CSI for secrets, (5) Container Insights + Prometheus for monitoring, (6) Separate node pools with autoscaler, (7) Velero backups, (8) Blue-green upgrade strategy, (9) Ingress controller with TLS, (10) Cost tags on resources. Automate cluster creation with Terraform or Bicep. Use GitOps with Flux or Argo CD for application deployment.

resource "azurerm_kubernetes_cluster" "prod" {
  name                = "prod-aks"
  location            = azurerm_resource_group.prod.location
  resource_group_name = azurerm_resource_group.prod.name
  dns_prefix          = "prod-aks"

  default_node_pool {
    name       = "systempool"
    node_count = 2
    vm_size    = "Standard_D2s_v3"
  }

  identity {
    type = "UserAssigned"
    identity_ids = [azurerm_user_assigned_identity.aks.id]
  }

  network_profile {
    network_plugin = "azure"
    network_policy = "calico"
  }

  azure_active_directory_role_based_access_control {
    managed            = true
    azure_rbac_enabled = true
    admin_group_object_ids = [azuread_group.aks_admins.object_id]
  }

  oms_agent {
    log_analytics_workspace_id = azurerm_log_analytics_workspace.prod.id
  }

  azure_policy_enabled = true

  key_vault_secrets_provider {
    secret_rotation_enabled = true
    secret_rotation_interval = "2h"
  }
}

13. GitOps with Flux and Argo CD for Application Delivery

GitOps is the standard for managing Kubernetes configurations in production. Two tools dominate: Flux v2 and Argo CD. Flux is lighter, integrates natively with AKS (via the gitops add-on), and uses a pull-based model with source controllers. Argo CD provides a rich UI, SSO integration, and sync strategies (e.g., blue-green sync waves). For production, use GitOps to manage both infrastructure (cluster add-ons, policies) and applications. Store Kubernetes manifests in Git repositories, and let the GitOps controller reconcile the cluster state. Enable the GitOps add-on on AKS with az aks enable-addons --addons gitops. For multi-cluster management, use Azure Kubernetes Fleet Manager to propagate GitOps configurations across clusters. A common mistake is committing generated files (Helm output) to Git instead of using Helm releases via GitOps. Use Flux's HelmRelease CRD or Argo CD's Helm support to deploy from Helm charts directly.

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: my-app
  namespace: flux-system
spec:
  interval: 5m
  chart:
    spec:
      chart: my-app
      sourceRef:
        kind: HelmRepository
        name: my-repo
        namespace: flux-system
      interval: 1m
  values:
    replicaCount: 3
    image:
      tag: 1.2.3
    ingress:
      host: app.example.com

14. Azure Front Door + Private Link: The Recommended Ingress Architecture

Microsoft's recommended AKS ingress architecture in 2026 uses Azure Front Door Premium with Web Application Firewall (WAF), Private Link to an internal load balancer, and NGINX Ingress Controller inside the cluster. This pattern provides global load balancing, DDoS protection, WAF, and private connectivity without exposing the cluster to the internet. The path is: Internet -> AFD Premium + WAF -> Private Link -> ILB -> NGINX Ingress -> Services. Azure Front Door handles SSL termination at the edge, WAF blocks OWASP top 10 attacks, and Private Link ensures traffic never traverses the public internet. To implement, create an internal load balancer (ILB) for the NGINX ingress, disable private link service network policies, and configure AFD with a private link origin. For egress filtering, use Azure Firewall Premium. This pattern is documented in Microsoft's AKS reference architecture. For simpler setups, AFD -> Public IP -> NGINX Ingress works but exposes the cluster IP to the internet.

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress \
  --set controller.service.type=LoadBalancer \
  --set controller.service.annotations.'service\.beta\.kubernetes\.io/azure-load-balancer-internal'=true \
  --set controller.service.annotations.'service\.beta\.kubernetes\.io/azure-load-balancer-health-probe-request-path'=/healthz

15. Container Image Security and Supply Chain Protection

Securing the container supply chain is critical for AKS production. Start with Azure Container Registry (ACR) with geo-replication for resilience. Enable ACR Tasks for automated image builds and vulnerability scanning. Use Microsoft Defender for Containers to scan images on push and at runtime. Integrate with Azure Policy to allow only signed images from approved registries (using notary or cosign). For CI/CD, scan images for vulnerabilities early—fail the build if critical CVEs are found (using Trivy, Snyk, or ACR scan). For runtime security, use Azure Defender for Kubernetes to detect threats like crypto mining or privilege escalation. Enable admission controllers (Azure Policy or OPA Gatekeeper) to block deployments that use images with critical vulnerabilities. Store image pull secrets in Azure Key Vault and use the Secrets Store CSI driver. Regularly update base images and rebuild containers—weekly scans ensure you don't accumulate vulnerabilities. Microsoft recommends using distroless base images to minimize attack surface.

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sACRImageVulnerability
metadata:
  name: block-critical-vulnerabilities
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
  parameters:
    severity: "critical"
    action: "deny"
    excludedNamespaces:
      - kube-system
      - gatekeeper-system