HPA Flapping — Cold-Start Pods Trigger 15-Second Cycles

Every production system eventually hits the same wall: traffic is unpredictable, and over-provisioning is expensive while under-provisioning is catastrophic. A Black Friday spike, a viral tweet, a nightly batch job — any of these can kneecap a statically-sized deployment in minutes.

Kubernetes Horizontal Pod Autoscaler (HPA) solves the reactive scaling problem by continuously watching resource metrics and adjusting pod replica counts to match demand. But the naive 'just set CPU threshold to 80%' approach breaks in subtle and painful ways in production — flapping deployments, ignored metrics, race conditions with the Cluster Autoscaler, and custom metrics that silently stop working.

This is not a getting-started guide. It is for engineers who need to understand the HPA algorithm at the formula level, how stabilization windows prevent flapping, how to wire up custom metrics via Prometheus and KEDA, how HPA interacts with VPA and Cluster Autoscaler, and the production mistakes that wake senior engineers at 3am.

What is Kubernetes HPA — Autoscaling?

Kubernetes HPA (Horizontal Pod Autoscaler) is a control loop that automatically adjusts the number of pod replicas in a Deployment, ReplicaSet, or StatefulSet based on observed metrics. It is the primary mechanism for reactive horizontal scaling in Kubernetes. HPA does not add nodes — it adds pods. If pods cannot be scheduled due to insufficient node capacity, the Cluster Autoscaler is responsible for adding nodes.

# Basic HPA with CPU utilization target
# Package: io.thecodeforge.kubernetes
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: payment-service-hpa
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: payment-service
  minReplicas: 3
  maxReplicas: 50
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 80
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 120
      policies:
        - type: Percent
          value: 50
          periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Percent
          value: 25
          periodSeconds: 120

Metrics Server Install & Troubleshoot Guide

The Kubernetes Metrics Server is the backbone for HPA CPU/memory scaling. It collects resource metrics (CPU and memory usage) from Kubelets and exposes them through the metrics.k8s.io API. Without it, HPA cannot compute utilization percentages and will show <unknown> for resource metrics.

Installation (standard method): ``bash kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml ` This installs the Metrics Server in the kube-system` namespace. For production clusters, customize the deployment with resource limits and readiness probes.

Cloud-specific variations: - Amazon EKS: Set --kubelet-insecure-tls if using a self-signed CA (common with EKS managed node groups). - Azure AKS: Enable the addon via CLI: az aks enable-addons --addons monitoring --name <cluster> --resource-group <rg>; Metrics Server is included with Azure Monitor. - Google GKE: GKE provides a managed Metrics Server automatically. No manual install needed.

Troubleshooting checklist: 1. Verify the Metrics Server deployment is running: kubectl get deployment metrics-server -n kube-system 2. Check pod logs: kubectl logs deployment/metrics-server -n kube-system — look for TLS or authentication errors. 3. Test metric collection: kubectl top pods and kubectl top nodes. If empty, Metrics Server is not collecting. 4. Confirm API service is healthy: kubectl get apiservice v1beta1.metrics.k8s.io — status should be True. 5. If using a custom CA, pass --kubelet-preferred-address-types=InternalIP and --kubelet-insecure-tls flags. 6. Ensure each node has at least 1 vCPU and 1 GB memory – Metrics Server can be resource-hungry on large clusters. 7. For clusters with hundreds of nodes, increase the Metrics Server request limit: ``yaml resources: requests: cpu: 100m memory: 200Mi limits: cpu: 200m memory: 500Mi ``

# Custom Metrics Server deployment with TLS flags
# Package: io.thecodeforge.kubernetes
apiVersion: apps/v1
kind: Deployment
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
    k8s-app: metrics-server
spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
  template:
    metadata:
      labels:
        k8s-app: metrics-server
    spec:
      containers:
      - args:
        - --kubelet-insecure-tls
        - --kubelet-preferred-address-types=InternalIP
        - --metric-resolution=15s
        name: metrics-server
        image: registry.k8s.io/metrics-server/metrics-server:v0.6.4
        ports:
        - containerPort: 4443
          name: https
          protocol: TCP
        readinessProbe:
          httpGet:
            path: /readyz
            port: https
            scheme: HTTPS
          initialDelaySeconds: 20
          periodSeconds: 10
        livenessProbe:
          httpGet:
            path: /livez
            port: https
            scheme: HTTPS
          initialDelaySeconds: 20
          periodSeconds: 10

The HPA Algorithm: How Desired Replicas Are Calculated

The HPA algorithm is proportional, not binary. It does not simply 'add one pod' when a threshold is breached. Instead, it calculates the ratio of current metric to target metric and scales proportionally. This means high load causes rapid scale-up (doubling or more), while moderate load causes gradual adjustments.

// Simplified HPA algorithm implementation
// Package: io.thecodeforge.kubernetes.autoscaling
package io.thecodeforge.kubernetes.autoscaling;

public class HpaAlgorithm {

    /**
     * Core HPA formula:
     * desiredReplicas = ceil[currentReplicas * (currentMetricValue / targetMetricValue)]
     *
     * For multiple metrics, HPA computes the max across all metric sources.
     */
    public static int calculateDesiredReplicas(
            int currentReplicas,
            long currentMetricValue,
            long targetMetricValue,
            int minReplicas,
            int maxReplicas
    ) {
        if (targetMetricValue == 0) {
            throw new IllegalArgumentException("Target metric value cannot be zero");
        }

        double ratio = (double) currentMetricValue / targetMetricValue;
        int desired = (int) Math.ceil(currentReplicas * ratio);

        // Clamp to min/max bounds
        return Math.max(minReplicas, Math.min(desired, maxReplicas));
    }

    public static void main(String[] args) {
        // Example: 10 replicas, CPU at 1500m, target 1000m
        // desired = ceil(10 * (1500/1000)) = ceil(15) = 15
        int desired = calculateDesiredReplicas(10, 1500, 1000, 3, 50);
        System.out.println("Desired replicas: " + desired);  // 15

        // Example: 10 replicas, CPU at 500m, target 1000m
        // desired = ceil(10 * (500/1000)) = ceil(5) = 5
        int scaledDown = calculateDesiredReplicas(10, 500, 1000, 3, 50);
        System.out.println("Scaled down replicas: " + scaledDown);  // 5
    }
}

Stabilization Windows: Preventing Flapping

Stabilization windows are the primary mechanism for preventing HPA flapping — the rapid oscillation between scale-up and scale-down. During the stabilization window, HPA considers only the most conservative metric value (max for scale-up, min for scale-down) from the history of collected samples.

# HPA with tuned stabilization windows
# Package: io.thecodeforge.kubernetes
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-gateway-hpa
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api-gateway
  minReplicas: 5
  maxReplicas: 100
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 65
  behavior:
    # Scale UP: Aggressive but dampened
    scaleUp:
      stabilizationWindowSeconds: 60    # Consider max metric over last 60s
      policies:
        - type: Percent
          value: 100                      # Can double replicas per period
          periodSeconds: 60
        - type: Pods
          value: 10                       # Or add 10 pods, whichever is more
          periodSeconds: 60
      selectPolicy: Max                   # Use the policy allowing most scaling
    # Scale DOWN: Conservative and slow
    scaleDown:
      stabilizationWindowSeconds: 600    # Consider min metric over last 10 minutes
      policies:
        - type: Percent
          value: 10                       # Remove at most 10% per period
          periodSeconds: 120
      selectPolicy: Min                   # Use the most conservative policy

Custom and External Metrics: Beyond CPU and Memory

CPU and memory are often poor proxies for actual application load. A web server might be CPU-bound during image processing but network-bound during API calls. HPA supports custom metrics (per-pod metrics from Prometheus) and external metrics (cluster-external signals like SQS queue depth or Pub/Sub backlog) through the Kubernetes API aggregation layer.

# HPA with custom metrics from Prometheus Adapter
# Requires: prometheus-adapter installed and configured
# Package: io.thecodeforge.kubernetes
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: order-processor-hpa
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: order-processor
  minReplicas: 2
  maxReplicas: 40
  metrics:
    # Custom metric: requests per second from Prometheus
    - type: Pods
      pods:
        metric:
          name: http_requests_per_second
        target:
          type: AverageValue
          averageValue: "100"    # Scale when avg RPS per pod exceeds 100
    # External metric: SQS queue depth
    - type: External
      external:
        metric:
          name: sqs_queue_length
          selector:
            matchLabels:
              queue: "order-processing"
        target:
          type: AverageValue
          averageValue: "50"     # Scale when queue depth per pod exceeds 50
    # Keep CPU as fallback
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 75

Custom Metrics (Prometheus Adapter) Guide

The Prometheus Adapter is the most common way to expose application-level metrics to HPA. It implements the custom.metrics.k8s.io API and translates Prometheus queries into metric values that HPA can consume. Below is a complete guide to installing and configuring the adapter for production use.

Installation using Helm (recommended): ``bash helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm install prometheus-adapter prometheus-community/prometheus-adapter \ --namespace monitoring \ --set prometheus.url=http://prometheus-server.monitoring.svc \ --set rules.custom[0]=default=true ``

Configuration via ConfigMap: The adapter uses a series of rules that define which Prometheus series become custom metrics. Each rule specifies a seriesQuery (PromQL to find matching time series) and template transformations for pods and nodes.

``yaml # ConfigMap for prometheus-adapter rules apiVersion: v1 kind: ConfigMap metadata: name: prometheus-adapter-config namespace: monitoring data: config.yaml: | rules: - seriesQuery: 'http_requests_total' resources: overrides: namespace: {resource: "namespace"} pod: {resource: "pod"} name: matches: "^(.*)_total$" as: "${1}_per_second" metricsQuery: 'rate(<<.Series>>{<<.LabelMatchers>>}[2m]) / <<.GroupBy>>' ``

Testing custom metrics: Once the adapter is running, check the available metrics: ``bash kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 | jq . ` This should return a list of metric names. Then verify a specific metric for a pod: `bash kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/http_requests_per_second | jq . ``

Common issues: - No metrics appear: The adapter cannot connect to Prometheus. Check prometheus.url and network policies. - <unknown> in HPA: The metric query returns no data. Verify the label selectors in HPA match the pod labels. - High latency: The PromQL query is too expensive. Use aggregation (<<.GroupBy>>) and limit time range. - Adapter crashes: Memory or CPU limit too low. Increase resources and add --logtostderr for debug logs.

# Helm values for prometheus-adapter production setup
# Package: io.thecodeforge.kubernetes
prometheus:
  url: http://prometheus.monitoring.svc
  port: 9090
rules:
  default: false  # Disable default rules to avoid namespace/pod conflicts
  custom:
  - seriesQuery: 'http_requests_total'
    resources:
      overrides:
        namespace: {resource: "namespace"}
        pod: {resource: "pod"}
    name:
      matches: "^(.*)_total$"
      as: "${1}_per_second"
    metricsQuery: 'rate(<<.Series>>{<<.LabelMatchers>>}[2m]) / <<.GroupBy>>'
  - seriesQuery: 'nginx_ingress_controller_requests_total'
    resources:
      overrides:
        namespace: {resource: "namespace"}
        service: {resource: "service"}
        ingress: {resource: "ingress"}
    name:
      matches: "^(.*)_total$"
      as: "${1}_per_second"
    metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)'
resources:
  requests:
    cpu: 200m
    memory: 300Mi
  limits:
    cpu: 500m
    memory: 500Mi

HPA, VPA, and Cluster Autoscaler: The Scaling Stack

HPA, VPA (Vertical Pod Autoscaler), and Cluster Autoscaler are complementary but interact in non-obvious ways. HPA scales horizontally (more pods). VPA scales vertically (bigger pods). Cluster Autoscaler scales infrastructure (more nodes). Using them together requires careful configuration to avoid conflicts.

# VPA in 'Off' mode: Recommends but does not auto-apply resource requests
# This avoids conflict with HPA which also modifies replica counts
# Package: io.thecodeforge.kubernetes
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: payment-service-vpa
  namespace: production
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: payment-service
  updatePolicy:
    updateMode: "Off"      # CRITICAL: Do not use 'Auto' with HPA on CPU/memory
  resourcePolicy:
    containerPolicies:
      - containerName: '*'
        minAllowed:
          cpu: 100m
          memory: 128Mi
        maxAllowed:
          cpu: 4
          memory: 4Gi
        controlledResources: ["cpu", "memory"]
---
# Cluster Autoscaler: Scales nodes based on pending pods
# No YAML needed — it watches for unschedulable pods automatically
# Key flags:
# --scale-down-delay-after-add=10m
# --scale-down-unneeded-time=10m
# --max-node-provision-time=15m

VPA vs HPA vs Cluster Autoscaler Comparison Table

Understanding the differences and interactions between the three Kubernetes scaling components is critical for designing a robust autoscaling strategy. Below is a detailed comparison table.

When to combine: - HPA + Cluster Autoscaler: The most common pair. HPA adds pods; Cluster Autoscaler adds nodes when pods can't be scheduled. - VPA (Off mode) + HPA: Safe. VPA provides dashboards/charts for right-sizing requests; HPA handles actual scaling. - VPA (Auto mode) alone: Works for single-instance workloads but cannot horizontally scale.

Production pitfalls: - Running VPA in Auto mode with HPA on the same metrics causes oscillation (see callout in the scaling stack section). - Cluster Autoscaler may conflict with HPA if scale-down is too aggressive — pods terminated by HPA trigger node removal, causing new pods to be pending. - Total scaling delay = HPA reaction time + Cluster Autoscaler provisioning time. For bursty traffic, pre-provision nodes or use KEDA with scale-to-zero.

KEDA: Event-Driven Autoscaling Beyond HPA

KEDA (Kubernetes Event-Driven Autoscaler) extends HPA by enabling scale-to-zero and supporting a wider range of event sources (message queues, databases, cron schedules). KEDA acts as an HPA adapter — it creates and manages HPA resources internally, but provides a simpler API and more scaler options.

# KEDA ScaledObject: Scale based on SQS queue depth with scale-to-zero
# Package: io.thecodeforge.kubernetes
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: order-processor-scaler
  namespace: production
spec:
  scaleTargetRef:
    name: order-processor
  minReplicaCount: 0          # Scale to zero when no messages
  maxReplicaCount: 50
  pollingInterval: 15         # Check metrics every 15 seconds
  cooldownPeriod: 300         # Wait 5 minutes before scaling to zero
  triggers:
    - type: aws-sqs-queue
      authenticationRef:
        name: aws-credentials
      metadata:
        queueURL: "https://sqs.us-east-1.amazonaws.com/123456789/orders"
        queueLength: "5"        # 1 pod per 5 messages
        awsRegion: "us-east-1"
    - type: cron
      metadata:
        timezone: America/New_York
        start: "0 8 * * *"      # Pre-scale at 8 AM
        end: "0 20 * * *"       # End pre-scale at 8 PM
        desiredReplicas: "10"

Before You Begin: The Minimum Viable Cluster

You don't need a production-grade cluster to test HPA. But you do need a cluster that can actually run the Metrics Server. That means your nodes must have kubelet serving metrics on port 10250, and your CNI must allow pod-to-node traffic. Don't assume your local minikube works out of the box — verify it.

If you're using kind or k3s, check that the Metrics Server can resolve node names. This is the #1 reason people waste hours on HPA labs. The Metrics Server does not use DNS by default — it relies on node internal IPs. If your nodes don't have resolvable names, patch the Metrics Server deployment with --kubelet-use-node-status-port=false and --kubelet-preferred-address-types=InternalIP.

You also need a workload with resource requests set. HPA cannot calculate target utilization if pods don't declare requests. That turns a 5-minute setup into a 45-minute debugging session. Set CPU and memory requests on every container.

// io.thecodeforge — devops tutorial

apiVersion: apps/v1
kind: Deployment
metadata:
  name: php-apache
spec:
  replicas: 1
  selector:
    matchLabels:
      run: php-apache
  template:
    metadata:
      labels:
        run: php-apache
    spec:
      containers:
      - name: php-apache
        image: registry.k8s.io/hpa-example
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: 200m
            memory: 64Mi

HPA Status Conditions: Your Debugging Lifeline

When HPA misbehaves — and it will — Kubernetes gives you conditions that tell you exactly what's broken. Most engineers only check kubectl get hpa and see the target/current metrics. That's like checking your car's speedometer while the engine is on fire.

Run kubectl describe hpa YOUR_HPA_NAME. The Conditions section shows AbleToScale, ScalingActive, and ScalingLimited. When ScalingActive is False, it means the HPA can't read metrics. Common reasons: Metrics Server is down, or you're using a custom metric that isn't registered. When ScalingLimited is True, you've hit max or min replicas. That's not a bug — that's you not setting limits correctly.

Another hidden gem: the LastScaleTime field. If it's suspiciously old (like 5+ minutes), your HPA is stuck in backoff. HPA uses a cooldown window (usually 3-5 minutes) to prevent flapping. Check the --horizontal-pod-autoscaler-downscale-stabilization flag in the kube-controller-manager if you need to tune this.

// io.thecodeforge — devops tutorial

$ kubectl describe hpa php-apache

Name:                     php-apache
Namespace:                default
Labels:                   <none>
Annotations:              <none>
CreationTimestamp:        Mon, 20 Mar 2024 10:00:00 +0000
Reference:                Deployment/php-apache
Metrics:                  ( current / target )
  resource cpu on pods:   120m / 50%
Min replicas:             1
Max replicas:             10
Conditions:
  Type            Status  Reason              Message
  ----            ------  ------              -------
  AbleToScale     True    ReadyForNewScale    recommended size
  ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from cpu resource utilization
  ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
Events:
  Type    Reason             Age   From                       Message
  ----    ------             ----  ----                       -------
  Normal  SuccessfulRescale  3m    horizontal-pod-autoscaler  New size: 4; reason: cpu resource utilization (percentage of request) above target

Introduction: Why HPA Autoscaling Demands a Solid Foundation

Kubernetes Horizontal Pod Autoscaling (HPA) is not a set-and-forget feature. It is a control loop that continuously adjusts the number of pod replicas based on observed resource utilization or custom metrics. Without a proper grounding, teams often face cascading failures — from pod flapping under load to ignored scaling events during traffic spikes. HPA succeeds only when three conditions are guaranteed: reliable metrics collection, accurate target utilization thresholds, and a stable cluster with sufficient headroom. The goal is to match supply (pods) with demand (requests) while minimizing cost and resource waste. This guide begins at the very start: understanding what HPA needs to function and configuring the essential metrics pipeline. The following sections will take you from zero to a production-safe autoscaling setup, starting with the absolute prerequisites.

// io.thecodeforge — devops tutorial
// Minimum cluster requirements for HPA:
// 1. Kubernetes v1.23+ (v1.27 recommended)
// 2. kube-controller-manager with --horizontal-pod-autoscaler-sync-period=15s
// 3. Metrics Server v0.7+ installed and serving
// 4. Each node has at least 500m CPU and 512Mi allocatable
// 5. Resource requests set on ALL containers in target workloads
// 6. Cluster Autoscaler installed if using node-level scaling
// 7. RBAC: metrics.k8s.io API accessible by HPA controller
// 8. kubelet resource metrics endpoint enabled (default)
// 9. No resource quotas preventing replica creation
// 10. At least 2 nodes for high availability

Starter Kit: Metrics Server Configuration from the Official Helm Chart

The metrics-server is your primary source for CPU and memory metrics consumed by HPA. Incorrect configuration is the #1 reason autoscaling fails silently. Using the official Helm chart (v3.8.2+) ensures you get sane defaults for RBAC, certificates, and resource limits. The reference repository at github.com/kubernetes-sigs/metrics-server provides the canonical chart structure. Key configuration points include: enabling the KubeletInsecureTLS flag only in non-production environments (prefer CA-bundle mounting), setting --kubelet-use-node-status-port for kubelet port discovery, and configuring resource requests (minimum 50m CPU / 64Mi memory per replica). For high-availability, deploy two replicas with pod anti-affinity. The chart also supports external metrics API aggregation via --metric-resolution=60s (default 60s). Validate connectivity with kubectl get --raw /apis/metrics.k8s.io/v1beta1. Production hardening: enable --kubelet-preferred-address-types=InternalIP,ExternalIP to avoid DNS resolution failures in multi-cloud setups.

// io.thecodeforge — devops tutorial
// Helm values for metrics-server v3.8.2
metricsResolution: 60s
replicas: 2
podAntiAffinity:
  preferredDuringSchedulingIgnoredDuringExecution: true
extraArgs:
  - --kubelet-insecure-tls=false
  - --kubelet-use-node-status-port
  - --kubelet-preferred-address-types=InternalIP,ExternalIP
  - --metric-resolution=60s
resources:
  requests:
    cpu: 100m
    memory: 128Mi
  limits:
    cpu: 200m
    memory: 256Mi
service:
  type: ClusterIP
rbac:
  create: true
serviceAccount:
  create: true