GCP Instance Templates and Managed Instance Groups: Autohealing, Rolling Updates, and Regional MIGs

You've built a perfect microservice. It's stateless, containerized, and tested. Then your VM crashes at 3 AM, and you're paged because the instance didn't come back. That's the moment you realize: manual VM management is a recipe for disaster. GCP Managed Instance Groups (MIGs) exist to solve exactly this problem — they keep your fleet healthy, update it without downtime, and distribute it across zones for resilience. But MIGs are not magic; they require careful configuration of instance templates, health checks, and update strategies. Get it wrong, and you'll face cascading failures during rolling updates or silent autohealing loops. This article walks through the complete lifecycle: from crafting a production-grade instance template to deploying a regional MIG with autohealing and rolling updates. By the end, you'll know how to build a compute layer that survives real-world chaos.

Instance Templates: The Blueprint for Your VMs

An instance template is a frozen specification for creating VM instances. It defines the machine type, boot disk image, service account, metadata, startup script, and network tags. Once created, templates are immutable — you cannot edit them. To change a template, you must create a new version and update the MIG to use it. This immutability is intentional: it forces you to treat infrastructure as code and prevents configuration drift. In production, you should version your templates (e.g., my-app-template-v42) and store them in a source-controlled deployment pipeline. Common pitfalls: forgetting to set --shielded-secure-boot for compliance, using default service accounts with excessive permissions, or hardcoding secrets in metadata (use Secret Manager instead).

gcloud compute instance-templates create my-app-template-v42 \
  --machine-type=e2-standard-4 \
  --image-family=ubuntu-2204-lts \
  --image-project=ubuntu-os-cloud \
  --boot-disk-size=50GB \
  --boot-disk-type=pd-balanced \
  --service-account=my-app-sa@project.iam.gserviceaccount.com \
  --scopes=https://www.googleapis.com/auth/cloud-platform \
  --metadata=startup-script='#!/bin/bash
    apt-get update
    apt-get install -y nginx
    systemctl enable nginx
    systemctl start nginx' \
  --tags=http-server,my-app \
  --labels=env=prod,app=my-app

Managed Instance Groups: The Fleet Manager

A Managed Instance Group (MIG) takes an instance template and manages a group of identical VMs. It ensures the desired number of instances are running, automatically recreates failed instances (autohealing), and supports rolling updates, autoscaling, and regional distribution. MIGs are ideal for stateless workloads like web servers, API backends, and worker pools. They are not suitable for stateful workloads unless you use stateful MIGs (which preserve instance names and disks). When creating a MIG, you specify the template, target size, zone (for zonal MIG) or regions (for regional MIG), and optional health checks for autohealing. In production, always use a regional MIG for high availability, and set a reasonable target size (e.g., 3 per zone) to survive zone failures.

gcloud compute instance-groups managed create my-app-mig \
  --template=my-app-template-v42 \
  --size=3 \
  --zone=us-central1-a \
  --health-check=my-app-health-check \
  --initial-delay=120

Autohealing: Keeping Your Fleet Healthy

Autohealing automatically recreates VM instances that are deemed unhealthy based on a health check. The health check can be HTTP, HTTPS, TCP, or SSL. When an instance fails the health check for a configurable number of consecutive failures, the MIG terminates it and creates a new instance from the template. Autohealing is not a substitute for proper application health checks — it only detects when the VM is unresponsive. For application-level health, use a custom health check endpoint that validates dependencies (database, cache, etc.). Common mistakes: using a health check that is too permissive (e.g., just checking the process is running) or too strict (e.g., checking a downstream service that is flaky). Also, ensure the health check firewall rule allows traffic from the GCP health check ranges (35.191.0.0/16, 130.211.0.0/22).

gcloud compute health-checks create http my-app-health-check \
  --request-path=/healthz \
  --port=80 \
  --check-interval=10s \
  --timeout=5s \
  --unhealthy-threshold=3 \
  --healthy-threshold=2

Rolling Updates: Zero-Downtime Deployments

Rolling updates allow you to update the instance template of a MIG without downtime. You can control the maximum surge (how many extra instances are created) and maximum unavailable (how many instances can be down during the update). For zero-downtime, set max-surge to 1 or more and max-unavailable to 0. The MIG will create new instances with the new template, wait for them to become healthy, then delete old instances. You can also use a canary update strategy by specifying a target size for the new template (e.g., 10% of instances) to test before rolling out fully. In production, always use a rolling update with a canary step. Common pitfalls: not setting a sufficient initial-delay for the new template, or having a health check that is too slow to detect failures in the new version.

gcloud compute instance-groups managed rolling-action start-update my-app-mig \
  --version=template=my-app-template-v43 \
  --max-surge=1 \
  --max-unavailable=0 \
  --zone=us-central1-a \
  --min-ready=60

Regional MIGs: Surviving Zone Failures

A regional MIG distributes instances across multiple zones within a region. This provides high availability: if one zone goes down, the MIG automatically recreates instances in other zones to maintain the target size. Regional MIGs use a distribution policy (balanced, any, or narrow) to control how instances are spread. For production, use the default balanced distribution, which spreads instances evenly across zones. Regional MIGs also support autohealing and rolling updates across zones. However, they require a regional instance template (which is the same as a global template, but you specify the region when creating the MIG). Cost implication: you pay for inter-zone traffic if your instances communicate across zones. In production, always use regional MIGs for critical workloads.

gcloud compute instance-groups managed create my-app-regional-mig \
  --template=my-app-template-v42 \
  --size=9 \
  --region=us-central1 \
  --distribution-policy-zones=us-central1-a,us-central1-b,us-central1-c \
  --health-check=my-app-health-check \
  --initial-delay=120

Autoscaling: Dynamic Capacity Management

Autoscaling adjusts the number of instances in a MIG based on load metrics (CPU, memory, HTTP requests, or custom metrics). You define a target utilization (e.g., 80% CPU) and a cool-down period. Autoscaling works with both zonal and regional MIGs. For production, use multiple metrics to avoid oscillation, and set a minimum and maximum size. Common pitfalls: setting the cool-down period too short (causes thrashing) or too long (slow to scale up). Also, ensure your application can handle scale-in gracefully — instances are terminated without warning. Use a termination handler (e.g., systemd service) to drain connections before shutdown. Autoscaling is not a replacement for proper capacity planning; it handles traffic spikes but cannot compensate for a fundamentally under-provisioned fleet.

gcloud compute instance-groups managed set-autoscaling my-app-mig \
  --max-num-replicas=10 \
  --min-num-replicas=3 \
  --target-cpu-utilization=0.8 \
  --cool-down-period=120 \
  --zone=us-central1-a

Stateful MIGs: When You Need Persistent State

Stateful MIGs preserve instance names, disks, and metadata across autohealing and rolling updates. Use them for stateful workloads like databases, legacy applications, or any service that requires stable hostnames. Stateful MIGs support per-instance configurations: you can attach persistent disks to specific instances and preserve them on recreation. However, stateful MIGs have limitations: rolling updates are more complex (you must use a canary or manual update), and autoscaling is not supported (because stateful instances are not interchangeable). In production, use stateful MIGs sparingly — prefer stateless architectures with external state storage (Cloud SQL, Memorystore, etc.). If you must use stateful MIGs, ensure you have backups and a disaster recovery plan.

gcloud compute instance-groups managed create my-stateful-mig \
  --template=my-app-template-v42 \
  --size=3 \
  --zone=us-central1-a \
  --stateful-disk=device-name=my-disk,source=projects/my-project/zones/us-central1-a/disks/my-disk \
  --stateful-internal-ip \
  --stateful-external-ip

Named Ports: Service Discovery for Load Balancers

Named ports map a port name to a port number for a MIG. They are used by load balancers (HTTP(S), TCP/UDP) to forward traffic to the correct port on instances. For example, you can define a named port 'http' mapping to port 8080. When you create a backend service, you reference the named port instead of a hardcoded port. This decouples the load balancer configuration from the actual port, allowing you to change the port in the instance template without updating the load balancer. In production, always use named ports for flexibility. Common mistake: forgetting to update the named port when changing the application port, causing traffic to be sent to the wrong port.

gcloud compute instance-groups managed set-named-ports my-app-mig \
  --named-ports=http:8080,metrics:9090 \
  --zone=us-central1-a

Resize and Capacity Management

You can manually resize a MIG by setting a new target size. The MIG will create or delete instances to match. Resizing is useful for planned capacity changes (e.g., scaling down during off-peak hours). When resizing down, the MIG selects instances to delete based on a selection policy (default is to delete the newest instances first). You can also use a proactive selection policy to delete instances in a specific zone. In production, avoid resizing down too aggressively — leave a buffer to handle traffic spikes. Also, ensure your application can handle scale-in gracefully. For regional MIGs, resizing maintains the distribution policy across zones.

gcloud compute instance-groups managed resize my-app-mig \
  --size=5 \
  --zone=us-central1-a

Resize Requests: All-or-Nothing Provisioning

Resize requests provide all-or-nothing capacity provisioning for MIGs, ensuring that either all requested VMs are created successfully or none are. This is critical for GPU workloads, ML training jobs, and HPC clusters that require a specific number of accelerator VMs to start simultaneously. Without resize requests, a partial allocation could leave a training job with fewer GPUs than needed, causing it to fail or run slowly. Resize requests use the same MIG but add a guarantee that the requested number of VMs are provisioned atomically. Production insight: we use resize requests for all GPU-based MIGs. A training job that requires 8 A100 GPUs must have all 8 available before starting. Without resize requests, the MIG would create 5 VMs, start the job, and fail. Add a timeout to avoid indefinite waiting.

gcloud compute instance-groups managed resize-requests create my-gpu-mig-resize \
  --group=my-gpu-mig \
  --zone=us-central1-a \
  --size=8 \
  --resize-request-timeout=3600s

# Check status
gcloud compute instance-groups managed resize-requests describe my-gpu-mig-resize \
  --group=my-gpu-mig \
  --zone=us-central1-a

Unmanaged Instance Groups and MIG Comparison

Unmanaged instance groups (UIGs) contain heterogeneous instances that you add and remove manually. They do not support autoscaling, autohealing, rolling updates, instance templates, or multi-zone distribution. Use them only for legacy workloads that need load balancing across dissimilar VMs or for instances you need to manage individually. For everything else, use managed instance groups. A common mistake is creating a UIG when you need a MIG—you lose autohealing, scaling, and update capabilities. Production insight: we inherited a UIG with 50 manually managed VMs for a production service. Every VM had a different configuration. Migrating to a MIG with a consistent template reduced incidents by 90%.

# Create an unmanaged instance group (not recommended for production)
gcloud compute instance-groups unmanaged create legacy-group \
  --zone=us-central1-a \
  --description="Legacy instances, not auto-managed"

# Add existing instances
gcloud compute instance-groups unmanaged add-instances legacy-group \
  --zone=us-central1-a \
  --instances=old-vm-1,old-vm-2

# Note: no autoscaling, no autohealing, no rolling updates

Monitoring and Debugging MIGs

Monitoring MIGs is critical for production. Key metrics: instance count, target size, autoscaling metrics, health check status, and rolling update progress. Use Cloud Monitoring to set up alerts for unexpected instance count changes or high error rates. For debugging, use gcloud compute instance-groups managed list-instances to see the state of each instance (RUNNING, UNHEALTHY, etc.). Common issues: instances stuck in 'UNHEALTHY' state due to health check misconfiguration, or 'CREATING' state for too long due to slow startup scripts. Enable serial port logging to capture instance boot logs. In production, set up a dashboard that shows MIG health and autoscaling activity.

gcloud compute instance-groups managed list-instances my-app-mig \
  --zone=us-central1-a

MIG Stability: Understanding status.isStable

Every MIG has a status.isStable field that indicates whether all pending operations are complete. When isStable is false, changes are active or pending—instances are being created, deleted, updated, or redistributed. This is useful for automation: wait for isStable: true before proceeding with dependent operations (e.g., updating a load balancer after a rolling update). Stability is affected by: rolling updates, autoscaling, autohealing, resize operations, and zonal redistribution in regional MIGs. Production insight: never assume a MIG is stable immediately after a rolling update. Poll isStable in your CI/CD pipeline before running smoke tests. We once started integration tests while a MIG was still updating instances, causing flaky test results.

# Check if a MIG is stable
gcloud compute instance-groups managed describe my-app-mig \
  --zone=us-central1-a \
  --format="value(status.isStable)"

# Wait for stability in a script
while true; do
  STABLE=$(gcloud compute instance-groups managed describe my-app-mig \
    --zone=us-central1-a \
    --format="value(status.isStable)")
  if [ "$STABLE" = "True" ]; then
    echo "MIG is stable"
    break
  fi
  echo "Waiting for MIG to stabilize..."
  sleep 10
done

Best Practices for Production MIGs

1. Always use regional MIGs for production workloads to tolerate zone failures. 2. Use a canary rolling update strategy: update a small percentage of instances first, monitor for errors, then proceed. 3. Set initial-delay to at least your application's startup time plus a buffer (e.g., 120s). 4. Design health checks to reflect true application health, not just process liveness. 5. Use named ports for load balancer backends. 6. Implement graceful shutdown handlers for scale-in and rolling updates. 7. Store instance templates in a version-controlled repository and automate their creation. 8. Set up monitoring and alerts for MIG health and autoscaling events. 9. Test autohealing and rolling updates in a staging environment before production. 10. Document your MIG configuration and recovery procedures.

# Canary update: update 10% of instances first
gcloud compute instance-groups managed rolling-action start-update my-app-mig \
  --version=template=my-app-template-v43,target-size=1 \
  --version=template=my-app-template-v42 \
  --max-surge=1 \
  --max-unavailable=0 \
  --zone=us-central1-a \
  --min-ready=60

# After monitoring, update the rest
gcloud compute instance-groups managed rolling-action start-update my-app-mig \
  --version=template=my-app-template-v43 \
  --max-surge=1 \
  --max-unavailable=0 \
  --zone=us-central1-a