Kubernetes Backup and Disaster Recovery

A single misconfigured backup job wiped out three production clusters at a fintech company last year. The root cause? They trusted a snapshot tool that didn't capture etcd state, leaving them with running pods but no service discovery or RBAC. Kubernetes backup isn't about copying YAML files—it's about preserving the entire control plane and data plane in a consistent, restorable state. Most teams learn this the hard way when a kubectl delete namespace cascades into a multi-day recovery. This article walks you through production-grade backup strategies, tooling choices, and DR patterns that actually work when the pager goes off.

Why Kubernetes Backup Is Different from Traditional VM Backup

Traditional backup tools assume a static infrastructure: you back up a disk, restore it, and the machine boots. Kubernetes is dynamic—pods move, volumes attach and detach, and the control plane manages state via etcd. A VM backup of a node won't restore your deployments, services, or ConfigMaps. You need to capture cluster resources (YAML manifests) and persistent data (volume snapshots) in a coordinated way. The key difference: Kubernetes backup must be application-aware. Restoring a database pod without its PersistentVolumeClaim is useless. Restoring a Deployment without its Service breaks traffic routing. Production backup strategies treat the cluster as a distributed system, not a collection of VMs.

# Example: Using Velero to backup cluster resources and volumes
velero backup create daily-backup --include-namespaces production --snapshot-volumes --ttl 72h

# Verify backup contents
velero backup describe daily-backup --details

Choosing the Right Backup Tool: Velero, Kasten, or DIY

The three main paths are Velero (open-source, CNCF), Kasten K10 (commercial, feature-rich), or a custom script stack. Velero is the most popular choice for teams that want control without vendor lock-in. It supports volume snapshots via CSI drivers, schedules backups, and integrates with object storage. Kasten offers policy-based automation, application-consistent backups via Kanister, and a UI—but costs money. DIY approaches using etcd snapshots + kubectl get all are fragile and miss volume data. For production, start with Velero. It's battle-tested, extensible, and has a large community. Avoid rolling your own unless you have a dedicated SRE team.

# Install Velero CLI
curl -LO https://github.com/vmware-tanzu/velero/releases/download/v1.14.0/velero-v1.14.0-linux-amd64.tar.gz
tar -xzf velero-v1.14.0-linux-amd64.tar.gz
sudo mv velero /usr/local/bin/

# Install Velero server with AWS S3 backend
velero install \
    --provider aws \
    --plugins velero/velero-plugin-for-aws:v1.9.0 \
    --bucket my-cluster-backups \
    --backup-location-config region=us-west-2 \
    --snapshot-location-config region=us-west-2 \
    --use-volume-snapshots=true

Backing Up etcd: The Control Plane's Brain

etcd is the source of truth for your cluster. If etcd is corrupted, you lose everything—even if your nodes are healthy. Back up etcd regularly using etcdctl or the snapshot API. For managed clusters (EKS, AKS, GKE), the provider handles etcd backup, but you should still verify it works. For self-managed clusters, run etcd snapshots every 5-15 minutes and store them off-cluster. A common mistake is backing up etcd without testing restoration. The etcd restore process is different from a normal backup restore—you must stop the etcd member, restore the snapshot, and restart. Practice this in a staging environment.

# Create etcd snapshot (run on control plane node)
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /backup/etcd-snapshot-$(date +%Y%m%d-%H%M%S).db

# Verify snapshot
ETCDCTL_API=3 etcdctl snapshot status /backup/etcd-snapshot-*.db

Volume Snapshots: Consistent Stateful Data

Persistent volumes hold your application data. Use CSI volume snapshots to capture point-in-time copies. Velero triggers these snapshots during backup. For databases, ensure application-consistent snapshots by quiescing writes before the snapshot (e.g., using Kanister or a pre-hook). Without quiescing, you might restore a corrupted database. Also, consider cross-region replication of snapshots for DR. Most cloud providers support copying snapshots across regions. Test restore times—restoring a 1 TB volume from a snapshot can take hours. Plan for that in your RTO.

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: csi-snapclass
  annotations:
    snapshot.storage.kubernetes.io/is-default-class: "true"
driver: ebs.csi.aws.com
deletionPolicy: Delete
parameters:
  type: gp3

Backup Strategies: Frequency, Retention, and Scheduling

Define backup frequency based on your recovery point objective (RPO). For critical stateful workloads, aim for hourly backups. For stateless apps, daily is often enough. Retention policies should balance storage cost vs. recovery needs. A common pattern: hourly backups for 24 hours, daily for 30 days, weekly for 6 months. Use Velero's TTL to automate expiration. Also, schedule backups during low-traffic windows to reduce impact. Monitor backup success rates—a failed backup that goes unnoticed is a ticking time bomb.

apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: daily-backup
  namespace: velero
spec:
  schedule: "0 2 * * *" # 2 AM daily
  template:
    includedNamespaces:
    - production
    ttl: 168h # 7 days
    snapshotVolumes: true
    storageLocation: default

Disaster Recovery Patterns: Multi-Cluster and Cross-Region

For true DR, you need a secondary cluster in a different region. The pattern: back up from the primary cluster and restore to the secondary. Velero supports this with backup storage locations. Use a single object store accessible from both clusters. For cross-region, replicate the object store bucket. When disaster strikes, restore the backup to the secondary cluster. This requires careful planning: the secondary cluster must have matching resources (StorageClasses, namespaces, etc.). Also, consider using a GitOps approach (ArgoCD) to manage manifests separately, so you only need to restore volume data.

# On secondary cluster, restore from backup
velero restore create --from-backup daily-backup-20260712 \
  --namespace-mappings production:dr-production \
  --include-resources deployments,statefulsets,services,pvc

# Verify restore
kubectl get all -n dr-production

Testing Restores: The Only Way to Know Your Backup Works

A backup that hasn't been restored is a wish. Schedule regular restore drills—monthly for critical systems. Automate the process using CI/CD pipelines. For example, spin up a temporary cluster, restore the latest backup, run smoke tests, then tear down. This validates both the backup data and the restore procedure. Document the steps and expected duration. Track restore success rate as a SLO. If a restore fails, treat it as a P1 incident.

#!/bin/bash
# Automated restore test
CLUSTER_NAME="test-restore-$(date +%s)"
kind create cluster --name $CLUSTER_NAME
velero install --provider aws --bucket test-backups ...
velero restore create --from-backup production-latest --wait
kubectl run smoke-test --image=busybox -- wget -O- http://myapp.dr-production.svc.cluster.local
if [ $? -eq 0 ]; then
  echo "Restore test passed"
else
  echo "Restore test failed"
  exit 1
fi
kind delete cluster --name $CLUSTER_NAME

Securing Backups: Encryption and Access Control

Backups contain sensitive data—secrets, configs, volume data. Encrypt them at rest and in transit. Use server-side encryption on your object store (S3 SSE-S3 or SSE-KMS). For Velero, you can enable encryption with a KMS key. Also, restrict access to backup storage: use IAM roles with least privilege. Backup data should not be accessible from the primary cluster's node roles—use a separate service account. Finally, consider backup immutability to prevent ransomware from deleting backups.

apiVersion: velero.io/v1
kind: BackupStorageLocation
metadata:
  name: default
  namespace: velero
spec:
  provider: aws
  objectStorage:
    bucket: my-cluster-backups
  config:
    region: us-west-2
    serverSideEncryption: AES256
    kmsKeyId: arn:aws:kms:us-west-2:123456789012:key/abc123

Handling Secrets and ConfigMaps in Backup

Secrets are base64-encoded, not encrypted. If your backup storage is compromised, secrets are exposed. Use a secrets management tool (Vault, AWS Secrets Manager) and store only references in Kubernetes. For backup, exclude secrets from Velero backups using --exclude-resources secrets. Alternatively, use Velero's plugin for encrypting secrets before upload. ConfigMaps are less sensitive but may contain configuration data. Treat them with similar care. Always test that your backup doesn't inadvertently leak credentials.

# Create backup excluding secrets
velero backup create no-secrets-backup \
  --include-namespaces production \
  --exclude-resources secrets \
  --snapshot-volumes

# Or use a plugin for encryption
velero plugin add velero/velero-plugin-for-aws:v1.9.0
# Then configure KMS encryption in BackupStorageLocation

Monitoring Backup Health and SLAs

Backup is a critical system—monitor it like one. Track metrics: backup duration, size, success rate, and age of last successful backup. Set up alerts for failures, partial failures (some resources failed), and backups older than expected. Use Prometheus with Velero's metrics endpoint. Define SLAs: 99.9% backup success rate over 30 days, RPO of 1 hour, RTO of 4 hours. Report these to stakeholders. If you can't meet SLAs, invest in infrastructure or process improvements.

groups:
- name: velero-alerts
  rules:
  - alert: VeleroBackupFailed
    expr: velero_backup_failure_total{status="Failed"} > 0
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "Velero backup failed for {{ $labels.schedule }}"
  - alert: VeleroBackupStale
    expr: time() - velero_backup_last_successful_timestamp > 86400
    labels:
      severity: warning
    annotations:
      summary: "No successful backup in last 24 hours"

Advanced: GitOps-Driven Backup and Restore

Combine GitOps with backup for a robust DR strategy. Use ArgoCD to manage application manifests in Git. Then, backup only persistent volume data and etcd state. To restore, deploy manifests from Git and restore volumes from snapshots. This reduces backup size and speeds up restore. It also ensures your cluster state is version-controlled. The downside: you must keep Git in sync with actual cluster state. Use ArgoCD's sync status to detect drift. This pattern is ideal for teams already using GitOps.

# On new cluster, bootstrap ArgoCD
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# Sync applications from Git
argocd app sync --all

# Restore volumes from backup
velero restore create --from-backup volumes-only-backup \
  --include-resources persistentvolumeclaims

Common Pitfalls and How to Avoid Them

1. Not testing restores: The #1 mistake. Fix: automate monthly drills. 2. Ignoring etcd backups: Managed clusters handle this, but verify. 3. Inconsistent snapshots: Database corruption. Fix: use pre/post hooks. 4. Backup storage single point of failure: Use cross-region replication. 5. Not monitoring backups: Silent failures. Fix: Prometheus alerts. 6. Overlooking network policies: Restored pods may be isolated. Fix: include NetworkPolicy in backup. 7. Assuming backup includes everything: Velero excludes some resources by default (e.g., events). Fix: review backup contents.

# List all resources in a backup
velero backup describe daily-backup --details | grep -E "(Namespaces|Resources)"

# Check for missing resource types
kubectl api-resources --verbs=list -o name | while read r; do
  if ! velero backup describe daily-backup | grep -q "$r"; then
    echo "Missing: $r"
  fi
done