etcd Operations for Kubernetes — Backup, Restore, Tuning

Welcome to etcd Operations for Kubernetes — Backup, Restore, Tuning. We'll break this down from first principles.

Understanding etcd's Role in Kubernetes

etcd is the brain of Kubernetes. Every kubectl command, every pod scheduling decision, every API object — it all ends up as a key-value pair in etcd. The Kubernetes API server is the only component that talks to etcd directly; all other components go through the API server. This architecture means that etcd's health directly impacts cluster responsiveness and reliability. In production, etcd failures manifest as API server timeouts, node heartbeat loss, and eventual cluster unavailability. Understanding etcd's internal data model — hierarchical keys with leases and watch mechanisms — is essential for debugging and performance tuning. The default etcd deployment in Kubernetes uses a 3-node cluster for high availability, but many operators underestimate the operational complexity of maintaining quorum and handling network partitions.

#!/bin/bash
# Check etcd cluster health
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  endpoint health --cluster

etcd Backup Strategies: Snapshot vs. Streaming

Backing up etcd is non-negotiable. There are two primary methods: snapshot-based and streaming (e.g., Velero). Snapshots capture the entire etcd data store at a point in time and are the most reliable for disaster recovery. The etcdctl snapshot save command creates a consistent snapshot by coordinating with the raft consensus. Streaming backups, while useful for continuous protection, introduce complexity because they must capture every write and can miss state if the backup process lags. For production, always take periodic snapshots (e.g., every 2 hours) and store them off-cluster. Never rely solely on streaming backups for etcd. The snapshot file is a single file that can be restored on any etcd version that is compatible (same major.minor). Always test your restore process in a non-production environment before you need it.

#!/bin/bash
# Create etcd snapshot
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /backup/etcd-snapshot-$(date +%Y%m%d-%H%M%S).db

# Verify snapshot integrity
ETCDCTL_API=3 etcdctl --write-out=table snapshot status /backup/etcd-snapshot-*.db

Restoring etcd from Snapshot

Restoring etcd from a snapshot is a controlled process that requires stopping all etcd members, restoring data, and restarting with a fresh cluster. The key command is etcdctl snapshot restore, which creates a new data directory from the snapshot file. You must restore on a single node first, then add other members as new nodes. Never attempt to restore a snapshot onto an existing etcd member — it will refuse because the cluster ID and member ID don't match. The restore process resets the cluster ID, so all other members must be re-added. After restore, verify that the API server can connect and that all Kubernetes objects are present. Common pitfalls: using the wrong snapshot file (e.g., from a different cluster), mismatched etcd versions, and forgetting to update the etcd member's initial-cluster flag.

#!/bin/bash
# Stop etcd (systemd example)
systemctl stop etcd

# Restore snapshot to a new data directory
ETCDCTL_API=3 etcdctl snapshot restore /backup/etcd-snapshot-20260712-120000.db \
  --name etcd-0 \
  --initial-cluster etcd-0=https://10.0.0.1:2380 \
  --initial-cluster-token etcd-cluster-1 \
  --initial-advertise-peer-urls https://10.0.0.1:2380 \
  --data-dir /var/lib/etcd-restored

# Move restored data into place
mv /var/lib/etcd /var/lib/etcd.old
mv /var/lib/etcd-restored /var/lib/etcd

# Start etcd
systemctl start etcd

# Verify health
ETCDCTL_API=3 etcdctl endpoint health

etcd Performance Tuning: Disk I/O and Network

etcd is I/O sensitive. The single most important performance factor is disk latency — etcd requires a high-performance SSD with low p99 latency (under 10ms). Use dedicated disks for etcd data and WAL (write-ahead log) to avoid contention with other workloads. The etcd --fsync duration is a critical metric: if it exceeds 100ms, you risk raft election timeouts. Network latency between etcd members should be under 10ms round-trip; deploy all members in the same availability zone if possible. Tune the --heartbeat-interval (default 100ms) and --election-timeout (default 1000ms) for your network conditions. For large clusters (over 1000 nodes), increase --quota-backend-bytes to 8GB or more to handle the increased key count. Monitor etcd's db size — if it exceeds the quota, etcd will stop accepting writes.

# etcd configuration for high-performance
# Place in /etc/etcd/etcd.conf
name: 'etcd-0'
data-dir: /var/lib/etcd
wal-dir: /var/lib/etcd/wal
listen-client-urls: https://0.0.0.0:2379
listen-peer-urls: https://0.0.0.0:2380
initial-cluster: 'etcd-0=https://10.0.0.1:2380,etcd-1=https://10.0.0.2:2380,etcd-2=https://10.0.0.3:2380'
initial-cluster-state: new
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 8589934592  # 8 GB
max-request-bytes: 1572864  # 1.5 MB
auto-compaction-mode: periodic
auto-compaction-retention: '1h'

etcd Compaction and Defragmentation

etcd compacts history periodically to free up space. By default, Kubernetes enables auto-compaction with a retention of 1 hour. However, compaction only marks space as free; it does not reclaim it. Over time, the etcd database file grows due to fragmentation. Defragmentation (etcdctl defrag) reclaims this space and should be run periodically (e.g., daily) on each member. Defrag is a blocking operation — it stops serving requests on that member while running. In a multi-member cluster, you can defrag one member at a time without downtime. Monitor the db size via metrics; if it approaches the quota, defrag immediately. For clusters with heavy churn (e.g., frequent pod creations/deletions), consider increasing compaction retention or running defrag more often.

#!/bin/bash
# Defragment etcd member (run on each member sequentially)
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  defrag

# Check db size after defrag
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  endpoint status --write-out=table | awk '{print $4}'

Monitoring etcd Health and Metrics

etcd exposes Prometheus metrics on the client port (2379). Key metrics to monitor: etcd_server_has_leader (should be 1), etcd_server_leader_changes_seen_total (should be low), etcd_disk_wal_fsync_duration_seconds_bucket (p99 < 10ms), etcd_network_client_grpc_sent_bytes_total (traffic volume), and etcd_mvcc_db_total_size_in_bytes (db size). Set up alerts for: no leader for >10 seconds, high fsync latency, db size >80% of quota, and frequent leader changes. Use etcdctl endpoint status and etcdctl endpoint health for quick checks. Integrate with your monitoring stack (Prometheus + Grafana) for historical trends. A common pitfall is monitoring only the local etcd member — always check all members from an external perspective.

# Alert: etcd no leader
alert: EtcdNoLeader
expr: etcd_server_has_leader == 0
for: 1m
labels:
  severity: critical
annotations:
  summary: "etcd cluster has no leader"

# Alert: high fsync latency
alert: EtcdHighFsyncLatency
expr: histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])) > 0.1
for: 5m
labels:
  severity: warning
annotations:
  summary: "etcd WAL fsync latency is high"

# Alert: db size approaching quota
alert: EtcdDbSizeHigh
expr: etcd_mvcc_db_total_size_in_bytes / etcd_server_quota_backend_bytes > 0.8
for: 10m
labels:
  severity: warning
annotations:
  summary: "etcd db size is above 80% of quota"

etcd Security: TLS and Authentication

etcd supports TLS for client-to-server and peer-to-peer communication, as well as username/password authentication. In Kubernetes, etcd is typically deployed with TLS certificates issued by the cluster's CA. Never expose etcd's client port (2379) to the public internet. Use firewall rules to restrict access to only the API server nodes. Enable client certificate authentication to ensure only the API server can connect. For peer communication, use separate certificates to prevent unauthorized nodes from joining the cluster. Rotate certificates before they expire — expired certificates cause immediate cluster failure. Use etcdctl with --cacert, --cert, and --key flags for secure access. For additional security, enable role-based access control (RBAC) on etcd itself, though this is rarely needed in Kubernetes contexts.

#!/bin/bash
# Verify TLS certificates
openssl x509 -in /etc/kubernetes/pki/etcd/server.crt -text -noout | grep -E 'Subject:|Not After'

# Test TLS connection
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  endpoint health

Scaling etcd for Large Clusters

etcd performance degrades as the number of keys and watch clients grows. For clusters with over 1000 nodes, consider tuning: increase --quota-backend-bytes to 8GB or more, reduce --heartbeat-interval to 50ms, and increase --election-timeout to 2000ms to avoid false elections. Use a dedicated etcd cluster with 5 members for better fault tolerance (tolerates 2 failures instead of 1). Avoid putting etcd on the same node as other heavy workloads. For extremely large clusters (5000+ nodes), consider using a separate etcd cluster for events only (via --etcd-servers-overrides). Monitor the number of watchers — each pod and controller creates watches. If watchers exceed 10000, consider using a more efficient watch mechanism or sharding.

# etcd configuration for large clusters
name: 'etcd-0'
data-dir: /var/lib/etcd
wal-dir: /var/lib/etcd/wal
listen-client-urls: https://0.0.0.0:2379
listen-peer-urls: https://0.0.0.0:2380
initial-cluster: 'etcd-0=https://10.0.0.1:2380,etcd-1=https://10.0.0.2:2380,etcd-2=https://10.0.0.3:2380,etcd-3=https://10.0.0.4:2380,etcd-4=https://10.0.0.5:2380'
initial-cluster-state: new
heartbeat-interval: 50
election-timeout: 2000
quota-backend-bytes: 17179869184  # 16 GB
max-request-bytes: 1572864  # 1.5 MB
auto-compaction-mode: periodic
auto-compaction-retention: '30m'

Disaster Recovery: Full Cluster Restore

A full cluster restore is needed when all etcd members are lost (e.g., entire region failure). The process: provision new etcd nodes, restore the latest snapshot on one node, start it as a single-member cluster, then add other members. After restore, verify that the API server can connect and that all Kubernetes objects are present. If the snapshot is old, you may lose recent changes — always combine with application-level backups. Test your DR plan quarterly. A common mistake is restoring a snapshot from a different cluster (different cluster ID) — etcd will refuse to start. Use etcdctl snapshot status to verify the snapshot's cluster ID matches your backup records. Document the exact steps and practice them in a staging environment.

#!/bin/bash
# Full cluster restore on new nodes
# Step 1: Restore on first node
ETCDCTL_API=3 etcdctl snapshot restore /backup/etcd-snapshot-latest.db \
  --name etcd-0 \
  --initial-cluster etcd-0=https://10.0.0.1:2380 \
  --initial-cluster-token etcd-cluster-restore \
  --initial-advertise-peer-urls https://10.0.0.1:2380 \
  --data-dir /var/lib/etcd

# Start etcd on node 1
systemctl start etcd

# Step 2: On node 2, restore with different name
ETCDCTL_API=3 etcdctl snapshot restore /backup/etcd-snapshot-latest.db \
  --name etcd-1 \
  --initial-cluster etcd-0=https://10.0.0.1:2380,etcd-1=https://10.0.0.2:2380 \
  --initial-cluster-token etcd-cluster-restore \
  --initial-advertise-peer-urls https://10.0.0.2:2380 \
  --data-dir /var/lib/etcd

# Start etcd on node 2
systemctl start etcd

# Repeat for node 3...

# Verify cluster health
ETCDCTL_API=3 etcdctl --endpoints=https://10.0.0.1:2379,https://10.0.0.2:2379,https://10.0.0.3:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  endpoint health --cluster

etcd Operator and Automation

For large-scale deployments, consider using the etcd Operator (from CoreOS) to automate cluster management. The operator handles backups, restores, scaling, and upgrades. It uses Custom Resource Definitions (CRDs) to define etcd clusters. However, the operator adds complexity and is not recommended for small clusters. For Kubernetes clusters managed by kubeadm, the static pod approach is simpler. If you use the operator, ensure you understand its backup and restore mechanisms — they differ from manual etcdctl commands. The operator can also automate defragmentation and compaction. Always test operator upgrades in a staging environment first.

apiVersion: etcd.database.coreos.com/v1beta2
kind: EtcdCluster
metadata:
  name: my-etcd-cluster
spec:
  size: 3
  version: 3.5.9
  pod:
    resources:
      requests:
        cpu: 200m
        memory: 512Mi
      limits:
        cpu: 500m
        memory: 1Gi
    etcdEnv:
      - name: ETCD_AUTO_COMPACTION_RETENTION
        value: '1h'

Common etcd Failure Modes and Mitigations

etcd failures often stem from disk latency, network partitions, or misconfiguration. Common failure modes: (1) Disk full — etcd stops accepting writes. Mitigation: monitor disk usage and set up alerts. (2) Network partition — a minority of members become isolated, causing leader election failure. Mitigation: deploy all members in the same AZ or use a 5-member cluster. (3) Slow disk — fsync latency exceeds election timeout, causing frequent leader changes. Mitigation: use dedicated SSDs with provisioned IOPS. (4) Certificate expiry — members cannot communicate. Mitigation: automate certificate renewal. (5) Quota exceeded — etcd stops writes. Mitigation: increase quota or defrag. (6) Raft log corruption — rare but catastrophic. Mitigation: regular backups and restore drills. Each failure mode should have a documented runbook.

#!/bin/bash
# Check disk latency and space
echo "Disk latency (fsync):"
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  endpoint status --write-out=table | awk '{print $6}'

echo "Disk usage:"
df -h /var/lib/etcd

etcd Version Upgrades

Upgrading etcd requires careful planning. Always upgrade one minor version at a time (e.g., 3.4 to 3.5, not 3.4 to 3.6). The Kubernetes API server must be compatible with the etcd version — check the Kubernetes release notes for supported etcd versions. The upgrade process: (1) Take a snapshot. (2) Upgrade one member at a time: stop etcd, replace binary, start etcd. (3) Verify cluster health after each member. (4) After all members are upgraded, verify API server connectivity. Rolling upgrades are safe because etcd supports mixed-version clusters during the upgrade window. However, avoid making configuration changes during the upgrade. Downgrades are not supported — if an upgrade fails, restore from snapshot.

#!/bin/bash
# Upgrade etcd on a single member
# 1. Take snapshot
ETCDCTL_API=3 etcdctl snapshot save /backup/pre-upgrade-snapshot.db

# 2. Stop etcd
systemctl stop etcd

# 3. Replace binary
cp /usr/local/bin/etcd /usr/local/bin/etcd.old
cp /tmp/etcd-v3.5.9-linux-amd64/etcd /usr/local/bin/etcd

# 4. Start etcd
systemctl start etcd

# 5. Verify health
ETCDCTL_API=3 etcdctl endpoint health

# Repeat on other members