Redis MISCONF RDB Snapshot — Disk Fix and Persist

MISCONF Redis is configured to save RDB snapshots, but it's currently unable to persist to disk. Commands that may modify the data set are disabled. It appears during the night nobody watches: logs fill the volume, a deploy changes the data-dir owner, or a forked bgsave hits memory pressure — and Redis chooses safety over availability, refusing writes until persistence works again.

That refusal surprises teams: the server is up, reads work, RAM looks fine — but every write fails. The stop-writes-on-bgsave-error yes default is deliberate protection: without it, Redis would accept writes it can no longer snapshot, turning a disk problem into silent data-loss exposure on restart.

This guide diagnoses through INFO persistence, fixes disk and permission causes, restarts snapshots cleanly, handles the stop-writes flag honestly, adds AOF as a second durability leg, and monitors persistence so the next failure pages before writes halt. The same drill covers every node type — primaries, replicas, and sentinels alike.

Read the Error and the Persistence State

MISCONF names the policy (configured to save, currently unable) and the consequence (writes disabled) — but not the cause. INFO persistence supplies it: rdb_last_bgsave_status ok/err, rdb_last_bgsave_time_sec (-1 means never failed... actually time of last attempt), rdb_changes_since_last_save (how much is at risk), and rdb_bgsave_in_progress. Pair with LASTSAVE (Unix timestamp of the last good snapshot): err status plus a LASTSAVE hours old is a failing snapshotter, full stop.

Read the server log for the one-line cause BGSAVE prints on failure: 'Failed opening the RDB file dump.rdb for saving: Permission denied' (EACCES — ownership/mount), versus 'Can't save in background: fork: Cannot allocate memory' (memory/overcommit), versus write errors mid-save (ENOSPC). Each maps to a different section below, so capture the line before acting — the fix for permissions does nothing for a full disk.

Record the risk number too: rdb_changes_since_last_save counts writes since the last good snapshot, i.e., your exposure if the box dies now. Thousands of changes reframes the incident from 'writes halted' to 'writes halted AND prior writes unsnapshotted' — which sets the urgency for the disk fix versus the flag workaround correctly.

# State machine: status, last attempt, exposure since last good save
redis-cli -h cache-01 INFO persistence | grep -E \
  'rdb_last_bgsave_status|rdb_last_bgsave_time_sec|rdb_changes_since_last_save|rdb_bgsave_in_progress'

# Last successful snapshot (stale = failing snapshots)
redis-cli -h cache-01 LASTSAVE
date +%s  # compare: gap is unsnapshotted exposure

# The one-line cause BGSAVE printed on failure
sudo grep -iE 'background saving|failed opening|cannot allocate' \
  /var/log/redis/redis-server.log | tail -5

Disk Full or Permissions: The Two Usual Causes

ENOSPC arrives via sharing: app logs, journald, or backups on the same volume as dump.rdb, growing until the next forked save finds zero bytes. Diagnose with df -h on the CONFIG GET dir path (not / — Redis may live on a dedicated mount whose fullness root's df hides among healthy lines), then du-sort the volume to name the hog. Reclaim non-Redis bytes only: stale logs, vacuumed journals, orphaned temps. dump.rdb and appendonly files are recovery assets, never cleanup candidates.

EACCES arrives via ownership: config management, manual chowns, or container volume mounts leaving the data dir root-owned or read-only. ls -ld plus ls -l on the dir and file name it; chown -R redis:redis plus mode 750 repairs it; /proc/mounts reveals read-only mounts needing remounts. The deploy that broke ownership will re-break it nightly unless its template is fixed — audit the config management diff, not just the directory.

Fork failures are the third, rarer cause: Cannot allocate memory on huge instances without overcommit tuning (vm.overcommit_memory=1). BGSAVE needs copy-on-write headroom proportional to write churn during the save; check the log line, set overcommit, and consider active defrag plus smaller save frequencies for multi-GB instances. Memory-cause MISCONF masquerades as disk — the log line distinguishes them.

# Which volume? (CONFIG dir, not /) + who hogs it?
redis-cli -h cache-01 CONFIG GET dir
redis-cli -h cache-01 CONFIG GET dbfilename
df -h /var/lib/redis
du -sh /var/lib/redis/* 2>/dev/null | sort -rh | head

# ENOSPC: reclaim non-Redis bytes (never dump.rdb / appendonly files)
sudo journalctl --vacuum-size=500M
sudo find /var/log/app -name '*.log.*' -mtime +7 -delete

# EACCES: repair ownership + mode
ls -ld /var/lib/redis
sudo chown -R redis:redis /var/lib/redis
sudo chmod 750 /var/lib/redis

The stop-writes Flag: Protection, Not Punishment

stop-writes-on-bgsave-error yes is the default for a reason: it converts silent durability loss into a loud, pageable write halt. With the flag on, a failed snapshot stops writes — painful, visible, recoverable. With it off, Redis accepts writes it cannot snapshot, and a subsequent crash loses everything since the last good save with no prior warning. The flag doesn't cause the incident; it announces the incident that already happened.

Use CONFIG SET stop-writes-on-bgsave-error no strictly as a timeboxed bridge: writes resume while you fix the disk in the same window, incident stays open, revert within the hour via CONFIG SET ... yes plus CONFIG REWRITE. Document the window in the ticket with timestamps — 'flag off 22:04, disk fixed 22:31, flag on 22:32' — so the next review sees a controlled bridge, not a casual dismissal of durability.

Never accept 'leave it off' as the resolution, from anyone, for any convenience argument. Guides recommending permanent disable trade real durability for uptime theater: green dashboards over a snaptureless server. The correct permanent fix is always working persistence — disk, permissions, memory — with the flag on, verified by LASTSAVE advancing.

# Current posture (expect yes)
redis-cli -h cache-01 CONFIG GET stop-writes-on-bgsave-error

# BRIDGE ONLY: resume writes while fixing disk (timebox < 1h, ticket stays open)
redis-cli -h cache-01 CONFIG SET stop-writes-on-bgsave-error no

# ... fix disk/permissions NOW ...

# Restore protection + persist config
redis-cli -h cache-01 CONFIG SET stop-writes-on-bgsave-error yes
redis-cli -h cache-01 CONFIG REWRITE

Restart Persistence After the Fix

With cause fixed, restart the snapshotter deliberately: BGSAVE (expect 'Background saving started'), then poll LASTSAVE until the timestamp advances past the incident window. An advancing LASTSAVE is the only proof that matters — config reads and log optimism don't count. Confirm INFO persistence flips rdb_last_bgsave_status to ok and rdb_changes_since_last_save resets toward zero as the fresh snapshot absorbs the backlog.

If BGSAVE fails again immediately, re-read the log line — the second failure often differs from the first (permissions fixed, now ENOSPC from the backlog's fork needs; or the temp file colliding). Fix forward through each distinct cause; each BGSAVE attempt prints exactly one. Two different failures in sequence is normal progress, not a new incident.

Persist any config changes with CONFIG REWRITE so restarts keep them — runtime SETs evaporate on restart, and a failover that resurrects the old broken dir or the disabled flag reopens the incident on the new primary. Verify the rewritten redis.conf contains the corrected dir, dbfilename, and flag before closing the ticket. A failover that resurrects stale config reopens the incident on the new primary within hours.

# Trigger + prove: LASTSAVE must advance past the incident window
redis-cli -h cache-01 BGSAVE
sleep 5
redis-cli -h cache-01 LASTSAVE
sleep 10
redis-cli -h cache-01 LASTSAVE  # must be newer than before

# Status flip + backlog absorbed
redis-cli -h cache-01 INFO persistence | grep -E \
  'rdb_last_bgsave_status|rdb_changes_since_last_save'

# Persist runtime config so failovers keep the fix
redis-cli -h cache-01 CONFIG REWRITE

AOF Fallback: Durability Without Fork Storms

AOF (append-only file) logs every write and rewrites compactly on schedule — second-granularity durability (appendfsync everysec) without RDB's fork-per-save cost profile. Enabling it beside RDB gives two independent recovery legs: snapshots for fast restarts, logs for fine-grained replay. When RDB fork pressure (huge instance, write churn) is itself the MISCONF cause, AOF carries durability while you retune snapshot frequency.

Enable online: CONFIG SET appendonly yes starts logging immediately; BGREWRITEAOF compacts the initial bulk; CONFIG REWRITE persists both. Expect steady I/O (everysec fsync) instead of bursty fork load — size the volume's IOPS for the stream, not the snapshot. Monitor aof_last_rewrite_time_sec and aof_last_bgrewrite_status exactly like their RDB twins; a second leg needs the same watchdog.

Keep both legs monitored and tested: restart drills that recover from RDB, point-in-time replays from AOF, and corruption checks (redis-check-aof) quarterly. An untested durability leg is a hope, and hopes don't restore sessions at 2 AM. Two legs, both watched, both drilled — that is the posture that survives disks, forks, and failures together every single time without exception.

# Second durability leg, enabled online
redis-cli -h cache-01 CONFIG SET appendonly yes
redis-cli -h cache-01 BGREWRITEAOF
sleep 10
redis-cli -h cache-01 INFO persistence | grep -E \
  'aof_enabled|aof_last_bgrewrite_status|aof_last_rewrite_time_sec'

# Persist + pin in redis.conf for restarts
redis-cli -h cache-01 CONFIG REWRITE
grep -E '^(appendonly|appendfsync|dir|dbfilename)' /etc/redis/redis.conf

# Quarterly drill asset
redis-check-aof --fix /var/lib/redis/appendonly.aof

Prevention: Disk Alerts, Backups, and Replicas

Isolate persistence onto its own volume at provisioning: dump.rdb, AOF, and temp files share one disk sized for snapshots plus growth — never with app logs, never with backups staging. Alert at 75% with a 24-hour growth projection (df trends, not just thresholds); a volume filling 2 GB/day pages days before ENOSPC, not minutes. Logrotate configs for anything near Redis get monitored checksums — the incident's root cause was a silently disabled rotator.

Watch the full persistence triad per node: disk %, rdb_last_bgsave_status, and LASTSAVE age (stale beyond 2× the save interval pages). A cron watchdog over redis-cli INFO plus df covers all three in ten lines and pages before writes halt — the 12 green minutes of read-only health checks never recur. Cover replicas identically: a replica that can't persist restarts empty, and Sentinel won't warn you.

Ship snapshots off-box: rdb backups to object storage hourly, AOF segments with the same cadence, restore drills monthly. On-box persistence protects against crashes; off-box copies protect against everything else — volume loss, AZ failure, the rm -rf that no flag can stop. Durability is a chain from fork to off-box copy, and every link gets a monitor.

#!/bin/bash
# Watchdog: disk %, bgsave status, LASTSAVE age (page before writes halt)
set -euo pipefail
HOST=cache-01
PCT=$(df -h /var/lib/redis | awk 'NR==2{print $5}' | tr -d '%')
ST=$(redis-cli -h $HOST INFO persistence | grep rdb_last_bgsave_status | cut -d: -f2 | tr -d '\r')
AGE=$(python3 -c "import time; print(int(time.time())-$(redis-cli -h $HOST LASTSAVE))")
[ "$PCT" -ge 75 ] && { echo "PAGE: disk ${PCT}%"; exit 1; }
[ "$ST" != "ok" ] && { echo "PAGE: bgsave status $ST"; exit 1; }
[ "$AGE" -gt 7200 ] && { echo "PAGE: LASTSAVE ${AGE}s stale"; exit 1; }
echo "OK: disk ${PCT}% bgsave $ST lastsave ${AGE}s ago"