Kafka Leader Not Available After Broker Restart

You restart one broker for a routine patch, and within seconds the alerts start: LeaderNotAvailableException across three topics, producers throwing, consumers stalling. Your first instinct is that the restart broke something. In most cases it didn't — you're watching leader election do its job, and the scariest minute of Kafka operations is also the most normal.

Still, 'normal' covers a wide range. A healthy election finishes in under two minutes and nobody pages. A stuck one — no surviving ISR replica, a durability config that can't be satisfied, clients caching the leaderless state — sits at leader -1 for twenty minutes while you restart brokers that were fine, making everything worse.

This guide teaches you to tell those two apart fast. You'll learn what the leader field and ISR list actually say during an outage, when unclean leader election is (rarely) justified, how min.insync.replicas and acks interact under failure, and the exact recovery sequence — preferred-replica-election included — that restores balanced leadership instead of a fragile pile-up on survivors.

What LeaderNotAvailableException Actually Means

Every partition has exactly one leader — the replica that accepts produces and serves reads — plus followers that replicate. Clients learn the leader from metadata responses, and LeaderNotAvailableException is the client's way of saying the metadata listed no leader for that partition. It's a routing failure, not a data verdict: the records may be perfectly safe on disk with nobody currently authorized to serve them.

Two very different situations produce the same exception, and confusing them causes most of the damage in these incidents. In situation one, an election is in flight: the old leader is gone, a new one is being chosen from the in-sync replica set (ISR), and the leader field will repopulate in seconds. In situation two, no ISR member is alive, so no legitimate election can complete and the field sits at -1 indefinitely. The first needs patience; the second needs a replica restored.

Your first command decides which world you're in: topic describe shows Leader and Isr per partition. Leader -1 with live ISR members means wait and re-check. Leader -1 with all replicas offline means waiting is futile. Teams that skip this read and restart brokers instead reliably convert situation one into situation two, because each bounce kills the replicas that were about to win the election.

Broker Restarts and the Leader Election Timeline

A broker restart fires a precise sequence: the controller notices the missing heartbeat, marks its led partitions leaderless, and starts elections among ISR members. Followers that were in sync campaign, the controller picks winners, and the new leaders fetch metadata updates to clients. On a healthy cluster with replication factor 3, this completes in 30-120 seconds — the window where LeaderNotAvailable is expected and harmless.

The timeline stretches when each stage degrades. Slow log recovery on restart (replaying unflushed segments) delays the broker's return to ISR. A second restart mid-election kills candidates and restarts the whole cycle — this is how 90 seconds becomes 20 minutes. Network partitions between controller and replicas add detection delay on top, and clients with long metadata.max.age.ms keep erroring minutes after brokers recovered because nobody told them the news.

Operate restarts like landings: one broker down at a time, verify under-replicated partitions return to zero, then proceed. Automate the verification — a deploy pipeline that restarts broker 2 while 40 partitions are still under-replicated from broker 1 is just an outage with extra steps. The election window is normal; overlapping windows are self-inflicted.

# Watch an election resolve in real time (run twice, 60s apart)
bin/kafka-topics.sh --bootstrap-server $BROKERS \
  --describe --topic payments-authorized

# Recovery gauge: trending to zero means elections are landing
watch -n 30 'bin/kafka-topics.sh --bootstrap-server $BROKERS \
  --describe --under-replicated-partitions | wc -l'

# Rolling restart the safe way: one broker, verify, next
bin/kafka-server-stop.sh  # on broker 2 only
sleep 90
bin/kafka-topics.sh --bootstrap-server $BROKERS \
  --describe --under-replicated-partitions

Unclean Leader Election: When No ISR Survives

When no ISR replica is alive, Kafka faces a genuine dilemma: keep the partition offline (durable but unavailable) or elect an out-of-sync replica (available but lossy). The unclean.leader.election.enable flag picks the answer. False — the default and the correct choice for anything valuable — keeps leader at -1 until an ISR member returns. True elects whoever is alive, and whatever committed records they lack are gone forever.

Understand exactly what 'gone' means. The out-of-sync replica missed writes the old leader acknowledged — including records producers received success for under acks=all. After unclean election those records never come back; consumers see the log jump backward and downstream systems inherit a gap no retry can fill. One real incident lost 41 seconds of committed payments this way, and reconciliation took finance a full day.

Reserve true for genuinely expendable data: clickstreams, metrics, debug logs — topics where a gap is a shrug. Document the choice per topic in your topic registry, because the engineer flipping the flag at 3 AM won't otherwise know payments-authorized and clickstream-raw deserve opposite answers. For money topics, the correct response to leader -1 with no ISR is restoring a replica and accepting the downtime, painful as that feels mid-incident.

# Broker default (server.properties) — keep false for anything that matters
unclean.leader.election.enable=false

# Per-topic override inspection
bin/kafka-configs.sh --bootstrap-server $BROKERS \
  --entity-type topics --entity-name payments-authorized --describe

# Nuclear option for EXPENDABLE topics only (logs, metrics):
# bin/kafka-configs.sh --bootstrap-server $BROKERS \
#   --entity-type topics --entity-name clickstream-raw \
#   --alter --add-config unclean.leader.election.enable=true

min.insync.replicas vs acks: the Durability Contract

acks and min.insync.replicas are a pair, and misconfiguring one wastes the other. acks=all tells the producer to wait for every in-sync replica to acknowledge. min.insync.replicas sets the broker-side floor: if the ISR shrinks below it, acks=all produces fail with NotEnoughReplicas instead of risking durability. Together on RF=3 with min.insync.replicas=2, you survive one replica loss with full guarantees and fail loudly (safely) on two.

The classic misconfiguration is acks=all against min.insync.replicas=1, which pays full latency for single-replica durability — the worst of both worlds. The mirror mistake is lowering min.insync.replicas mid-incident to silence NotEnoughReplicas errors, converting a safe stall into silent single-replica writes that the next failure turns into loss.

Treat NotEnoughReplicas during an outage as protection working, not breakage. The correct response is restoring replicas (restart the downed broker, fix its disk, let it catch up) while producers retry with backoff. Monitor ISR shrink as a leading indicator: alert when under-replicated partitions stay nonzero for 5 minutes, so you start the day with a warning instead of ending it with an outage. When in doubt, page the topic owner before flipping the flag — a two-minute discussion beats a two-day reconciliation.

# The standard durable trio on RF=3 topics
# server.properties:
# min.insync.replicas=2
# producer: acks=all, retries=Integer.MAX_VALUE, max.in.flight.requests.per.connection=5

# Inspect effective topic config (topic overrides broker defaults)
bin/kafka-configs.sh --bootstrap-server $BROKERS \
  --entity-type topics --entity-name payments-authorized --describe

# Alert long before the breach: ISR shrink rate
# monitor: kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions
# warn if UnderReplicatedPartitions > 0 for > 5 min

preferred-replica-election: Restoring Balance

After an outage, leadership piles onto survivors. Broker 1 holds 100% of leaders while brokers 2 and 3 sit idle but healthy — and the next routine restart of broker 1 becomes a full outage because every leader lives there. Preferred-replica-election fixes this by moving each partition's leadership back to its preferred replica (the first in the replica list) once that replica is caught up.

The operation is safe by design: it only elects preferred replicas that are in sync, so no data risk attaches. Run it after every ISR recovery, not just incidents — any restart skews leadership a little, and skew compounds across months of patches until one broker secretly leads everything. The targeted JSON-file form lets you move leadership gradually during business hours if a full rebalance feels spicy.

Automate what humans forget. A 10-minute scheduler tick running preferred election (or your operator's auto-rebalance) means post-maintenance skew self-heals overnight. Verify with the leader histogram: roughly equal counts per broker is healthy, 90%-on-one is a pager waiting to happen. One command, run routinely, removes an entire class of 'routine restart becomes outage' surprises. Rehearse single-broker loss in staging quarterly so the first time you see ISR shrink is not during a real outage.

# Restore balance after ISR recovery
bin/kafka-preferred-replica-election.sh --bootstrap-server $BROKERS

# Targeted: only the partitions that need it (from a JSON file)
# bin/kafka-preferred-replica-election.sh --bootstrap-server $BROKERS --path-to-json-file election.json

# Confirm leaders spread evenly again
bin/kafka-topics.sh --bootstrap-server $BROKERS \
  --describe --topic payments-authorized | awk '{print $6}' | sort | uniq -c

Verifying ISR Recovery and Client Recovery

Broker-side recovery and client-visible recovery are two different events, and incidents stay open in the gap. Producers and consumers cache metadata — including which broker leads each partition — and refresh it on metadata.max.age.ms (default 5 minutes) or on demand after errors. A client that cached the leaderless state can keep throwing LeaderNotAvailable for minutes after the partition is fully healthy.

Close the gap deliberately. Shorten metadata.max.age.ms to 30 seconds on critical clients so post-recovery refresh happens fast, and after any leader incident run an end-to-end probe: produce one record to each affected topic and consume it back. That single round trip proves more than any dashboard — it exercises metadata fetch, leader routing, produce path, ISR write, and fetch path in one shot.

Make the probe part of your runbook's definition of done. 'Topic describe shows leaders' is a broker claim; 'produce plus consume round-trips' is client proof. If the probe fails while brokers look healthy, suspect stale metadata first (restart the client or force refresh) before reopening the broker investigation. Declaring victory from broker metrics alone is how teams close incidents that are still burning.

# Client-side: refresh metadata fast during incidents
# producer/consumer config:
# metadata.max.age.ms=30000

# End-to-end proof the partition serves again
bin/kafka-console-producer.sh --bootstrap-server $BROKERS --topic payments-authorized <<'EOF'
recovery-probe-1
EOF
bin/kafka-console-consumer.sh --bootstrap-server $BROKERS \
  --topic payments-authorized --from-beginning --max-messages 1 \
  --timeout-ms 15000