Kafka CommitFailedException — Stop Rebalance Loops

Your Kafka consumer logs CommitFailedException, then reprocesses 40,000 records it already handled — sending duplicate emails, double-counting metrics, and triggering fraud alerts on transactions that settled an hour ago. The rebalance that caused it took 90 seconds, reassigned all 12 partitions, and repeated 6 times before anyone noticed the loop.

This failure loves slow handlers: a consumer that calls a 2-second fraud API per record, processes 500 records per poll, and exceeds the 5-minute max.poll.interval.ms without calling poll again. The broker declares it dead, rebalances, and the revived consumer's commit lands on partitions it no longer owns. Each rebalance makes the backlog bigger, processing slower, and the next rebalance sooner — a death spiral.

The trap is catching the exception and continuing as if the commit succeeded. Offsets never committed, the new owner replays from the last commit, and your 'handled' records process twice. Combined with auto-commit enabled, teams get the worst of both worlds: commits firing mid-rebalance and duplicates everywhere.

By the end of this article you'll read rebalance logs like a timeline, size poll intervals from measured processing time, configure static membership and cooperative rebalancing, and build idempotent handlers plus transactional produces that make rebalances boring instead of catastrophic.

Why the Coordinator Revokes First and Asks Later

The group coordinator can't distinguish a dead consumer from a slow one — both stop calling poll. Its only evidence is silence past max.poll.interval.ms, and its only safe action is revocation: reassign the silent member's partitions to survivors so the group keeps making progress. Your consumer's late commit is then correctly rejected, because the new owner has already started fetching from the last committed offset and accepting yours would fork the offset history.

This design makes rebalances load-bearing for correctness. Offsets are per-partition progress markers shared by whoever owns the partition; two owners with divergent commit sequences would corrupt consumption guarantees for the whole group. The coordinator prefers duplicate processing (new owner replays from last commit) over lost processing (accepting a stale commit that skips records) — which is why every rebalance replays uncommitted work by construction, and why idempotent handlers are mandatory rather than nice-to-have.

Read rebalance logs as ownership transfers with timestamps: Revoked partitions [...] marks what you lost, Assigned partitions [...] marks the new map, and the gap between JoinGroup and SyncGroup is the stop-the-world pause (90s in this article's incident under eager rebalancing). Frequent revoke-assign cycles with no membership changes scream slow processing; cycles aligned with deploys scream missing static membership.

# Read the rebalance story from consumer logs (ownership timeline)
grep -E 'Revoked|Assigned|JoinGroup|SyncGroup|CommitFailed|Heartbeat failed' /var/log/app/consumer.log | tail -30
# Pattern A (slow processing): Revoked WITHOUT any deploy/restart nearby,
#   poll-start..poll-end gaps >> max.poll.interval.ms
# Pattern B (deploy storm): Revoked exactly at rollout timestamps, all members

# Measure the actual offense: time one poll batch end-to-end
# Add temporary markers (or use existing poll metrics):
# logger.info('poll-start', {count: records.count})
# ... process ...
# logger.info('poll-end', {elapsed_ms})
awk '/poll-start/{s=$1} /poll-end/{e=$1; print "batch:", e-s}' /var/log/app/consumer.log | tail -5
# Verdict: batch time must fit in max.poll.interval.ms with 2x+ margin

# Live lag view: which partitions stall while rebalances churn?
kafka-consumer-groups --bootstrap-server $BROKERS --describe --group orders

Size the Poll Loop: Records, Latency, and the 5-Minute Wall

The governing inequality is brutally simple: max.poll.records times worst-case seconds-per-record must fit inside max.poll.interval.ms with margin. Default 500 records at a 2-second downstream call needs 1,000 seconds against a 300-second wall — mathematically guaranteed rebalancing, no mystery, no broker fault. Every CommitFailedException of the slow-processing class is this arithmetic failing, and the fix is changing one of the three numbers: fewer records, faster records, or a longer wall.

Prefer fewer records first. Cutting max.poll.records to 50 turns the incident's 1,000s batch into 100s — inside budget with 3x margin — at the cost of more frequent polls (cheap) instead of rebalances (catastrophic). Speed up records second: timeouts on downstream calls, caching repeat verdicts, batching external calls. Lengthen the wall last: raising max.poll.interval.ms to 10 minutes accommodates genuinely slow work but delays failure detection for truly dead consumers — a real availability tradeoff, not a free knob.

Validate with production-like dependency latency, never mocks. The 50ms staging mock hid a 40x processing difference that only production traffic revealed. Load-test consumers with dependency latency injected at p99-plus, measure batch times at the 99th percentile (not the mean), and demand 2x margin under the interval before shipping.

# The governing arithmetic — run it with MEASURED numbers, not hopes
# batch_seconds = max.poll.records * p99_seconds_per_record  (must be < interval/2)
# Incident: 500 * 2.0s = 1000s vs 300s wall -> guaranteed rebalance
# Fixed:     50 * 0.8s = 40s   vs 300s wall -> 7x margin

# Consumer config that respects the arithmetic (properties style)
# max.poll.records=50
# max.poll.interval.ms=300000
# session.timeout.ms=10000
# max.poll.interval must exceed batch p99 with 2x+ margin

# Load-test with production-like dependency latency (NOT mocks)
# tc qdisc add dev eth0 root netem delay 2000ms  # staging consumer host
# Run consumer, measure batch p99 from poll-start/poll-end markers:
awk '/poll-start/{s=$NF} /poll-end/{print $NF-s}' /var/log/app/consumer.log | sort -n | awk '{a[NR]=$1} END{print "p99 batch:", a[int(NR*0.99)], "s"}'
# Ship only if p99 batch < max.poll.interval.ms / 2

Heartbeats vs Processing: Two Liveness Signals, Two Failures

Modern Kafka clients separate liveness into two channels: a background heartbeat thread governed by session.timeout.ms (dead in ~10s of silence) and the poll-call cadence governed by max.poll.interval.ms (dead in ~5 minutes without poll). Slow processing kills via the interval; GC pauses, event-loop blocks, and network stalls kill via heartbeats — even when processing itself is fast. Treating both as 'consumer slow' misdiagnoses half of all rebalance incidents.

Heartbeat deaths leave distinctive evidence: 'Heartbeat failed' and group-rebalancing warnings with healthy poll-batch timings, correlated with GC pause spikes, full event-loop lag metrics, or coordinator-connection resets. The consumer was doing fine work and got evicted for going quiet on the wrong channel. Fixes target the stall (GC tuning, unblocking the loop, connection resilience) plus modest session.timeout.ms headroom — not smaller poll batches, which change nothing for a heartbeat problem.

Monitor both channels independently. Track poll-batch p99 against the interval (processing health) and heartbeat success rate plus GC pause p99 (heartbeat health) on the same dashboard. When rebalances strike, the channel that degraded first names the cause — and the fix list for each channel shares almost nothing, which is why the distinction matters more than any single tuning value.

# Separate heartbeat death from processing death (same incident, different fixes)
# 1. Heartbeat evidence: failures with HEALTHY batch times?
grep -c 'Heartbeat failed' /var/log/app/consumer.log
awk '/poll-start/{s=$NF} /poll-end/{print $NF-s}' /var/log/app/consumer.log | sort -n | tail -3
# Heartbeat failures + small batches -> stall (GC/loop/network), NOT slow work

# 2. Correlate with GC pauses (JVM consumers)
grep 'Pause Young\|Pause Full' /var/log/app/gc.log | awk '{print $NF}' | sort -nr | head -5
# Pauses near session.timeout.ms (10s) evict members mid-processing

# 3. Heartbeat-side config (only after ruling out slow processing)
# session.timeout.ms=15000        # modest headroom, not 60s (delays real failure detection)
# heartbeat.interval.ms=3000     # ~1/3 of session timeout, the standard ratio
# max.poll.interval.ms=300000    # leave alone — this knob is for processing, not heartbeats

Static Membership and Cooperative Rebalancing for Calm Deploys

Every rolling deploy without static membership is a voluntary rebalance storm: each restarting consumer rejoins as a stranger, the coordinator revokes and reassigns, and with eager rebalancing the entire group pauses while every member reclaims partitions. Four consumers restarting sequentially means four stop-the-world pauses — during which lag grows, batches swell, and the slow-processing trap arms itself for the aftermath.

Static membership (group.instance.id set to a stable, unique-per-instance value) lets a restart rejoin as the same member and keep its partitions — no revocation, no reassignment, no storm. Kubernetes StatefulSets (stable pod names) or persisted instance IDs provide the stability; bare Deployments with random pod names need the ID plumbed from stable storage or the downward API. Pair it with the cooperativeSticky assignor so the rebalances that do occur migrate only the delta partitions while the rest keep consuming — turning 90s full pauses into sub-second partial ones.

Session timeout interacts here too: with static membership, a truly dead instance's partitions wait for session.timeout.ms before reassignment (availability cost), so keep it tight (10-30s) rather than inflating it as a rebalance band-aid. The combination — stable IDs, cooperative assignment, tight sessions — makes deploys invisible and reserves rebalances for genuine failures.

# Calm-deploy consumer config (properties style)
# group.instance.id=orders-consumer-${HOSTNAME}  # STABLE + UNIQUE per instance
# partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor
# session.timeout.ms=10000
# heartbeat.interval.ms=3000
# max.poll.interval.ms=300000

# Kubernetes: stable identity via StatefulSet pod name (deployments get random names)
# env:
# - name: HOSTNAME
#   valueFrom: { fieldRef: { fieldPath: metadata.name } }
# group.instance.id = f"orders-{HOSTNAME}"  # same pod name -> same member -> no revoke

# Verify: rolling restart should show (almost) zero revocations
kubectl rollout restart statefulset orders-consumer
grep -cE 'Revoked|CommitFailed' /var/log/app/consumer.log  # expect ~0 during rollout
kafka-consumer-groups --bootstrap-server $BROKERS --describe --group orders  # lag flat

Idempotent Handlers: Replays Are Guaranteed, Duplicates Are Optional

Kafka's delivery contract under rebalancing is at-least-once: the new owner replays from the last committed offset, so every uncommitted record processes twice. This isn't a bug to eliminate — it's the protocol's chosen tradeoff (duplicates over data loss). Your handler design decides whether replays are harmless no-ops or $18,400 double-charge incidents. Idempotency is therefore not an optimization; it's the correctness layer the transport deliberately leaves to you.

Build guards keyed on the record's natural idempotency key (order ID, event ID): conditional writes (DynamoDB attribute_not_exists, SQL INSERT ... ON CONFLICT DO NOTHING), processed-set lookups with TTLs covering the maximum replay window, or state-machine transitions that refuse to re-apply terminal states. The guard and the mutation must be atomic — check-then-act across two calls races when two owners briefly overlap during the handoff.

Commit strategy complements but never replaces guards. commitSync after each batch gives clean replay boundaries (replay from batch start) at throughput cost; async commits need rebalance-listener hooks committing on revoke to shrink the replay window. Auto-commit on a timer fires mid-batch and widens replay unpredictably — disable it for any handler with side effects. But remember the hierarchy: guards make duplicates safe, commits merely make replays smaller.

# Idempotency guard shape (DynamoDB conditional write — atomic check+act)
# key = record.order_id; the write itself refuses replays:
# table.put_item(
#   Item={'order_id': oid, 'status': 'CHARGED', 'ts': now},
#   ConditionExpression='attribute_not_exists(order_id)')
# except ConditionalCheckFailedException:
#   metrics.incr('duplicate_suppressed'); return  # replay absorbed, zero side effects

# Commit hygiene for side-effect handlers (disable timer auto-commit)
# enable.auto.commit=false
# commitSync() after each successfully guarded batch
# rebalance listener: on_partitions_revoked -> commitSync() (shrink replay window)

# Verify replays are absorbed, not just committed (downstream metrics, not lag)
# duplicates_suppressed rate should RISE during rebalances while charges stay flat
kafka-consumer-groups --bootstrap-server $BROKERS --describe --group orders | awk '{print $1, $5, $6}'
# Lag draining + flat charge counts + rising suppressed = healthy replay handling

Exactly-Once With Idempotent Producers and Transactions

For consume-process-produce pipelines (read order, write receipt to another topic), idempotent handlers alone leave a gap: the consume offset and the produced record commit separately, so a crash between them duplicates the produce or loses the offset. Kafka transactions close the gap by atomically committing consumed offsets and produced records together — the downstream sees the receipt exactly when the offset advances, and aborted transactions are invisible to read_committed consumers.

The setup has four mandatory pieces: enable.idempotence=true on the producer (sequence-checked, duplicate-free writes within a session), a unique transactional.id per producer instance (stable across restarts for fencing zombies), sendOffsetsToTransaction inside the consume loop coupling offsets to output, and isolation.level=read_committed on every downstream reader (otherwise uncommitted transactional records leak through). Miss any piece and you have complexity without the guarantee — the worst outcome.

Adopt transactions where duplicates cost money (payments, inventory, billing events) and accept their price: added end-to-end latency, transaction-coordinator load, and operational surface (stuck transactions need abort monitoring, zombie fencing needs stable IDs). For everything else — notifications, metrics, search indexing — idempotent handlers plus read-committed consumers deliver the practical safety at a fraction of the cost. Match the guarantee to the damage.

# Transactional consume-process-produce checklist (all four or nothing)
# Producer: enable.idempotence=true
#           transactional.id=orders-tx-${STABLE_INSTANCE_ID}  # unique + stable (fences zombies)
# Consumer: isolation.level=read_committed  (on EVERY downstream reader)
# Loop: producer.beginTransaction()
#         -> produce receipt record
#         -> producer.sendOffsetsToTransaction(offsets, group_id)
#       producer.commitTransaction()   # offsets + output commit atomically
#       # abortTransaction() on ANY exception — never leave transactions dangling

# Ops: watch for stuck/old transactions (abort + alert, they block read_committed readers)
kafka-transactions --bootstrap-server $BROKERS --describe --all 2>/dev/null | head -20
# Downstream lag on read_committed readers that never drains often means
# an open transaction pinning the log-start offset — find and abort it.