Heartbeats and Failure Detection: Stop Guessing, Start Detecting Node Failures

Your cluster just split-brained because a node went silent for 2 seconds during a GC pause. The load balancer kept sending traffic to a dead process. Your pager went off at 3 AM. This is the real cost of naive failure detection. Heartbeats and failure detection are the nervous system of distributed systems — get them wrong and your system becomes fragile, flappy, or worse, silently corrupts data. By the end of this article, you'll know how to implement adaptive failure detection, tune timeouts for your latency profile, and avoid the production pitfalls that burn teams who treat heartbeats as an afterthought.

Why Heartbeats? The Problem Without Them

Without heartbeats, you're blind. A node crashes silently — no error, no signal. Your load balancer keeps sending requests to a black hole. Clients hang waiting for responses that never come. The old hack was TCP keepalives — but those take 2 hours to detect a dead peer by default. Heartbeats give you control: you decide the detection speed and the cost. In a checkout service, a 30-second detection delay means 30 seconds of failed transactions. That's revenue on fire. Heartbeats are the minimal signal that says 'I'm still here and processing.'

# io.thecodeforge — System Design tutorial

import socket
import time

HEARTBEAT_INTERVAL = 5  # seconds
MONITOR_HOST = ('monitor.local', 9999)

def send_heartbeat():
    sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    while True:
        # Send a simple 'alive' message with process ID
        message = f'ALIVE:{os.getpid()}'
        sock.sendto(message.encode(), MONITOR_HOST)
        time.sleep(HEARTBEAT_INTERVAL)

if __name__ == '__main__':
    send_heartbeat()

Failure Detection: The Art of Deciding Someone Is Dead

Heartbeats alone are useless without a decision rule. The simplest: if no heartbeat for N seconds, declare dead. But N is a minefield. Too small: false positives from GC, network jitter, or clock skew. Too large: slow failover, angry users. The production reality is that you need adaptive thresholds. The Phi Accrual failure detector used in Cassandra and Akka models the arrival time of heartbeats as a normal distribution. It computes a suspicion level (phi) — how likely it is that the node is dead given the history. When phi exceeds a threshold, you act. This adapts to the actual network behavior.

// io.thecodeforge — System Design tutorial

import java.util.concurrent.ConcurrentLinkedDeque;

public class PhiAccrualDetector {
    private final ConcurrentLinkedDeque<Long> intervals = new ConcurrentLinkedDeque<>();
    private final int windowSize = 100; // keep last 100 intervals
    private final double threshold = 8.0; // phi threshold

    public double computePhi(long lastHeartbeatMs) {
        long now = System.currentTimeMillis();
        long elapsed = now - lastHeartbeatMs;
        double mean = intervals.stream().mapToLong(Long::longValue).average().orElse(1000);
        double variance = intervals.stream().mapToDouble(i -> Math.pow(i - mean, 2)).average().orElse(1);
        double stddev = Math.sqrt(variance);
        if (stddev < 1) stddev = 1; // avoid division by zero
        double phi = -Math.log10(1 - normalCdf((elapsed - mean) / stddev));
        return phi;
    }

    public boolean isSuspect(long lastHeartbeatMs) {
        return computePhi(lastHeartbeatMs) > threshold;
    }

    public void recordHeartbeat() {
        long now = System.currentTimeMillis();
        if (!intervals.isEmpty()) {
            long interval = now - intervals.getLast();
            intervals.addLast(interval);
            if (intervals.size() > windowSize) intervals.removeFirst();
        }
        intervals.addLast(now); // store timestamp for next interval calc
    }

    private double normalCdf(double x) {
        return 0.5 * (1 + erf(x / Math.sqrt(2)));
    }

    private double erf(double x) {
        // approximation from Abramowitz and Stegun
        double a1 =  0.254829592;
        double a2 = -0.284496736;
        double a3 =  1.421413741;
        double a4 = -1.453152027;
        double a5 =  1.061405429;
        double p  =  0.3275911;
        double sign = 1;
        if (x < 0) sign = -1;
        x = Math.abs(x);
        double t = 1.0 / (1.0 + p * x);
        double y = 1.0 - (((((a5 * t + a4) * t) + a3) * t + a2) * t + a1) * t * Math.exp(-x * x);
        return sign * y;
    }
}

Gossip-Style Heartbeats: Scaling to Thousands of Nodes

All-to-all heartbeats don't scale. With 1000 nodes, each sending a heartbeat per second, that's 1 million messages per second. Your network drowns. Gossip protocols like SWIM (Scalable Weakly-consistent Infection-style Membership) solve this. Each node periodically picks a random node and sends its membership list. The receiver merges and propagates. Failure detection is piggybacked: if a node hasn't heard from another in a while, it starts an indirect probe — asks a few peers to ping the suspect. If all fail, the suspect is declared dead. This is how Cassandra, Consul, and Serf handle large clusters.

// io.thecodeforge — System Design tutorial

package main

import (
	"fmt"
	"math/rand"
	"sync"
	"time"
)

type Node struct {
	ID      string
	Alive   bool
	Version int64
}

type Membership struct {
	mu    sync.RWMutex
	nodes map[string]*Node
}

func (m *Membership) Gossip() {
	for {
		time.Sleep(time.Duration(rand.Intn(1000)+500) * time.Millisecond)
		m.mu.RLock()
		// Pick a random peer
		var peer *Node
		for _, n := range m.nodes {
			if n.ID != "self" {
				peer = n
				break
			}
		}
		m.mu.RUnlock()
		if peer == nil {
			continue
		}
		// Send membership to peer (simulated)
		fmt.Printf("Gossiping to %s\n", peer.ID)
	}
}

func main() {
	m := &Membership{nodes: make(map[string]*Node)}
	m.nodes["self"] = &Node{ID: "node1", Alive: true, Version: time.Now().UnixNano()}
	m.nodes["node2"] = &Node{ID: "node2", Alive: true, Version: time.Now().UnixNano()}
	m.nodes["node3"] = &Node{ID: "node3", Alive: true, Version: time.Now().UnixNano()}
	go m.Gossip()
	time.Sleep(5 * time.Second)
}

When Heartbeats Lie: The Split-Brain Problem

Heartbeats can only detect failures, not prevent them. If a network partition splits your cluster, both sides think the other is dead. Both sides start serving writes. When the partition heals, you have conflicting data. This is split-brain. Heartbeats alone cannot solve this — you need a quorum or a lease. In a 3-node cluster, require 2 nodes to agree on a leader. If a node can't reach the majority, it must step down. Never trust a single heartbeat timeout to make decisions that affect data consistency.

# io.thecodeforge — System Design tutorial

class QuorumLeader:
    def __init__(self, total_nodes):
        self.total = total_nodes
        self.alive = set()

    def on_heartbeat(self, node_id):
        self.alive.add(node_id)
        if len(self.alive) > self.total // 2:
            return True  # I can be leader
        else:
            return False  # I must step down

# Usage: if not quorum.on_heartbeat('node1'):
#     step_down_as_leader()

Tuning Heartbeat Intervals for Production

The heartbeat interval and timeout are a trade-off between detection speed and overhead. Rule of thumb: set heartbeat interval to 1/3 of the desired detection time. For a 15-second detection, heartbeat every 5 seconds. Timeout = 3 * interval. But this is for fixed timeouts. With adaptive detection, you set the suspicion threshold instead. In practice, for a microservice mesh, 1-second heartbeats with a phi threshold of 8 works well. For a cross-datacenter link with 100ms latency, use 5-second intervals. Always add jitter to prevent thundering herd when all nodes heartbeat at once.

# io.thecodeforge — System Design tutorial

import random
import time

BASE_INTERVAL = 5.0  # seconds
JITTER = 0.5  # +/- 0.5 seconds

def jittered_sleep():
    jitter = random.uniform(-JITTER, JITTER)
    time.sleep(BASE_INTERVAL + jitter)

while True:
    send_heartbeat()
    jittered_sleep()

Production Monitoring: What to Watch

You need to monitor heartbeats themselves. Track: heartbeat arrival jitter (standard deviation), missed heartbeats per minute, phi values over time, and false positive rate. A sudden increase in jitter often precedes a node failure — the node is struggling but still alive. Set alerts on phi > threshold for more than 2 consecutive windows. Also monitor the heartbeat sender's thread pool — if it's exhausted, heartbeats stop. I've seen a payments service go down because the heartbeat thread got stuck on a slow DNS lookup.

# io.thecodeforge — System Design tutorial

# Check heartbeat arrival times from logs
# Assumes heartbeat logs contain 'heartbeat from node X at timestamp'
awk '/heartbeat from/ {print $NF}' /var/log/heartbeat.log | uniq -c | sort -n

# Check for gaps > 10 seconds
awk '/heartbeat from/ {now=$NF; if (prev && now-prev>10) print "Gap:", now-prev, "seconds"} {prev=$NF}' /var/log/heartbeat.log

When Not to Use Heartbeats

Heartbeats are overkill for two-node systems — just use a TCP keepalive with a short timeout (e.g., 5 seconds). For single-node systems, obviously not needed. For systems where failure detection latency can be minutes (e.g., batch processing), heartbeats add unnecessary complexity. Also, if your network is extremely unreliable (e.g., satellite links), heartbeats will cause constant false positives. In that case, use a circuit breaker pattern instead — assume the node is alive until proven otherwise, and handle failures reactively.