ConnectionResetError 104: Fix Abrupt Peer Disconnects

Your service was healthy for weeks. Then checkout latency spiked, the logs filled with ConnectionResetError: [Errno 104] Connection reset by peer, and every traceback pointed at a different line — a database driver here, an HTTP call there, an upload somewhere else. Nothing deployed. Nothing obviously broke. Just a wave of peers hanging up without saying goodbye.

That scatter is the signature of errno 104. Unlike a clean close, a TCP reset carries no explanation: the peer's process may have crashed, a load balancer may have culled an idle connection, or your own upload may have outrun the server's patience. Your code discovers it only when it next touches the socket, so the traceback marks the victim, not the cause.

This article closes that gap. You'll learn what a RST actually is, why keep-alive mismatches manufacture resets on a timer, how load-balancer idle timeouts became the most common source, what large uploads do to impatient servers, exactly which failures deserve a retry with backoff, and the client-hardening checklist that turns the rest into background noise.

What RST Means: the Peer Hung Up Without Goodbye

A TCP connection normally ends with manners: one side sends FIN, the other acknowledges, both agree the conversation is over. A reset skips the etiquette. The peer sends a RST packet that says this connection no longer exists — now — and both sides' kernels tear down state immediately. Any bytes in flight are discarded, and your next read or write on that socket raises ConnectionResetError with errno 104.

RSTs come from a short list of senders. A process that crashes or gets OOM-killed leaves the kernel holding sockets it never closed cleanly; the kernel RSTs them. A socket closed abortively with SO_LINGER set to zero discards unsent data and emits RST by design. Firewalls, NAT gateways, and load balancers send RSTs (or spoof them) when they reap idle or half-open connections. An overloaded server whose accept queue overflows may RST new arrivals rather than handshake.

The critical consequence is timing: you learn about the reset only when you touch the socket. The RST may have arrived seconds ago, but your code meets it at the next recv or sendall — which is why the traceback frames a random line instead of the true cause. Debugging therefore starts away from the traceback: which peer, what was its state, and what sits between you. The snippet below reproduces an abortive close locally so you can see both the read and write symptoms deterministically.

import socket
import struct

a, b = socket.socketpair()
b.send(b"unread-bytes")
# Abortive close: discard unsent data and emit RST, like a crashed peer
a.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, struct.pack("ii", 1, 0))
a.close()
try:
    data = b.recv(4096)
    print("recv returned:", data)
except ConnectionResetError as exc:
    print("ConnectionResetError:", exc)
try:
    b.sendall(b"more")
    print("second write buffered")
except (ConnectionResetError, BrokenPipeError) as exc:
    print(type(exc).__name__, "on write:", exc)
finally:
    b.close()

Server and Keep-Alive Mismatch: Idle Connections Get Killed

HTTP keep-alive exists to reuse TCP connections across requests, and pooled clients hold idle connections ready for the next call. Every holder sets a limit: your pool keeps a conn for 120 s, the server closes idle conns after 75 s, the load balancer reaps after 60 s. The moment your pool's limit exceeds anyone else's, it starts handing out corpses — connections the far side already closed — and the first request on each one eats a reset.

The failure has a telltale rhythm. Resets arrive in a sawtooth that tracks idle gaps: quiet minutes breed stale connections, the next request after each lull fails, then traffic flows until the next pause. Peak hours look clean because connections never sit idle long enough to die. Teams routinely misread this as provider flakiness that strikes when load is low, which sends them interrogating vendors instead of their own pool settings.

The fix has two halves. First, set your keep-alive below the shortest upstream idle timeout — 45 s against a 60 s balancer leaves a comfortable margin — and validate-before-reuse where the client supports it. Second, add exactly one retry for idempotent reads that fail with a reset, since even aligned settings lose an occasional race. The snippet models the checkout decision so the margin rule becomes concrete rather than folklore.

IDLE_TIMEOUT = 60.0  # load balancer reaps idle conns after 60 s

def checkout(idle_for, timeout):
    if idle_for > timeout:
        return "stale: far side already closed — first use earns a reset"
    return "fresh: safe to reuse"

print(checkout(75.0, IDLE_TIMEOUT))
print(checkout(12.0, IDLE_TIMEOUT))
print("rule: hold pools below the shortest upstream timeout, e.g. 45 s")

Load Balancer Idle Timeouts: the 60-Second Killer

If keep-alive mismatches are the mechanism, load-balancer idle timeouts are the usual loaded gun. Managed balancers ship with short defaults — AWS ALB and NLB default to 60 s in many configurations, cloud NATs often lower — and those defaults are invisible until they fire. Your dashboards show healthy targets, your pools look warm, and every minute-old connection is quietly dead.

The failure concentrates exactly where you'd least expect: low-traffic services and off-peak minutes. A checkout path doing three requests a minute holds each pooled connection idle for twenty seconds on average — safe — until a two-minute gap appears and the next request reuses a corpse. High-throughput paths never notice because their connections never rest. That inverse correlation with traffic is the fingerprint: when quieter means broker, suspect the balancer's idle timer before anything else.

Remediation is arithmetic, not heroics. Read the balancer's idle timeout from its attributes, set client keep-alive clearly beneath it, and enable TCP keep-alive probes so truly dead peers surface before reuse. Where the platform allows, raise the balancer timeout to match your slowest legitimate idle gap instead — but document the chosen numbers in the service catalog, because the next engineer will otherwise re-tune one side and revive the bug. The snippet computes the safe ceiling from real settings.

lb_idle_timeout = 60.0  # from describe-load-balancer-attributes
server_idle_timeout = 75.0
MARGIN = 15.0

safe_keepalive = min(lb_idle_timeout, server_idle_timeout) - MARGIN
print("LB idle:", lb_idle_timeout, "s")
print("server idle:", server_idle_timeout, "s")
print("safe pool keep-alive:", safe_keepalive, "s")
assert safe_keepalive > 0, "margins must leave a positive keep-alive"
print("pool setting: keep-alive=45s, retry once on stale reads")

Large Uploads and Slow Readers: Writes That Outrun the Peer

Small requests sail through while large uploads reset midway, and the asymmetry has a physical cause. Servers bound request sizes, enforce auth, and validate headers before reading bodies. When your 200 MB upload violates any of those, the server responds with an error and closes the connection — while your client is still streaming chunk forty of four hundred. Your next chunk lands on a closed socket and raises ConnectionResetError, which reads as a network failure but is really a rejected upload.

Slow readers create the mirror image. A client that trickles bytes at dial-up speed can outlast the server's per-request deadline; the server gives up, closes, and the client's remaining chunks die the same death. Either direction, the signature is identical: resets correlated with body size or transfer duration, small payloads unaffected, server access logs showing 413 or 400 responses your client never read.

The efficient fix is to ask permission first. HTTP's Expect: 100-continue lets the client send headers, wait for a go-ahead, and stream the body only after approval — rejections arrive before any bytes move. Where that header isn't supported, validate size and credentials client-side and read the error response body before any retry. The snippet simulates an early rejection so the stop-early logic is testable without a real server.

CHUNK = 65536
payload = b"x" * (CHUNK * 4)

sent = 0
peer_gone_after = CHUNK * 2  # server rejected the upload early

def fake_send(chunk):
    global sent
    if sent >= peer_gone_after:
        raise ConnectionResetError(104, "Connection reset by peer")
    sent += len(chunk)

try:
    for i in range(0, len(payload), CHUNK):
        fake_send(payload[i:i + CHUNK])
    print("upload complete:", sent, "bytes")
except ConnectionResetError as exc:
    print("upload cut at", sent, "bytes:", exc)
    print("action: read the error response body before retrying")

Retry With Backoff: Which Resets Deserve a Second Try

Some resets deserve a second chance and some don't, and the line is idempotency. A GET that reset on a stale pooled connection can simply run again — the server never saw it, or saw it fully, and repeating it changes nothing. The same holds for HEAD, idempotent PUTs, and DELETEs. These are the requests where one retry with backoff converts a visible error into an invisible hiccup.

Writes are the opposite. A POST that resets may have executed fully before the connection died; retrying blindly double-charges, double-orders, or double-enqueues. The safe shape is an idempotency key: the client generates one key per logical operation, sends it with every attempt, and the server deduplicates. Stripe, payment gateways, and queue producers all work this way because the network makes exactly-once delivery impossible and keys fake it convincingly.

Backoff discipline matters as much as the retry decision. Retry once or twice, cap the delay, add jitter so a fleet of reset clients doesn't thundering-herd the recovering peer, and only retry the reset errors — timeouts and 5xx responses follow different policies. The snippet below shows the full shape: attempt, catch, exponential delay with jitter, and a final raise that preserves the original error when retries run out.

import random
import time

random.seed(7)
failures_left = 2

def flaky_call():
    global failures_left
    if failures_left > 0:
        failures_left -= 1
        raise ConnectionResetError(104, "Connection reset by peer")
    return "200 OK"

def call_with_backoff(fn, retries=4, base=0.01):
    for attempt in range(retries):
        try:
            return fn()
        except ConnectionResetError as exc:
            if attempt == retries - 1:
                raise
            delay = base * (2 ** attempt) + random.uniform(0, base)
            print(f"attempt {attempt + 1} reset ({exc}); retry in {delay:.3f}s")
            time.sleep(delay)

print(call_with_backoff(flaky_call))

Harden the Client: Timeouts, Pools, and Graceful Shutdown

Retries treat the symptom; hardening treats the disease. A production HTTP client needs five settings reviewed together: timeouts that bound every wait, pools sized to the timeout budget, keep-alive aligned beneath upstream idle limits, retry rules scoped to idempotent operations, and graceful shutdown that drains connections instead of RSTing them mid-flight.

Timeouts come first because unbounded waits turn one dead peer into a pool-wide freeze. Set connect, read, and total timeouts on every client — five seconds each is a sane starting point for internal services — and size pools so the worst case still fits your thread budget. A pool of fifty connections with thirty-second timeouts can pin fifteen hundred thread-seconds; that arithmetic should be explicit in code review, not discovered during an outage.

Then close the loop operationally. Log resets with peer addresses so patterns (one bad host, one availability zone, one deploy) jump out. Dashboard reset rates beside latency so product sees the trade-off. Drain connections on shutdown with a grace period so deploys stop manufacturing RSTs. The snippet captures the configuration shape with the standard library — real clients like httpx or urllib3 take the same numbers under different names, and the values matter far more than the library.

import http.client

conn = http.client.HTTPConnection("example.com", 80, timeout=5)
print("timeout:", conn.timeout)
print("rules: timeout=5, pool maxsize=10, keep-alive below LB idle")
print("retries: 1x on idempotent GET after reset, keys on POST")
conn.close()
print("client hardened: no unbounded waits remain")