502 Bad Gateway in Nginx: Fix the Failing Upstream

Checkout is down. Every browser shows the same stark page: 502 Bad Gateway. Nginx is up, the servers are up, CPU graphs are flat — yet every request through the proxy dies. Someone suggests restarting Nginx. Someone else blames the CDN. Meanwhile the actual culprit, a crashed app server one hop behind the proxy, sits quietly untouched.

A 502 is Nginx telling you the truth about someone else: it asked the upstream for a response and got garbage, silence, or a slammed door. The fix is almost always behind the proxy, not in it — a dead process, a proxy_pass pointing at last month's port, a socket with wrong permissions, or an app slower than the proxy's patience. One log file read correctly splits those cases in under a minute: connect() failed, prematurely closed, timed out, too big header.

This guide gives you that minute back. You'll classify any 502 from its error.log line, split backend death from config drift with a direct curl, apply production proxy blocks with keepalive, retries, and buffers set sanely, and stop confusing 502 with its cousins 503 and 504. Check the backend first, then the target, then the timeouts — every step uses commands you can run as-is.

502 Means the Proxy Got a Bad Answer From Upstream

A 502 Bad Gateway is the proxy reporting a failed conversation, not a failed proxy. Nginx accepted your request, opened a connection to the upstream named in proxy_pass, and received something it couldn't use: a refused connection, an empty reply, a truncated response, or headers so malformed they can't be forwarded. HTTP semantics pin the blame precisely — the gateway received an invalid response from the inbound server — which is why seasoned operators read a 502 as upstream trouble until proven otherwise.

That conversation breaks at distinct moments, and error.log names each one. connect() failed (111: Connection refused) means nothing listens at the configured address. connect() to unix:/run/myapp/app.sock failed (2: No such file or directory) means the app server isn't running; the same line with (13: Permission denied) is SELinux, AppArmor, or socket group membership blocking Nginx. upstream prematurely closed connection and recv() failed (104: Connection reset by peer) mean the app accepted then hung up mid-response. upstream sent too big header means the response outgrew your proxy buffers. no live upstreams means every peer tripped max_fails and got marked dead.

Don't debug the wrong proxy. A Cloudflare-branded 502 page means your origin handed Cloudflare a 502; a bare unbranded 502 means Cloudflare itself couldn't reach or parse your origin — dead tunnel, broken gzip, partial HTTP/2 support. An AWS ALB splits it the same way with HTTPCode_ELB_502_Count versus target 502s. Confirm the 502 in your own Nginx error.log first: if it isn't there, the fault lives a layer above you.

The payoff is where you spend your first five minutes: past the proxy, not at it. Restarting Nginx never revives a dead app server, and purging CDN caches never fixes a stale proxy_pass. What follows makes the correct reflex concrete — the full phrase catalog, the bypass curl, sane proxy blocks with keepalive and buffers, and status codes that tell the truth.

Backend Down vs Proxy Misconfigured: Tell Them Apart

Every 502 investigation starts with one split: dead backend or misdirected proxy? Curl the backend directly with the proxy removed from the path — a 200 means the app's alive and Nginx is looking in the wrong place, while connection refused means the process is down no matter what the config says. For socket upstreams, ls -l the .sock file: missing means the app server never started, present-but-unreachable screams permissions.

A dead backend announces itself loudly once you look. systemctl status shows inactive or failed; docker ps shows the container gone or restarting, with docker logs ending in an OOM kill or an unhandled exception. In Kubernetes, kubectl get pods shows CrashLoopBackOff while describe pod reports Liveness probe failures and exit 137 — probes murdering pods mid-request manufacture 502s from healthy code. Fix the crash, the leak, or the probe threshold first. No Nginx directive revives a dead process.

A living backend behind a persistent 502 means Nginx is asking the wrong address. Diff proxy_pass against ss -ltnp, and in Docker use the service name — proxy_pass http://backend:3000 works where http://127.0.0.1:3000 fails, because localhost inside the Nginx container isn't your app container. When error.log shows (13: Permission denied), suspect SELinux or AppArmor: check ausearch -m avc and enable httpd_can_network_connect. Port moves, hostname renames, and upstream blocks copied between environments all produce healthy apps with 502s on top.

Lock in the order: curl the backend, check the process, then read the config. Editing config over a dead process wastes the exact minutes checkout is down. File one more distinction now: upstream timed out blames app slowness against proxy_read_timeout, while upload stalls belong to client_body_timeout on the client leg. This whole split runs in under thirty seconds in one terminal.

# Ask the backend directly, bypassing Nginx entirely
curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:3001/health

# Time it: slow-but-answering (tune app/timeout) vs refused (dead backend)
curl -w 'connect:%{time_connect} start:%{time_starttransfer} total:%{time_total}\n' -o /dev/null http://127.0.0.1:3001/health

# Is the app process even alive, and on which port?
systemctl status myapp --no-pager
ss -ltnp | grep 3001

# Unix-socket upstreams: socket present? who may touch it?
ls -l /run/myapp/app.sock
namei -l /run/myapp/app.sock

# Containers: running state plus recent crash output
docker ps --filter name=myapp
docker logs --tail 50 myapp

# Kubernetes: restarts and probe kills manufacturing 502s
kubectl get pods -l app=myapp
kubectl describe pod -l app=myapp | grep -A 5 -i 'probe\|last state'

Read Nginx error.log Like a Map to the Fault

Three phrases cover most 502s, but production throws eight — learn them all and you'll classify any incident in seconds. connect() failed (111: Connection refused) means nothing listens at the configured address: dead process or wrong port, settled by the direct curl. connect() to unix:/run/myapp/app.sock failed (2: No such file or directory) means the app server isn't running; the same line with (13: Permission denied) is SELinux, AppArmor, or socket group membership — check namei -l and ausearch -m avc before touching anything else.

upstream prematurely closed connection means the app accepted then hung up: a crash between accept and reply, a Gunicorn worker murdered by --timeout mid-request, or a keepalive race where the backend closed an idle connection just as Nginx reused it. recv() failed (104: Connection reset by peer) is the same story one layer down. upstream sent too big header means the response outgrew your proxy buffers — that's a sizing fix, not a restart. no live upstreams means every peer tripped max_fails, so check the fleet, not one box.

Pair the phrase with the live targets. nginx -t validates syntax without touching traffic. nginx -T dumps the effective running config with includes resolved, so grepping proxy_pass shows true targets instead of the file you wish were loaded. When disk and -T disagree, you've found the sites-enabled symlink or include-path problem — the classic edit-a-file-Nginx-never-loads outage. In Docker, a missing resolver turns service-name lookups into host-not-found 502s, so pin resolver 127.0.0.11 valid=30s.

Lock in the reflex: tail, grep the phrase, dump live targets, run the diagnostic script below. Alert on error.log upstream phrases rather than bare 5xx rates — connect() failed pages the app owner while status-code dashboards just count damage. Repair without that frame is how one-digit typos survive for 38 minutes.

# Fresh upstream errors with context, newest last
sudo tail -n 200 /var/log/nginx/error.log | grep -E 'upstream|connect\(\)|timed out|recv\(\)|no live upstreams|too big header'

# Classify at a glance: count each failure mode in the last 5k lines
sudo tail -n 5000 /var/log/nginx/error.log | grep -oE 'connect\(\) failed|prematurely closed|upstream timed out|recv\(\) failed|no live upstreams|too big header' | sort | uniq -c | sort -rn

# Validate syntax before touching anything live
sudo nginx -t

# Print the running config's proxy targets (includes resolved)
sudo nginx -T | grep -B 2 -A 6 'proxy_pass|upstream '

# One-shot triage: log verdict + live target + backend answer
echo '--- error.log verdict ---'
sudo tail -n 100 /var/log/nginx/error.log | grep -Eo 'connect\(\) failed.*|upstream prematurely closed[^,]*|upstream timed out[^,]*' | tail -5
echo '--- live targets ---'
sudo nginx -T | grep proxy_pass
echo '--- backend answer ---'
curl -s -o /dev/null -w "%{http_code} in %{time_total}s\n" http://127.0.0.1:3001/health

Fix proxy_pass and Upstream Blocks Without Guessing

The minimal correct proxy block has five ingredients: an upstream naming the real host and port, a location forwarding to it, the two headers backends need for correct URLs and client IPs, and HTTP/1.1 with an empty Connection header so keepalive to upstream actually works. The upstream block earns its place the day you add a second app server — one name to change instead of five location blocks. Until then it documents intent: this name is the app, and the app lives here.

Port accuracy is the whole game. The port in the upstream or proxy_pass line must equal the port the app bound, verified with ss -ltnp rather than memory. Render the port from the same variable or service-discovery entry the app uses so the two can't drift. In Docker, point at the Compose or K8s service name (http://backend:3000), never 127.0.0.1, which is the Nginx container talking to itself. Respect trailing-slash semantics too: proxy_pass http://app_backend; under location /api/ preserves the full path, while proxy_pass http://app_backend/v1/; strips the matched prefix. Mixing them up converts a fixed 502 into a wave of 404s — curl one real endpoint after every change.

Healthy stacks still 502 without keepalive alignment. Nginx reuses idle upstream connections while backends close them on their own schedule — Gunicorn defaults to a 2s keepalive against Nginx's 60s assumption, so Nginx keeps sending requests down dead sockets and logging prematurely closed. Set keepalive 32 with keepalive_timeout 60s and keepalive_requests 1000 in the upstream block, keep the backend's idle timeout above Nginx's, and add proxy_next_upstream error timeout http_502 with tries capped at 2 so idempotent requests retry once on a sibling instead of failing the user.

Size buffers before fat headers bite: proxy_buffer_size 16k with proxy_buffers 8 16k absorbs chunky auth cookies and JWTs that otherwise die as upstream sent too big header. Apply changes with nginx -t then nginx -s reload, and confirm with nginx -T that live matches intent. When 502s persist past a correct reload, the fault was never the config: return to the backend, because a perfect proxy still 502s against a dead app.

upstream app_backend {
    server 127.0.0.1:3001 max_fails=3 fail_timeout=30s;
    keepalive 32;
    keepalive_timeout 60s;
    keepalive_requests 1000;
}

server {
    listen 80;

    location /api/ {
        proxy_pass http://app_backend;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        proxy_connect_timeout 10s;
        proxy_send_timeout 60s;
        proxy_read_timeout 60s;

        proxy_next_upstream error timeout http_502;
        proxy_next_upstream_tries 2;
        proxy_next_upstream_timeout 10s;

        proxy_buffer_size 16k;
        proxy_buffers 8 16k;
        proxy_busy_buffers_size 32k;
    }
}
# Unix-socket variant: proxy_pass http://unix:/run/myapp/app.sock:/;
# Docker variant:    server backend:3000;  (service name, never 127.0.0.1)

Timeouts vs Slow Apps: Tune Without Hiding Outages

Nginx carries three patience knobs guarding three phases. proxy_connect_timeout caps the TCP handshake — keep it at 5–10s so dead backends fail fast instead of stacking connections. proxy_send_timeout caps pushing the request body uphill; 60s covers all but pathological uploads. proxy_read_timeout caps the gap between response bytes, and it's the one slow endpoints actually hit — 60s default, raised deliberately per location. Error.log mirrors them: connection timed out versus upstream timed out.

The order is non-negotiable: prove the app slow before extending patience. You need upstream timed out in error.log plus direct curls that eventually succeed — the backend answers, just slower than the current read timeout. Scope every raise to that location block, as below, never globally: a blanket 300s read timeout turns each future backend death into connections that hang for five minutes, filling worker pools with doomed waits. Timeouts shape slow success; they can't create success from refusal, and raising them over connect() failed converts crisp 502s into a site that freezes instead of erroring.

Bound your retries with the same discipline. proxy_next_upstream error timeout lets a failed request try the next peer, with proxy_next_upstream_tries 3 and proxy_next_upstream_timeout 10s capping the blast radius — unbounded, one slow fleet multiplies every user's wait by your peer count. Keep non_idempotent off the list unless you've thought hard: retrying a POST that already charged a card is how timeouts become double-billing incidents.

Align the app's own clocks above the proxy's. Gunicorn's --timeout 30 murders workers and their in-flight requests a full 90 seconds before a 120s proxy_read_timeout gives up — set the app timeout strictly higher. Node servers behind keepalive need keepAliveTimeout above Nginx's keepalive_timeout. And keep slow-client knobs separate: client_body_timeout and client_max_body_size govern the user-to-Nginx leg, proxy_read_timeout governs Nginx-to-app. When the report endpoint still takes 90 seconds after all this, paginate it, cache it, or background it — a 120s timeout blessed forever is latency debt.

# Default patience for the API: fail fast on dead, firm on slow
location /api/ {
    proxy_pass http://app_backend;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_connect_timeout 10s;
    proxy_send_timeout 60s;
    proxy_read_timeout 60s;
    proxy_next_upstream error timeout http_502;
    proxy_next_upstream_tries 2;
    proxy_next_upstream_timeout 10s;
}

# Generous patience, scoped to the one endpoint proven legitimately slow
location /api/reports/ {
    proxy_pass http://app_backend;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_connect_timeout 10s;
    proxy_send_timeout 120s;
    proxy_read_timeout 180s;
    # No next-upstream retry here: report POSTs are not idempotent
}

502 vs 503 vs 504: Return the Status That Tells the Truth

These three statuses describe three different truths, and returning the wrong one sends the next incident down the wrong path. A 502 says the upstream answered badly: refused connection, truncated reply, garbage headers, or a vanished socket. It indicts the conversation between proxy and app, and its remedy lives in process health, proxy targets, or buffer sizing. Count 502s as broken conversations — a spike means a bad deploy, a dead fleet, or config drift, and error.log already names which.

A 503 says the service knowingly declined: overloaded, in maintenance, or shedding load, ideally with Retry-After telling clients when to return. It indicts capacity or intent, and its remedy is scaling, draining, or waiting. Have backends emit honest 503s instead of hanging until Nginx manufactures a 502 — balancers retry 503s elsewhere by design, while a 502 looks like a poisoned conversation they'll keep hammering. Your maintenance pages and overload shedding must return 503 deliberately.

A 504 says the upstream never answered in time: the proxy waited the full proxy_read_timeout and gave up. It indicts slowness, and its remedy is faster handlers or deliberately longer waits. The error.log distinction stays crisp — refused and prematurely closed versus upstream timed out — so classify from the log line whenever the status code and the story seem to disagree.

Map the edge layers too. Behind Cloudflare, a branded 502 is your origin's own 502 passed through; an unbranded one means Cloudflare can't reach or parse your origin — broken gzip with a stale content-length, partial HTTP/2 support, or a dead tunnel. On AWS, split ELB_502 counts (the balancer's complaint) from target 502s (your app's). gRPC upstreams need grpc_pass with a matching HTTP version, and WebSocket locations need Upgrade plus Connection headers with a long read timeout, or quiet sockets die as 502s with zero app-side trace. Dashboard 502 and 504 trends separately: blended 5xx graphs hide a deploy-driven 502 spike inside normal 504 noise for exactly as long as it takes to miss the rollback.