502 Bad Gateway in Nginx: Fix the Failing Upstream

Checkout is down. Every browser shows the same stark page: 502 Bad Gateway. Nginx is up, the servers are up, CPU graphs are flat — yet every request through the proxy dies. Someone suggests restarting Nginx. Someone else blames the CDN. Meanwhile the actual culprit, a crashed app server one hop behind the proxy, sits quietly untouched.

A 502 is Nginx telling you the truth about someone else: it asked the upstream for a response and got garbage, silence, or a slammed door. The fix is almost always behind the proxy, not in it — a dead process, a proxy_pass pointing at last month's port, or an app slower than the proxy's patience. Reading one log file correctly splits those cases in under a minute.

This guide shows how to confirm which one you have with error.log and a direct curl, fix the backend first, and only then tune proxy_pass and timeouts. You'll also learn to stop confusing 502 with its cousins 503 and 504, so your status codes tell the truth during the next incident. Every step uses commands you can run as-is.

502 Means the Proxy Got a Bad Answer From Upstream

A 502 Bad Gateway is the proxy reporting a failed conversation, not a failed proxy. Nginx accepted your request, opened a connection to the upstream named in proxy_pass, and received something it couldn't use: a refused connection, an empty reply, a truncated response, or headers so malformed they can't be forwarded. HTTP semantics pin the blame precisely — the gateway received an invalid response from the inbound server — which is why seasoned operators read a 502 as upstream trouble until proven otherwise.

That conversation can break at three distinct moments. Before it starts, when nothing listens on the configured host and port and the kernel answers with connection refused. Midway, when the app accepts the connection but crashes or hangs up early, leaving Nginx holding half a response it must discard. Or at the content layer, when the upstream sends bytes Nginx can't forward, such as headers larger than its buffers or a body that ends mid-chunk. Each moment leaves a different fingerprint in error.log, which is why that file — not access.log — is the primary witness.

The most important consequence is where you spend your first five minutes. Restarting Nginx, purging CDN caches, and tweaking DNS all target machinery that is demonstrably working: Nginx is healthy enough to produce a well-formed 502. The fault sits one hop behind it in the app process, its port binding, or its response time. Teams that internalize this aim every first command past the proxy instead of at it.

The rest of this guide makes that reflex concrete. You'll learn the three error.log phrases that split every 502 into dead, misconfigured, or slow; the direct-curl test that bypasses the proxy in seconds; minimal proxy_pass and timeout blocks that are correct without being clever; and the 502 vs 503 vs 504 distinction that keeps your status codes honest.

Backend Down vs Proxy Misconfigured: Tell Them Apart

Every 502 investigation starts with one split: is the backend dead, or is the proxy misdirected? The direct curl answers it in seconds. Point curl at the backend's host and port with the proxy removed from the path — a 200 means the app is alive and Nginx is looking in the wrong place, while connection refused means the process is down regardless of what any config says. This single test eliminates half the hypothesis space before you've opened a single config file.

A dead backend usually announces itself loudly once you look. systemctl status shows inactive or failed with the last few log lines pointing at the crash; docker ps shows the container missing or restarting, and docker logs ends with an out-of-memory kill or an unhandled exception. These are application or capacity problems wearing a proxy costume. Fix the crash, the leak, or the failed deploy — no Nginx directive addresses a process that isn't running.

A living backend behind a persistent 502 points the other way: Nginx is asking the wrong address. Compare the port in proxy_pass against the port the app actually bound, which ss -ltnp reports authoritatively. Deploys that move ports, hosts files that changed, and upstream blocks copied between environments all produce a healthy app and a 502 at once. The proxy is faithfully forwarding to nowhere.

Make the order a habit: curl the backend, check the process, then read the config. Skipping straight to config edits while the process is dead wastes the exact minutes when checkout is failing. The commands above fit in one terminal and run in under thirty seconds, which is the right budget for triage when every checkout returns 502.

# Ask the backend directly, bypassing Nginx entirely
curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:3001/health

# Is the app process even alive?
systemctl status myapp --no-pager

# Or for containers: running state plus recent crash output
docker ps --filter name=myapp
docker logs --tail 50 myapp

Read Nginx error.log Like a Map to the Fault

Nginx's error.log is the authoritative witness for every 502, and three phrases cover nearly all of them. connect() failed (111: Connection refused) while connecting to upstream means nothing listened at the configured address — dead process or wrong port, settled by the direct curl. upstream prematurely closed connection means the app accepted then hung up mid-response: a crash between accept and reply, a response larger than buffers, or a keepalive race. upstream timed out means the app never finished in time, which is slowness rather than death.

Each phrase demands a different next move, which is why reading before acting matters. Refused sends you to the process list and the port comparison. Premature close sends you to app crash logs and response sizes around the failing endpoint. Timed out sends you to latency measurements and timeout math, never to a restart. Teams that grep the phrase first pick the right fix on the first try; teams that guess restart first sometimes never learn which fault they had.

The config commands beside the log matter just as much. nginx -t validates syntax without touching traffic, catching the missing semicolon before it becomes an outage. nginx -T dumps the effective running configuration — the files Nginx actually loaded, includes resolved — so grep on proxy_pass shows the true targets instead of the file you wish were loaded. When the on-disk edit and the -T output disagree, you've found the sites-enabled symlink or include-path problem.

Build the reflex as tail, grep the phrase, dump the live targets. Those three outputs — the failure mode, the claimed target, the real target — frame every 502 completely. Everything after them is repair, and repair without that frame is how one-digit typos survive for 38 minutes.

# Fresh upstream errors with context, newest last
sudo tail -n 100 /var/log/nginx/error.log | grep -E 'upstream|connect\(\)|timed out'

# Validate syntax before touching anything live
sudo nginx -t

# Print the running config's proxy targets
sudo nginx -T | grep -B 2 -A 2 proxy_pass

Fix proxy_pass and Upstream Blocks Without Guessing

The minimal correct proxy block has four ingredients: an upstream naming the real host and port, a location forwarding to it, and the two headers backends need for correct URLs and client IPs. The upstream block earns its place the day you add a second app server — one name to change instead of five location blocks. Until then it documents intent: this name is the app, and the app lives here.

Port accuracy is the whole game. The port in the upstream or proxy_pass line must equal the port the app bound, verified with ss -ltnp rather than memory. Deploys move ports, environment copies preserve stale ones, and typos turn one digit into a total outage for that location. Render the port from the same variable or service discovery entry the app uses so the two can never drift again — a single source of truth beats any amount of careful typing.

Watch the trailing-slash semantics when you write proxy_pass with a URI part. proxy_pass http://app_backend; under location /api/ preserves the full path, while proxy_pass http://app_backend/v1/; strips the matched prefix and rewrites it. Both are correct in different designs; mixing them up converts a fixed 502 into a wave of 404s. Test one real endpoint path with curl after every proxy change and compare what the app's access log received against what you sent.

Apply changes with nginx -t followed by nginx -s reload, then confirm with nginx -T that the live config matches your edit. The reload swaps configs without dropping connections, and the -T dump closes the loop between intention and reality. If 502s persist past a correct reload, the fault was never the config — return to the backend, because a perfect proxy still 502s against a dead app.

upstream app_backend {
    server 127.0.0.1:3001;
}

server {
    listen 80;

    location /api/ {
        proxy_pass http://app_backend;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

Timeouts vs Slow Apps: Tune Without Hiding Outages

Nginx carries three separate patience knobs, and each guards a different phase. proxy_connect_timeout caps the TCP handshake with the upstream — keep it short, around seconds, so dead backends fail fast instead of piling up connections. proxy_send_timeout caps pushing the request body uphill, which matters for uploads. proxy_read_timeout caps the wait between bytes of the response, and it is the one slow endpoints actually hit.

The order of operations is non-negotiable: prove the app slow before extending patience. An upstream timed out line plus direct curls that eventually succeed is the evidence — the backend answers, just slower than the current read timeout. Raising proxy_read_timeout for that endpoint, scoped to its location block as shown here, is then a deliberate capacity decision. Raising it globally because one report endpoint is slow grants every endpoint patience nobody reviewed.

Never tune timeouts over a dead backend. Extending read timeouts while error.log shows connect() failed converts instant, debuggable 502s into connections that hang for two minutes before failing identically. Users experience it as the site freezing instead of erroring, and your connection pools fill with doomed waits. Timeouts shape slow success; they cannot create success from refusal.

Prefer fixing the app's latency over extending the proxy's patience. A report endpoint taking 90 seconds usually needs pagination, caching, or a background job — not a 120s read timeout blessed forever. When slowness is genuinely legitimate, scope the generous timeout to that location, alert on its latency separately, and revisit it quarterly so temporary patience doesn't fossilize into permanent debt.

location /api/reports/ {
    proxy_pass http://app_backend;
    proxy_connect_timeout 10s;
    proxy_send_timeout 60s;
    proxy_read_timeout 120s;
}

502 vs 503 vs 504: Return the Status That Tells the Truth

These three statuses describe three different truths, and returning the wrong one sends the next incident down the wrong path. A 502 says the upstream answered badly: refused connection, truncated reply, or garbage headers. It indicts the conversation between proxy and app, and its remedy lives in process health or proxy targets. Monitoring that counts 502s is counting broken conversations.

A 503 says the service knowingly declined: overloaded, in maintenance, or shedding load, ideally with a Retry-After header telling clients when to return. It indicts capacity or intent, and its remedy is scaling, draining, or waiting. Configure your maintenance pages and overload shedding to return 503 deliberately — a load balancer that sees 503 retries elsewhere, while one that sees 502 may keep hammering a dead conversation.

A 504 says the upstream never answered in time: the proxy waited the full proxy_read_timeout and gave up. It indicts slowness, and its remedy is faster handlers or deliberately longer waits. The error.log distinction is crisp — prematurely closed or refused versus upstream timed out — so classify from the log line rather than the status code when they seem to disagree.

Get this taxonomy into your runbooks and your app code. Backends should return 503 with Retry-After when they choose to decline, letting the proxy forward honesty instead of manufacturing a 502 from a timeout. Review 502 and 504 trends separately in dashboards: 502s spiking means broken deploys or dead processes, while 504s climbing means growing latency. One graph blending them hides both stories at once.