Nginx 504 Gateway Timeout — Fix Slow Upstream Fast

Your users stare at a loading spinner for exactly 60 seconds, then get a stark 504 Gateway Timeout page. Not a crash, not an error message from your app — a timeout notice from Nginx, delivered with metronomic precision at the same second every time. That precision is the clue: something is waiting on a fixed timer, and the timer always wins.

504s surface when real work gets slow: a report query that scans 40 million rows, a downstream payment API that hangs during their incident, a PHP worker stuck on a locked database row. Development never shows it because datasets are small and dependencies are local. Production combines all three at scale.

The trap is treating every 504 as 'increase the timeout.' Timeouts buy patience, not performance — raising proxy_read_timeout from 60s to 300s turns a 60-second failure into a 5-minute failure while worker connections pile up behind it. The timeout change is sometimes right, but only after you've proven what the upstream was doing with those seconds.

By the end of this article you'll separate 502s from 504s in one glance at the logs, measure exactly where upstream seconds go with timing variables, trace slow queries and hung downstream calls, and set each Nginx timeout (including fastcgi_read_timeout for PHP) to a value your system can actually honor.

502 vs 504: Read the Timing Before Anything Else

The single most valuable diagnostic in any gateway incident is the failure's timing signature. A 502 arrives fast — milliseconds — because the upstream refused the connection, reset it, or returned garbage immediately. A 504 arrives exactly at the timeout — 60.000 seconds with default settings — because Nginx waited the full window and gave up. Two incidents with identical user-facing 'the site is broken' reports have opposite causes, and the timestamp delta tells them apart before you open a single config file.

Make this distinction structural by logging it. The default Nginx access format hides upstream timing; adding $upstream_response_time, $upstream_connect_time, and $upstream_status turns every log line into a pre-diagnosed incident. Upstream_status 504 with response_time equal to proxy_read_timeout is the canonical slow-upstream fingerprint. Upstream connection errors with tiny response times are dead-upstream fingerprints. When both appear together, the dead upstream is usually a downstream casualty of the slow one (crashed under pileup), so chase the 504 first.

Error.log corroborates independently. 'upstream timed out (110: Connection timed out) while reading response header from upstream' is the 504 smoking gun; 'connect() failed (111: Connection refused)' and 'upstream prematurely closed connection' are 502-class. Grep both patterns in the incident window and count — the ratio tells you whether you're fighting one slow dependency or a dead tier.

# Classify the incident in 60 seconds: timing + error-log pattern
# 1. Do 504s land exactly on the timeout? (slow upstream fingerprint)
awk '$9==504 {print $NF}' /var/log/nginx/access.log | sort -n | uniq -c | tail -5
# Expect: failures clustered at 60.000 (your proxy_read_timeout value)

# 2. Count 504-class vs 502-class evidence in error.log
sudo grep -c 'upstream timed out' /var/log/nginx/error.log
sudo grep -cE 'connect\(\) failed|connection refused|prematurely closed' /var/log/nginx/error.log

# 3. Permanent fix: log upstream timing on every request
# In nginx.conf http block:
# log_format timed '$remote_addr - $status $request_time '
#   'upstream=$upstream_response_time connect=$upstream_connect_time '
#   'ustatus=$upstream_status';
# access_log /var/log/nginx/access.log timed;
# Then: sudo nginx -t && sudo nginx -s reload

The Three Proxy Timers and What Each One Covers

Nginx splits gateway patience into three independent timers, and production configs routinely set the wrong one. proxy_connect_timeout governs the TCP handshake to the upstream (default 60s — generous; a healthy connect takes milliseconds, so a connect firing means network or DNS trouble). proxy_send_timeout governs writing the request body upstream (fires on huge uploads over slow links). proxy_read_timeout governs the gap between response bytes (the one that fires in nearly every 504 you'll ever see). Each defaults to 60s, and each must be set in the context that actually proxies — http, server, or location.

The subtlety is that proxy_read_timeout is an idle-between-bytes timer, not a total-request timer. A response that streams one byte every 59 seconds never trips a 60s read timeout even over an hour — while a response silent for 61 seconds trips it once. For SSE streams, long-polling, and slow file downloads, this means the correct fix is often a much larger read timeout on that specific location, not a global increase. Scope generous values to the streaming location and keep interactive endpoints tight.

Size timers from measured latency, not hope. If your p99 upstream time is 800ms, a 10s read timeout gives 12x headroom while still failing fast enough to protect worker pools. A checkout that legitimately needs 90s is a broken checkout — fix the query instead of blessing 120s. Every timeout second multiplied by concurrent slow requests is worker connections held hostage, which is how a slow endpoint becomes a full outage.

# Size from measured p99, scope per location — never one global value
upstream app_backend {
    server 10.0.1.11:3000 max_fails=3 fail_timeout=30s;
    server 10.0.1.12:3000 max_fails=3 fail_timeout=30s;
    keepalive 32;
}

server {
    listen 80;
    server_name app.example.com;

    # Interactive API: tight — slow here is broken, fail fast
    location /api/ {
        proxy_pass http://app_backend;
        proxy_connect_timeout 5s;    # healthy connects take ms
        proxy_send_timeout 10s;
        proxy_read_timeout 30s;      # 12x+ headroom over 800ms p99
    }

    # Report export: legitimately slow, scoped generously
    location /api/reports/export {
        proxy_pass http://app_backend;
        proxy_connect_timeout 5s;
        proxy_send_timeout 60s;
        proxy_read_timeout 300s;     # only where minutes are legitimate
    }
}
# Verify: sudo nginx -t && sudo systemctl reload nginx

Trace the Slow Query: Database Locks and Missing Indexes

The database is the most frequent owner of Nginx's wasted 60 seconds. Two patterns dominate: the missing-index full scan, where a query that was instant on 10,000 dev rows grinds through 40 million production rows, and the lock wait, where your request isn't slow at all — it's queued behind another session's lock and burning the whole timeout standing still. Both look identical from Nginx (60.0s, then 504), and pg_stat_activity separates them in one query.

Run the active-session query during the incident, not after. Sessions with identical query_start ages and a wait_event_type of Lock are queued, not computing — find the blocking pid and the lock holder's query, which is usually a report, a migration, or an uncommitted idle transaction someone's laptop is holding open. Sessions with no lock wait but huge durations are genuinely slow — EXPLAIN ANALYZE them and look for Seq Scan on large tables, which names the missing index directly.

Fix at the source in priority order: kill the blocker to restore service, add the index to fix the plan, then move analytics off the write primary (read replica) so the pattern can't recur. Only when the query is genuinely as fast as physics allows and still exceeds the timer should the timer move — and then only for that endpoint's location block.

# Catch the guilty query LIVE during the 504 window (Postgres)
psql -c "SELECT pid, now() - query_start AS duration, wait_event_type, left(query, 120) FROM pg_stat_activity WHERE state = 'active' AND now() - query_start > interval '5 seconds' ORDER BY duration DESC;"
# wait_event_type = Lock -> queued behind a blocker, not slow itself
# wait_event_type null + huge duration -> genuinely slow, EXPLAIN it

# Find what holds the lock (run when wait events show Lock)
psql -c "SELECT blocked.pid AS blocked_pid, blocking.pid AS blocking_pid, left(blocking.query, 100) FROM pg_stat_activity blocked JOIN pg_stat_activity blocking ON blocking.pid = ANY(pg_blocking_pids(blocked.pid));"

# Prove the plan is the problem, then fix with the index it names
psql -c "EXPLAIN ANALYZE SELECT * FROM orders WHERE status='pending' AND created_at > now() - interval '7 days';"
# Seq Scan on orders (cost=... rows=40000000) -> CREATE INDEX CONCURRENTLY
psql -c "CREATE INDEX CONCURRENTLY idx_orders_status_created ON orders (status, created_at);"
# Verify: re-run EXPLAIN ANALYZE, expect Index Scan and ms-scale time

Hung Downstream Calls: Timeouts, Breakers, and X-Ray Traces

When your app calls a payment provider, a fraud API, or another internal service that hangs, Nginx's 60 seconds get spent entirely inside your app waiting on someone else. App CPU stays idle, app logs show nothing until the downstream finally responds, and every diagnostic pointed at your own code comes back clean — because your code is innocent and parked on a socket read. Distributed tracing is the only honest witness: the trace shows your 60s span with 59s inside a single downstream child span.

Every outbound call needs its own timeout strictly shorter than Nginx's timer, or the downstream's hang becomes your 504 by construction. If Nginx allows 30s, downstream HTTP clients must time out at 5–10s with one retry at most — the remaining budget covers your own processing plus a clean error response. An outbound call with no timeout (still the default in several HTTP clients) can hang forever, holding a worker and eventually the whole pool.

Circuit breakers convert repeated downstream hangs from cascading 504s into fast, controlled failures. After N consecutive downstream timeouts, the breaker opens and fails immediately for a cooldown window — your endpoint returns 503-with-retry-after in milliseconds instead of 504ing for a minute per request. Users see a brief degradation; without the breaker they see a full outage plus a connection pileup that takes the database down with it.

# Prove the downstream owns the seconds (no app changes needed)
# 1. Time your app directly, bypassing Nginx
time curl -s http://10.0.1.11:3000/api/checkout -X POST -d '{}' -o /dev/null
# real 0m59.8s -> the app itself waited ~60s

# 2. Time the downstream dependency alone
time curl -s https://payments-provider.example.com/health -o /dev/null
# real 0m55.1s -> 55 of your 60 seconds belong to the provider

# 3. Confirm in the trace (X-Ray): find the 60s segment's longest child
aws x-ray get-trace-summaries --start-time $(date -u -d '15 min ago' +%s) --end-time $(date -u +%s) --filter-expression 'service("checkout") AND duration > 50' --query 'TraceSummaries[0].Id'
# Open that trace: your span 60s, child span 'POST payments-provider' 55s

# 4. Fix shape (Node example): downstream timeout << Nginx timer
# const ctrl = new AbortController();
# const t = setTimeout(() => ctrl.abort(), 8000); // 8s vs Nginx 30s
# await fetch(providerUrl, { signal: ctrl.signal });

PHP and fastcgi_read_timeout: The Timer Everyone Forgets

PHP-FPM deployments don't use proxy_read_timeout at all — they proxy over FastCGI, which has its own parallel timer family: fastcgi_connect_timeout, fastcgi_send_timeout, and fastcgi_read_timeout. Teams migrating timeout knowledge from Node/Python guides set proxy_read_timeout, reload, change nothing, and conclude 'timeouts don't fix it' — because the directive they set governs a module their traffic never touches. If your location uses fastcgi_pass, only fastcgi_* timers apply.

The rest of the 504 playbook is identical once the right timer is named: failures at exactly the fastcgi_read_timeout value are slow-PHP fingerprints, and PHP's own slow log (request_slowlog_timeout plus slowlog path in the FPM pool config) names the exact function and stack burning the seconds. A 60s fastcgi timer with a 60s PHP max_execution_time is a race you'll lose half the time — size FPM's execution limit comfortably above the FastCGI read timer so PHP logs its own fatal error instead of being silently cut off mid-request.

Also check FPM pool saturation, which masquerades as slowness. When pm.max_children is exhausted, requests queue inside FPM and burn the FastCGI timer waiting for a worker — the PHP code is fast but never gets to run. The FPM status page (pm.status_path) shows the listen queue directly; a nonzero queue with idle CPU means add children (within memory budget), not seconds. After setting the timer, reproduce the slow endpoint and confirm failures (if any) now land past the old 60s mark with slow-log entries naming the function — proof the FastCGI timer governs the path.

server {
    listen 80;
    server_name shop.example.com;
    root /var/www/shop/public;

    location ~ \.php$ {
        include fastcgi_params;
        fastcgi_pass unix:/run/php/php8.3-fpm.sock;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        # The FastCGI timer family — proxy_read_timeout does NOTHING here
        fastcgi_connect_timeout 5s;
        fastcgi_send_timeout 10s;
        fastcgi_read_timeout 120s;   # scoped: only .php locations
    }
}
# PHP-FPM pool: log the guilty function (/etc/php/8.3/fpm/pool.d/www.conf)
# request_slowlog_timeout = 10s
# slowlog = /var/log/php-fpm/www-slow.log
# Check saturation: curl http://localhost/fpm-status | grep 'listen queue'
# Verify: sudo nginx -t && sudo systemctl reload nginx php8.3-fpm

Prevention: Alerts on Upstream Latency, Not on 504s

Paging on the 504 rate means paging on user-visible failure — you're already late. The leading indicator is upstream response-time percentiles: alert when p99 crosses a fraction of the timeout (5s against a 30s timer) and you get hours of warning while users still see success. Track per-endpoint, because a global average hides the one report route climbing toward the cliff while checkout stays flat.

Make timeouts reviewable artifacts, not folklore. Every location's timers should carry a comment naming the measured p99 they were sized from and the date, so the next engineer knows whether 30s is data or superstition. CI can assert the basics: no location without explicit timers, no timer exceeding the worker-pool budget, fastcgi locations using fastcgi_* directives.

Finally, rehearse the slow-dependency scenario the way you rehearse failover. Chaos-test a 10x downstream latency injection quarterly and watch: do breakers open, do retries back off, do queues drain? The teams that survive provider incidents aren't the ones with the biggest timeouts — they're the ones whose systems fail fast, shed load, and recover the moment the dependency heals. Review the timer-comment dates in every post-incident: a sizing note older than two traffic doublings is stale data dressed as engineering, and re-measuring p99 takes less time than the next 504 page.

# Leading-indicator alert: page on latency, not on user-facing 504s
# Prometheus + nginx-prometheus-exporter (or parse $upstream_response_time):
# ALERT UpstreamP99NearTimeout
#   IF histogram_quantile(0.99, nginx_upstream_seconds) > 5
#   FOR 10m  (timeout is 30s — pages with 6x headroom remaining)

# Weekly audit: every proxied location must set explicit timers
sudo grep -rL 'proxy_read_timeout\|fastcgi_read_timeout' /etc/nginx/sites-enabled/ /etc/nginx/conf.d/
# Any file listed proxies without an explicit timer — fix before it pages you

# Quarterly chaos drill: inject 10x downstream latency in staging, verify:
# 1. Breaker opens (fast 503s, not 60s 504s)
# 2. Frontend retries back off (no lock-queue pileup)
# 3. Recovery is instant when latency normalizes (no stuck connections)
tc qdisc add dev eth0 root netem delay 5000ms  # staging only, remove after