PyMySQL 2003 Can't Connect — Network Fix

Your app boots, tries to open its database connection, and dies instantly: pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on 'db' ([Errno 111] Connection refused)"). No query ever ran. No credential was ever checked. The client never reached a server at all.

Error 2003 is purely a reachability failure — the TCP connection to MySQL's port never completed. That simplicity is deceptive, because the break can sit in any of five layers: MySQL isn't running, it's bound to the wrong interface, the host or port is wrong, a firewall eats the packets, or a container name doesn't resolve. Each layer produces the same 2003 with a slightly different errno, and guessing the layer wrong burns hours.

The stakes are practical: 2003 is the most common database error in fresh deploys, Docker moves, and cloud migrations. It always strikes at the worst moment — when you have the least diagnostic tooling and the most time pressure. A systematic outside-in check (server up, listening, reachable, resolvable, allowed) resolves it in minutes; random config edits can eat a day.

This guide gives you that order: confirming the server listens, fixing host and port, opening the network path, wiring Docker names correctly, and keeping 1045 grants work out of a 2003 problem. You'll also get timeouts and healthchecks that turn future 2003s into fast, obvious failures.

Read the 2003 Tuple: Host, Port, and Errno

PyMySQL raises OperationalError 2003 when socket.create_connection to the MySQL host and port fails. The message packs three facts: the hostname it tried, the errno from the OS, and sometimes the resolved IP. Errno 111 (connection refused) means SYN packets reached the host but nothing listens on that port. Errno 110 or 113 (timed out / no route) means packets died in transit — firewall, security group, or a wrong address. Errno -2 or -3 means DNS resolution itself failed, which is a naming problem, not a MySQL problem.

Read the full tuple before touching anything: (2003, "Can't connect to MySQL server on 'db' ([Errno 111] Connection refused)"). The quoted name is the exact host value your config supplied — compare it against the topology you intended. If it says localhost while you meant a remote host, the bug is in config loading, not networking. If it shows a stale IP, suspect cached DNS or an env file from another environment.

Crucially, 2003 happens before authentication. No password was sent, no grant was evaluated, no plugin negotiated. Any fix involving GRANT, ALTER USER, or password rotation cannot help a 2003 — those address 1045, a later failure. Logging the attempted host, port, and errno with every 2003 turns a mystery into a map: the numbers tell you which layer to inspect first.

import pymysql

# Fail fast with diagnostics: short timeout, explicit host/port, full error context.
try:
    conn = pymysql.connect(
        host="db", port=3306, user="app",
        password="secret", database="shop",
        connect_timeout=5, read_timeout=10, write_timeout=10,
    )
except pymysql.err.OperationalError as exc:
    code, msg = exc.args[0], exc.args[1]
    print(f"mysql connect failed: code={code} msg={msg}")
    print("host=db port=3306 -- check listener, firewall, DNS in that order")
    raise

Is MySQL Up and Listening on TCP?

A surprising share of 2003s mean MySQL simply isn't running — a failed upgrade, an OOM kill, a data-directory permission change after a restore. Check from the DB host itself: systemctl status mysql (or mysqld on RHEL-family) tells you whether the service is alive, and the error log — usually /var/log/mysql/error.log — tells you why it died. Look for InnoDB corruption lines, port-in-use lines, or permission denials on the data directory.

Next confirm the listener: ss -ltn | grep 3306 should show a LISTEN line. Its address matters — 127.0.0.1:3306 accepts loopback only, 0.0.0.0:3306 accepts all interfaces, and a missing line with a running server means skip-networking is set or the port is remapped. Match this against bind-address in my.cnf (grep -R bind-address /etc/mysql/) and the port setting, remembering that Docker -p mappings and managed-instance parameter groups can override file config.

Then prove the handshake without credentials: mysqladmin -h 127.0.0.1 -P 3306 ping expects mysqld is alive. If ping succeeds locally but the app fails remotely, the server is fine and the fault lies between hosts — firewall, routing, or wrong address. If ping fails locally too, stay on the server: fix the service before chasing networks.

Never restart blindly. A restart that clears a full disk or a deadlock works for a day and hides the cause; read the error log first so the fix addresses why it stopped.

# On the DB host: is MySQL alive, listening, and answering?
sudo systemctl status mysql --no-pager
mysqladmin -h 127.0.0.1 -P 3306 ping
ss -ltn | grep 3306
# Expect: LISTEN 0  151  0.0.0.0:3306  0.0.0.0:*

grep -R "bind-address\|skip-networking\|^port" /etc/mysql/ 2>/dev/null
sudo tail -n 50 /var/log/mysql/error.log

Wrong Host or Port: Test the Exact Endpoint

With the server proven, verify the address the app dials. Print the effective config — not the file you think it read, but the values after env substitution: grep -R MYSQL_HOST .env docker-compose.yml k8s/ and echo the DSN the app logs at startup. Then test that exact host and port from the app host with nc -zv db-host 3306, which performs the same TCP connect PyMySQL will attempt, minus credentials.

The localhost trap bites here. Most MySQL clients treat localhost as the Unix socket, but PyMySQL connects over TCP to 127.0.0.1 — so CLI success via localhost proves nothing about PyMySQL's path. Test both explicitly: mysql -h localhost for the socket path and mysql -h 127.0.0.1 --protocol=TCP for the TCP path. If the socket works but TCP fails, the server runs with skip-networking or binds the socket only.

Ports get remapped more often than teams expect: Docker -p 3307:3306, managed replicas on nonstandard ports, and SSH tunnels that forward a local port. Confirm the server's real port from my.cnf and from ss -ltn, then make the app's MYSQL_PORT match exactly. A connect to the wrong port on a live host returns refused fast — indistinguishable from a dead database unless you check.

Adopt one rule: the startup log prints the exact host, port, and connection type on every boot. When 2003 strikes, that line tells you what was dialed before you reproduce anything.

# From the APP host: test the exact endpoint the app will dial.
nc -zv db-host 3306
python3 -c "import socket; print(socket.create_connection(('db-host', 3306), timeout=5))"

# Socket vs TCP: prove which path works.
mysql -h localhost -u app -p -e 'SELECT 1'          # socket path
mysql -h 127.0.0.1 --protocol=TCP -u app -p -e 'SELECT 1'  # TCP path

# Show what the app really configured (after env substitution).
grep -RH "MYSQL_HOST\|MYSQL_PORT\|DB_HOST" .env docker-compose.yml

Firewalls and Security Groups: Prove the Split

When local connects succeed but the app host fails, packets are dying between machines — firewall, security group, or routing. Confirm the split cleanly: run nc -zv db-host 3306 on the DB host (expect success) and on the app host (observe timeout). Success-here plus timeout-there is the signature of a network policy drop, not a server fault.

Inspect each enforcement point in path order. Host firewalls: iptables -L -n or ufw status numbered on the DB host, looking for an INPUT rule on 3306. Cloud security groups: the DB instance's inbound rules must admit 3306 from the app's security group or subnet — a rule scoped to a stale CIDR after a VPC renumber is a classic silent killer. Corporate egress proxies and Kubernetes NetworkPolicies can also swallow 3306 while SSH sails through, so verify the port specifically, not general connectivity.

Fix narrowly and durably: open 3306 for the app subnet or reference the app security group by ID rather than CIDR, so future rescaling keeps working. Commit the rule to Terraform or the equivalent — a console click that fixes today's 2003 becomes next quarter's mystery outage when the environment is rebuilt without it. Then re-run nc from the app host until it succeeds before restarting the app.

Never answer a firewall 2003 by binding MySQL wider or disabling the firewall. Reachability should come from precise allow-rules, not from removing layers.

# Prove the split: same test, two vantage points.
# On DB host (expect: succeeded):
nc -zv 127.0.0.1 3306
# On APP host (observe: timeout = policy drop):
nc -zv db-host 3306

# Inspect host firewall and listening scope on the DB host.
sudo iptables -L INPUT -n --line-numbers | grep 3306
sudo ufw status numbered
ss -ltn | grep 3306

# After fixing the rule, verify from the app host before restarting anything.
nc -zv db-host 3306 && echo REACHABLE

Docker Service Names: localhost Is Yourself

Container networking gives localhost a new meaning, and 2003 is how teams learn it. Inside a container, localhost is that container — so an app configured with MYSQL_HOST=localhost dials itself on 3306 and gets refused by its own loopback. The database lives at its Compose service name (db), resolvable via Docker's embedded DNS only when both containers share a network.

Wire it correctly: set the app's DB host to the service name, keep both services on the same user-defined network (Compose does this by default per project), and expose the DB port to the app through the internal listener — no host port publishing required for container-to-container traffic. Verify from inside the app container: docker compose exec api getent hosts db should print the DB container's IP, and nc -zv db 3306 should succeed.

Then eliminate the boot race. depends_on alone only orders container start, not readiness — the app can still dial while MySQL initializes and log a 2003 that looks like a config bug. Add a healthcheck to the db service running mysqladmin ping and gate the app on service_healthy (Compose v2 syntax) or a wait-for-it entrypoint. Retries with backoff in the app's connect path cover the residual window.

Kubernetes rhymes: localhost is the pod, the DB host is a Service DNS name like mysql.prod.svc.cluster.local, and NetworkPolicies play the firewall role. Same outside-in checks, same resolution.

1045 Is a Different Error: Switch Runbooks

Error 1045 — access denied for user@host — is the error you get after beating 2003, and confusing the two wastes entire incidents. 2003 means no TCP connection; no password was ever sent. 1045 means the connection succeeded and the server rejected the credentials, the host grant, or the auth plugin. Password rotations, GRANT statements, and plugin changes fix 1045 and cannot fix 2003.

Use 1045 as your proof of reachability. When mysql -h db-host -u app -p starts returning 1045 instead of 2003, the network path works — celebrate, then switch tools: SELECT user, host, plugin FROM mysql.user to find the account rows, SHOW GRANTS FOR 'app'@'app-host' to read its privileges, and ALTER USER to align the password and plugin. The host part matters because 'app'@'%' and 'app'@'10.0.0.5' are different accounts that match different source addresses.

Keep the runbooks separate. The 2003 runbook is network: listener, host/port, firewall, DNS, container names. The 1045 runbook is identity: user@host row exists, password matches, plugin compatible, privileges granted, FLUSH PRIVILEGES if edited directly. Running the wrong runbook is how a 20-minute network fix becomes a 3-hour credential rotation with the same error at the end.

One operational habit prevents the confusion: log the numeric error code with every database failure. Codes don't lie — 2003 sends you to the network, 1045 sends you to grants.

# 2003 defeated when this stops refusing and starts denying:
mysql -h db-host -P 3306 -u app -p -e 'SELECT 1'
# ERROR 1045 (28000): Access denied for user 'app'@'10.0.0.5' -- GOOD: network works.

-- Now (and only now) inspect identity:
SELECT user, host, plugin FROM mysql.user WHERE user = 'app';
SHOW GRANTS FOR 'app'@'10.0.0.5';

-- Typical fix: create the exact user@host row the app connects from.
-- CREATE USER 'app'@'10.0.0.5' IDENTIFIED BY 'secret';
-- GRANT SELECT, INSERT, UPDATE, DELETE ON shop.* TO 'app'@'10.0.0.5';