x509 Signed by Unknown Authority — Fix TLS Trust

Your deploy goes green, your app boots, and then every outbound HTTPS call dies with x509: certificate signed by unknown authority. Your monitoring lights up, your pods crash-loop, and the certificate itself is perfectly valid — it expires in 300 days, the domain matches, and browsers accept it without complaint. The problem isn't the certificate. It's the chain of trust behind it.

This error strikes in the gaps between systems: a Go microservice calling an internal API behind a private CA, a Jenkins agent pulling from a registry with a fresh intermediate, a Python ETL job that worked on your laptop but fails in a slim container missing the OS bundle. Each runtime carries its own trust store, so a chain that's complete for Chrome can be broken for Java, Node, or curl in the same environment.

The dangerous 'fix' is everywhere on forums: set InsecureSkipVerify, NODE_TLS_REJECT_UNAUTHORIZED=0, or verify=False. That silences the error by disabling the check that protects your traffic — the equivalent of firing the security guard instead of showing ID.

By the end of this article you'll diagnose chain problems with openssl in under a minute, serve complete chains from your servers, install private CAs into every store that matters, and know exactly why verification must never be skipped.

How Chain Building Fails: Leaf, Intermediate, and Root

TLS trust is a signature chain with exactly three roles. The leaf (your server's certificate) is signed by an intermediate, which is signed by a root that clients ship in their trust stores. The client verifies each signature link by link until it lands on a root it already trusts. Break any link — omit the intermediate, or terminate at a root nobody trusts — and the whole handshake dies with 'signed by unknown authority'.

Servers are responsible for sending everything except the root: leaf plus all intermediates. Roots are never sent because the client must already have them; a server that sends its root is misconfigured (harmless, but a smell). The classic failure is serving cert.pem (leaf only) instead of fullchain.pem (leaf plus intermediate). Public CAs issue both files precisely because this mistake is so common, yet renewal scripts still grab the wrong one.

Clients differ in forgiveness. Browsers cache intermediates seen on other sites and can fetch missing ones via AIA URLs, so a broken chain often looks fine in Chrome. Go's crypto/x509, Java's PKIX validator, Python's ssl module, and curl against minimal bundles build strictly from what's served plus local roots. That's why the signature symptom of this bug is 'works in the browser, fails everywhere else' — and why server-side verification with strict settings is the only honest test. Keep a known-good reference: run the same s_client command against a host serving fullchain correctly and compare side by side — the missing intermediate block is obvious once you've seen a healthy chain.

# See exactly what the server sends (strict clients see only this)
openssl s_client -connect api.example.com:443 -servername api.example.com -showcerts </dev/null 2>/dev/null | grep -E 's:|i:|BEGIN CERT'

# s: = subject (who the cert is for), i: = issuer (who signed it)
# Healthy public chain shows 2+ certs:
#   0 s:CN = api.example.com / i:C = US, O = Intermediate CA
#   1 s:O = Intermediate CA / i:O = Public Root CA
# Broken chain shows only cert 0 — the intermediate never left the server

# Verify the served chain against the system bundle (strict, like Go/Java)
openssl s_client -connect api.example.com:443 -servername api.example.com -showcerts </dev/null 2>/dev/null | awk '/BEGIN CERT/,/END CERT/' > /tmp/chain.pem
openssl verify -CAfile /etc/ssl/certs/ca-certificates.crt /tmp/chain.pem
# Want: /tmp/chain.pem: OK. Anything else names your broken link.

Serve the Full Chain: fullchain.pem on Every Terminator

The server-side fix is almost always one filename: point your TLS terminator at the full chain file instead of the leaf. Certbot and most ACME clients write fullchain.pem (leaf plus intermediate) alongside cert.pem (leaf only) and chain.pem (intermediate only) — the renewal bug is grabbing cert.pem out of habit. Nginx, Apache, HAProxy, and ingress controllers all want the bundle; only the exact directive name differs.

Verify from the client's perspective after every change, not by re-reading your config. Config files say what you intended; s_client shows what you served. Count the BEGIN CERTIFICATE blocks (2 for a standard public chain, more for cross-signed roots), verify strict OK against the system bundle, and confirm the issuer of cert 0 names the intermediate rather than the root or itself.

Automate this into renewals. The deploy hook that installs a renewed certificate should run the s_client depth check and roll back on failure — a 10-second gate that would have prevented the 52-minute outage in this article's incident. Manual 'it renewed fine' checks don't survive the 3 AM auto-renewal six months later. Keep the previous known-good fullchain file versioned alongside the new one during rotation week, so a bad renewal rolls back with a single symlink flip instead of a reissue scramble.

# WRONG (leaf only — strict clients fail, browsers forgive):
# ssl_certificate /etc/letsencrypt/live/api.example.com/cert.pem;

# RIGHT (leaf + intermediate — every client can build the chain):
server {
    listen 443 ssl;
    server_name api.example.com;
    ssl_certificate /etc/letsencrypt/live/api.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/api.example.com/privkey.pem;
    ssl_trusted_certificate /etc/letsencrypt/live/api.example.com/chain.pem;
}

# Post-renewal gate (run in your --deploy-hook before reload):
# test "$(openssl s_client -connect localhost:443 -servername api.example.com -showcerts </dev/null 2>/dev/null | grep -c 'BEGIN CERTIFICATE')" -ge 2 || exit 1
# nginx -t && systemctl reload nginx

Private CAs: Install the Root Into the OS Trust Store

Internal PKI, corporate TLS-intercepting proxies, and dev clusters issue certificates from a company root that no public bundle contains. Every client rejects these by design until you install the root — that's the trust model working correctly, not a bug. The OS store is the foundation: most Linux tools (curl, wget, git, apt) read the system bundle, so installing there fixes the broadest set of clients in one move.

The procedure is distribution-specific but conceptually identical: drop the root .crt into the anchors directory, run the update tool, and verify with openssl verify. On Debian/Ubuntu that's /usr/local/share/ca-certificates plus update-ca-certificates; on RHEL it's /etc/pki/ca-trust/source/anchors plus update-ca-trust. Containers need the same steps baked into the image — a root installed on the host doesn't propagate into containers, which is why the app 'works on the VM but fails in the pod'.

Corporate proxies deserve special mention. When a proxy re-signs external traffic with a company root, every client behind it — including your CI runners and language package managers — needs that root. The symptom is bewildering ('even google.com fails verification') until you realize the issuer on every cert names your company proxy, not a public CA.

# Debian/Ubuntu: install a private root into the OS store
sudo cp corp-root-ca.crt /usr/local/share/ca-certificates/corp-root-ca.crt
sudo update-ca-certificates
# Expected: "1 added" — bundle rebuilt at /etc/ssl/certs/ca-certificates.crt

# RHEL/CentOS/Fedora equivalent
sudo cp corp-root-ca.crt /etc/pki/ca-trust/source/anchors/
sudo update-ca-trust extract

# Verify the root is trusted for your leaf (strict check, no app involved)
openssl verify -CAfile /etc/ssl/certs/ca-certificates.crt /tmp/internal-leaf.pem
# Want: /tmp/internal-leaf.pem: OK

# Bake into containers — host stores never propagate into images
# Dockerfile:
# COPY corp-root-ca.crt /usr/local/share/ca-certificates/
# RUN apt-get update && apt-get install -y ca-certificates && update-ca-certificates

Language Stores: Java cacerts, Node, and Python certifi

Fixing the OS store and declaring victory is the second most common way this incident recurs — because Java, Node.js, and Python each carry trust stores that ignore (partially or fully) the system bundle. Java is the strictest outlier: the JVM reads only its own cacerts file and never consults the OS store, so a root installed via update-ca-certificates remains invisible to every Java service on the box until you keytool -import it.

Node.js uses a bundled Mozilla root list compiled into the binary and ignores the OS store by default. The supported override is NODE_EXTRA_CA_CERTS pointing at a PEM bundle — note it replaces nothing, it appends your extra CAs to the built-in list. Python's requests library goes through certifi (its own pinned bundle via pip), while the stdlib ssl module typically uses the OS store — so the same host can have Python scripts that pass with urllib and fail with requests, depending purely on which bundle each path loads.

Go is the pleasant exception: crypto/x509 reads the OS store on Linux (and the system keychain on macOS/Windows), so OS-level installation suffices. Know your runtime's source of truth before you start installing — check keytool -list, NODE_EXTRA_CA_CERTS, and certifi.where() first, and you'll fix the right store on the first attempt instead of the third. When two runtimes on one host disagree, capture both probes' outputs before changing anything — the pair (system curl OK, JVM FAIL) is the fastest possible proof of a store mismatch and ends all debate about whose config is wrong.

# Java: import the private root into the JVM's own store (OS store is ignored)
sudo keytool -import -trustcacerts -alias corp-root-ca \
  -file corp-root-ca.crt -cacerts -storepass changeit -noprompt
keytool -list -cacerts -storepass changeit 2>/dev/null | grep -i corp-root

# Node: append extra CAs without replacing the built-in Mozilla list
export NODE_EXTRA_CA_CERTS=/etc/ssl/certs/ca-bundle-with-corp.crt
node -e "console.log(process.env.NODE_EXTRA_CA_CERTS)"
node -e "require('https').get('https://internal.example.com', r => console.log(r.statusCode))"

# Python: find which bundle each path uses, then pin the right one
python3 -c "import certifi; print(certifi.where())"
REQUESTS_CA=/etc/ssl/certs/ca-certificates.crt python3 probe.py
pip install --upgrade certifi  # stale certifi bundles cause phantom failures

# Go: uses the OS store — verify the binary sees your root
go run probe.go  # crypto/x509 loads /etc/ssl/certs on Linux automatically

Curl, Python, and Go: Reproduce Per-Runtime Before You Fix

Reproduce the failure with the smallest possible client before changing anything — a one-line probe per runtime that isolates trust from application logic. If curl fails with 'unable to get local issuer certificate' while openssl verify passes, the difference is which bundle each loaded, and that gap is your diagnosis. Probes also give you a regression test: the same one-liner becomes the deploy gate and the monitoring check.

Curl's --cacert flag lets you test candidate bundles without installing anything, which is perfect for confirming 'this bundle would fix it' before you roll it fleet-wide. Python probes should exercise both requests (certifi path) and urllib (stdlib path) since they can disagree on the same host. A Go probe using crypto/x509 with SystemCertPool mirrors exactly what your microservices do at startup, including the strict no-AIA-fetching behavior that makes Go the canary for chain problems.

Keep these probes in your runbook as copy-paste blocks. During an incident, the engineer on call shouldn't be composing TLS test harnesses from memory — they should paste, run, and read OK versus FAIL within sixty seconds. After the fix, re-run every runtime probe from the same runbook block and paste the outputs into the incident record — per-runtime OKs are the only closure evidence that survives the next on-call rotation.

# curl: test a candidate bundle WITHOUT installing it fleet-wide
curl --cacert /tmp/candidate-bundle.crt https://internal.example.com/health -s -o /dev/null -w '%{http_code}\n'
# 200 = this bundle fixes it. 60 (exit code) = still untrusted.
curl -v https://internal.example.com/health 2>&1 | grep -iE 'issuer|verify|subject'

# Python: exercise BOTH trust paths (they can disagree on one host)
python3 - <<'EOF'
import ssl, socket, requests
# stdlib path (OS store)
ctx = ssl.create_default_context()
with ctx.wrap_socket(socket.socket(), server_hostname='internal.example.com') as s:
    s.connect(('internal.example.com', 443))
    print('stdlib OK:', s.getpeercert()['subject'])
# requests path (certifi bundle)
r = requests.get('https://internal.example.com/health', timeout=10)
print('requests OK:', r.status_code)
EOF

Prevention: Renewal Gates, Depth Monitoring, and CA Rotation Drills

x509 outages are almost never novel — they're renewals that dropped the intermediate, rotations that forgot a store, or base images that lost a root. All three are preventable with the same trio: verify at deploy time, monitor the live chain, and drill rotations. The deploy gate runs s_client depth plus strict openssl verify against every TLS endpoint you serve, in CI and in the renewal hook, and blocks promotion on any non-OK.

Live monitoring needs two probes per endpoint: a browser-like check (uptime, expiry date) and a strict check (Go or openssl verify with no AIA fetching). The incident in this article stayed invisible for 20 minutes precisely because only the forgiving check existed. Alert on depth changes too — a chain that drops from 2 certificates to 1 is a renewal bug even if nothing has failed yet.

Finally, drill private-CA rotations like the outage they are. Rotating a corp root touches OS bundles, JVM cacerts, Node env vars, certifi pins, and container images simultaneously; each missed store is a future page. Maintain the trust matrix, rotate in staging first with per-runtime probes, and keep the old root trusted during a crossover window so slow-to-update clients don't hard-fail at cutover.

# Deploy gate: every endpoint must serve a verifiable chain (CI + renewal hook)
for host in api.example.com internal.example.com registry.example.com; do
  depth=$(openssl s_client -connect "$host:443" -servername "$host" -showcerts </dev/null 2>/dev/null | grep -c 'BEGIN CERTIFICATE')
  result=$(openssl s_client -connect "$host:443" -servername "$host" -showcerts </dev/null 2>/dev/null | awk '/BEGIN CERT/,/END CERT/' | openssl verify -CAfile /etc/ssl/certs/ca-certificates.crt /dev/stdin 2>&1)
  echo "$host depth=$depth verify=$result"
done

# Expiry early-warning (page at 21 days, not at midnight of expiry day)
openssl s_client -connect api.example.com:443 </dev/null 2>/dev/null | openssl x509 -noout -checkend 1814400 || echo 'RENEW WITHIN 21 DAYS'

# Cron/Blackbox: strict probe every minute, alert on anything but OK
# blackbox.yml tls_config uses the same system bundle your Go services load