Rust linking with cc failed: toolchain and linker fixes

You've just run cargo build on a fresh laptop, a new CI runner, or a slimmed-down Docker image, and instead of a binary you get a wall of text ending in error: linking with cc failed: exit status: 1. If you're new to Rust, that line feels like a verdict on your code. It isn't. Your code compiled. rustc did its job, produced every object file it was asked for, and then handed the final assembly step to a completely different program that wasn't ready.

That other program is your system's C linker, invoked under the name cc. Rust doesn't ship its own linker driver for this step — it borrows the one from your operating system, the same toolchain C and C++ projects rely on. When that toolchain is missing, when a system library's headers were never installed, when pkg-config can't find a path, or when a cross-compilation target needs a different linker, the link dies and cargo blames the messenger by name: cc.

What makes this error eat whole afternoons is misdirection. The last line names cc, so teams reinstall Rust, downgrade crates, or reboot machines — none of which touch the actual layer that's broken. The real diagnosis sits dozens of lines higher in the log: a cannot find -lssl line, an undefined reference to PQconnectdb line, or a plain cc: command not found. Each of those points at a different shelf in your environment, and each has a different fix.

You'll learn a layered way of reading these failures that works every time. First you'll see how rustc hands off to cc so the split between code bugs and environment bugs becomes instinct. Then you'll install the toolchain per operating system, decode undefined-symbol lines like a map, satisfy -sys crates through pkg-config or vendored builds, pin cross and musl linkers, speed up slow links with modern linkers, and clear stale artifacts without torching your lockfile.

By the end you'll hold the definitive mental model plus a production runbook: a Docker builder pattern that prevents the whole class, CI guards that catch it in seconds, and an OpenSSL case study you can mirror the next time a build goes red at 5 PM on a Friday.

How rustc invokes cc: the two-phase build you must picture

Every Rust build runs in two phases owned by two different programs, and the cc error lives entirely in the second one. In phase one, rustc compiles each crate — your code plus every dependency — into object files and Rust archives called rlibs. If your code has a type error or a borrow error, the build dies here with a rustc diagnostic pointing at your source. When rustc succeeds, its job is done. Phase two hands those objects to the platform linker driver, invoked as cc, which merges them with system C libraries into your finished binary. The error linking with cc failed names that handoff: compilation passed, assembly of the final binary did not.

Cargo builds the cc command line from metadata you rarely see: library search paths as -L flags, libraries to pull in as -l flags, the target triple, optimization and debuginfo settings, and extra arguments contributed by build scripts. Run cargo build -vv once and the veil lifts — the log prints the exact cc invocation with every flag, plus the environment the probe saw. Pipe it to a file with cargo build -vv 2>&1 | tee /tmp/build.log so you can search it instead of scrolling a terminal. That file is your incident artifact: it records compiler paths, search directories, and linked libraries exactly as cargo saw them, which beats any guess about what the machine contains.

Exit status 1 from the linker family means unresolved inputs, not a crashed tool. The linker ran correctly and reported that pieces are missing: an object references a symbol nobody provided, a -l flag names a library the search paths can't find, or the driver binary itself doesn't exist. This is why reinstalling Rust changes nothing — rustc already proved itself by producing the objects the linker is now complaining about. The skill is reading the complaint at face value instead of treating the whole dump as one mysterious failure.

Build scripts blur the phase boundary and deserve a clear picture. Crates ending in -sys ship a build.rs program that runs before linking: it probes the system with pkg-config or directory hunts, prints cargo directives like cargo:rustc-link-lib=ssl, and sometimes compiles bundled C sources with the cc compiler. Those directives shape the final link command, so a bad probe poisons the link minutes later. When openssl-sys prints a probe failure, you are watching phase-one reconnaissance fail, and the phase-two link error that follows is just the consequence arriving late.

Internalize one sentence and most of these failures get boring fast: rustc errors are code bugs, cc errors are environment bugs. Pure-Rust crates often link quietly with the default driver, which is why developers can use Rust for months without knowing cc exists — then one reqwest or diesel dependency pulls in C code and the hidden dependency surfaces. From here on, every section fixes one shelf of that environment in the order you'd check it during a real incident.

The name cc is a stand-in that resolves differently per platform, and knowing the stand-in saves confusion when logs differ between machines. On Linux it usually reaches gcc driving GNU ld. On macOS it reaches clang driving ld64. On Windows MSVC targets the driver is link.exe reached through rustc's MSVC handling rather than a cc binary at all. Same cargo error text, three different programs underneath — which is why a fix verified on Ubuntu says nothing about the Mac build until the Mac log is read too.

Make the -vv log a habit rather than a rescue move. The failing cc invocation lists every -L search directory and -l library in order, so copying those flags into a directory listing answers cannot-find questions in seconds. Pair that with rustup show to confirm the active toolchain matches the target you think you're building — developers switching between stable and nightly, or between MSVC and GNU on Windows, often debug the wrong toolchain's paths. The log plus the toolchain readout ends that class of confusion before it starts.

One last habit seals the model: narrate the phases out loud when triaging with a teammate. Stating rustc passed, the cc command failed on a named input turns a sprawling thread into a focused hunt. Shared language is shared speed.

# Capture the real evidence: full verbosity, saved to a searchable file.
cargo build -vv 2>&1 | tee /tmp/build.log

# Triage the log: the summary is last, the cause sits above it.
grep -n "linking with" /tmp/build.log
grep -nE "cannot find -l|undefined reference|cc: error|pkg-config|failed to run" /tmp/build.log | head -20

# Prove which layer is missing before installing anything.
which cc gcc pkg-config
cc --version | head -2
rustc --version && cargo --version

Missing C toolchain per OS: build-essential, Xcode CLT, and VS Build Tools

The most common cc failure on fresh machines is also the simplest: no C toolchain is installed. Rustup ships rustc, cargo, and the standard libraries — not gcc, not the platform linker, not C headers. Minimal Docker images, brand-new laptops, and bare CI runners all lack cc until someone installs it. The signature is unmistakable once you've seen it: every crate with a build script or C dependency fails identically, often with cc: command not found or unable to find tool, while pure-Rust crates sail through. That split behavior is your proof the Rust side is healthy.

On Debian and Ubuntu the fix is one metapackage: sudo apt install -y build-essential pkg-config pulls in gcc, make, libc6-dev, and the linker driver in a single step. Fedora and RHEL users run sudo dnf groupinstall -y Development Tools plus pkg-config, and Arch users install base-devel the same way. Always verify with both cc --version and gcc --version afterward, because some build scripts probe gcc directly while cargo invokes cc — a machine where one name resolves and the other doesn't will fail confusingly on alternating crates. Add pkg-config in the same command; slim images that lack cc almost always lack the probe tool too.

macOS hides the same gap behind friendlier branding. The Command Line Tools, installed with xcode-select --install, provide clang behind the cc name plus the SDK headers every native link needs. After a major macOS upgrade, links that worked for a year can break because the SDK moved — re-running the CLT installer and accepting the license is the standard repair. Homebrew users should also brew install pkg-config, since several -sys probes shell out to it and Apple's tools don't include it. Confirm with xcrun --show-sdk-path that an SDK is actually present, not just the compiler binary.

Windows splits into two toolchains you must not mix. MSVC targets, which rustup installs by default, need the Visual Studio Build Tools with the C++ workload and the Windows SDK; the rustup-init program even links to them during setup. GNU targets need MinGW-w64 with gcc on PATH instead. Check your active toolchain with rustup show: if it says stable-x86_64-pc-windows-msvc, install Build Tools, and if it says gnu, install MinGW-w64. Mixing a GNU linker with an MSVC rustc is a classic source of link errors that mention incompatible formats rather than missing files.

Bake the toolchain into every base so nobody pays this tax twice. Devcontainer Dockerfiles, CI base images, and onboarding scripts should install the compiler set before Rust code is ever fetched — a three-line install in the image saves every future hire the same lost afternoon. Pair that with a CI guard step running cc --version before cargo build: a missing toolchain then fails the pipeline in 5 seconds with an obvious message instead of after a 9-minute compile with a cryptic one.

Slim Docker images deserve special attention because they fail in a way that looks intermittent. The rust slim variants never promised a compiler, yet older tags carried gcc through transitive layers that later rebuilds removed — a routine image refresh can delete cc from a Dockerfile that changed nothing. Pin base images by digest, install the toolchain explicitly in the builder stage, and run cc --version as a build step so the failure points at the image instead of masquerading as a Rust problem.

On Windows, match the toolchain to the triple and keep the two from mixing on PATH. MSVC builds want the Build Tools C++ workload plus a recent Windows SDK, with linking handled through the Visual Studio environment rustup helps configure. GNU builds want MinGW-w64 gcc reachable on PATH with a gnu rustup toolchain such as stable-x86_64-pc-windows-gnu. Verify with rustc -vV and gcc --version from the same shell that runs cargo — mismatched shells, one launched from a VS Developer Prompt and one not, are a steady source of works-here link mysteries.

Record the verified toolchain in the repo so the fix survives turnover. A short doc naming the packages per OS plus the two version commands gives every new hire a green build on day one. Tribal knowledge evaporates; checked-in prerequisites endure.

# Debian/Ubuntu: the one metapackage that ends most fresh-machine failures.
sudo apt update && sudo apt install -y build-essential pkg-config
cc --version | head -1 && gcc --version | head -1
pkg-config --version

# Fedora/RHEL family:
# sudo dnf groupinstall -y 'Development Tools'
# sudo dnf install -y pkg-config gcc

# macOS: Command Line Tools provide cc/clang plus SDK headers.
# xcode-select --install
# xcrun --show-sdk-path

# Verify the exact toolchain cargo will use:
rustup show active-toolchain
ls -la "$(which cc)"

Read the log bottom-up: undefined symbols are the real error

Undefined-reference output terrifies by volume: hundreds of lines like undefined reference to SSL_new or undefined reference to PQconnectdb scroll past, and the instinct is to treat the wall as one giant failure. Volume is an illusion. One missing library produces one error per symbol it should have provided, so a single absent libssl can generate 300 lines that all mean the same thing. The diagnosis is always the first undefined line: it names the missing provider, and everything below it is the same absence repeated per symbol. Train yourself to read bottom-up — glance at the cc summary to confirm the phase, then jump to the first undefined reference and stop there.

Map the symbol prefix to the library and the fix writes itself. Symbols starting with SSL_, TLS_, BIO_, EVP_, or X509_ mean OpenSSL's libssl and libcrypto. PQ prefixes mean the Postgres client libpq. sqlite3_ means SQLite, z_ with inflate and deflate means zlib, and mysql_ means the MySQL client library. This prefix fluency covers nearly every production case you'll meet. When a prefix is unfamiliar, ask the system directly: nm -D on a candidate shared library lists the symbols it provides, so nm -D /usr/lib/x86_64-linux-gnu/libssl.so | grep SSL_new confirms the provider in one line. The ldd command on a previously working binary shows which provider a healthy link chose, which settles arguments about which of two installed libraries should win.

Two subtleties trap even experienced developers, so learn them now. Static-versus-shared mismatches fail with undefined lines despite files being present: a -sys crate expecting a static archive finds only shared objects, or a musl build demands static libs on a glibc machine carrying only .so files. Check for the right archive flavor, not just the library name. Versioned symbols are the second trap: a crate needing OpenSSL 3 APIs against OpenSSL 1.1 headers fails only on the new symbols, which looks like random breakage until you compare openssl version against the crate's documented minimum in one step.

The cannot find -l variant is the same diagnosis wearing different clothes. Where undefined reference means the library was searched but the symbol wasn't inside, cannot find -lssl means the linker never found the library file at all — the -L search paths in the -vv log don't include its directory. Copy the -L flags from the failing cc invocation, list those directories, and the gap becomes visible. Either way the workflow is identical: capture with -vv, isolate the first missing piece, verify the provider with nm or the path with ls, install or point at it, clean the -sys crate, rebuild.

Treat the symbol lines as the real error and the cc summary as its messenger from today on. Teams that learn this read resolve link failures in minutes; teams that keep starting at the last line keep reinstalling compilers that were never broken. The rest of this guide is just the fix catalog for each provider the symbols can name.

Walk one real wall to cement the habit. Suppose the log shows 200 lines starting with undefined reference to SSL_library_init, followed by neighbors naming SSL_load_error_strings and EVP_PKEY_new. A beginner sees 200 problems; you see one provider — OpenSSL's crypto library — missing from the link, with every line another symbol from the same absent archive. The second line never matters until the first is fixed, and after libssl-dev lands, all 200 vanish in a single rebuild. Volume was the disguise, never the diagnosis.

C++ symbols add one wrinkle: name mangling turns missing functions into alphabet soup. Pipe the symbol through c++filt to demangle it into a readable signature before mapping it to a library — one quick command prints the original declaration. Rust symbols demangle with rustfilt where installed, though most Rust-side undefined references trace back to a native library boundary anyway. Demangle first, map second, install third, and the exotic-looking failure becomes an ordinary missing package.

# Isolate the true error: first undefined symbol, not the summary.
grep -n 'undefined reference' /tmp/build.log | head -5
grep -nE 'cannot find -l' /tmp/build.log | head -5

# Map symbol prefix to provider library on this machine.
nm -D /usr/lib/x86_64-linux-gnu/libssl.so 2>/dev/null | grep -m2 'SSL_new'
nm -D /usr/lib/x86_64-linux-gnu/libpq.so 2>/dev/null | grep -m2 'PQconnectdb'

# Inspect what a healthy binary linked against (run on a working build).
# ldd target/debug/myapp | grep -E 'ssl|crypto|pq'

# After fixing the provider, force the probe to rerun:
# cargo clean -p openssl-sys && cargo build 2>&1 | tail -5

-sys crates, pkg-config, and vendored builds: feeding the probe

Crates ending in -sys are Rust wrappers around system C libraries: openssl-sys for TLS, pq-sys for Postgres, libsqlite3-sys, libz-sys, and dozens more. Their build.rs scripts run before linking and probe the machine for headers and libraries, emitting cargo directives that shape the final cc command. When headers are absent, the probe fails and the link dies with it. The classic paradox — the library is installed but the build fails — dissolves once you learn the packaging split: runtime packages carry shared objects your app needs at run time, while development packages carry the headers and .pc files the build needs at compile time. Production servers want the first; builders need both.

The fix order runs from simplest to most self-contained. First, install the development package: libssl-dev on Debian and Ubuntu, openssl-devel on Fedora, openssl and libpq through Homebrew on macOS. Second, when libraries live in odd prefixes, point the probe explicitly instead of moving files: OPENSSL_DIR for the whole tree, OPENSSL_LIB_DIR plus OPENSSL_INCLUDE_DIR for split layouts, PQ_LIB_DIR for Postgres. Third, sidestep the system entirely with the crate's vendored feature — openssl with vendored, or SQLite's bundled option — which compiles C sources from crates.io. Vendored builds trade longer compile times and larger binaries for zero system drift, which is exactly the trade you want in locked-down CI or minimal images.

pkg-config is the phone book between probes and libraries, and it fails in two boring ways. Either the tool itself is missing from slim images, or its search path PKG_CONFIG_PATH doesn't include the directory holding the .pc file — the canonical Homebrew case, where .pc files sit under the openssl prefix outside every default path. Diagnose outside cargo: a bare pkg-config --libs --cflags openssl should print flags, and Package openssl was not found reproduces the build failure in one line. Locate the real file, export the corrected path, and re-run the bare command until it prints flags before touching cargo again.

After changing anything, force the probe to rerun or you'll replay stale failure. Build scripts cache aggressively, so headers installed without cargo clean -p on the -sys crate produce the identical error and convince teams the fix didn't work. Watch the fresh probe output in a new -vv log to confirm it found the new paths — the confirmation line is the actual proof, not the absence of the old error. Then record system dependencies as code: a README section, a small build-deps script, or apt lines in the Dockerfile beside Cargo.toml. Unrecorded dependencies are future incidents wearing invisibility cloaks, and the next image bump will find them.

Choose vendored versus system deliberately per environment. Servers with patched system OpenSSL should link the system library so OS security updates flow through automatically. Hermetic CI, cross builds, and minimal containers should prefer vendored or rustls-based TLS to cut the system cord. Either choice is defensible; drifting between them per machine is what creates the cannot find surprises at deploy time.

Concrete variables beat vague advice, so here is the shape of each escape hatch. A custom OpenSSL under /opt/openssl answers to OPENSSL_DIR pointing at that prefix, or to the split pair OPENSSL_LIB_DIR plus OPENSSL_INCLUDE_DIR when libraries and headers diverge. Postgres clients answer to PQ_LIB_DIR pointing at the directory holding libpq. Export them in the same place for every consumer — shell profile, direnv file, Dockerfile ENV, and CI variables — because a fix present in your shell but absent in CI fails every second build and teaches the team to distrust the documented repair.

Read each -sys crate's docs before fighting it, since probes differ in honored variables and minimum versions. The openssl-sys and pq-sys pages list exactly which environment variables each release respects, and crate READMEs state minimum system versions that settle is-my-OpenSSL-too-old questions instantly. Skim the features list too: vendored, bundled, and static options live there with their build-time costs noted. Ten minutes in the docs replaces an hour of probe guessing, and the habit compounds across every native dependency you will ever meet.

# Debian/Ubuntu: runtime is not enough — install the headers + probe tool.
sudo apt update && sudo apt install -y libssl-dev libpq-dev pkg-config

# Reproduce the probe outside cargo (expect -L/-I flags, not an error).
pkg-config --libs --cflags openssl
pkg-config --libs --cflags libpq

# Homebrew macOS: the .pc files live outside the default search path.
# brew install openssl libpq pkg-config
# export PKG_CONFIG_PATH="$(brew --prefix openssl)/lib/pkgconfig:$PKG_CONFIG_PATH"
# pkg-config --libs --cflags openssl

# Force the cached probe to rerun after any environment change.
cargo clean -p openssl-sys
cargo build -vv 2>&1 | grep -iE 'openssl|pkg-config' | head -10

Cross-compilation linkers: pinning the right cc per target

Cross-compilation fails at link whenever the host cc is asked to emit a foreign binary. Rust's rustc handles codegen per target beautifully, but the final link needs a cross-capable linker plus target libraries — pieces rustup deliberately doesn't install. Building on x86_64 for aarch64 with the default linker produces architecture-mismatch or linker-not-found lines after a flawless compile, and the mismatch surprises people because everything up to the last step looked perfect. The compile was never the problem; the linker's architecture was.

Assemble three parts and the failure class disappears. First, the standard library for the target: rustup target add aarch64-unknown-linux-gnu, or the musl triple for static builds. Second, the linker binary itself: gcc-aarch64-linux-gnu on Debian and Ubuntu, the LLVM MinGW toolchain for Windows targets, or musl-gcc for musl. Third, the pin that ties them together: a target section in .cargo/config.toml setting linker to the cross binary, checked into the repo so every machine and CI runner agrees. Without the checked-in pin, each developer hand-configures a linker and version drift becomes its own incident category within a month.

Then chase host-path leakage, the chronic follow-up failure. Build scripts and pkg-config probes run on the host and happily return host include and library paths that the cross linker can't use — an x86_64 libssl path handed to an aarch64 link. Scope paths per target with TARGET_-prefixed environment variables, or run the build inside a maintained cross image where host and target sysroots are separated by construction. The cross-rs project images handle exactly this separation, and standardizing on one image beats per-developer linker installs for any team past three people.

Verify the artifact, not just the exit code. A binary that links but targets the wrong architecture is a runtime mystery waiting for deploy night — run file on the output and confirm it reports the target architecture, and check linkage expectations with the matching readelf or ldd. Keep a tiny matrix in the repo docs: each supported target, its std component, its linker package, and the config stanza. When the next engineer adds riscv64 or an Apple target, the matrix tells them the three parts instead of letting them rediscover the pattern through failures.

Apple targets and Windows targets each add their own wrinkle worth knowing before you need them. Building for Apple silicon or Intel Macs from Linux is poorly supported because Apple's SDK and linker don't travel — plan to cross to Apple targets from a Mac runner instead of fighting the toolchain gap. Windows targets cross best with LLVM-flavored MinGW toolchains pinned per triple exactly like the ARM example. The principle never changes: the linker must understand the target's object format, and hope is not a linker.

Scope host-path leakage with per-target environment variables rather than global exports. A global PKG_CONFIG_PATH pointing at host OpenSSL poisons every cross link, while target-prefixed overrides steer only the cross probe and leave host builds untouched. The maintained cross images enforce the same separation structurally: host tools in one sysroot, target libraries in another, with the config.toml pin selecting between them. Either mechanism works; what fails is the single global export serving two architectures from one path.

Document the matrix per target the day the first cross build lands. Each row names the rustup target component, the linker package and version, the config.toml stanza, and the CI job that exercises it. When the second target arrives — riscv64, an Android triple, a second ARM variant — the matrix turns a day of rediscovery into twenty minutes of following the established pattern. Review the matrix quarterly, because cross linker packages update on their own schedule and a pinned-but-ancient cross gcc becomes its own subtle bug source.

Rehearse the cross failure once before it counts. A scheduled CI run building every supported target weekly catches toolchain drift while it is still cheap. Green rehearsals are the evidence the matrix works.

# Pin one linker per target so every machine and CI runner agrees.
[target.aarch64-unknown-linux-gnu]
linker = "aarch64-linux-gnu-gcc"

[target.x86_64-unknown-linux-musl]
linker = "musl-gcc"

# Optional: per-target flags live here too, not in shell history.
# [target.aarch64-unknown-linux-gnu]
# rustflags = ["-C", "link-arg=-fuse-ld=mold"]

musl static builds: musl-tools, static linking, and libc trade-offs

Static musl builds are the deployment dream — a single binary with no libc dependency to mismatch on the target — and a linking puzzle until the musl pieces are in place. The musl target triple x86_64-unknown-linux-musl swaps glibc for musl libc, which changes what the linker may pull in: musl links demand static archives or musl-compatible objects where glibc builds happily consume shared ones. A workspace that links perfectly for gnu fails for musl with undefined C symbols, and the gap confuses because the Rust code is identical. The libc changed under the link, so the link's inputs must change too.

Set up the musl toolchain in three moves. Install musl-tools for the musl-gcc linker wrapper and confirm it prints a version. Add the target standard library with rustup target add x86_64-unknown-linux-musl. Pin linker to musl-gcc under the musl triple in .cargo/config.toml. Then rebuild and inspect: file should report a statically linked binary, which is the whole point of the exercise. Alpine-based images pair naturally with musl since Alpine already runs it; glibc Ubuntu hosts crossing to musl need the most care, because every -sys probe must be steered away from glibc paths.

The -sys crates need musl-compatible inputs, and vendored or static features are the pragmatic answer. OpenSSL through openssl-sys wants vendored against musl headers, SQLite prefers its bundled build, and ring's prebuilt assembly needs attention on some targets. An increasingly popular escape is avoiding C TLS entirely with rustls, a pure-Rust TLS stack that removes the OpenSSL link from the graph — reqwest with rustls-tls instead of native-tls turns a whole category of musl failures into non-events. Measure the trade: vendored builds lengthen compile times noticeably, while rustls changes your TLS implementation and deserves its own security review.

Watch binary size and DNS behavior before declaring victory. Static binaries run larger than dynamic ones, and musl's resolver behaves differently from glibc's in edge cases around search domains — both fine in practice, both worth one staging check. Keep musl as a dedicated CI job building from the same lockfile rather than a laptop-only ritual, and store the produced artifact's file output in the job log. When a musl link breaks months later, that log plus the pinned config tells the next engineer exactly which of the three pieces moved.

The openssl-src route deserves a concrete picture since it powers most vendored musl builds. Enabling the vendored feature pulls in a source-build crate that compiles a fixed OpenSSL release against musl headers during your build. Clean builds gain several minutes and the target directory gains hundreds of megabytes of C objects — budget both in CI timeouts and cache sizes before enabling it fleet-wide. The payoff is hermetic: the same lockfile links the same TLS stack on every runner regardless of what the host image carries, which is precisely what minimal containers need.

Alpine builders make the pairing concrete. A builder based on an Alpine image with rustup carries musl natively, so probes find musl headers without steering and the musl-gcc pin just works. The runtime stage can then be scratch with the static binary copied in — no shell, no libraries, minimal attack surface. One staging check remains mandatory: exercise DNS-heavy paths under musl before promoting, since resolver edge cases around search domains differ from glibc behavior. Pass that check once and the static artifact becomes the most boring, most reliable deploy in the fleet.

Track the size and time trade openly so the choice sticks. Static binaries run larger — expect roughly half again the size of dynamic ones for TLS-heavy services — and vendored clean builds run longer on every cold cache. Both costs are fixed and predictable, unlike the unbounded cost of 2 AM missing-library pages. Teams that publish the numbers stop relitigating vendored against system every quarter and get back to shipping.

# musl static setup: wrapper, target std, pin, then verify.
sudo apt update && sudo apt install -y musl-tools pkg-config
musl-gcc --version
rustup target add x86_64-unknown-linux-musl

# .cargo/config.toml must contain:
# [target.x86_64-unknown-linux-musl]
# linker = "musl-gcc"

cargo build --target x86_64-unknown-linux-musl
file target/x86_64-unknown-linux-musl/debug/myapp
# expect: ELF 64-bit LSB executable, x86-64, statically linked

Link speed: mold, wild, lld, and -C link-arg without breakage

Rust links slowly for a structural reason: monomorphized generics and link-time optimization hand the linker enormous inputs, and the default GNU linker processes them single-threaded. On large workspaces the link phase dominates incremental rebuilds — 40-second compiles followed by 3-minute links are a normal complaint, not a broken setup. The fix is swapping the linker backend rather than your code: modern linkers like mold, wild, and LLVM lld parallelize the work and routinely cut link times by multiples. This is a safe, well-trodden upgrade as long as you keep the default as fallback.

Pick per platform and install deliberately. Linux gets mold or wild for maximum speed; both are available as packages on recent distributions and as release binaries otherwise. LLVM targets and many macOS setups use ld64.lld or lld, while Windows MSVC builds can use lld-link. Confirm availability with mold --version or ld.lld --version before configuring anything — pointing cargo at a linker absent from half your CI runners trades slow links for broken ones, which is worse in every dimension.

Point cargo at the fast linker through -C link-arg so the change is visible and versioned. A workspace .cargo/config.toml entry with rustflags carrying link-arg=-fuse-ld=mold applies to every build from that checkout, while RUSTFLAGS in the environment scopes the experiment to one shell or one CI job. Start scoped, measure with time cargo build on a clean and an incremental build, then promote to the config file once numbers justify it. Teams commonly see full-workspace links drop from minutes to tens of seconds; incremental links improve less dramatically but still noticeably.

Treat debug info and LTO as the second dial once the linker is fast. Split debuginfo keeps incremental links lean, and understanding that release LTO moves cost from link to an even longer link prevents surprise when release profiles behave differently from dev. Never trade correctness flags for speed on release artifacts: keep the fast linker plus standard settings, and gate the change on the existing test suite passing bit-identical behavior. Speed is a feature only when the binary stays correct.

Split debuginfo is the second dial once the linker is fast. A dev profile carrying split debuginfo keeps incremental links lean by leaving debug info out of the hot path, while release profiles keep full fidelity where profiling matters. Pair that with default incremental compilation for dev and leave codegen units alone until measured — changing codegen units for link speed without profiling trades runtime performance for build speed blindly. Measure each dial separately: linker swap first, debuginfo second, and only then anything exotic.

Cache wisely and the fast linker compounds. Dependency-only prebuilds in Docker — copying manifests first, building dependencies, then copying sources — keep the linker warm across source edits instead of relinking the world. Shared compilation caches trim the compile side so the link side dominates less of the total. Keep the default linker configured as fallback wherever the fast one might be absent: a config that breaks on runners without mold is slower in the only metric that counts, builds completed.

Choose the backend deliberately rather than by hype. Mold leads on raw link speed for large Linux workspaces and is the default recommendation for teams starting out. Wild is the newer contender posting even faster numbers on supported targets, worth an experiment where nightly-level tooling is acceptable. LLVM lld is the steady choice where LLVM toolchains already dominate, including setups sharing flags with clang. Any of the three beats the default GNU linker by a wide margin; the wrong choice is staying on the slow default out of inertia.

Gate every linker change on the existing suite passing unchanged. Linkers affect layout, not semantics, but layout-adjacent tooling such as profilers, debuginfo consumers, and sanitizers can notice the swap. Run the full test suite plus one sanitizer or profiling smoke pass where the team relies on them before promoting the config from experiment to default. Speed the team trusts beats speed the team fears, and the gate is what earns the trust.

# Measure first, then experiment scoped to one shell.
time cargo build
sudo apt update && sudo apt install -y mold
mold --version

# Scoped experiment: fast linker for this build only.
RUSTFLAGS="-C link-arg=-fuse-ld=mold" time cargo build

# Promote to the workspace once numbers justify it (.cargo/config.toml):
# [target.x86_64-unknown-linux-gnu]
# linker = "clang"
# rustflags = ["-C", "link-arg=-fuse-ld=mold"]

Stale artifacts and lockfile conflicts: cargo clean without regrets

Build-script caching turns yesterday's environment into today's failure. When a -sys probe ran against missing headers, cargo cached the verdict, and installing the headers afterward doesn't automatically invalidate it — the identical error replays and convinces teams the fix didn't work. The surgical answer is cargo clean -p with the failing -sys crate name, which forces just that probe to rerun while leaving the rest of the cache warm. Reserve full cargo clean for cases where the target directory itself is suspect, such as a corrupted artifact after a killed build or a toolchain swap. Full cleans on 300-crate workspaces cost tens of minutes; per-package cleans cost seconds and fix the same probe problem.

The target directory deserves the same discipline as any cache. Stale fingerprints after restoring target/ from CI cache across different images produce link errors that reference libraries the current image never had. Scope cache keys by image digest, Rust version, and target triple so a restored cache always matches its environment. When suspicion falls on the cache, one full clean rebuild that goes green is cheaper than three days of theorizing — but make it a deliberate experiment with the result recorded, not a ritual performed before every build.

Lockfile conflicts are the merge-time cousin of stale state. After a branch merge, Cargo.lock can carry duplicate entries or mixed versions that resolve to a dependency set nobody tested, and the link step is where incompatible native versions surface. Resolve with cargo update -p naming the single crate, review the lockfile diff the way you'd review code, and commit the result. Deleting Cargo.lock is the tempting reset that unlocks every transitive dependency at once — the link error clears and three new version surprises arrive. Binaries should commit the lockfile; libraries may leave it to consumers, but either way the file is the reproducibility contract.

Checksum and yank surprises round out the category. A yanked crate version pinned in the lockfile blocks fresh checkouts while old target dirs keep building, which looks exactly like a machine-specific link failure until someone clones fresh. cargo update -p past the yank, or cargo fetch on a clean runner, exposes it in seconds. Keep one clean-runner CI job that builds without cache weekly: it catches yanked, stale, and drifted inputs that warm caches hide, and it pays for itself the first time it fires before a release instead of during one.

Suspect the target directory itself when errors reference libraries no installed package owns. Fingerprints inside target/ record absolute paths from previous environments, and a cache restored across different images replays those paths as fact. A shared CARGO_TARGET_DIR across workspaces multiplies the hazard: one workspace's probe artifacts leak into another's link. When the log's -L flags point at directories that don't exist on this machine, stop debugging packages and clean the target tree — the map is from a different territory.

Expose version drift with cargo tree before blaming the linker. A lockfile that resolves two major versions of a -sys crate, or a native dependency yanked from the registry after your last fetch, looks exactly like a machine-specific failure until a fresh clone or an uncached job reveals it. Run cargo tree for the suspect crate, compare against the last green build's lockfile, and the surprise upgrade or yank shows itself in seconds. Fresh eyes beat stale caches for this class every time.

Make clean rebuilds a recorded experiment, not a superstition. Note the hypothesis — stale probe, corrupt artifact, drifted lockfile — then run the clean build and record whether it confirmed the theory. Teams that log outcomes learn which failures are cache-shaped and which are real; teams that clean reflexively learn nothing and pay full rebuild prices forever. The experiment record is what turns cargo clean from a ritual into a diagnostic instrument. Keep it beside the runbook.

# Surgical: rerun one failed probe, keep the rest of the cache warm.
cargo clean -p openssl-sys
cargo build -vv 2>&1 | tee /tmp/rebuild.log

# Lockfile discipline: update one crate, review the diff, commit it.
# cargo update -p openssl-sys
# git diff Cargo.lock | head -40

# Full reset only when the target dir itself is suspect:
# cargo clean && cargo build 2>&1 | tail -5

# Inspect what the lockfile actually resolved:
# grep -A3 'name = "openssl-sys"' Cargo.lock

Case study: the classic openssl-sys failure, start to finish

No crate teaches this error like openssl-sys, so walk one failure end to end and keep the pattern for everything after. The setup is ordinary: a service depends on reqwest with default TLS, which pulls openssl-sys, whose build script probes for system OpenSSL. On a fresh slim container the build dies with a probe complaint plus the familiar cc failed summary, and buried between them sit cannot find -lssl lines. The developer's code is a ten-line HTTP fetch that compiled fine — every frame of the failure belongs to the environment, which is exactly why newcomers find it so disorienting.

Diagnosis follows the layered read with no shortcuts. First capture cargo build -vv into a file and confirm rustc compiled the workspace crates before the link — that single check rules out code bugs permanently. Then grep the log for the probe section: the build script reports what it searched and where, and the missing piece is usually headers absent while the runtime library is present. Confirm with the package manager — dpkg -l showing libssl3 installed but no libssl-dev is the canonical split — and reproduce the probe standalone with pkg-config --libs --cflags openssl failing while the .so file sits on disk. Four commands, one consistent story, zero guessing.

Three fixes compete and the environment picks the winner. Installing libssl-dev is the standard repair on servers that already track system OpenSSL: one package, probe rerun, done. Pointing OPENSSL_DIR at a custom prefix suits machines where OpenSSL lives outside system paths and moving it isn't an option. Enabling the vendored feature compiles OpenSSL from source and suits hermetic CI or minimal images where system drift is the enemy — at the price of several extra minutes per clean build. A fourth option changes the game instead: switching reqwest to rustls-tls drops C TLS from the graph entirely, which many teams adopt deliberately after their second OpenSSL incident rather than their fifth.

Verification closes the loop the way it opened: rerun the probe standalone until it prints flags, clean the -sys crate, rebuild with -vv, and watch the fresh probe succeed. Record the numbers that matter — clean build time before and after, incremental time, binary size — because the next debate about vendored versus system will be settled by data, not memory. One team logged 8 min 12 s vendored clean builds against 2 min 5 s system-linked ones on identical runners, which justified system OpenSSL on servers and vendored only in hermetic CI. That's the case study earning its keep: not just a fix, but the measurements that make the next decision obvious.

Keep this pattern as your template for pq-sys, libsqlite3-sys, and every -sys crate to come. The names change — libpq-dev instead of libssl-dev, PQ_LIB_DIR instead of OPENSSL_DIR — but the five-step rhythm never does: capture, confirm compilation, reproduce the probe, apply the environment-appropriate fix, verify the fresh probe. Engineers who internalize the openssl-sys walkthrough triage the next -sys failure in minutes, because they've already debugged it once under a different name.

The pq-sys mirror case proves the template travels. Same symptom shape — undefined references, this time PQ-prefixed — same layered read, same three fixes with different names: libpq-dev instead of libssl-dev, PQ_LIB_DIR instead of OPENSSL_DIR, and the corresponding build features. Engineers who run the openssl-sys rhythm here finish in minutes because the steps are muscle memory. Collect these mirrors in the team runbook: each new -sys entry is five lines naming the package, the variables, and the verify command, and the collection compounds into coverage of the whole native surface.

Close the case study with the measurement habit that prevents repeats. The team kept a small table — clean seconds, incremental seconds, binary megabytes — for system-linked, vendored, and rustls builds on identical runners. The next TLS debate opened that table instead of opinions, and the decision took ten minutes. Evidence recorded once decides many arguments later.

# End-to-end openssl-sys triage on the failing machine.
cargo build -vv 2>&1 | tee /tmp/build.log
pkg-config --libs --cflags openssl || echo "PROBE FAILED"
dpkg -l | grep -E 'libssl|openssl' || rpm -qa | grep -i openssl

# Fix layer 1: system headers, then rerun the probe.
sudo apt update && sudo apt install -y libssl-dev pkg-config
cargo clean -p openssl-sys && cargo build 2>&1 | tail -3

# Fix layer 2 (alternative): vendored OpenSSL in Cargo.toml:
# openssl = { version = "0.10", features = ["vendored"] }
# Fix layer 3 (alternative): pure-Rust TLS, no C OpenSSL in the graph:
# reqwest = { version = "0.12", default-features = false, features = ["rustls-tls"] }

Production hardening: Docker builders, CI guards, and the runbook

Prevention beats triage, and the builder pattern prevents this entire failure class for the price of one Dockerfile stanza. Split the image into a builder stage carrying the full toolchain — gcc, pkg-config, and every -dev package the workspace needs — and a lean runtime stage carrying only shared libraries the binary loads. The builder is where cc lives; the runtime stays small and patchable. Pin both stages to digest-pinned base images so a silent upstream layer removal can't reintroduce the slim-image incident. Check a SYSTEM_DEPS manifest into the repo beside Cargo.toml listing every apt package, and have the Dockerfile install from it — image bumps then diff against a visible list instead of tribal memory.

Guard the pipeline so the next failure announces itself in seconds. A CI step running cc --version, pkg-config --version, and the bare pkg-config probe for each -sys library belongs before cargo build in every job, including cross and musl variants with their own linkers asserted. Follow it with a lockfile check that fails on merge markers and unexpected duplicate entries. Cache target/ keyed by image digest, Rust toolchain version, and target triple so restored artifacts always match their environment. These guards cost under 30 seconds per run and convert 9-minute cryptic failures into 5-second obvious ones — the highest-leverage CI change most Rust teams never make.

Keep the 15-minute runbook on one page where on-call can find it at night. Minute zero to two: capture cargo build -vv into a file and confirm rustc compiled before the link. Minutes two to five: grep the first undefined reference or cannot find line and name the layer. Minutes five to ten: apply the layer fix — toolchain install, -dev package, PKG_CONFIG_PATH export, or linker pin. Minutes ten to fifteen: clean the -sys crate, rebuild, verify the fresh probe and the artifact with file. If the runbook step for the layer is missing, that's the documentation gap to close in the postmortem, not during the incident.

Assign ownership so the environment stays fixed. One team owns the builder image and the SYSTEM_DEPS list, one rotation owns the CI guards, and every service owner records its -sys dependencies in its own README. Review lockfile diffs like code, rebuild from scratch weekly to catch drift warm caches hide, and test musl and cross artifacts with file before deploy. Link failures are environment bugs, and environments with owners stop drifting — which is why teams with this hardening in place describe cc failures in the past tense.

Write the SYSTEM_DEPS manifest as data, not prose, so tooling can consume it. One package per line with a comment naming the -sys crate that needs it turns the file into both documentation and install script input. The Dockerfile reads it, the onboarding script reads it, and the CI guard diffs installed packages against it. When a developer adds a dependency on a new native library, the review checklist asks for the manifest line beside the Cargo.toml line — two files changed together, zero surprise build failures later.

Shape the guard job to fail loud and early on every target the fleet builds. Host jobs assert cc, pkg-config, and each library probe; cross jobs assert their cross linker binary and the config pin; musl jobs assert musl-gcc and the static expectation. Emit the tool versions into the job log so postmortems compare failing and passing runs directly. Under 30 seconds of guards per run buys the team its evenings back, which is a trade any incident review would sign twice.

Run the postmortem on the environment, not the person, after every escape. Each incident that reaches the runbook stage should add one guard, one manifest line, or one runbook row — never just a story. Environments improve one checked-in line at a time, and the teams that treat every escape as a gap to close watch the incident class shrink quarter over quarter until the guide you are reading becomes history. Start tonight with the guard job.

# Builder carries the toolchain; runtime stays lean. Pin digests in real use.
FROM rust:1.76 AS builder
RUN apt-get update && apt-get install -y \
    gcc pkg-config libssl-dev \
    && rm -rf /var/lib/apt/lists/*
RUN cc --version && pkg-config --libs openssl
WORKDIR /app
COPY Cargo.toml Cargo.lock ./
COPY src ./src
RUN cargo build --release

FROM debian:bookworm-slim AS runtime
RUN apt-get update && apt-get install -y libssl3 ca-certificates \
    && rm -rf /var/lib/apt/lists/*
COPY --from=builder /app/target/release/myapp /usr/local/bin/myapp
CMD ["myapp"]