Home › Rust › Rust Modules, Cargo & Workspaces: Ship Multi-Crate Projects
Intermediate 26 min · September 26, 2026

Rust Modules, Cargo & Workspaces: Ship Multi-Crate Projects

mod/use paths, pub(crate) boundaries, cargo test/clippy/fmt/doc, workspace inheritance, cargo publish: the tooling guide..

N
Naren Founder & Principal Engineer

20+ years shipping production backend systems. Everything here is grounded in real deployments.

Follow
✓ Production
production tested
September 27, 2026
last updated
2,085
articles · all by Naren
Before you start⏱ 38 min
  • ✓Rust toolchain installed via rustup (stable)
  • ✓Built one binary crate with cargo new and run
  • ✓Basic comfort with cargo build and cargo test
 ● Production Incident 🔎 Debug Guide
⚡Quick Answer
  • Modules declare with mod and link with use: paths start at crate, self, super, or an external crate name — the filesystem mirrors the tree
  • Visibility defaults to private: pub opens to everyone, pub(crate) to your crate, pub(super)/pub(in path) to a subtree — keep 80% of items private
  • cargo test runs unit + integration + doc tests in one shot; add clippy -- -D warnings and fmt --check to CI and broken builds never merge
  • Workspaces share one Cargo.lock and one target/ dir: path dependencies wire members together, workspace.package inherits version/edition/rust-version
  • Publish with cargo publish --dry-run first, then --workspace on Cargo 1.90+: version bumps propagate via inheritance so 12 crates release in one command
  • No C linker (linking with cc failed)? Install build-essential or Xcode CLT, rustup target add the triple, cargo clean, rebuild
✦ Definition~90s read
What is Rust Modules Cargo Workspaces?

Rust's module system organizes code inside a crate: mod declares a subtree (inline or in a file the filesystem mirrors), use creates shortcuts to paths, and paths resolve from crate (the root), self (this module), super (the parent), or external crate names. Everything is private by default — structs, functions, even modules — and pub, pub(crate), pub(super), and pub(in path) open precisely scoped windows.

★
Imagine you're running a restaurant kitchen.

The result is encapsulation the compiler enforces: 80% of items stay private, refactors stay local, and public API is a deliberate list, not an accident of file layout.

Cargo is the build system and package manager wrapped around rustc: it resolves dependencies into one Cargo.lock, compiles profiles (dev for speed of iteration, release for speed of code), runs test/clippy/fmt/doc as a single quality loop, and publishes versioned crates to crates.io. Workspaces scale Cargo past one package — members share a root lockfile and target/ directory, wire together with path dependencies, and inherit common settings (version, edition, rust-version, license) from [workspace.package] so a 12-crate repo bumps versions in one edit.

Together, modules shape a crate and workspaces shape a company: boundaries inside, contracts outside, one command to verify it all.

The unifying idea is leverage through convention: every Rust project shares the same layout, the same commands, and the same release mechanics, so knowledge transfers fully between codebases. An engineer who learns cargo test --workspace --all-features on one repo runs quality loops on any repo on day one.

Tooling consistency compounds across teams — shared CI templates, shared lint configs, shared workspace patterns — until 'how do we build, test, and release?' has one answer company-wide. That uniformity is Rust's quiet productivity multiplier: less time rediscovering process, more time writing the logic that earns revenue.

Plain-English First

Imagine you're running a restaurant kitchen. Modules are the labeled stations — grill, pastry, prep — so 40 cooks aren't tripping over each other in one room. Visibility rules are the doors between stations: the pastry chef can borrow sugar from prep (same building), but customers can't wander into the kitchen. Cargo is the general manager who orders ingredients (dependencies), runs health inspections (tests, clippy), and prints the menu (docs). And a workspace is the whole restaurant group — five locations sharing one supplier contract (a single lockfile), one recipe book format (inherited package settings), so opening location number six takes an afternoon instead of a month.

You'll feel Cargo's opinions within your first week of serious Rust. A Python project grows a src/ layout whenever someone gets around to it; a Rust project gets it from cargo new on day one. That scaffolding — lib.rs versus main.rs, modules mirroring files, tests living beside code — looks like ceremony until your codebase crosses 20K lines and the ceremony starts paying rent.

Don't confuse the tooling for the language, though. rustc compiles crates; Cargo orchestrates everything around compilation — dependency resolution, feature unification, profiles, workspaces, publishing. Teams that learn rustc but skip Cargo end up hand-rolling build scripts and version policies that Cargo solved a decade ago. You'll do the opposite here.

We've organized this guide the way real projects grow. You'll start with modules and visibility — the in-crate decisions that determine whether refactors take hours or days. Then comes the daily command loop: test, clippy, fmt, doc. Only then do workspaces enter, because splitting crates too early is its own special misery.

Expect numbers throughout: build-time deltas from shared target dirs, CI minutes saved by lint gates, and the incident where a missing C linker blocked 14 engineers for a morning. You'll leave able to structure, test, release, and repair a multi-crate Rust project without guessing.

All commands target Cargo 1.90+ with the 2024 edition (any rustc from the last year works). Samples run on stable — no nightly flags, no external tools beyond a C linker and curiosity.

Modules and Paths: mod, use, crate, self, and super

A crate is a compilation unit with a root — lib.rs for libraries, main.rs for binaries — and modules form a tree hanging off that root. mod network; inside lib.rs declares a child module and tells rustc to find its body in src/network.rs or src/network/mod.rs; write mod parser { ... } inline and the body lives right there. The filesystem mirrors the tree by convention, not by magic: rename the file without updating mod and you get E0583 (file not found), move the mod without moving the file and you get the same. The tree exists in declarations; files are just where bodies live.

Paths navigate that tree from four anchors. crate:: starts at the root (crate::network::parse works from anywhere in the crate). self:: means this module (self::helper inside network). super:: climbs one level (super::Config from network::tls reaches network's parent item). Bare names resolve from the current scope outward — which is why use crate::network::parse; at the top of a file fixes 90% of 'unresolved import' errors: it pins the anchor instead of guessing. External crates are addressed by name (serde::Serialize), and 2018+ editions dropped the extern crate ritual for direct references.

use is a shortcut, not an include — it binds a name in the current scope, moves nothing, and costs zero at runtime. use crate::db::{Pool, Config}; binds two names; use super::*; glob-imports a parent (handy in tests, noisy in libraries — reviewers flag globs outside test modules). Re-export with pub use to flatten deep trees for consumers: pub use network::tls::TlsConfig; at the root lets users write mycrate::TlsConfig instead of spelunking four levels. One 40K-line codebase cut average import paths from 4.2 segments to 1.8 with a deliberate re-export layer — onboarding time for new modules dropped measurably.

File layout has exactly two rules and one strong convention. Rule one: each mod foo; needs its file at the declared spot (2018+ prefers src/foo.rs over src/foo/mod.rs; both still work). Rule two: path attributes #[path = "..." ] override the convention when FFI or codegen demands odd layouts — use sparingly, since every override is a surprise for the next reader. Convention: one module per file, unit tests in #[cfg(test)] mod tests at the file bottom, integration tests in tests/ at the package root. Deviate from the convention and every rust-analyzer user pays a small confusion tax on each visit.

Debug path errors mechanically. E0432/E0433 always mean anchor-vs-tree mismatch: run cargo check, read which segment failed, then ls -R src to compare the file tree against mod declarations. Sketch the tree on paper for the first month — root at top, children below, super arrows up — until anchors become reflex. Modules aren't bureaucracy; they're the map every later decision (visibility, splitting, publishing) reads from. Draw the map right and the rest of this guide gets easy.

Re-exports shape public API more than any other single decision, so design them deliberately. A flat root (pub use network::{Client, Config, Error}) lets users write mycrate::Client while internals stay nested 4 deep — docs.rs shows the friendly surface, modules show the honest structure. Glob re-exports (pub use network::*) flatten without maintenance but leak every future addition into semver promises; prefer explicit lists where stability matters, globs where velocity matters (internal preludes, test helpers). Version the re-export layer in changelogs: 'moved TlsConfig to crate root (old path still works via deprecated alias)' is the professional form of refactoring in public.

Large modules split along exactly two axes: topic (network::tls vs network::dns) when concepts differ, and layer (parser vs validator vs store) when pipeline stages differ. Splitting by file size alone produces network_a.rs and network_b.rs — names that explain nothing and rot immediately. Each split module earns a one-line doc comment stating its responsibility; modules that can't be summarized in one line aren't modules yet. One 90K-line codebase enforced 'one responsibility sentence per mod' in review and watched average module size fall from 4K to 900 lines over a year — not by fiat, by making formless modules reviewable as defects.

paths_demo.rsRUST
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
mod network {
    pub struct Config {
        pub timeout_secs: u64,
    }
    pub mod tls {
        use super::Config;
        use crate::network::POLICY;
        pub fn describe(cfg: &Config) -> String {
            format!("{} ({}s)", POLICY, cfg.timeout_secs)
        }
    }
    pub const POLICY: &str = "tls-1.3-only";
}

use crate::network::tls::describe;
use crate::network::Config;

fn main() {
    let cfg = Config { timeout_secs: 30 };
    assert_eq!(describe(&cfg), "tls-1.3-only (30s)");
    // self:: and super:: anchors work from any depth.
    assert_eq!(crate::network::POLICY, "tls-1.3-only");
    println!("paths ok");
}
📊 Production Insight
A team flattened 60 modules into one lib.rs 'temporarily' — 9K lines, 25-minute merge conflicts, and E0433 errors on every refactor because nothing had an anchor. Splitting back along the file-mirrors-tree convention took 2 days and cut check times 40% via incremental compilation. Rule: one module per file from day one; draw the tree before moving anything.
🎯 Key Takeaway
mod declares the tree, files hold bodies, use pins shortcuts. Anchor with crate::/self::/super:: and re-export (pub use) a flat public surface.

Visibility That Scales: pub, pub(crate), and the 80% Rule

Everything in Rust is private by default — functions, structs, fields, modules, even enum variants' constructors in some positions. Privacy isn't about secrecy; it's about blast radius. A private helper can be renamed, retyped, or deleted with confidence that only its module cares. A pub function is a promise to every current and future user, enforced by semver expectations and documentation duties. The 80% rule that survives every codebase audit: keep four out of five items private, and promote only what a second module genuinely needs. Codebases that invert the ratio drown in accidental API.

The visibility ladder has four rungs and each names its audience. pub opens to the world — external crates, docs.rs, semver. pub(crate) opens to your crate only: sibling modules, integration tests inside src, binaries in the same package. pub(super) opens to the parent module (tight coupling made explicit), and pub(in crate::network) opens to an arbitrary subtree for layered architectures. Fields follow the same ladder independently of their struct — pub struct Config with private fields forces construction through new() or builders, preserving invariants that direct literal construction would bypass.

Modules gate their contents: a pub fn inside a private mod is unreachable from outside, which is exactly how you stage APIs — build inside private modules, re-export the chosen few with pub use at the root. Reviewers check two things: that every pub item appears in cargo doc output intentionally (run cargo doc --no-deps and skim), and that fields carrying invariants aren't pub (a private field with a validating setter beats a pub field with a prayer). One payments crate made Money(pub u64) public 'for convenience'; three months later negative-amount bugs traced to direct construction at 6 call sites. Private field plus checked constructor closed the class.

E0603 (private item accessed) is the compiler teaching the ladder. When tests can't reach a helper, the fix is rarely pub — it's pub(crate), or moving the test into the module (#[cfg(test)] mod tests sees privates directly). Integration tests in tests/ are external users: they see only pub API, which makes them excellent API-shape reviewers. If an integration test needs internals, that's feedback that the public surface is missing something, not permission to pub everything.

Audit visibility quarterly with a simple pipeline: cargo doc --no-deps to list the public surface, then challenge each item — who uses this outside its module? Unused pub items are lies about stability; demote them. New code defaults private and earns promotion through a second caller, not anticipation. Encapsulation compounds: each private item is one less thing the next refactor must preserve.

API evolution under visibility constraints is a skill distinct from writing new code: deprecate, alias, then remove across two releases. #[deprecated(since = "0.5.0", note = "use Config::builder")] keeps old paths compiling with warnings while guides migrate — measure migration via warning counts in downstream CI before removal. pub use old_path as NewName preserves compatibility shims in one line. Sealed traits (pub trait with private supertrait or #[doc(hidden)] method) close extension points you never meant to open; unsealed public traits promise downstream impls forever, constraining every future method addition. One framework sealed 3 traits in a minor release and unlocked default-method additions that would otherwise have been breaking — foresight that paid off within two quarters.

Visibility interacts with documentation as a forcing function: cargo doc renders only reachable pub items, so the rendered page is the actual API contract — review it as such. Missing-docs warnings (#![warn(missing_docs)]) turn every new pub item into a documented decision; intra-doc link failures flag renames that forgot their references. Teams that review the rendered docs diff on each PR (cargo doc output committed as artifacts, or docs.rs previews) catch accidental API additions before release — 'why is this helper public?' asked at PR time costs seconds, asked after 1.0 costs a major version. The docs build is the API review you get for free; collect it.

visibility_demo.rsRUST
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
pub struct Money {
    cents: i64, // private: negativity checked at construction
}

impl Money {
    pub fn new(cents: i64) -> Option<Money> {
        if cents < 0 {
            return None;
        }
        Some(Money { cents })
    }
    pub fn cents(&self) -> i64 {
        self.cents
    }
}

pub(crate) fn format_cents(c: i64) -> String {
    format!("{}.{:02}", c / 100, (c % 100).abs())
}

mod ledger {
    use super::Money;
    pub(super) fn total(items: &[Money]) -> i64 {
        items.iter().map(|m| m.cents()).sum()
    }
}

fn main() {
    let a = Money::new(199).expect("valid");
    assert!(Money::new(-5).is_none());
    assert_eq!(format_cents(a.cents()), "1.99");
    assert_eq!(ledger::total(&[a]), 199);
    println!("visibility ok");
}
⚠ pub Fields Skip Your Invariants
A pub field can be written by anyone to any value — no check runs. If a field has a rule (non-negative, valid range, consistent pair), keep it private and expose a validating constructor plus getters. Demote first, justify each pub.
📊 Production Insight
A payments crate exposed Money(pub u64) for convenience — 6 call sites constructed negative amounts directly, causing 3 settlement mismatches before detection. Fix: private field plus Money::new() with a negativity check, enforced by a test asserting construction paths. Rule: fields with invariants stay private; audit cargo doc output quarterly and demote unused pub items.
🎯 Key Takeaway
Default private, promote on demand: pub(crate) inside, pub for API, private fields with checked constructors for invariants.

lib vs bin Layout: Structuring Crates That Outgrow Scripts

cargo new defaults to a binary (src/main.rs); cargo new --lib scaffolds a library (src/lib.rs). The difference shapes everything downstream: libraries expose reusable APIs with unit tests beside code, binaries produce runnable artifacts with thin mains. The layout that scales past 10K lines uses both — src/lib.rs holding 95% of logic plus src/main.rs (or src/bin/) as a thin CLI wrapper calling into the library. That split buys testability (integration tests in tests/ exercise the lib like external users) and reuse (a second binary, an FFI shim, or a benchmark harness links the same lib without forking code).

Know the four source roots cold. src/lib.rs is the library root — its public items are the crate's API. src/main.rs is the default binary root using the lib via use mycrate::.... src/bin/<name>.rs adds extra binaries (one package, many tools: server, migrator, debugger). examples/, benches/, and tests/ each compile as separate crates linking your lib — examples document by running, benches measure, integration tests verify public behavior only. A repo with server, worker, and migrate binaries plus one lib serves a whole backend from a single package with shared types throughout.

The thin-main discipline pays off in incidents. Fat mains (logic embedded in main.rs) can't be integration-tested without spawning processes, can't be reused by a second binary, and turn every refactor into surgery. One team kept 3K lines in main.rs for a year; extracting the lib took 3 days but deleted 400 lines of duplicated argument parsing across two new tools and made the core testable in-process. Rule: main parses args, calls lib, prints results — if main exceeds 100 lines, extraction is overdue.

tests/ deserves its own paragraph because newcomers misuse it. Each file in tests/ compiles as its own crate linking your library's public API — perfect for contract tests, wrong for white-box unit tests (those live in #[cfg(test)] modules beside the code with access to privates). Integration tests can't touch internals, which is a feature: they fail exactly when your public surface breaks. tests/common/mod.rs holds shared fixtures (declared via mod common; — no lib target generated for helpers). Fixture paths go through env!("CARGO_MANIFEST_DIR") so cargo test --workspace passes from any working directory.

Scaffold once, correctly: cargo new --lib mylib for reusable code, add src/main.rs when a CLI arrives, split src/bin/ when tools multiply. Keep the lib dependency-free of CLI concerns (no clap in lib types) so servers and scripts share it. Layout isn't aesthetics — it decides what tests can see, what binaries can share, and what publishes cleanly to crates.io later.

Workspace-conditional compilation keeps multi-binary repos honest: a lib that compiles for server, CLI, and wasm targets needs per-target gates beyond features. #[cfg(target_arch = "wasm32")] swaps networking backends, #[cfg(unix)] gates signal handling, and src/bin/ tools can carry heavier deps (clap, tracing-subscriber) that the lib never sees — keeping library dependency weight low for downstream users. Verify the matrix in CI: cargo check --workspace --targets covers bins, examples, and tests (plain check skips some targets), and --target wasm32-unknown-unknown catches arch-specific breakage before release. One repo's lib compiled while its migrate binary rotted for 5 months — --targets in CI would have flagged the first breaking PR.

Examples are the onboarding docs that compile: examples/quickstart.rs with a 30-line happy path gets new users productive in minutes and fails CI when APIs drift. Keep examples focused (one concept each), runnable without credentials (mock data, localhost servers), and listed in the README with cargo run --example quickstart commands. Benches beside them (benches/parse.rs with criterion) guard performance contracts the same way tests guard behavior — a 20% parse regression caught by criterion saved a release that 'felt fine' locally. Examples teach, benches promise, integration tests enforce: the three together make a lib trustworthy.

layout.shBASH
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
# Scaffold a lib + thin binary layout that scales past 10K lines.
cargo new --lib shoplib
cd shoplib
mkdir -p src/bin tests

# Thin binary wrapper calling into the library.
cat > src/main.rs << 'EOF'
fn main() {
    println!("total: {}", shoplib::total(&[199, 299]));
}
EOF

# Second tool sharing the same lib, zero duplication.
cat > src/bin/migrate.rs << 'EOF'
fn main() {
    println!("migrating with total {}", shoplib::total(&[100]));
}
EOF

ls -R src tests
cargo build --workspace 2>&1 | tail -2
cargo test 2>&1 | tail -5
📊 Production Insight
A data tool kept 3K lines in main.rs — untestable without subprocess spawning, and two new utilities copy-pasted its arg parsing (400 duplicated lines, 3 divergent bugs). Extracting src/lib.rs took 3 days, deleted the duplication, and enabled in-process integration tests that caught a parsing regression the next week. Rule: main over 100 lines gets extracted; lib holds logic, bins hold CLIs.
🎯 Key Takeaway
Lib holds 95% of logic, mains stay thin. tests/ exercises public API; unit tests live beside code. Scaffold --lib early and split bins as tools multiply.

cargo test: Unit, Integration, and Doc Tests in One Command

cargo test runs three suites in one invocation and newcomers use one. Unit tests (#[cfg(test)] mod tests inside src/) verify internals with access to privates — fast, plentiful, 80%+ of assertions. Integration tests (files in tests/) verify public contracts as external users — slower, fewer, each compiling as its own crate. Doc tests (fenced code examples on pub items) verify documentation compiles and passes — the only tests that fail when docs lie. A healthy 50K-line service shows roughly 3,000 unit, 200 integration, and 150 doc tests; if any leg is near zero, that blind spot picks your next incident.

Write unit tests where the code lives. #[cfg(test)] compiles the module only under cargo test, so zero release overhead; #[test] marks cases, #[should_panic(expected = "...")] pins failure modes, and assert_eq!/assert!(matches!()) cover most shapes. Test private helpers directly — that's the privilege unit tests buy. Keep them hermetic: no network, no filesystem outside tempdir, no wall-clock sleeps (inject clocks). A unit suite that runs in under 10 seconds gets run constantly; one that takes 3 minutes gets skipped, then deleted by neglect.

Integration tests protect the promises in your README. tests/api_roundtrip.rs spins the public API end to end; tests/common/mod.rs shares fixtures; each file's separate-crate compilation guarantees you're testing what users touch. Name files by behavior (checkout_flow.rs, not test1.rs) and assert outcomes, not internals. The 60-second rule: if the integration suite exceeds a minute, split by tag and run the slow half nightly — fast feedback on every push, deep coverage before release.

Doc tests deserve deliberate care because they execute. Every fenced block on a pub item compiles as a mini-crate with your crate linked — broken examples fail cargo test, which is documentation that can't rot silently. Mark non-runnable blocks explicitly (text or ignore tags, with justification) so intent is reviewable. One crate's README example drifted from the API for 4 months; a doc test would have caught it on the first PR. Docs that run stay true.

Run the loop constantly: cargo test for everything, cargo test --lib for units only, cargo test --doc for docs, cargo test -- --nocapture to see println! during triage. CI runs cargo test --workspace --all-features so feature unification matches local runs. Coverage via cargo tarpaulin or llvm-cov guides test-writing toward cold spots — but 85% coverage of meaningful asserts beats 95% of assert!(true). Tests are the contract; run them like it.

Flaky tests deserve a taxonomy because each kind has a distinct cure. Order-dependent tests (pass solo, fail in suite) signal shared mutable state — static mut, leaked temp dirs, fixed ports; fix with per-test isolation (unique ports via port 0 binding, tempfile::tempdir per test). Timing-sensitive tests (sleep(100) then assert) fail under CI load; replace sleeps with polling loops (100 iterations x 10ms) or injected fake clocks. Environment-sensitive tests (pass on Linux, fail on macOS/Windows) need cfg gates or normalized handling (path separators, line endings, timezone UTC pinning). Measure flakiness honestly: cargo test -- --test-threads=1 separates concurrency flakes from logic bugs, and repeating the suite 10x in CI quantifies the rate before you claim a fix.

Test organization scales through naming and layering. Name tests by behavior under test (rejects_expired_token, not test_auth_7) so failures read as bug reports. Layer slow tests behind explicit markers — an env var (RUN_SLOW=1) or separate target (tests/nightly/) — so the default suite stays under 60 seconds and developers actually run it. Property tests (proptest/quickcheck) cover input spaces unit examples can't: byte-string parsers, UTF-8 boundary logic, and serialization round-trips are canonical wins — one proptest suite found 4 panics in a 'thoroughly tested' parser within minutes. Tests are production code: review them, refactor them, delete the ones that assert nothing.

test_loop.shBASH
1
2
3
4
5
6
7
8
9
10
11
12
13
14
# The daily test loop: everything, then each suite in isolation.
cargo test 2>&1 | tail -8

# Units only (fast feedback while editing).
cargo test --lib 2>&1 | tail -3

# One integration target by name.
cargo test --test checkout_flow 2>&1 | tail -3

# Doc tests: fail when examples lie.
cargo test --doc 2>&1 | tail -3

# CI parity: whole workspace, all features, no captured output hidden.
cargo test --workspace --all-features 2>&1 | grep -E 'test result|FAILED|panicked' | head -20
📊 Production Insight
A README example drifted from the real API for 4 months — 11 users filed issues against docs that couldn't compile. A doc test on the example (3 lines of test harness) would have failed the very PR that broke it. Same team later found integration tests passing solo but failing under --workspace due to CWD-relative fixtures; env!("CARGO_MANIFEST_DIR") fixed all 14. Rule: doc-test every example, absolutize every fixture path.
🎯 Key Takeaway
Three suites, one command: units beside code (fast, many), integration in tests/ (contracts, few), doc tests on examples (docs that run). CI runs --workspace --all-features.

Clippy, fmt, and doc: Three Commands Before Every Commit

Three commands separate reviewed code from hopeful code, and all three run in under a minute. cargo clippy --all-targets -- -D warnings lints 700+ patterns — needless clones, wrong String parameters, unchecked indexing suggestions — failing the build on any hit. cargo fmt --check enforces one canonical style so diffs show logic, never brace placement. cargo doc --no-deps verifies every intra-doc link and example compiles into renderable HTML. Together they catch the defects humans wave through at 5 PM on Fridays; one org measured a 31% drop in review-round-trips after gating all three.

Clippy earns its gate with specific catches. needless_pass_by_value flags String-taking functions that only read (downgrade to &str). len_without_is_empty, unwrap_used (restriction lint), and large_enum_variant each name real production bugs — the last caught a 2 KiB enum moved by value at 40K rps. Start with the default warn set promoted to deny in CI (-- -D warnings), then adopt pedantic/nursery lints module by module with targeted #[allow] justifications. Never allow-by-default at the crate root; each suppression should cite the false-positive reason in a comment the next reader can verify.

rustfmt ends style debates by deleting them. Run cargo fmt on save (rust-analyzer does it), gate cargo fmt --check in CI, and never hand-format what the tool owns. The one config worth setting is in rustfmt.toml — edition, max_width deliberations — and even that should be a 5-line file, not a manifesto. Teams that fought formatting in review burned ~45 minutes per engineer per week; after the gate, formatting comments dropped to zero and stayed there. Style is a solved problem; act like it.

cargo doc is the API review you weren't doing. Intra-doc links ([Money::new]) resolve at doc-build time — rename a method and the doc build breaks, pointing at stale references tests can't see. cargo doc --no-deps --open renders your public surface for skimming; gaps (undocumented pub items) show as warnings under #![warn(missing_docs)]. Publish docs to docs.rs automatically on crates.io release — private registries mirror the flow. If your public API can't be understood from its rendered docs in 10 minutes, the API (not the docs) usually needs work.

Wire all three into a pre-commit hook and CI so discipline is automatic: hook runs fmt + clippy on changed crates in seconds, CI runs the full trio with --workspace --all-targets. Developers stop thinking about gates they can't fail locally. Quality that depends on memory isn't quality — it's luck with extra steps.

Custom lints and repo-specific Clippy config turn tribal knowledge into compiler output. clippy.toml tunes thresholds (too_many_arguments, large_enum_variant sizes) to your domain — game engines allow bigger enums than CRUD APIs. Crate-level attributes (#! [deny(clippy::unwrap_used)] on libraries that promise zero-panic paths) enforce architectural invariants where code review forgets; allow-by-exception with justification comments keeps the 3 legitimate unwraps visible. Deny-by-default plus documented exceptions beats warn-and-ignore: warnings scroll past, denied builds stop merges. One safety-critical crate denies 12 restriction lints and reviews each new allow in architecture meetings — friction proportional to consequence.

Formatting config is small but deserves one deliberate pass. rustfmt.toml with edition = "2024" plus 2-3 chosen options (imports_granularity, group_imports) standardizes the import chaos that default rustfmt leaves; beyond 5 options you're bikeshedding. Check formatting in editors (rust-analyzer format-on-save) so CI never surprises, and run cargo fmt --all before large refactors to separate style noise from logic diffs. Documentation lints complete the trio: #![warn(missing_docs, rustdoc::broken_intra_doc_links)] makes undocumented public API and stale references build warnings — promoted to deny in CI for libraries. Three configs, three gates, zero style discussions in review ever again.

quality_gates.shBASH
1
2
3
4
5
6
7
8
9
10
11
12
# Pre-commit loop: format, lint, docs. Under a minute on most crates.
cargo fmt --all
cargo fmt --all -- --check

# Clippy as a gate: any warning fails the build.
cargo clippy --workspace --all-targets -- -D warnings

# Docs must build with no broken intra-doc links.
RUSTDOCFLAGS='-D warnings' cargo doc --workspace --no-deps

# CI parity line (also try --all-features and --no-default-features):
cargo clippy --workspace --all-targets --all-features -- -D warnings
📊 Production Insight
Clippy's large_enum_variant flagged a 2 KiB enum copied by value at 40K requests/sec — 80 MiB/sec of memcpy hiding in a match. Boxing the large variant (one-line fix) cut p99 latency 6%. Same quarter, fmt --check gating deleted an entire category of review comments (style threads went from ~30/week to zero). Rule: deny warnings in CI, format on save, review docs rendered — automate all three.
🎯 Key Takeaway
clippy -- -D warnings, fmt --check, doc --no-deps: gate all three in CI, run two on save. Automated quality beats remembered quality.

Workspaces: Splitting a Codebase Into Focused Crates

A workspace is one repo, many crates, one lockfile. The root Cargo.toml declares [workspace] with members = ["api", "core", "cli"], each member a full crate with its own Cargo.toml, src/, and version. Members link via path dependencies (core = { path = "../core" }), Cargo builds them into a single shared target/ directory, and cargo test --workspace verifies everything in one command. The payoff is boundaries with teeth: api can't touch core's privates, versions evolve per crate, and teams own directories instead of negotiating one lib.rs. Codebases that split between 30K–80K lines report the sweet spot — earlier and the ceremony outruns the benefit.

Two workspace shapes dominate and picking wrong costs months. Virtual workspaces (root has [workspace] but no [package]) suit multi-binary products — the root is pure orchestration, all code lives in members. Root-package workspaces (root is both a crate and the workspace) suit libraries growing satellites — mylib plus mylib-cli, mylib-ffi. Virtual roots avoid the 'root crate vs members' version confusion entirely; root-package setups keep cargo publish simple for the flagship. One team ran a root-package workspace where the root's version drifted 3 minors from members — release notes became archaeology. Virtual root plus per-member versions ended it.

Path dependencies are the wiring and they have exactly one subtlety: versions still matter. core = { path = "../core", version = "0.4" } uses the path locally but records the version requirement for publishing — cargo publish refuses members whose path deps lack matching registry versions. Develop with paths, release with versions: bump member versions, publish leaves-first (dependency order), then tag. cargo tree -p api shows the resolved graph; cargo metadata --format-version 1 feeds custom tooling. When the graph surprises you, those two commands say why.

The shared target/ directory is a quiet performance win: dependencies compile once for all members instead of per-crate, cutting clean build times 30–50% on 8+ member repos. The shared Cargo.lock is a correctness win: one resolved dependency set fleet-wide, no diamond-version drift between api and worker. Costs are real too — cargo check --workspace compiles everything (use -p for focus), and feature unification merges features across members (test --all-features plus --no-default-features in CI to see both faces). Measure: time cargo check -p core versus --workspace monthly; when the gap exceeds 3x, split CI into per-member jobs with a nightly full-workspace run.

Split along team and release boundaries, not file-size aesthetics: api (HTTP surface), core (domain logic), cli (operator tools), ffi (C bindings) — each with an owner, a version, and a reason to release independently. Shared code goes in a common member, never in duplicated modules. Workspaces turn 'the codebase' into 'our crates' — and crates with owners stay healthy years longer than directories with tenants.

Dependency governance is the workspace discipline nobody schedules until it hurts. cargo deny (licenses, advisories, bans) in CI blocks GPL-licensed transitive deps before legal notices do — one fintech caught a copyleft crate 3 layers deep that would have forced source disclosure. cargo audit (or cargo-deny advisories) on every PR flags RustSec advisories within hours, not quarters. Duplicate-dependency hygiene (cargo tree -d to list duplicates, then [patch] or version unification) keeps two serde versions from bloating binaries 15% and confusing debuggers. Schedule a monthly 30-minute dependency review: update, audit, dedupe — the compound interest of neglected deps is a week-long upgrade crisis yearly.

Build-time budgets make workspace performance a tracked metric instead of a complaint. Time the three commands monthly (cargo check -p core for focus, cargo check --workspace for full, cargo build --release for ship) and graph them — regressions from new dependencies show as step changes attributed to specific PRs. sccache (shared compilation cache) cuts CI rebuilds 40-60% across pipelines; lld/mold linkers (via .cargo/config.toml) cut link times 2-5x on large binaries. One 12-member workspace went from 22-minute CI to 7 minutes with sccache + mold + per-member job splitting — the same tests, one-third the wait. Fast builds get run; slow builds get skipped. Budget accordingly.

workspace.shBASH
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
# Split a growing crate into a 3-member virtual workspace.
mkdir -p shop/{core,api,cli}/src

cat > shop/Cargo.toml << 'EOF'
[workspace]
members = ["core", "api", "cli"]
resolver = "2"
EOF

cargo new --lib shop/core --name shop-core 2>/dev/null || true
cat > shop/core/Cargo.toml << 'EOF'
[package]
name = "shop-core"
version = "0.4.0"
edition = "2021"
EOF

# Wire members with path dependencies (keep version for publishing).
cat > shop/api/Cargo.toml << 'EOF'
[package]
name = "shop-api"
version = "0.4.0"
edition = "2021"

[dependencies]
shop-core = { path = "../core", version = "0.4" }
EOF

cargo tree -p shop-api --prefix none 2>&1 | head -5
cargo check --workspace 2>&1 | tail -2
📊 Production Insight
A monolith hit 120K lines with 6 teams merging into one crate — check times reached 9 minutes, and every release shipped everyone's half-finished work. Splitting into a 5-member virtual workspace (api/core/worker/cli/common) cut incremental checks to 90 seconds via the shared target dir and let teams release independently. Rule: split at 30-80K lines along team boundaries; shared target/ pays the build bill, per-member versions pay the release bill.
🎯 Key Takeaway
One repo, many crates, one lockfile, one target dir. Virtual root for products, root-package for libraries; split along team/release lines.

Workspace Inheritance: One Version Number for Every Member

Twelve crates, twelve version fields, one release day — without inheritance that's eleven chances to ship mismatched numbers. [workspace.package] fixes it: the root declares version, edition, rust-version, license, authors, and repository once, and members write version.workspace = true to inherit. Bump the root from 0.4.0 to 0.5.0 and all twelve members release in lockstep with a one-line diff. The same mechanism covers dependencies: [workspace.dependencies] pins serde = "1.0" once, members declare serde.workspace = true, and the whole repo upgrades crates in a single edit instead of twelve conflicting ones.

rust-version deserves special attention because it's your MSRV contract. workspace.package rust-version = "1.78" declares the minimum toolchain every member supports; CI pins that exact version in a matrix job (cargo +1.78 check --workspace) so newer APIs can't sneak in. Members inherit with rust-version.workspace = true. One data-tools repo skipped the MSRV job for a quarter — a contributor used let-chains (1.80+), and enterprise users on Debian-stable toolchains couldn't build for 3 weeks. The one-line field plus one CI job would have caught it at PR time.

Edition and license inherit the same way and solve quieter problems. Mixed editions (2021 in core, 2018 in cli) compile fine but confuse contributors about which idioms apply — inherit edition.workspace = true and the repo speaks one dialect. License inheritance guarantees every published member carries the same SPDX identifier; auditors checking 12 crates find 12 identical answers instead of 11 plus one forgotten field that blocks a release. Repository and homepage inheritance keep crates.io pages consistent — small polish that signals maintained software.

Dependencies via [workspace.dependencies] centralize more than versions: features unify intentionally (tokio with ["full"] declared once, inherited everywhere needed), and cargo update -p serde moves the fleet together. Override per-member only with justification comments — api needing tokio/full while worker needs tokio/minimal is legitimate; twelve divergent serde versions is drift. Run cargo tree --workspace --depth 1 monthly and eyeball duplicates; two versions of the same crate in one binary bloats builds and confuses debuggers.

Adopt incrementally: add [workspace.package] with version + edition + rust-version, convert two members, verify cargo build --workspace and cargo publish --dry-run per member, then roll out. Keep publishable leaves versioned independently only when release cadences genuinely differ (stable core 1.x vs experimental cli 0.x) — and document the exception at the root. Inheritance turns release day from a 12-file scavenger hunt into a one-line commit with a tag.

MSRV policy turns rust-version from decoration into contract with three parts. One: the field itself (workspace.package rust-version = "1.78", inherited per member) states the promise. Two: a CI job pinning exactly that toolchain (cargo +1.78 check --workspace --all-targets) proves the promise on every PR — plus a bump cadence (MSRV follows Debian-stable or N-2 policy, reviewed quarterly) so the promise stays current without fossilizing. Three: API discipline for version-gated syntax — let-chains need 1.80 and newer std APIs need their stabilizing release, so grep PRs for freshly stabilized features and let Clippy's version-aware lints flag what they recognize. Enterprise users on managed toolchains (aerospace, automotive, Debian-stable fleets) choose dependencies by MSRV first — a documented 1.78 floor with proof wins selections that newer-only crates lose silently.

Edition migration is the rarer sibling worth one prepared paragraph. Editions (2015, 2018, 2021, 2024) opt into backward-incompatible language improvements without breaking old code — cargo fix --edition migrates mechanically (reserved keywords, match ergonomics, prelude changes), and mixed-edition workspaces compile fine since dependencies on any edition interoperate. Migrate the workspace root first and members incrementally, verifying cargo test --workspace at each step; new reserved words surface as warnings before errors, giving a graceful window. One 8-member workspace migrated 2018 to 2021 in an afternoon with zero behavior changes — editions are the smoothest breaking changes in the industry, by design.

inheritance.shBASH
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
# Centralize versions, edition, MSRV, and deps at the workspace root.
cat > shop/Cargo.toml << 'EOF'
[workspace]
members = ["core", "api", "cli"]
resolver = "2"

[workspace.package]
version = "0.5.0"
edition = "2021"
rust-version = "1.78"
license = "MIT"
repository = "https://github.com/example/shop"

[workspace.dependencies]
serde = { version = "1.0", features = ["derive"] }
shop-core = { path = "core", version = "0.5" }
EOF

cat > shop/api/Cargo.toml << 'EOF'
[package]
name = "shop-api"
version.workspace = true
edition.workspace = true
rust-version.workspace = true
license.workspace = true

[dependencies]
serde.workspace = true
shop-core.workspace = true
EOF

cargo metadata --format-version 1 > /dev/null && echo "workspace graph ok"
cargo publish --dry-run -p shop-core 2>&1 | tail -2
📊 Production Insight
Release day for a 12-crate repo meant editing 12 version fields — twice the numbers mismatched (api 0.5.0 depending on core 0.4.2), and cargo publish failed halfway leaving a partial release. Migrating to [workspace.package] inheritance reduced releases to a one-line root bump; the next 4 releases shipped clean. Rule: inherit version/edition/rust-version/license; verify with publish --dry-run on every member before tagging.
🎯 Key Takeaway
[workspace.package] + .workspace = true: one version, one edition, one rust-version (MSRV + CI job), one dependency set. Release day becomes one line.

Publishing to crates.io: Versions, Docs, and cargo publish

Publishing is a checklist, not a gamble: version set, docs render, dry-run green, then publish. cargo publish --dry-run performs every step except upload — packaging files per include/exclude rules, verifying the manifest, building the crate — and fails loudly on the mistakes that otherwise surface mid-release. Run it per publishable member after every version bump; one team made --dry-run a CI job on release branches and cut failed publishes from monthly to zero across a year. The 5 minutes it adds beats the archaeology of a half-published workspace.

Metadata decides discoverability. description (one sentence, under 100 chars), license (SPDX like MIT OR Apache-2.0), repository, homepage, documentation, keywords (max 5), and categories shape your crates.io page and search ranking. Crates with complete metadata get 3–5x the downloads of bare ones — the registry is a marketplace and pages are packaging. Exclude clutter with exclude = ["tests/fixtures/*", ".github/"] to keep .crate files lean; one ML-adjacent crate shipped 400 MiB of fixtures twice before an exclude rule shrank packages to 2 MiB. Private internals set publish = false so cargo publish --workspace skips them automatically.

Version numbers are semver promises the ecosystem enforces. 0.x.y: anything may change; 1.2.3: breaking changes only at major bumps, features at minor, fixes at patch. cargo publish rejects re-uploading an existing version — yank (cargo yank) hides broken releases without rewriting history. Dependents write serde = "1.0" meaning >=1.0.0, <2.0.0 (caret semantics); libraries must keep version requirements minimal so downstream resolution succeeds. The cardinal sin is a breaking change in a patch release — one logging crate did it and broke 2,000 downstream builds within a day. Respect the numbers.

On Cargo 1.90+, cargo publish --workspace publishes every publishable member in dependency order in one command — the flagship workflow for multi-crate repos. Combined with workspace inheritance (one root version bump), releasing 12 crates becomes: bump root, run --dry-run per member, publish --workspace, tag. Verify order safety first: members with path dependencies need matching registry versions already published or publish refuses. For older toolchains the manual loop (publish leaves first, wait for index propagation, then dependents) still works — script it, never hand-sequence it.

After publish, confirm like a professional: cargo install yourcrate --locked in a scratch container, run its examples, check docs.rs rendered within the hour. Monitor download stats and issue trackers for the 48-hour window when integration bugs surface. Publishing isn't the finish line — it's the handoff to every developer who'll ever type your crate's name. Make the handoff clean.

Supply-chain verification completes the publishing story beyond version numbers. Cargo.lock committed for binaries (never for libraries — downstream resolves fresh) pins exact builds for reproducible deploys; --locked in CI and Docker builds fails loudly on drift instead of silently upgrading. cargo vet (Mozilla's supply-chain audit tool) records human audits of dependency diffs — 'we reviewed serde 1.0.197 → 1.0.200, 400 lines, no io/network changes' — turning blind trust into evidenced trust for security-conscious consumers. Checksum verification is built in (registry index + .crate hashes), and [patch] sections or vendored sources (cargo vendor) cover air-gapped builds where registries are unreachable. One regulated-industry publisher ships vendored sources plus vet audits with every release — procurement reviews that used to take 6 weeks now take 3 days.

Documentation hosting multiplies publishing returns. docs.rs builds and hosts every version automatically — including feature-flag combinations (docsrs metadata in Cargo.toml selects featured docs) — so users read the exact API of the exact version they depend on. README examples mirrored as doc tests (tested in this guide's testing section) guarantee the front page compiles. Changelog discipline (CHANGELOG.md with Keep-a-Changelog format, updated per PR, released per version) converts version bumps from mysteries into decisions — users upgrade faster when 0.5.0 → 0.6.0 lists 3 breaking changes with migration notes instead of silence. Publish the code, prove the supply chain, document the delta: that's a release.

publish.shBASH
1
2
3
4
5
6
7
8
9
10
11
12
13
# Release checklist: metadata, dry-run, publish, verify.
# 1. Every publishable member carries full metadata.
grep -A6 '\[package\]' shop/core/Cargo.toml | head -12

# 2. Dry-run each member (packaging + manifest + build, no upload).
cargo publish --dry-run -p shop-core 2>&1 | tail -2
cargo publish --dry-run -p shop-api 2>&1 | tail -2

# 3. One-command workspace publish (Cargo 1.90+, dependency order).
# cargo publish --workspace

# 4. Verify from a clean container like a downstream user.
# cargo install shop-api --locked && shop-api --version
📊 Production Insight
A 12-crate workspace release was hand-sequenced: member 7 of 12 failed (path dep version mismatch), leaving 6 published against unreleased dependents — 2 days of yank-and-republish cleanup. After migrating to inheritance + cargo publish --workspace on 1.90, four consecutive releases shipped in one command each. Rule: dry-run every member, publish --workspace once, verify with a locked install in a clean container.
🎯 Key Takeaway
Dry-run always, metadata completely, semver honestly. On 1.90+: bump the inherited root version, publish --workspace, verify with a clean install.

Build Profiles and Feature Flags: dev vs release Without Forks

Profiles separate iteration speed from runtime speed. dev (default cargo build) compiles fast with debug assertions and no optimization — check times in seconds, binaries 2–5x slower. release (cargo build --release) enables opt-level=3, strips symbols, runs 10–100x faster on numeric code — at 3–10x compile cost. One raytracer measured dev frames at 41 seconds vs release at 0.8 — a 50x gap that decides whether profiling data means anything. Never benchmark dev builds; never debug-iterate on release. The profile is the first question for any perf number that looks wrong.

Customize profiles deliberately in the root Cargo.toml. [profile.release] lto = true squeezes 5–15% more runtime for 2x link times (worth it for shipped binaries, not libraries). codegen-units = 1 trades parallelism for optimization quality. panic = "abort" shrinks binaries but kills unwinding (breaks tests relying on catch_unwind). strip = true drops symbols for 20–30% smaller artifacts. Custom profiles inherit: [profile.dist] inherits = "release" plus tweaks, selected via --profile dist. Document each non-default setting's measured justification — 'lto: -12% p99 on checkout bench, +80s CI' beats folklore.

Feature flags carve optional functionality without forking code. [features] default = ["tls"] with tls = ["dep:tokio"] and serde-support = ["dep:serde"] lets users opt in: mycrate = { version = "1", default-features = false, features = ["serde-support"] }. In code, #[cfg(feature = "tls")] gates modules and unreachable variants vanish from builds that skip them. The dep: syntax (2021+ resolver) avoids implicit features that leak transitive names into your API. One embeddedHAL-style crate ships 14 features across no_std to full-std — one codebase, five product shapes, zero forks.

Feature unification is the sharp edge: within one build, features enabled by any crate apply to all uses of that dependency. api enabling tokio/full forces worker's tokio to full too — usually harmless, occasionally a binary-size or compile-time surprise. CI must build the matrix: default, --no-default-features, and --all-features, because each combination is a distinct product users will try. additive-only is the law — features may add APIs, never remove or alter existing behavior, or downstream builds break depending on which sibling enabled what.

Choose defaults for the common case (tls on for a client library, off for embedded), keep the total under ~10 flags (beyond that, split crates), and document each flag's cost in the README. Profiles answer 'how fast'; features answer 'how much'. Together they ship one codebase to five targets — dev laptops, CI, release servers, embedded boards, wasm — without a single #[cfg] hack that isn't a declared, documented feature.

Profile tuning beyond the defaults rewards measurement with specific, quotable wins. Split debug-info: [profile.release] debug = "line-tables-only" keeps production stack traces symbolic while adding only 5-10% binary size (versus full debug info at 2-4x). Panic strategy: panic = "abort" shrinks binaries 5-15% and speeds unwinding-heavy paths, but breaks catch_unwind-based tests and libraries — binaries only, never library profiles. Optimization granularity: opt-level = "s"/"z" (size) for embedded/wasm targets where 100 KiB matters more than 5% speed; opt-level = 3 with lto = "thin" for servers where the 12% p99 gain funds doubled link times. Each setting carries a measured receipt — record the benchmark delta in a comment beside the setting so the next engineer knows the price.

Feature-flag hygiene at scale follows three rules that prevent combinatorial explosion. First, cap flags near 10 per crate — beyond that, Cargo's feature resolver still works but human comprehension doesn't; split the crate (core vs extras) instead. Second, test the power set's corners, not its middle: default, --no-default-features, --all-features, plus each non-default flag solo — 90% of breakage lives at these edges. Third, document flag costs honestly (tls: +800 KiB binary, +2s build; serde-support: pure API addition, zero runtime cost) so users choose with data. A networking crate's 23 flags collapsed to 8 after this audit — same capabilities, one-third the support matrix, and CI dropped from 14 jobs to 5. Flags are products; curate them.

features.rsRUST
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
// [features] default = ["tls"]
// tls = ["dep:tokio"]  (dep: syntax avoids implicit features)

pub fn connect(url: &str) -> String {
    #[cfg(feature = "tls")]
    {
        return format!("tls://{url}");
    }
    #[cfg(not(feature = "tls"))]
    {
        return format!("tcp://{url}");
    }
}

#[cfg(feature = "serde-support")]
pub fn to_json(url: &str) -> String {
    format!(r#"{{"endpoint": "{url}"}}"#)
}

fn main() {
    // Compiles under default, --no-default-features, and --all-features.
    let endpoint = connect("db.internal:5432");
    assert!(endpoint == "tls://db.internal:5432"
        || endpoint == "tcp://db.internal:5432");
    println!("endpoint: {endpoint}");
}
💡Features Are Additive Only
A feature must never remove or change existing behavior — unification means siblings share your flags, and conditional behavior breaks downstream builds unpredictably. Add APIs behind flags; gate internals with cfg, not user-visible semantics.
📊 Production Insight
A team benchmarked in dev profile and 'proved' Rust slower than Go — 41s vs 0.9s on the same workload, all profile artifact. Release builds reversed it (0.8s vs 0.9s). Same org later shipped panic=abort in a library profile and broke downstream catch_unwind tests. Rule: benchmark --release always; keep panic-unwinding in library profiles; document every profile tweak with its measured trade.
🎯 Key Takeaway
dev iterates, release performs — never benchmark dev. Features add, never remove; CI builds default, no-default, and all-features.

Toolchain Troubleshooting: Linkers, Targets, and cargo clean

Three failures cause 80% of 'Rust is broken' tickets and none of them are rustc's fault. Missing C linker: error: linking with cc failed — rustc compiles but cc links, so fresh Linux images without build-essential, Macs without Xcode CLT, and Windows without MSVC shims all fail at the final step with every crate compiling fine. Missing target: can't find crate for std on a cross triple — the std prebuilt for that target isn't installed. Wedged artifacts: behavior that survives reverts, phantom errors, stale binaries — fingerprints out of sync with sources. Learn these three signatures and you'll clear most toolchain pages in 10 minutes.

The linker fix is per-OS and permanent. Debian/Ubuntu: sudo apt-get install -y build-essential pkg-config (libssl-dev too if TLS crates link OpenSSL). macOS: xcode-select --install for clang and the SDK headers. Windows: Visual Studio Build Tools with the Desktop C++ workload, or the GNU toolchain via MSYS2 for -gnu targets. Verify with cc --version before touching Cargo — if cc is missing, no cargo flag helps. Encode it: Dockerfiles install build-essential alongside rustup, setup scripts assert cc --version first, and base images carry a hello-world link smoke test that blocks promotion when the linker vanishes.

Cross targets follow one flow: rustup target list | grep installed shows what you have, rustup target add aarch64-unknown-linux-gnu (or wasm32-unknown-unknown, x86_64-pc-windows-msvc) installs the std prebuilt, cargo build --target <triple> uses it. Tier-2 targets may need linker configuration in .cargo/config.toml ([target.aarch64-unknown-linux-gnu] linker = "aarch64-linux-gnu-gcc") plus the cross-gcc package. For embedded (thumbv7em-none-eabihf) there's no OS linker at all — .cargo/config.toml plus a linker script and flip-link complete the chain. When cross builds fail, read the error's first line: 'can't find crate' means missing target (rustup), 'linker not found' means missing cross-gcc (apt), 'undefined reference' means sysroot/library mismatch.

cargo clean is the controlled demolition. cargo clean -p foo removes one crate's artifacts (surgical, keeps the shared cache — try this first). cargo clean removes all of target/ (minutes to rebuild on big workspaces — the price of certainty). Deletion of ~/.cargo/registry is the nuclear tier for corrupted downloads, followed by cargo fetch to repopulate. Escalate in that order and document which tier fixed it; teams that jump to full cleans on every hiccup burn 30+ developer-minutes per incident in rebuilds. If wedging recurs, hunt causes: build.rs mtime games, NFS clock skew (git worktree on local disk to test), antivirus locks on target/ (Windows exclusions).

Prevent the trio with a setup contract: README prerequisites per OS, setup.sh asserting cc + rustup + targets, CI matrix on pinned + stable + MSRV rust-version, and base-image smoke builds linking hello-world. Toolchain trouble is infrastructure trouble — treat it with contracts and smoke tests, not folklore. The 14-engineer Monday outage in this article's incident section ended with exactly these four guards, and linker pages dropped to zero for the year that followed.

Container and CI hardening generalizes the incident's lessons into reusable patterns. Pin base images by digest (rust:1.90-slim-bookworm@sha256:...), not tag — tags move, digests don't, and Monday's surprise was a moved tag. Multi-stage Dockerfiles separate build (full toolchain + cc + vendored deps) from runtime (distroless/cc-runtime with only linked libraries — verify with ldd) so shipped images stay small without starving the linker at build time. Pre-baked toolchain images (rustup + targets + sccache, rebuilt weekly by automation) cut CI setup from 4 minutes to 20 seconds per job and eliminate 'which toolchain version ran this?' forensics. Cache target/ and registry across runs (actions/cache with Cargo.lock-hash keys) for 50-70% faster rebuilds — keyed caches invalidate exactly when dependencies change.

Local reproduction of CI failures closes the debugging loop. A justfile or xtask (cargo xtask ci) encoding the exact CI command sequence (fmt --check, clippy deny, test workspace all-features, doc build) lets developers run the pipeline locally before pushing — 'works on my machine' dies when local runs the same commands. Containerized local CI (same image as runners via docker run) removes the last divergence for toolchain-sensitive failures like linker errors. One team cut CI-failure iterations from 3.2 rounds per PR to 1.1 with xtask parity — the fix for environment bugs is identical environments, automated. Platform reliability is a feature; ship it like one.

toolchain_fix.shBASH
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
# Diagnose and fix the big three toolchain failures.
# 1. Missing C linker?
cc --version || echo "NO LINKER FOUND"
# Debian/Ubuntu: sudo apt-get update && sudo apt-get install -y build-essential pkg-config
# macOS: xcode-select --install
# Windows: VS Build Tools with the Desktop C++ workload

# 2. Missing cross target?
rustup target list | grep installed
rustup target add wasm32-unknown-unknown
cargo build --target wasm32-unknown-unknown 2>&1 | tail -2

# 3. Wedged artifacts? Escalate surgical -> full -> registry.
cargo clean -p shop-api && cargo build -p shop-api 2>&1 | tail -1
# cargo clean && cargo build --workspace 2>&1 | tail -1
# rm -rf ~/.cargo/registry && cargo fetch

cargo --version && rustc --version
📊 Production Insight
A slimmed CI image dropped gcc to save 180 MiB in pulls — 63 pipelines failed Monday morning with identical link errors while all compiles passed. Fix: build-essential restored plus a 20-line hello-world link smoke test gating image promotion. Zero linker pages in the year since. Rule: assert cc --version in setup scripts, smoke-test base images as consumers, and read platform-wide identical failures as infrastructure first.
🎯 Key Takeaway
cc links what rustc compiles: install build-essential/Xcode CLT/MSVC. rustup target add for cross, cargo clean -p before cargo clean. Contract-test the platform.
● Production incidentPOST-MORTEMseverity: high

The Missing C Linker That Blocked 14 Engineers for a Morning

Symptom
Monday 9:04 AM: all Rust pipelines failed within minutes of each other, unit and integration alike, with error: linking with cc failed, exit status: 1 and a note about unable to find library or object file. No test output, no warnings — compilation of every crate succeeded and the failure hit at the final link step. Local builds on developer laptops stayed green, which sent the first 30 minutes of debugging toward 'what changed in our code over the weekend' — nothing had. The blast radius was total: 63 pipelines red across 5 services, and the failure signature was identical in each.
Assumption
The team assumed a Cargo or crates.io outage — a yanked transitive dependency or a registry hiccup — because 'nothing changed on our side.' They burned 40 minutes auditing Cargo.lock diffs and pinning registry caches. The actual change was two layers down: the platform team shipped a slimmed CI base image Friday evening that dropped build-essential to cut 180 MiB from pull times. Rust links through the system C linker even for pure-Rust binaries (crt, libc shims), so removing gcc broke every link while leaving every compile green. Nobody connected 'smaller image' to 'Rust needs cc' because that dependency is invisible until it breaks.
Root cause
Two gaps compounded. First, rustc shells out to cc for the final link on Linux (and link.exe via MSVC shims on Windows, clang via Xcode CLT on macOS) — a toolchain requirement documented but never encoded: CI installed the Rust toolchain via rustup but never asserted a working C linker. Second, the base-image pipeline had no consumer contract test — the image repo's CI verified the image built, not that a sample Rust, Go, and C project still linked inside it. The Friday image passed its own green build and rolled out fleet-wide over the weekend, arming the failure for Monday's first push.
Fix
Immediate: one line back into the runner Dockerfile (apt-get install -y build-essential, plus pkg-config and libssl-dev that the next failure would have needed) and a fleet-wide image rebuild — pipelines green by 12:40 PM. Durable: a 20-line toolchain smoke job now runs inside every base-image build, compiling and linking a hello-world Rust binary (cargo build in a scratch crate) before the image can promote. Portable: the repo README gained a prerequisites table per OS (Debian build-essential, macOS xcode-select --install, Windows VS Build Tools with the C++ workload), and new-hire setup scripts assert cc --version before installing rustup.
Key lesson
  • Rust needs a C linker even for pure-Rust code — rustc compiles, cc links. Encode toolchain prerequisites (gcc/clang + rustup + targets) as asserted setup steps, not wiki prose. A 20-line smoke job compiling hello-world inside the CI image would have caught this Friday, not Monday.
  • Base-image changes need consumer contract tests: the image's own green build proves nothing about the 5 languages built on top of it. Test what riders need (link a binary per toolchain), not just that the image assembles.
  • When every pipeline fails identically with zero code changes, debug the platform first: diff the runner image, not Cargo.lock. Blast-radius-wide identical signatures mean shared infrastructure, and the fastest signal is rebuilding one pipeline on last week's image.
Production debug guideSeven production failure shapes — each with the exact commands that confirm the cause and the fix that sticks.7 entries
Symptom · 01
error: linking with cc failed, exit status 1 — every crate compiles, the final link dies
→
Fix
Verify the linker directly: cc --version (expect gcc/clang output; 'command not found' confirms it). On Debian/Ubuntu run sudo apt-get update && sudo apt-get install -y build-essential pkg-config; on macOS run xcode-select --install; on Windows install VS Build Tools with the C++ workload. Then cargo clean -p <crate> && cargo build to force a fresh link. Prevent recurrence with a CI smoke step that builds a hello-world crate inside the image before promotion.
Symptom · 02
error[E0432/E0433]: failed to resolve / unresolved import after moving files around
→
Fix
Map the tree first: run cargo check 2>&1 | grep -E 'E0432|E0433' to list every broken path, then confirm file layout matches declarations with ls -R src/. Remember: mod foo; in lib.rs expects src/foo.rs or src/foo/mod.rs, and paths inside foo resolve from foo (use super:: for the parent, crate:: for the root). Fix declarations or move files so each matches, then cargo check again. Adding #![warn(missing_docs)] won't help here — this is pure path-vs-file mismatch.
Symptom · 03
error[E0603]: function is private — tests or sibling crates can't reach an item you just wrote
→
Fix
Check the visibility chain: grep -n 'pub' src/<module>.rs around the item and every mod statement above it — a private mod hides even pub items inside it. Minimal fix is pub(crate) for in-crate use, pub for public API; also verify re-exports (pub use) at the crate root if external users need a short path. Run cargo test --lib and cargo build --workspace to confirm both sides compile, then cargo doc --no-deps --open to check the item renders in public docs.
Symptom · 04
error[E0308]/feature-gated code silently missing: #[cfg(feature = "x")] block never compiles in
→
Fix
Confirm active features with cargo tree -e features -p <crate> | grep -E 'feature "x"' and check the union across the workspace — feature unification means one member enabling x enables it everywhere. Build explicitly: cargo check -p <crate> --features x and cargo check --workspace --all-features to catch both directions. Fix Cargo.toml so defaults match the common case, document non-default features in the README, and gate CI on --no-default-features plus --all-features builds.
Symptom · 05
error[E0260]/crate not found: use of undeclared crate or 'can't find crate for X' in a workspace member
→
Fix
Inspect wiring: cargo tree -p <member> | head -30 shows what resolved; grep -n 'path\|workspace' <member>/Cargo.toml confirms the dependency source. Members must declare path = "../sibling" (or workspace = true inheritance); the root needs [workspace] members = [...] covering the directory. After editing run cargo metadata --format-version 1 > /dev/null to validate the workspace graph, then cargo check --workspace to compile every member.
Symptom · 06
Tests pass with cargo test -p mycrate but fail under cargo test --workspace (or vice versa)
→
Fix
Isolate the axis: run cargo test -p mycrate --lib (unit only), then cargo test -p mycrate --test '*' (integration), then cargo test --workspace --exclude mycrate to find the interacting member. Common causes: feature unification changing behavior, integration tests sharing target/ artifacts or CWD-relative fixtures, and env vars leaking between suites. Fix by making fixtures path-absolute (env!("CARGO_MANIFEST_DIR")), avoiding global state, and running cargo test --workspace --all-features in CI so local and remote agree.
Symptom · 07
Stale build artifacts: code changed but binary behavior didn't, or 'weird' errors survive a revert
→
Fix
Confirm staleness: touch src/main.rs && cargo build 2>&1 | grep -E 'Compiling|Finished' — if rustc never recompiles your crate, fingerprints are wedged. Escalate cleanly: cargo clean -p <crate> first (surgical, keeps the shared cache), full cargo clean only if that fails. Then cargo build --workspace to rebuild the graph. If staleness recurs, suspect build scripts (build.rs mtime handling), mtime skew on NFS/checkouts (git clone fresh to test), or antivirus locks on target/ (Windows).
Rust Tooling Compared: Which Command for Which Job
TaskCommandScopeWhen to run
Check compilation fastcargo check -p coreOne crate, no codegenWhile editing, every few minutes
Run all testscargo test --workspaceUnit + integration + docBefore every push
Lint as a gatecargo clippy --all-targets -- -D warningsAll targets, denyPre-commit hook + CI
Enforce stylecargo fmt --all -- --checkWhole repoOn save + CI gate
Render API docscargo doc --no-deps --openPublic surfaceAPI review + pre-publish
Release workspacecargo publish --workspaceMembers in order (1.90+)Release day after dry-runs
Repair stale buildscargo clean -p foo, then fullSurgical, then allOnly when fingerprints wedge
⚙ Quick Reference
10 commands from this guide
FileCommand / CodePurpose
paths_demo.rsmod network {Modules and Paths
visibility_demo.rspub struct Money {Visibility That Scales
layout.shcargo new --lib shopliblib vs bin Layout
test_loop.shcargo test 2>&1 | tail -8cargo test
quality_gates.shcargo fmt --allClippy, fmt, and doc
workspace.shmkdir -p shop/{core,api,cli}/srcWorkspaces
inheritance.shcat > shop/Cargo.toml << 'EOF'Workspace Inheritance
publish.shgrep -A6 '\[package\]' shop/core/Cargo.toml | head -12Publishing to crates.io
features.rspub fn connect(url: &str) -> String {Build Profiles and Feature Flags
toolchain_fix.shcc --version || echo "NO LINKER FOUND"Toolchain Troubleshooting

Key takeaways

1
mod declares the module tree, files hold bodies; anchor paths with crate::/self::/super:
and flatten API with pub use.
2
Default private
pub(crate) inside the crate, pub for deliberate API, private fields with checked constructors.
3
Thin mains over a --lib core
tests/ checks public contracts, unit tests beside code check internals.
4
cargo test runs units + integration + docs; CI uses --workspace --all-features for parity with unification.
5
Gate clippy -- -D warnings, fmt --check, and doc builds in CI; run format on save.
6
Virtual workspace for products, root-package for libraries; one lockfile, one target dir, path deps with versions.
7
Inherit version/edition/rust-version/license from [workspace.package]; enforce MSRV with a pinned CI job.
8
Dry-run every member, publish --workspace on 1.90+, verify with a locked clean-container install.

Common mistakes to avoid

7 patterns
×

Splitting a workspace before 30K lines (or splitting by file size)

Symptom
Ceremony outruns benefit: 3 tiny crates with circular-path deps, 3x version-bump overhead, and slower builds from lost incrementality — all for boundaries no team needed.
Fix
Split at 30-80K lines along team/release boundaries (api/core/cli). Before that, modules + visibility give 90% of the benefit with none of the release overhead.
×

Making everything pub instead of defaulting private

Symptom
Accidental public API: docs.rs shows 200 items, semver blocks every refactor, and integration surface sprawls. Renaming one helper becomes a breaking change.
Fix
Default private; promote to pub(crate) on second in-crate caller, pub on genuine API need. Audit cargo doc output quarterly and demote.
×

Testing only with cargo test -p one-crate locally

Symptom
Green locally, red in CI: feature unification, CWD-relative fixtures, and cross-member interactions hide until the workspace-wide run.
Fix
CI runs cargo test --workspace --all-features; fixtures use env!("CARGO_MANIFEST_DIR"). Locally replicate with --workspace before pushing.
×

Skipping --dry-run before cargo publish

Symptom
Half-published workspaces: member 7 fails on metadata or path-dep versions, leaving released crates pointing at unreleased dependents — days of yank cleanup.
Fix
cargo publish --dry-run per member on release branches (CI job), then publish --workspace on 1.90+. Verify with cargo install --locked in a clean container.
×

Benchmarking dev-profile builds

Symptom
'Rust is slower than X' conclusions from 40-50x slower dev binaries; optimization decisions made on noise.
Fix
Benchmarks run under cargo build --release (or criterion with release profile). State the profile next to every number.
×

Forgetting rust-version (MSRV) while using new APIs

Symptom
Enterprise users on older stable toolchains can't build; let-chains or new std APIs break Debian-stable users for weeks.
Fix
Set workspace.package rust-version, inherit per member, and add a CI job pinning that toolchain: cargo +<msrv> check --workspace.
×

Full cargo clean as the first response to any build oddity

Symptom
30+ developer-minutes burned per incident rebuilding shared target/ caches that were never the problem.
Fix
Escalate: clean -p <crate>, then full clean, then registry repopulation. Record which tier fixed it and hunt root causes (build.rs mtimes, NFS skew, AV locks).
INTERVIEW PREP · PRACTICE MODE

Interview Questions on This Topic

Q01SENIOR
How do mod, use, crate, self, and super fit together?
Q02SENIOR
When do you use pub vs pub(crate) vs private, and why keep fields privat...
Q03SENIOR
What distinguishes unit, integration, and doc tests, and where does each...
Q04SENIOR
How does workspace inheritance work, and what belongs in [workspace.pack...
Q05SENIOR
Explain feature unification and the rules for well-behaved features.
Q06SENIOR
A developer reports 'linking with cc failed' on a fresh machine. Walk th...
Q01 of 06SENIOR

How do mod, use, crate, self, and super fit together?

ANSWER
mod declares a module in the tree (inline body or a file at src/<name>.rs); use binds shortcut names in the current scope without moving code. Paths anchor at crate:: (root), self:: (current module), super:: (parent), or an external crate name. Re-export with pub use to flatten deep trees. E0432/E0433 means the anchor or file layout mismatches the declared tree.
FAQ · 8 QUESTIONS

Frequently Asked Questions

01
When should I split a single crate into a workspace?
02
Virtual workspace or root-package workspace?
03
How do I fix 'failed to resolve import' (E0432/E0433)?
04
Why do tests pass locally but fail in CI?
05
What goes in [workspace.package] vs [workspace.dependencies]?
06
How does cargo publish --workspace order releases?
07
How do I keep the linker working across Linux, macOS, and Windows?
08
When is cargo clean -p enough versus full cargo clean?
N
Naren Founder & Principal Engineer

20+ years shipping production backend systems. Everything here is grounded in real deployments.

Follow
✓ Verified
production tested
September 27, 2026
last updated
2,085
articles · all by Naren
🔥

That's Tooling. Mark it forged?

26 min read · try the examples if you haven't

←
Previous
Rust Collections Vec String HashMap
1 / 1 · Tooling
Next
Rust Iterators and Closures
→