Rust Tokio Async: 10 Patterns That Hold p99 at 190ms

Your first async Rust service will feel like a superpower. You write straight-line code with await, boot a Tokio runtime with twelve workers, and watch it hold 40k idle connections while using less memory than your old thread-per-connection prototype used for 400. You'll wonder why every backend isn't written this way.

Then you will hit the stall you did not see coming. One std::thread::sleep left in a retry path, one std::sync::Mutex guarding a cache across an await, one refresh task that parses a 4MB blob while holding a lock, and all twelve workers park at once. Throughput drops from 22k to 300 requests per second. CPU looks idle. Error rates stay zero because nothing fails, everything just arrives late. You haven't got a logic bug, you've got an executor bug, and no amount of adding threads will fix it.

That gap between demo async and production async is what this guide closes. You will learn how futures, the poll model, and the work-stealing scheduler actually fit together, why async closures needed the new AsyncFn traits in Rust 2024, and when the current_thread runtime beats multi_thread. You will overlap vendor calls with join!, enforce deadlines with select!, and wire backpressure through bounded channels so bursts flatten instead of OOMing the pod.

You will also see the failure modes that page teams at 2 AM. A metadata refresh that held a std mutex for two seconds and spiked p99 from 190ms to 2100ms every five minutes. A fan-out that awaited thirty vendor calls in a loop and missed SLA by 3x until join! cut wall time from 612ms to 210ms. A test suite that passed for months and never caught any of it because it never ran under tokio::test with paused time.

By the end you'll have ten production patterns you can apply this week. You will know which channel type fits each shape, how to push CPU work to spawn_blocking without starving the blocking pool, why Send bounds reject your spawn, and how to prove overlap with tokio-console instead of guessing. Your p99 will stop being a mystery and start being a budget you'll enforce with timeouts.

Futures and the Executor: The Poll Model You Can't Skip

Every async fn in Rust returns an inert future, which is a state machine describing work that has not started yet. Calling fetch_user(42) runs zero IO. It builds a value that implements Future, meaning it can be polled until it yields Poll::Ready(output). Until something polls it, nothing happens. That laziness surprises every newcomer once, usually as an unused future warning next to a database call that never ran.

The executor is the thing that polls. Tokio's multi_thread scheduler owns a set of worker threads, each with a local queue plus a shared global queue, and uses work stealing to balance load. When your code hits await on a socket read with no data ready, the future returns Poll::Pending and hands the runtime a waker. The worker parks that task and polls another one immediately. When epoll reports the socket readable, the waker fires, the task goes back on a queue, and a worker resumes it right after the await point. No thread ever blocks waiting.

This model explains the numbers teams brag about. A future costs hundreds of bytes, while an OS thread costs an 8MB stack plus kernel bookkeeping. Twelve workers polling 100k parked futures use less memory than 2k threads idling on blocking reads. Context switches drop because wakeups move tasks between userspace queues instead of trapping into the kernel. That is how a 16-core box holds 40k connections at 31% CPU while the thread-per-connection rewrite of the same service OOMs at 9k.

The contract cuts both ways. Poll must never block, and neither must the code between awaits. A 200ms CPU loop inside an async fn holds its worker for the full 200ms while 4000 parked tasks wait behind it. The runtime cannot preempt you. It can only poll the next task when you yield at await. Treat every await as a yield point you owe the scheduler, keep the gaps between them short, and push anything long to the blocking pool where it cannot starve pollers.

You can prove the model in five minutes with tokio-console. Spawn 500 sleep tasks, watch them all sit parked on one worker timeline, then add a single std sleep and watch every lane freeze. That picture rewires your intuition faster than any doc page. Futures describe, the runtime drives, and your job is to never hold the wheel while parked.

Buffer sizing follows directly from the poll model. Each parked task holds its future state plus any socket buffers the OS keeps, so capacity planning starts with measuring parked bytes per connection under realistic payloads. A JSON gateway holding 2KB of decode scratch per task consumes 200MB at 100k connections before application state enters the picture. Teams that graph parked memory against connection count spot the linear slope early and budget buffers before launch instead of discovering the OOM during the first traffic spike.

Cooperative scheduling also shapes latency reasoning. Because workers never preempt, a 5ms CPU burst delays every task behind it by exactly 5ms, and ten such bursts per second add 50ms to p99 without touching averages. That is why p99-first teams profile max poll time rather than mean: the longest gap between awaits sets the tail. Keep every gap under a millisecond on hot workers, move the rest to the blocking pool, and the tail collapses toward the median.

Waker mechanics repay one level of depth. Each Pending return stores a Waker cloned from the polling context, and the IO driver calls wake() when readiness arrives, moving the task from the parked set back to a run queue. Cloning wakers per poll is cheap but not free, so futures that return Pending in a tight loop without new readiness waste CPU re-registering interest. Collapse redundant wakeups by awaiting a single readiness event instead of polling in a loop, and instrument wake counts when a task shows high poll counts with zero progress. In practice, sketch the task lifecycle on a whiteboard once: spawn, Pending with waker, readiness, wake, Ready. Engineers who can draw that loop diagnose stalls in minutes instead of hours.

use std::time::Duration;

#[tokio::main]
async fn main() {
    // Lazy future: nothing runs until awaited or spawned.
    let fut = fetch_user(42);
    let (a, b) = tokio::join!(fut, heartbeat());
    println!("{a} {b}");
}

async fn fetch_user(id: u64) -> String {
    tokio::time::sleep(Duration::from_millis(50)).await;
    format!("user-{id}")
}

async fn heartbeat() -> String {
    tokio::time::sleep(Duration::from_millis(20)).await;
    String::from("alive")
}

async/await Syntax: What Desugars and What It Costs

Rust's async syntax looks like scripting but compiles to a state machine with zero heap allocation in the common case. An async fn returning String becomes a struct holding each live variable across await points, with poll() advancing an internal discriminant. Await points become states. Locals that die before the next await vanish from the struct entirely, which is why holding a 4KB buffer across three awaits bloats every parked task while dropping it before await costs nothing.

The 2024 edition tightened several edges you will feel daily. Temporary lifetimes across await are checked more precisely, so code that borrowed a local into a future and awaited something else now fails fast instead of smuggling a dangling reference. Never-type fallback and gen-keyword reservations rarely bite application code, but the async desugar output did get leaner discriminant encoding, shaving a few bytes per future. At 100k concurrent tasks those bytes turn into real megabytes, so upgrading the edition on an IO-bound service is a free memory win.

Boxing is the cost lever you control. A plain async fn future is an anonymous sized type, and combinators like join! poll them inline with no allocation. Recursion, trait objects, or futures held in a Vec need dyn Future, which forces Box::pin. Pinning is not scary: Box::pin(fut) heap-allocates once and guarantees the state machine never moves, which self-referential borrows require. The mistake is boxing everything by habit. Profile parked-task memory first, box only the recursive or heterogeneous paths, and keep the hot 95% unboxed.

Borrowing across await has one rule worth memorizing: the borrow lives as long as the await chain needs it, and the compiler tracks it across yield points. Holding a &str from a HashMap while awaiting a socket write is fine if the map outlives the future, but mutating that map on another task while the borrow is parked is a data race the borrow checker rejects. When two tasks truly share mutable data, stop borrowing and switch to Arc plus a tokio lock or channel, which moves the sharing from stack lifetimes to runtime coordination.

Read one desugared future with cargo expand on a toy example and the magic evaporates in the best way. You will see the state enum, the waker plumbing, and exactly which locals survive each await. Engineers who do this once stop fearing async and start budgeting it: small states park cheap, big states park dear, and dropping buffers before await is the cheapest optimization in the whole runtime.

Lifetime errors around await deserve a calm reading habit. When the compiler complains that a borrow does not live long enough, it names the await point where the borrow is still held and the scope where the owner drops. The repair is usually ownership, not annotation: clone the header map into an owned struct before the socket write, or restructure so the borrow ends before the first await. Fighting the error with wider lifetimes almost always backfires; narrowing the borrow to the pre-await region resolves it on the first try.

Allocation discipline compounds at scale. An extra Box per request at 20k rps means 20k allocations per second hitting the allocator hot path, which shows up as 3 to 5% CPU in jemalloc profiles. Replacing boxed trait objects with concrete futures or static dispatch on the hot path returns that CPU directly to request handling. Reserve dyn Future for plugin boundaries and heterogeneous handler tables where the flexibility earns its cost, and keep the core proxy loop fully static.

Pinning guarantees the state machine never moves in memory after first poll, which self-referential futures need for safety. Box::pin provides that guarantee on the heap, while stack futures pinned by macros like tokio::pin work for scoped joins. Moving a pinned future is a compile error, not a runtime hazard, and that strictness is what lets RawWaker pointers stay valid across yields. Respect pin, never mem::swap a polled future, and self-referential borrows stay sound.

use std::future::Future;
use std::pin::Pin;
use std::time::Duration;

#[tokio::main]
async fn main() {
    let fut = read_with_deadline("orders/42");
    println!("{}", fut.await);
    // Boxed heterogeneous set: only box where the types differ.
    let mixed: Vec<Pin<Box<dyn Future<Output = String> + Send>>> =
        vec![Box::pin(fetch("a")), Box::pin(fetch("b"))];
    for f in mixed {
        println!("{}", f.await);
    }
}

async fn read_with_deadline(key: &str) -> String {
    // Buffer is dropped before await: it never inflates parked state.
    let owned = key.to_owned();
    drop(key);
    tokio::time::timeout(Duration::from_millis(250), fetch(&owned))
        .await
        .unwrap_or_else(|_| String::from("fallback"))
}

async fn fetch(key: &str) -> String {
    tokio::time::sleep(Duration::from_millis(10)).await;
    format!("row:{key}")
}

Async Closures and AsyncFn Traits in Rust 2024

Rust 2024 finally gave async closures first-class status with the AsyncFn, AsyncFnMut, and AsyncFnOnce traits. Before this, passing an async closure into a generic helper meant awkward boxing or nightly-only tricks, because Fn traits return synchronously and async closures return futures. Now async || works like sync ||: the compiler infers the right AsyncFn* bound from how the closure captures, and generic code can accept either an async closure or an async fn with the same bound. Retry helpers, middleware chains, and test harnesses all get simpler signatures.

The capture rules mirror sync closures exactly, which makes them easy to predict. A closure borrowing state gets AsyncFn, one mutating captured state gets AsyncFnMut, and one consuming state gets AsyncFnOnce. The returned future borrows the closure for as long as the call needs, so retry(|| fetch(key)) holds the borrow across each attempt but releases it between attempts at await points. That precision lets the borrow checker approve patterns that used to demand owned clones everywhere, cutting Arc traffic on hot paths by half in several codebases we reviewed.

Generic bounds are where the new traits pay rent. A helper declared as async fn with_retry<F, Fut>(mut f: F) where F: AsyncFnMut() -> Fut works for closures, function pointers, and async fns without a single Box. Callers pass || async { client.get(url).await } directly. Inside, each f() call produces a fresh future you can select! against a timeout or join! with a fallback. The old workaround needed Pin<Box<dyn Future>> at every layer, adding an allocation per attempt that showed up at 20k retries per second.

There are two sharp edges to memorize. First, an async closure's future captures the closure borrow, so holding that future across a scope that mutates the closure is a borrow error, not a runtime bug. Await each attempt before the next iteration mutates state. Second, spawning an async closure's future still demands Send, so Rc captures that compiled fine in a retry loop will fail at tokio::spawn with a long note. Capture Arc and cloned handles inside the closure body, keep the closure itself Send, and spawns compile cleanly.

Use async closures for the glue: retries, timeouts, per-item mapping over streams, and route handlers built in loops. Keep named async fns for the units you test. That split gives you terse call sites with named, mockable cores, and the AsyncFn* bounds make the glue generic without a single nightly feature or boxing tax.

Testing generic AsyncFn helpers stays pleasant because the bounds accept plain async fns as well as closures. A retry helper tested with a counting async fn needs no mock framework: pass an async fn that fails twice then succeeds, assert the attempt count and the virtual-time deadline separately. The same helper accepts a capturing closure in production without changing a line. That duality makes AsyncFn bounds the rare abstraction that simplifies both call sites and test suites at once.

Migration from boxed-future helpers is mechanical. Replace Pin<Box<dyn Future>> parameters with impl AsyncFnMut plus a Fut generic, delete the boxing at each call site, and let type inference carry the output type. Compiler errors during migration point at the exact closure that captures too much, which is usually an Rc or a borrowed buffer that should have been an Arc or an owned String. Fix the capture, not the bound, and the migration finishes in an afternoon for most services.

Work-stealing details shape tail behavior under imbalance. Idle workers steal half the local queue of a busy sibling, which balances fast but migrates cache lines across cores. Tasks touching large per-connection buffers pay a cache-miss cost after each steal, so pinning hot loops with modest worker counts can beat wide pools on NUMA boxes. Measure steal rates alongside poll times: high steals with low poll times mean the pool is too wide, while deep local queues with zero steals mean tasks are pinned unintentionally.

use std::time::Duration;

#[tokio::main]
async fn main() {
    let client = FakeClient { n: 0 };
    let out = with_retry(3, || async { client.get("orders/7").await }).await;
    println!("{out}");
}

#[derive(Clone)]
struct FakeClient {
    n: u8,
}

impl FakeClient {
    async fn get(&self, key: &str) -> Result<String, String> {
        tokio::time::sleep(Duration::from_millis(20)).await;
        Ok(format!("ok:{key}"))
    }
}

async fn with_retry<F, Fut>(tries: u32, mut f: F) -> String
where
    F: AsyncFnMut() -> Fut,
    Fut: Future<Output = Result<String, String>>,
{
    let mut last = String::from("empty");
    for _ in 0..tries {
        match tokio::time::timeout(Duration::from_millis(100), f()).await {
            Ok(Ok(v)) => return v,
            Ok(Err(e)) => last = e,
            Err(_) => last = String::from("timeout"),
        }
    }
    last
}

Runtime Flavors: multi_thread vs current_thread

Tokio ships two schedulers and picking the wrong one is a silent tax. The multi_thread flavor runs a work-stealing pool with one worker per core by default, which is what servers want: tasks migrate to idle cores, blocking pool threads absorb spawn_blocking, and throughput scales with cores until the NIC or database gives out. The current_thread flavor runs everything on one thread and drives timers and IO inline, which is what tests, CLIs, and tiny edge agents want: deterministic ordering, tiny binaries, and no cross-thread surprises.

Construction has three idioms you should know cold. Bare #[tokio::main] gives you multi_thread with core-count workers and both drivers enabled, which fits 90% of services. Attribute tuning like #[tokio::main(flavor = "current_thread")] or #[tokio::main(flavor = "multi_thread", worker_threads = 8)] covers the next 9%. Full Builder::new_multi_thread() with enable_all, worker_threads, max_blocking_threads, and thread_name covers the last 1%: gateways that pin workers to cores, batch jobs that cap blocking threads at 64 to protect a fragile NFS mount, or services that name threads for flame graphs.

Worker count deserves measurement, not superstition. Start at the default, then load-test at 1x, 0.5x, and 2x core count while watching tokio-console mean poll time and schedule delay. IO-bound APIs usually peak near core count because workers spend their lives parked on wakers. CPU-mixed services peak lower because long polls starve the queue. One team moved from 32 workers to 12 on a 16-core box and gained 18% throughput because fewer workers meant less stealing overhead and warmer caches. More threads is not more speed once polls get long.

Current_thread has a trap that bites everyone once: it only drives tasks inside block_on, so spawned tasks freeze the moment block_on returns. Tests that spawn a background refresher then exit block_on will see zero refreshes and blame the channel. Either await the JoinHandle inside block_on or switch the test to multi_thread. The reverse trap hits servers: current_thread in production caps you at one core, and p99 looks fine until 3k rps when the single lane saturates and every request queues behind one poll loop.

Name your runtime choice in a comment at main. Future readers should see why this binary uses current_thread with start_paused in tests, or multi_thread with 12 workers and 256 blocking threads in production. Runtime flavor is load-bearing architecture, not trivia, and the comment saves the next engineer from retuning what you already measured.

Feature flags interact with flavor choice in ways worth stating. The rt-multi-thread flag unlocks the work-stealing scheduler, while the base rt flag covers current_thread plus timers and IO. Minimal binaries for embedded gateways enable only what they use and boot current_thread in under a millisecond. Full servers enable full and accept the larger binary for the complete driver set. Matching flags to flavor keeps compile times and binary sizes honest, and docs.rs feature tables confirm which Builder method needs which flag before you chase a missing-method error.

Observability differs by scheduler in one practical way. Multi_thread exposes per-worker poll times, steal counts, and queue depths through tokio-console and runtime metrics, which is how you prove balance versus contention. Current_thread collapses all of that to a single lane, so diagnosis reduces to reading one ordered trace. Debug scheduler bugs on current_thread where ordering is reproducible, then confirm the fix under multi_thread load where stealing reorders everything.

Driver enablement trims both binary size and surprise. enable_all boots IO plus time for typical servers, while granular enable_io plus enable_time lets lean agents skip signal handling they never use. Forgetting the time driver produces a panic naming the missing driver at the first sleep call, which confuses newcomers who blame the scheduler. Read that panic as a configuration checklist, enable what the code path needs, and gate the combination in a smoke test that sleeps once at startup.

use std::time::Duration;

fn main() {
    // Production shape: pinned workers, named threads, capped blocking pool.
    let rt = tokio::runtime::Builder::new_multi_thread()
        .worker_threads(8)
        .max_blocking_threads(64)
        .thread_name("api-worker")
        .enable_all()
        .build()
        .expect("runtime builds");
    rt.block_on(async {
        let h = tokio::spawn(serve());
        tokio::time::sleep(Duration::from_millis(50)).await;
        h.abort();
    });
}

async fn serve() {
    tokio::time::sleep(Duration::from_millis(10)).await;
    println!("served");
}

#[cfg(test)]
mod tests {
    #[tokio::test(flavor = "current_thread", start_paused = true)]
    async fn deterministic_timer() {
        let f = tokio::time::sleep(std::time::Duration::from_secs(30));
        tokio::pin!(f);
        assert!(!f.is_elapsed());
    }
}

Spawn, JoinHandle, join! and try_join!

Spawning is how independent work overlaps. tokio::spawn takes a Send future, schedules it immediately even before you await the handle, and returns a JoinHandle you later await for the output. tokio::join! polls a fixed set of futures inline on the current task with no spawn overhead, returning a tuple of outputs. tokio::try_join! is the fallible twin that short-circuits on the first Err. Pick by shape: fixed known set goes to join!, fallible fixed set goes to try_join!, dynamic or detached work goes to spawn.

The wall-time math is the whole pitch. Three vendor calls costing 200ms each finish in 612ms sequentially, 210ms under join!, and roughly the same under spawn with slightly more scheduling overhead. One checkout team measured exactly that: 612ms to 210ms, p99 SLA met without touching a single vendor. The overhead gap between join! and spawn only matters past a few hundred fan-outs per request, where spawn's queue hops add microseconds each. Below that, pick readability.

JoinHandle discipline decides whether failures hide. Awaiting a handle yields Result<T, JoinError>: Ok for completion, Err for panic or cancellation via abort. Dropping a handle detaches the task, which keeps running but reports nothing, so panics vanish into the log void. The production rule is simple: every spawn site either awaits its handle, stores it in a JoinSet for later collection, or explicitly aborts with a logged reason. Unawaited handles are how a payment capture task dies silently while the handler returns 200.

JoinSet is the workhorse for dynamic fan-out. Spawn thirty shard queries into a set, then loop join_next().await to collect as they finish, aborting the remainder on the first fatal error. Unlike a Vec of handles polled in order, JoinSet returns whichever finishes first, so one slow shard never head-blocks twenty-nine fast ones. Cap concurrency with a semaphore before spawning when the fan-out is unbounded, or a 10k-item backlog will spawn 10k tasks and blow the local queue's 256-slot soft limit into the global queue.

Abort semantics deserve respect. Handle::abort cancels at the next await point, and the task's Drop impls run, so locks release but partial writes may persist. Design spawned work to be idempotent or checkpointed: write-then-rename files, conditional database writes, or compensating actions on JoinError::is_cancelled. Teams that treat abort as free rollback eventually debug a half-written ledger at midnight.

Error propagation across fan-out deserves a deliberate policy. Join! returns each result independently, so partial failure handling lives with you: log which shard failed, substitute cached values per shard, and decide whether one failure fails the request. Try_join! bails on the first Err and cancels nothing already spawned, which suits all-or-nothing writes. JoinSet with join_next lets you abort siblings the moment a fatal error lands, which suits hedged reads where the first success wins. Write the policy next to the spawn call so reviewers see it without hunting.

Backpressure before fan-out prevents self-inflicted overload. Spawning 10k tasks for a 10k-row import queues 10k futures instantly and spills the local queues into the global one, adding scheduling latency to every other request on the box. A semaphore with 32 permits around the spawn loop holds at most 32 in flight while the rest wait cheaply as permits, keeping p99 flat for neighboring routes. The import takes the same total time but stops punishing unrelated traffic, which is the difference between a batch job and an incident.

Timeout accounting closes the loop on fan-out policy. Every join site should document its worst-case wall time as the max branch plus scheduling slack, and every spawn site should state whether sibling failure aborts the set. Without those two comments, reviewers guess at deadline behavior and timeouts drift. One checkout service added worst-case comments to 14 fan-out sites and found 3 with no effective deadline at all, each fixed with a single timeout wrapper.

use std::time::Duration;

#[tokio::main]
async fn main() {
    // Fixed set: join! overlaps with zero spawn overhead.
    let (a, b, c) = tokio::join!(vendor("a"), vendor("b"), vendor("c"));
    println!("{a} {b} {c}");
    // Dynamic set: JoinSet collects in finish order.
    let mut set = tokio::task::JoinSet::new();
    for i in 0..5u32 {
        set.spawn(vendor(&format!("shard-{i}")));
    }
    while let Some(r) = set.join_next().await {
        match r {
            Ok(v) => println!("done {v}"),
            Err(e) => eprintln!("task failed: {e}"),
        }
    }
}

async fn vendor(name: &str) -> String {
    tokio::time::sleep(Duration::from_millis(50)).await;
    format!("ok:{name}")
}

select!, Timeouts, and Cancellation Safety

Select! is a race: it polls several branches concurrently and runs the winner, dropping the losers. The canonical shape pairs real work against a sleep branch, turning any dependency into a bounded operation. A pricing call that occasionally takes 8 seconds becomes a 250ms timeout returning cached data with a stale flag. Without that branch, one slow vendor holds a worker and a pool connection until the client gives up, and at 3k rps the pileup parks every worker within minutes.

Timeouts come in two flavors with different ergonomics. tokio::time::timeout(dur, fut) wraps one future and returns Err on expiry, which is perfect for single calls. Select! with a sleep branch handles multi-way races, fallbacks that need cleanup, and biased polling via biased; to prefer the fast path. Prefer timeout() for simple deadlines because its error type forces you to handle expiry at the call site. Reach for select! when losers need explicit handling, like aborting a spawned upload or recording which branch won for timeout tuning dashboards.

Cancellation safety is the concept that decides whether dropping a loser corrupts state. A future is cancellation-safe if dropping it mid-poll leaves shared state consistent. tokio::time::sleep is safe. tokio::sync::mpsc send is safe. tokio::fs::read is safe. tokio::sync::oneshot receive is safe, but the sender side can observe cancellation. tokio-postgres queries and many database futures are not safe: dropping them mid-poll can leave the connection in a half-read state. The docs mark this per API, and ignoring the marks is how a timeout introduces connection corruption that outlives the incident.

The safe pattern for unsafe futures is indirection: spawn the risky operation, select! on the JoinHandle with a timeout, and abort on expiry. The spawned task owns the connection interaction, and aborting it keeps the half-read state inside a task boundary where the pool can reset the connection. Yes, this costs a spawn per call. At 20k rps that overhead is measurable but far cheaper than poisoned pool connections that fail 2% of queries for an hour after each vendor blip.

Log which branch wins every time. A timeout counter tagged by dependency and tier turns tuning from folklore into arithmetic: 250ms with a 0.3% timeout rate beats 500ms with 0.05% when the fallback is cached data, but loses when the fallback is an error page. Teams that graph select! outcomes per route converge on per-tier deadlines in two weeks. Teams that do not argue about timeouts forever.

Tiered deadlines beat single global timeouts in every deployment we reviewed. Edge-facing routes get 250ms with cached fallbacks, internal aggregation gets 800ms with partial responses, and batch reconciliation gets 10 seconds with retries. One global 5-second timeout either starves interactive users or lets batch work hold workers far too long. Encode the tiers as constants next to each client constructor, graph timeout rates per tier, and adjust quarterly as vendor behavior drifts. The constants file becomes the latency budget made executable.

Idempotency decides whether a timeout retry is safe. Retrying a GET price fetch after expiry is harmless, while retrying a POST capture after an ambiguous timeout can double-charge. Mark each client method safe-to-retry or exactly-once at the type level, and let the select! fallback branch check the mark before retrying. Teams that skip this step turn every vendor blip into duplicate side effects, then blame the timeout layer for a contract bug.

Cancellation logs turn silent drops into tunable data. Record a counter each time the timeout branch wins, tagged by route and dependency, and review the top five weekly. A branch winning 2% of the time at 250ms is a healthy guardrail; the same branch winning 15% means the deadline is wrong or the vendor degraded. Without the counter, both situations look identical from p99 alone, and tuning debates run on anecdote instead of counts. Start with per-vendor defaults derived from p99 plus 50% headroom, then tighten monthly as fallbacks improve. Documented tiers survive reorgs; tribal timeout knowledge does not.

use std::time::Duration;

#[tokio::main]
async fn main() {
    println!("{}", price_with_fallback("widget").await);
    // Direct timeout wrapper: simplest deadline for one call.
    match tokio::time::timeout(Duration::from_millis(100), slow_vendor()).await {
        Ok(v) => println!("vendor {v}"),
        Err(_) => println!("vendor timed out"),
    }
}

async fn price_with_fallback(sku: &str) -> String {
    tokio::select! {
        biased;
        v = live_price(sku) => format!("live:{v}"),
        _ = tokio::time::sleep(Duration::from_millis(250)) => {
            format!("cached:{sku}:stale=true")
        }
    }
}

async fn live_price(sku: &str) -> String {
    tokio::time::sleep(Duration::from_millis(40)).await;
    format!("9.99:{sku}")
}

async fn slow_vendor() -> String {
    tokio::time::sleep(Duration::from_secs(5)).await;
    String::from("slow")
}

Channels: mpsc, oneshot, broadcast, watch

Channels are how async tasks communicate without shared locks, and Tokio ships four types because four shapes dominate. Bounded mpsc gives you a multi-producer pipeline with backpressure: send().await parks the producer when the buffer fills, which keeps memory flat during bursts. Oneshot carries exactly one reply, perfect for request-response fan-out where each spawned task reports home once. Broadcast fans one value to many subscribers with lagging-receiver semantics. Watch publishes the latest snapshot to every reader, which is ideal for config and metadata that changes rarely but reads constantly.

Backpressure is the reason bounded beats unbounded in production. An unbounded mpsc queue grows until the OOM killer intervenes: one team watched 2M messages pile to 3GB during a 40-second downstream stall. The same pipeline with channel(1000) parked producers at depth 1000, held memory at 40MB, and shed excess with try_send plus a 503 metric. Size buffers from load tests, export depth as a gauge, and alert when depth sits above 70% for two minutes. A full channel is a signal to scale consumers or degrade, never to grow memory.

Each type has a misuse that pages someone. Mpsc with a single sender and receiver used as a snapshot bus replays stale history to every new reader; watch fits because new subscribers immediately see the latest value. Broadcast with 200 slow subscribers and a capacity of 16 drops lagging receivers constantly; either raise capacity to cover the slowest consumer's burst or switch slow consumers to mpsc. Oneshot senders dropped without sending make receivers hang until timeout; always pair oneshot receives with a deadline so a panicked worker becomes a 250ms fallback instead of a hung request.

Shutdown choreography leans on channel closure. Dropping all senders closes an mpsc channel so the receiver loop ends cleanly; dropping the receiver makes sends fail fast so producers stop. Watch's mark_changed plus borrow_and_update lets readers skip intermediate states during rapid config flaps. Broadcast's sender.send fails only when zero receivers exist, which doubles as a zero-subscriber probe. Design each pipeline's close path first: who drops what, who observes closure, and what metric fires when closure happens unexpectedly.

Measure depth, send latency, and lag events from day one. Depth near zero with high throughput means the buffer is oversized but harmless. Depth pinned at capacity with rising send latency means consumers lag and timeouts will follow. Broadcast lagged errors above 0.1% mean capacity or consumer speed is wrong. These three gauges predict every channel incident we have reviewed, usually 10 minutes before p99 moves.

Ordering guarantees vary by type and your code should depend only on what is promised. Mpsc preserves send order per sender but interleaves across senders, so sequence reconstruction needs explicit offsets. Broadcast delivers in send order to every connected receiver but drops lagging ones rather than slowing the sender. Watch always shows the latest value, skipping intermediates freely. Depending on unpromised ordering, like assuming cross-sender mpsc order, creates bugs that pass every test and fail only under production interleaving.

Capacity planning for channels starts from burst math, not averages. A pipeline averaging 2k messages per second with 5-second downstream stalls needs at least 10k buffer or a shed path, not the 100-slot default from the example. Multiply peak rate by worst stall duration, add 30% headroom, then verify with a chaos test that pauses the consumer and watches depth, memory, and shed metrics. Buffers sized from averages overflow on the first real stall; buffers sized from bursts ride through it.

Consumer scaling policy completes the channel story. When depth holds above 70% for minutes, add consumers horizontally before raising buffer size, since deeper buffers only delay the same overload. Autoscale rules keyed on channel depth react faster than CPU rules for IO-bound pipelines, often 3 minutes sooner in reviewed incidents. Pair the scale rule with a shed rule so depth spikes during scale-up still degrade gracefully instead of queueing unbounded work.

use std::time::Duration;

#[tokio::main]
async fn main() {
    // Bounded mpsc pipeline with shed-on-full.
    let (tx, mut rx) = tokio::sync::mpsc::channel::<u32>(1000);
    let producer = tokio::spawn(async move {
        for i in 0..10u32 {
            match tx.try_send(i) {
                Ok(()) => {}
                Err(_) => eprintln!("shed {i}: queue full"),
            }
        }
    });
    // Watch snapshot for config readers.
    let (wtx, mut wrx) = tokio::sync::watch::channel(String::from("v1"));
    wtx.send(String::from("v2")).expect("watch send");
    println!("config {}", wrx.borrow_and_update().clone());
    // Oneshot reply from a worker.
    let (otx, orx) = tokio::sync::oneshot::channel::<String>();
    tokio::spawn(async move {
        let _ = otx.send(String::from("done"));
    });
    let reply = tokio::time::timeout(Duration::from_millis(250), orx)
        .await
        .expect("timeout")
        .expect("reply");
    println!("{reply}");
    producer.await.expect("producer");
    while let Ok(v) = rx.try_recv() {
        println!("piped {v}");
    }
}

spawn_blocking and the Never-Block Rule

The never-block rule is one sentence: no worker thread may run blocking code. That means no std::thread::sleep, no std::sync::Mutex or RwLock, no filesystem reads, no DNS resolution, no CPU loops past a few hundred microseconds, and no subprocess waits on a worker. Each violation parks a worker that 4000 tasks share. Two violations park two workers. Twelve violations on a twelve-worker pool freeze the entire service while CPU graphs look reassuringly idle, which is why executor stalls evade every CPU alert you own.

Spawn_blocking is the escape hatch. It runs a FnOnce closure on a dedicated blocking pool that defaults to 512 threads, returning a JoinHandle you await like any future. Move JSON parsing of 4MB blobs, bcrypt verification, image resizing, and legacy blocking client calls there. The closure must be Send and static, so move owned data in and return owned results out. Keep each closure under a few seconds: the blocking pool grows on demand but unbounded queues of 30-second closures still exhaust 512 threads and stall subsequent spawn_blocking calls behind them.

Block_in_place is the specialist alternative available only on multi_thread. It transitions the current worker into a blocking thread temporarily, letting the scheduler migrate its tasks elsewhere without a pool hop. That saves a context switch for sub-millisecond blocking operations like brief file metadata checks. It panics on current_thread, must never nest, and still counts against blocking capacity. Most teams should default to spawn_blocking and reserve block_in_place for measured hot paths where the pool hop shows up in profiles.

The audit that finds violations is mechanical. Grep for std::thread::sleep, std::sync::Mutex, std::fs, and blocking library calls inside async fns, then add a clippy gate denying them in async contexts. One 200-route service found 9 sites in an afternoon: two sleeps in retry paths, three std mutexes in caches, two CSV parses on workers, and two blocking S3 client calls. Moving all nine to spawn_blocking or async clients removed the last recurring p99 spike with zero hardware changes, and the clippy gate has caught 14 regressions since.

Size the blocking pool deliberately. The 512 default fits most services, but NFS-backed workloads and slow subprocess farms need explicit caps via max_blocking_threads plus queue-depth metrics. When spawn_blocking latency climbs while worker poll times stay flat, the blocking pool is the bottleneck, not the scheduler. That split diagnosis saves hours: worker metrics point at async code, blocking-pool metrics point at offloaded work, and conflating them sends you tuning the wrong pool.

Filesystem and DNS work belongs off workers even when it looks fast. A p50 stat call at 40us becomes p99 at 12ms under disk pressure, and that 12ms holds a worker shared by thousands of tasks. Tokio's async fs and ToSocketAddrs already route through the blocking pool internally, so prefer them over their std twins in async code. Direct std::fs reads inside handlers are the most common audit finding after sleeps and mutexes, and each one is a latency landmine waiting for the first slow disk.

CPU intensity has a threshold worth internalizing. Anything under roughly 100us, like small JSON serialization or header hashing, is cheaper inline than paying the spawn_blocking hop. Anything over a millisecond, like bcrypt, image decode, or 4MB parses, belongs on the blocking pool without debate. The band between is judgment: profile both placements under load and keep whichever holds p99 lower. Document the decision next to the call so the next engineer does not re-litigate it during a refactor.

Pool saturation signals differ from worker stalls in exactly one metric: spawn_blocking queue latency climbs while worker poll times stay flat. Export both, and alert on each independently so the page names the right pool. Raising max_blocking_threads without checking downstream capacity merely moves the queue into the database or NFS server, so pair pool growth with downstream concurrency limits. The blocking pool absorbs bursts; it cannot create capacity the dependency lacks.

use std::time::Duration;

#[tokio::main]
async fn main() {
    // CPU-heavy parse leaves workers free: it runs on the blocking pool.
    let parsed = tokio::task::spawn_blocking(|| heavy_parse(&vec![1u8; 4096]))
        .await
        .expect("blocking task");
    println!("entries {parsed}");
    // Async sleep parks the task; std sleep would park the worker.
    tokio::time::sleep(Duration::from_millis(10)).await;
}

fn heavy_parse(bytes: &[u8]) -> usize {
    // Simulate serde_json / bcrypt / image work: blocking, owned in and out.
    let mut n = 0usize;
    for (i, b) in bytes.iter().enumerate() {
        n = n.wrapping_add((*b as usize).wrapping_mul(i % 97));
    }
    n % 61000
}

Send Across .await: Locks, Guards, and Primitives

Tokio::spawn demands Future + Send because workers steal tasks across threads. A future is Send only if every live value held across an await is Send. That single rule explains every spawn rejection you will meet. Rc and RefCell are !Send. std::sync::MutexGuard is !Send because unlocking must happen on the locking thread in some implementations. A tokio::sync::MutexGuard is Send because it unlocks from any thread. Hold the wrong guard across await and the whole future turns !Send, failing at the spawn call with an error note longer than your function.

The fix pattern is always the same: shrink the !Send region. Scope the Rc computation before the first await, clone into owned Strings, and await with only Send values live. Replace Rc with Arc for shared ownership, RefCell with tokio::sync::Mutex or RwLock for shared mutation, and std guards with tokio guards when the critical section must span await. Drop guards explicitly before await when the protected data is not needed past that point. Re-run cargo check after each change: the Send error evaporates the moment no !Send value crosses an await.

Primitive choice follows the hold pattern. Use tokio::sync::Mutex when the critical section awaits, tokio::sync::RwLock when reads dominate and occasionally span await, std::sync::Mutex only for microsecond sections with no await inside, and parking_lot only if you have measured contention and accept its blocking semantics off workers. For counters, prefer AtomicU64 over any lock. For one-writer-many-reader snapshots, prefer watch channels over locks entirely: readers clone an Arc without ever acquiring anything. Locks coordinate; channels publish; atomics count. Mixing the idioms causes the stalls.

Deadlock shapes in async code look different from threaded code. Awaiting while holding a tokio mutex that the awaited task also needs deadlocks deterministically, and single-threaded runtimes make it permanent. Lock ordering across two mutexes in different orders deadlocks under contention. The defenses are boring and effective: never hold a lock across a call that acquires another lock, establish a global lock order documented at the state struct, and add try_lock with timeout metrics on hot paths so ordering bugs surface as error-rate blips instead of frozen workers.

When the compiler rejects your spawn, read the note bottom-up: it names the !Send type and the await that crosses it. That note is a gift. In C++ the same bug ships and corrupts memory in production quarters later. In Rust it fails in four seconds with a file and line. Thank the borrow checker, scope the guard, and move on.

Scoped threads and borrowed data interact badly with spawn, and the compiler tells you so. Spawn requires static futures, which means borrowed stack references cannot cross into a spawned task. Either move owned data in or use scoped task patterns within a single function body where join! borrows safely. Reaching for unsafe to smuggle a reference into spawn trades a four-second compiler error for unbounded production risk, and every incident review that found such code wished the author had cloned the 200 bytes instead.

Lock granularity follows the same shrink instinct. A single global RwLock around all tenant state serializes every request through one cache line, while per-tenant locks or sharded maps spread contention across cores. Measure lock hold time with histogram timers: holds above 50us at high rps deserve sharding or snapshot publication via watch. The fastest lock is the one never acquired, and watch snapshots plus atomics eliminate entire lock categories rather than merely shortening them.

Atomic-first design removes whole lock categories. Request counters, sequence numbers, and shutdown flags fit in Atomics with Ordering semantics weaker engineers fear needlessly: Relaxed for pure counters, Acquire-Release for flags guarding data. Replacing a hot mutex-guarded counter with fetch_add cut a gateway hot path from 180ns to 24ns per request in one profile. Reach for locks only when invariants span multiple fields, and even then consider a channel-owned state task first.

use std::sync::Arc;
use std::time::Duration;

#[tokio::main]
async fn main() {
    let state = Arc::new(tokio::sync::RwLock::new(Cache {
        entries: 61000,
    }));
    let h = tokio::spawn(reader(state.clone()));
    // Short write: guard dropped before any await.
    {
        let mut w = state.write().await;
        w.entries += 1;
    }
    println!("{}", h.await.expect("reader"));
}

#[derive(Clone)]
struct Cache {
    entries: u64,
}

async fn reader(state: Arc<tokio::sync::RwLock<Cache>>) -> String {
    // Clone out, drop guard, then await freely: future stays Send.
    let n = { state.read().await.entries };
    tokio::time::sleep(Duration::from_millis(10)).await;
    format!("entries={n}")
}

Testing Async Code: tokio::test, Paused Time, Mocks

Async tests need a runtime, and #[tokio::test] builds one per test with minimal fuss. Default flavor is current_thread, which gives deterministic task ordering that multi_thread scrambles. Declare #[tokio::test(flavor = "multi_thread", worker_threads = 2)] only for tests that genuinely need cross-thread spawning or Send-boundary coverage. Keep the rest on current_thread so failures reproduce instead of flaking once per twenty runs on CI.

Paused time is the cheat code for timer-heavy logic. With #[tokio::test(start_paused = true)], sleeps, timeouts, and intervals never touch the wall clock: advance them explicitly with tokio::time::advance(Duration::from_secs(30)).await. A retry suite with exponential backoff that took 47 seconds of real sleeps finishes in 300ms of virtual time with exact assertions on each deadline. Without paused time those suites either sleep for real and slow CI to a crawl, or skip timer coverage and miss the timeout bug that pages you.

Structure tests around three layers. Unit-test pure logic synchronously with no runtime at all: parsers, deadline math, backoff tables. Runtime-test orchestration with tokio::test and mocked IO: channels instead of sockets, oneshot replies instead of vendors, watch snapshots instead of config services. Integration-test the real stack sparingly with axum::serve on ephemeral ports or testcontainers for databases, asserting timeouts and shutdown rather than business logic. The pyramid keeps the suite under 10 seconds while still covering the race that matters.

Flakiness almost always traces to real time or shared global state. Replace tokio::time::sleep in assertions with timeout-wrapped joins that fail fast instead of hanging CI for 10 minutes. Give each test its own channel pair and state instance instead of sharing a static pool. Run with --test-threads=4 and repeat five times before declaring victory: cargo test -- --test-threads=4 followed by four reruns catches the ordering bug that passes solo. Tracing subscriber init must use try_init or a per-test dispatcher, or parallel tests panic on global logger setup.

Assert the concurrency properties directly. Spawn three mocked vendors, join!, and assert wall time under 300ms to lock overlap in. Race a hung mock against a 100ms timeout and assert the fallback branch won. Fill a bounded channel to capacity and assert try_send fails instead of growing memory. These are the tests that would have caught every incident in this guide before it shipped, and each one runs in milliseconds under paused time.

Property-style concurrency tests catch what example-based tests miss. A test that spawns N producers against a bounded channel with random consumer delays and asserts memory stays under a cap will find the backpressure hole that three hand-written scenarios overlooked. Tokio's test runtime plus deterministic seeds for the delay generator keeps such tests reproducible. Run them nightly rather than per-commit if they take seconds, and keep the fast deterministic trio of overlap, fallback, and shed assertions in the per-commit suite.

CI configuration for async suites is small but load-bearing. Pin the Tokio version with the rest of the lockfile so timer semantics never drift silently, run the suite with a fixed test-thread count to stabilize ordering-sensitive tests, and set a global timeout that fails fast instead of hanging the pipeline for an hour on a deadlocked channel test. One team cut median CI from 11 minutes to 4 by splitting the slow integration job with real sockets from the sub-second unit job, keeping per-commit feedback tight while nightly covered the full matrix.

Load-shape coverage beats line coverage for async suites. A suite with 90% line coverage but zero tests above 10x burst load misses every backpressure bug by construction. Add one burst test per pipeline that multiplies producers tenfold for 5 seconds and asserts memory caps plus shed counts. Those tests run in seconds under paused time with mocked consumers, and they catch the queue-growth bug that line coverage never sees.

fn backoff(attempt: u32) -> std::time::Duration {
    // Pure logic: unit-test without any runtime.
    std::time::Duration::from_millis(50 * 2u64.pow(attempt.min(5)))
}

#[cfg(test)]
mod tests {
    use super::*;
    use std::time::Duration;

    #[test]
    fn backoff_table() {
        assert_eq!(backoff(0), Duration::from_millis(50));
        assert_eq!(backoff(1), Duration::from_millis(100));
        assert_eq!(backoff(5), Duration::from_millis(1600));
    }

    #[tokio::test(start_paused = true)]
    async fn retry_respects_deadline() {
        let op = async {
            tokio::time::sleep(Duration::from_secs(30)).await;
            "slow"
        };
        tokio::time::advance(Duration::from_secs(31)).await;
        assert_eq!(op.await, "slow");
    }

    #[tokio::test]
    async fn fallback_wins_race() {
        let out = tokio::select! {
            v = hung_vendor() => v,
            _ = tokio::time::sleep(Duration::from_millis(50)) => String::from("fallback"),
        };
        assert_eq!(out, "fallback");
    }

    async fn hung_vendor() -> String {
        tokio::time::sleep(Duration::from_secs(60)).await;
        String::from("hung")
    }
}

fn main() {
    println!("backoff(2) = {:?}", backoff(2));
}