Rust Axum API: Router, State, and Tower Layers That Scale

Your first Axum service will feel suspiciously easy. A dozen lines map GET /health to an async function, cargo run binds port 3000, and curl returns ok before your coffee cools. You will add a JSON route, derive Deserialize, and watch the compiler reject malformed handlers before they ever boot. It feels like the framework does the boring parts for you, and it doesn't until traffic arrives.

Then production traffic arrives and the boring parts bite back. A greedy /{id} route swallows /health and your Kubernetes probes fail. A vendor slows to 8 seconds, the router has no timeout layer, and 512 workers park until the autoscaler triples your bill. Clients post valid JSON without the content-type header and eat 415s nobody reproduces locally. None of these fail in unit tests, because none of them are handler bugs. They're wiring bugs, and Axum leaves wiring to you.

This guide closes that gap between demo router and production API. You will learn how method routing and nesting compose, why extractor order is load-bearing, and how Arc state with FromRef substates keeps pools cheap. You will stack TraceLayer and TimeoutLayer in the order that survives incidents, map every error variant to HTTP with thiserror plus IntoResponse, and inject sqlx pools so handlers stay pure functions of typed inputs.

You will also see the outage that motivates each rule. A pricing API with no TimeoutLayer climbed from 120ms to 840ms p99 over 26 minutes while 5xx alerts stayed silent. A per-request pool clone added 12ms per call and exhausted 100 connections at 2k rps. A missing graceful shutdown cut in-flight checkouts on every deploy. Each story ends with the exact layer, extractor, or shutdown hook that prevents the sequel.

By the end you'll have a complete build checklist: versioned routers, validated extractors, uniform errors, bounded bodies, pooled databases, drain-on-shutdown, and tower ServiceExt tests that lock every path. Ship that stack and your API stays boring while traffic triples, even when vendors don't cooperate.

Router and Method Routing: Nesting That Survives Growth

Axum's Router is a macro-free radix tree mapping method plus path to handlers, and its 0.8 path syntax uses curly braces: /users/{id} captures one segment, /files/{*rest} captures the remainder. Method routers compose with get, post, put, patch, and delete on the same path, so .route("/users", get(list).post(create)) keeps one path with two verbs. Routers nest with .nest("/v1", v1_router) for versioned prefixes and merge with .merge(other) for feature modules, which lets teams own sub-routers independently and compose them at main without merge conflicts.

Route ordering is the rule that prevents midnight pages. Axum matches in registration order among overlapping patterns, so a greedy /{id} registered before /health captures the health probe and returns item-handler 404s. Kubernetes then marks every pod unready and the deploy stalls. Register static segments first, parameterized segments next, and wildcards last, then add a route-table test asserting /health, /metrics, and /readyz each return their own handler. That test costs six lines and has prevented more outages than any other test in reviewed codebases.

State typing threads through every composition choice. Router<S> carries its missing-state type until .with_state(state) provides it, producing the Router<()> that axum::serve accepts. Combining routers with nest or merge requires identical state types, so define one AppState for the binary and use FromRef substates for modules needing slices. Annotate inner builders as Router<AppState> when defined in separate functions so inference never silently defaults to Router<()> and fails at the final serve call with a confusing trait error.

Method routing also covers the unglamorous verbs that production needs. Register HEAD alongside GET for link checkers, OPTIONS via CorsLayer for browsers, and a fallback with .fallback(not_found) returning structured JSON instead of an empty 404 body. MethodNotAllowed surfaces automatically when the path matches but the verb does not, which clients can distinguish from true 404s. Explicit fallbacks turn mystery empty responses into parseable errors with request IDs attached.

Versioning strategy deserves a decision before the second release. Path versioning with /v1 and /v2 nests keeps routing explicit and lets old and new handlers share extractors and state. Header versioning hides the split and complicates every log query. Most Axum teams path-version major boundaries and evolve minor changes additively inside one version, deprecating fields with serde defaults rather than forking routers. Whatever you pick, write it in the API guide so clients never guess.

Serve wiring closes the chapter with two lines that matter. Bind tokio::net::TcpListener on 0.0.0.0:3000, then axum::serve(listener, app).await drives Hyper with graceful IO. That pair replaces the old Server::bind builder from pre-0.6 releases, and upgrading codebases sometimes still carry the legacy shape behind compatibility shims. Standardize on the listener plus serve pair, add with_graceful_shutdown around it, and the router you tested is exactly the router that serves.

Growth hygiene keeps the tree fast past 200 routes. Split domain routers per module file, merge them once at main, and keep per-route layers minimal since each layer wraps matching cost. Benchmarks show routing overhead stays under 2us per request even at 500 routes, so slowness past that scale points at extractors or middleware, never the tree. When a route misbehaves, enable matched-path logging to see exactly which pattern won before blaming the handler.

Fallback routing handles the long tail of client mistakes gracefully. A .fallback handler returning AppError::NotFound JSON with the requested path echoed plus a link to versioned docs turns dead ends into guidance. Log fallback hits by path prefix weekly: a rising /v1/users-list slug reveals clients guessing plurals, fixed with a redirect rather than a ticket. Small courtesies at the routing edge compound into fewer integration threads.

use axum::{Router, http::StatusCode, routing::{get, post}};

#[tokio::main]
async fn main() {
    let v1 = Router::new()
        .route("/health", get(health))
        .route("/users", get(list_users).post(create_user));
    // Specific routes first, params after, wildcards last.
    let app = Router::new()
        .nest("/v1", v1)
        .route("/files/{*rest}", get(serve_file))
        .fallback(not_found);
    let listener = tokio::net::TcpListener::bind("127.0.0.1:3000")
        .await
        .expect("bind");
    axum::serve(listener, app).await.expect("serve");
}

async fn health() -> &'static str {
    "ok"
}

async fn list_users() -> &'static str {
    "[]"
}

async fn create_user() -> StatusCode {
    StatusCode::CREATED
}

async fn serve_file(axum::extract::Path(rest): axum::extract::Path<String>) -> String {
    format!("file:{rest}")
}

async fn not_found() -> (StatusCode, String) {
    (StatusCode::NOT_FOUND, String::from("unknown route"))
}

Extractors: Path, Query, Json, and Rejection Codes

Extractors are Axum's signature idea: handler arguments implementing FromRequest or FromRequestParts that parse the request into typed values before your code runs. Path<Uuid> deserializes segments, Query<Filter> parses query strings, Json<Order> buffers and deserializes bodies, and State<Arc<AppState>> injects shared resources. Each rejection maps to an HTTP status automatically: bad path types give 400 or 404, bad query gives 400, missing JSON content-type gives 415, oversized bodies give 413. Validation that Express spreads across middleware lives in the type signature.

Ordering rules are load-bearing because the body streams once. Extractors running left to right must place body consumers last: method, headers, State, and Query implement FromRequestParts and can sit anywhere, while Json, String, Bytes, and Form implement FromRequest and must be the final argument. Axum enforces this at compile time, so swapping Json before State fails to build rather than corrupting data. Internalize the chant of parts before body and every handler signature writes itself correctly the first time.

Path shapes deserve deliberate types. Single-segment Path<Uuid> parses and validates the identifier format in one step, multi-segment Path<(u32, String)> destructures pairs, and RawPathParams preserves undecoded text for proxying. Prefer narrow types like Uuid over String plus manual parsing: the extractor rejects malformed IDs with 400 before your database sees them, deleting a validation branch and its tests. Document each path's shape in OpenAPI from the same struct so docs never drift from parsing.

Query handling replaces hand-rolled validation most teams regret. A Filter struct with Option<String> search, Option<u32> page, and serde defaults turns ?search=shoe&page=abc into a 400 with a clear message instead of a panic on parse().unwrap(). Extra query keys are ignored by default, which keeps clients forward-compatible, while deny_unknown_fields opts strict endpoints into exact contracts. One team deleted 300 lines of manual checks by moving to Query structs and watched a whole 400-error triage class vanish from support queues.

Json errors need client-friendly mapping because the default 415 and 422 bodies confuse integrators. Missing content-type yields 415, malformed syntax yields 422 with a terse message, and the fix is usually a header the client forgot. Wrap Json in a custom extractor or map its rejection to your AppError with an example curl in the message body. Integrators copy the example, headers get fixed in minutes, and your on-call stops explaining content-types at midnight.

Custom extractors extend the pattern without breaking it. Implement FromRequestParts for header-derived auth contexts and FromRequest for validated bodies like a CheckedJson<T> that enforces business rules after serde. Keep each extractor to one concern so rejection codes stay unambiguous: parsing failures versus business-rule failures should map to different statuses. Two focused extractors compose better than one clever extractor with four failure modes.

Testing extractors directly locks contracts cheaply. Unit-test Query deserialization from sample strings, assert Path rejects malformed UUIDs, and post real JSON with and without content-type headers through ServiceExt::oneshot. These tests run in milliseconds, need no sockets, and catch the boundary regressions that integration suites find only after deploy. Every extractor in production deserves at least one rejection test proving the bad input fails loudly.

Nested rejection handling keeps extractor errors consistent across versions. A single rejection-to-AppError mapper shared by /v1 and /v2 routers guarantees identical 415 and 422 shapes everywhere, so mobile clients parse once. Test the mapper with table-driven cases per extractor rather than per route: one case per rejection kind covers every route using that extractor. Shared mapping plus table tests is how three engineers hold 200 routes without drift.

use axum::extract::{Path, Query, State};
use axum::{Json, Router, routing::get};
use serde::{Deserialize, Serialize};
use std::sync::Arc;
use uuid::Uuid;

#[derive(Clone)]
struct AppState {
    version: &'static str,
}

#[derive(Deserialize)]
struct Filter {
    search: Option<String>,
    page: Option<u32>,
}

#[derive(Serialize)]
struct Item {
    id: Uuid,
    name: String,
}

async fn show(
    State(s): State<Arc<AppState>>,
    Path(id): Path<Uuid>,
    Query(f): Query<Filter>,
) -> Json<Item> {
    let _ = (s.version, f.page);
    Json(Item { id, name: String::from("demo") })
}

#[tokio::main]
async fn main() {
    let state = Arc::new(AppState { version: "1.4.2" });
    let app = Router::new()
        .route("/items/{id}", get(show))
        .with_state(state);
    let listener = tokio::net::TcpListener::bind("127.0.0.1:3000")
        .await
        .expect("bind");
    axum::serve(listener, app).await.expect("serve");
}

Shared State: Arc, FromRef Substates, and Pool Wiring

Shared state in Axum lives in one struct injected once with .with_state and read per request through the State extractor. The canonical shape is struct AppState holding a sqlx PgPool, a reqwest Client, typed Config, and a tokio broadcast Sender for events, wrapped in Arc so each request clones a pointer in nanoseconds. Handlers declare State(state): State<Arc<AppState>> and stay pure functions of typed inputs plus shared resources. Construction happens at startup where failures are loud, never per request where they add latency.

FromRef substates keep modules decoupled without splitting state. A billing module needing only DbPool declares State(pool): State<DbPool> while the router holds full AppState, bridged by impl FromRef<AppState> for DbPool cloning the pool field. Derive it with #[derive(FromRef)] under the macros feature for single-field cases. Modules test against minimal substates instead of the whole world, and main still wires one struct. The alternative of five separate extension layers drifts within a quarter; one state struct with substates stays coherent.

Extension versus State is a choice with a clear default. State is compile-checked: extracting an unprovided type fails to build, which catches wiring bugs in CI. Extension is runtime-checked: a missing layer yields 500s in production, which catches nothing until users complain. Prefer State for pools and config, reserve Extension for request-scoped data like trace IDs inserted by middleware. Teams that standardize on State delete a class of deploy-time 500s that Extension allowed through review.

Mutability inside state follows Tokio rules because handlers run on a multi_thread pool. Counters use AtomicU64, caches use tokio::sync::RwLock with short holds, and hot snapshots publish through watch channels instead of locks. Never hold std::sync::Mutex across await inside a handler: the future turns !Send and either fails to compile or stalls workers. Clone Arc generously, lock briefly, await freely, and contention profiles stay flat to 20k rps.

Pool sizing ties state to capacity planning. A sqlx PgPool with max_connections 20 serves roughly 5k simple-query rps before wait queues grow; size from load tests measuring pool wait time, not gut feel. Keep one pool per database role, share it through AppState, and never build pools per request: construction costs 12ms and exhausts 100 connections at 2k rps. Export pool idle, size, and wait gauges so the dashboard shows pressure before p99 moves.

Startup wiring order prevents the half-ready serve. Build config from env, connect pools with a 10-second connect timeout, run migrations, warm caches, then bind the listener and serve. Fail fast before binding so orchestrators restart cleanly instead of probing a half-wired pod. Log pool sizes and feature flags at boot: the first 20 log lines should let any engineer reconstruct exactly what this binary can do.

Testing with state stays fast by substituting fakes at the substate boundary. A handler needing State<DbPool> tests against a sqlx pool pointed at a test database or a trait-object store, constructed once per suite. Pure logic extracts into free functions taking plain values, tested synchronously with no runtime. The handler becomes a thin typed shell that integration tests cover once while unit tests cover the logic fifty times.

Connection-level state like rate-limit counters belongs in layers rather than AppState clones. A GovernorLayer or custom semaphore around expensive routes shares permits independently of handler state, keeping quotas consistent across AppState refactors. Size permits from load tests on the constrained resource, typically GPU inference or legacy SOAP backends, and return 429 with retry-after when exhausted. State holds resources; layers hold policies.

Config reload without restarts completes the state story: watch channels carrying new limits reach handlers in milliseconds while pools stay connected. Readers borrow the latest snapshot without locks, writers publish atomically, and reload tests assert convergence under 100ms.

use axum::{Router, extract::State, routing::get};
use std::sync::Arc;

#[derive(Clone)]
struct AppState {
    pool_label: &'static str,
    config: Config,
}

#[derive(Clone)]
struct Config {
    max_items: usize,
}

impl axum::extract::FromRef<AppState> for Config {
    fn from_ref(s: &AppState) -> Config {
        s.config.clone()
    }
}

async fn version(State(s): State<Arc<AppState>>) -> String {
    format!("pool={} max={}", s.pool_label, s.config.max_items)
}

async fn budget(State(c): State<Config>) -> String {
    format!("max={}", c.max_items)
}

#[tokio::main]
async fn main() {
    let state = Arc::new(AppState {
        pool_label: "primary",
        config: Config { max_items: 100 },
    });
    let app = Router::new()
        .route("/version", get(version))
        .route("/budget", get(budget))
        .with_state(state);
    let listener = tokio::net::TcpListener::bind("127.0.0.1:3000")
        .await
        .expect("bind");
    axum::serve(listener, app).await.expect("serve");
}

Middleware: Tower Layers Ordered for Survival

Axum has no proprietary middleware system: it uses tower::Service layers stacked with .layer(), which means timeouts, tracing, compression, auth, and CORS compose from the shared Tower ecosystem. Each layer wraps the inner service, observing requests on the way in and responses on the way out. Because layers are ordinary Tower services, middleware written for Hyper or Tonic stacks onto Axum unchanged, and Axum middleware works beyond Axum. That portability is the framework's core bet, and it pays off every time a team reuses a vetted auth layer instead of writing one.

Ordering decides what survives the incident you most need to observe. Tracing goes outermost so every outcome including timeouts logs exactly once. Timeout goes outside auth so slow credentials shed instead of holding workers through a broken identity provider. Compression sits innermost near the handler where body sizes are known. CORS precedes auth so preflight succeeds without credentials. Reverse tracing and timeout and your logs go blind precisely when timeouts fire; reverse timeout and auth and a hung IdP parks workers that timeouts were meant to protect.

The survival stack fits in four layers for most APIs. TraceLayer::new_for_http logs method, path, matched route, status, and latency per request with tunable redaction. TimeoutLayer::new(Duration::from_secs(5)) caps every handler including middleware inside it. CorsLayer scopes allowed origins per environment. CompressionLayer shrinks JSON payloads 60 to 80% for browser clients. Add RequestBodyLimitLayer where uploads exist and AuthLayer where identity gates routes. Each layer is one line; the order is the architecture.

Custom middleware has three idioms with rising power. from_fn wraps an async function around requests for auth checks, header injection, and request IDs with minimal boilerplate. from_fn_with_state threads AppState into that function for DB-backed auth. Full Layer plus Service implementations handle buffering, retries, and protocol upgrades needing poll-level control. Start with from_fn, graduate only when the function shape cannot express the behavior, and keep custom layers stateless so they clone cheaply per request.

Scope layers deliberately instead of stacking everything globally. Health and readiness probes skip auth and heavy logging so orchestrators get millisecond answers under load. Webhook receivers get signature verification plus tight body limits but skip compression. Admin routes nest under their own router with IP allowlisting plus audit logging. Per-route .route_layer applies to single paths while .layer covers the router, and mixing the two keeps cross-cutting concerns global with exceptions local.

Middleware testing asserts behavior, not just presence. Send a request through the stacked router with ServiceExt::oneshot and assert the trace header exists, the timeout sheds a hung handler at 5 seconds, and oversized bodies fail with 413. Assert layer order by firing a slow-auth request and confirming the timeout wins: if auth wins, the order is wrong. These tests run in milliseconds and catch the misordering that code review misses because each layer looks correct alone.

Performance cost of layers stays negligible when scoped sanely. Trace overhead lands near 1 to 2us per request, timeout near 200ns when idle, compression pays only on compressible bodies above 1KB. The bill that matters is behavioral: a missing timeout costs 26-minute pileups, a missing trace layer costs 40-minute blind debugs. Measure layer latency with histogram timers per layer during load tests, then stop worrying and keep the survival stack on every public router.

Path prefix stripping with nest plus StripPrefixBuffer matters for mounted sub-APIs. A /admin nest served by a separately versioned router sees trimmed paths internally while outer logs keep full paths, which keeps both routing and observability correct. Test nesting with full external URIs through oneshot rather than trimmed internals so the test matches what clients send. Misaligned prefix tests pass locally and fail behind the real ingress.

use axum::{Router, routing::get};
use std::time::Duration;
use tower_http::{compression::CompressionLayer, timeout::TimeoutLayer, trace::TraceLayer};

async fn health() -> &'static str {
    "ok"
}

async fn slow() -> &'static str {
    tokio::time::sleep(Duration::from_secs(30)).await;
    "slow"
}

#[tokio::main]
async fn main() {
    // Outermost first: trace sees everything, timeout sheds before auth stalls.
    let app = Router::new()
        .route("/health", get(health))
        .route("/slow", get(slow))
        .layer(TraceLayer::new_for_http())
        .layer(TimeoutLayer::new(Duration::from_secs(5)))
        .layer(CompressionLayer::new());
    let listener = tokio::net::TcpListener::bind("127.0.0.1:3000")
        .await
        .expect("bind");
    axum::serve(listener, app).await.expect("serve");
}

Errors to HTTP: thiserror plus IntoResponse

Uniform errors start with one enum and one trait impl. Define AppError with variants like NotFound, Validation(String), Unauthorized, Conflict, UpstreamTimeout, and Db(sqlx::Error), derive thiserror::Error for display plus Debug, then implement IntoResponse mapping each variant to a status code plus a JSON body with code, message, and request ID. Handlers return Result<Json<T>, AppError> everywhere, the ? operator converts library errors through From impls, and clients parse one schema regardless of route. Response-shape drift ends the day the enum ships.

Status mapping follows HTTP semantics teams can defend. Validation and serde rejections become 400, missing rows become 404, auth failures become 401 or 403 by credential versus permission, conflicts become 409, timeouts become 503 with retry-after hints, and unexpected failures become 500 with a redacted message plus a full server-side log. Never leak sqlx error text or backtraces to clients: log the chain with tracing::error and return a stable public message. Support tickets drop measurably once 503s say retry in 5s instead of dumping driver internals.

Thiserror ergonomics keep the enum cheap to grow. Each variant carries exactly the context its mapping needs: Validation(String) for the field message, UpstreamTimeout { dependency: &'static str } for dashboards, Db(#[from] sqlx::Error) for automatic conversion. Adding a variant means adding one match arm plus one test asserting its status and body shape. Without thiserror the same enum costs a manual Display impl reviewers dread touching, so variants accrete as stringly-typed 500s instead.

Rejection integration closes the extractor loop. Json, Query, and Path rejections convert into AppError through a from_rejection constructor or a dedicated rejection handler, so boundary failures speak the same dialect as business failures. A missing content-type becomes AppError::Validation with an example curl in the message rather than a bare 415. Clients fix headers in minutes because the error teaches, and your 415 ticket queue empties permanently.

Logging and metrics key off the same enum without string parsing. Emit a counter app_error_total{variant=..} per mapping and a histogram on handler latency by route plus outcome. Alert on 5xx rate per variant rather than global status codes: a Db spike pages the data team while an UpstreamTimeout spike pages integrations. Structured logs carry variant, request ID, and latency so any error reconstructs in one query. The enum becomes the observability taxonomy, not just the response shape.

Client guidance ships with the mapping, not after it. Document retry behavior per code: 400 never retries, 404 retries only after input changes, 409 retries with new idempotency keys, 503 retries with backoff honoring retry-after. Mobile teams encode these rules once and every endpoint inherits correct behavior. Without the guide each client invents its own retry policy and your 503 becomes a retry storm; with it the same 503 becomes polite backoff.

Testing the mapping is pure and fast. Unit-test each variant's IntoResponse for status plus body code with no runtime, then integration-test two handlers end to end proving the ? operator preserves variants through await chains. Snapshot the JSON schema once so refactors cannot silently rename code to error_code. Sixteen assertions lock the whole dialect, and every future handler inherits coverage by returning the same Result type.

OpenAPI generation from extractor types closes the docs loop. Deriveutoipa ToSchema alongside Deserialize and Serialize so path params, query structs, and error bodies render into one spec file. Serve the spec at /docs with Redoc or Scalar and assert spec freshness in CI by diffing generated output. Docs generated from types cannot drift; docs written by hand always do within two releases.

Pagination envelopes round out response design: data plus next_cursor plus total keep clients stateless across releases. Cursor-based paging over offset paging avoids skipped rows during concurrent writes, which matters most on the feeds clients poll hardest.

use axum::{Json, http::StatusCode, response::IntoResponse};
use serde::Serialize;
use thiserror::Error;

#[derive(Debug, Error)]
enum AppError {
    #[error("item not found")]
    NotFound,
    #[error("invalid input: {0}")]
    Validation(String),
    #[error("upstream timed out")]
    UpstreamTimeout,
}

#[derive(Serialize)]
struct ErrorBody {
    code: &'static str,
    message: String,
}

impl IntoResponse for AppError {
    fn into_response(self) -> axum::response::Response {
        let (status, code) = match &self {
            AppError::NotFound => (StatusCode::NOT_FOUND, "not_found"),
            AppError::Validation(_) => (StatusCode::BAD_REQUEST, "validation"),
            AppError::UpstreamTimeout => (StatusCode::SERVICE_UNAVAILABLE, "upstream_timeout"),
        };
        let body = Json(ErrorBody { code, message: self.to_string() });
        (status, body).into_response()
    }
}

async fn get_item() -> Result<Json<String>, AppError> {
    Err(AppError::NotFound)
}

fn main() {
    let _ = get_item;
    println!("error dialect ready");
}

JSON with serde: Schemas, Validation, and Body Limits

Serde derives turn JSON into typed contracts with two attributes doing most of the work. Derive Deserialize on every input with deny_unknown_fields on strict contracts and defaults on evolvable ones, derive Serialize on every output, and keep input and output structs separate so internal fields never serialize by accident. Field renames with rename_all = "camelCase" match JS clients while Rust fields stay snake_case. The compiler plus serde then validate more than hand-written checks ever will, and OpenAPI schemas generate from the same structs.

Validation splits into syntax versus semantics. Serde handles syntax: types, required fields, and shapes rejected with 422 before your code runs. The validator crate or a CheckedJson extractor handles semantics: email formats, range bounds, cross-field rules like start before end. Keep the layers distinct so error codes stay precise: 422 for malformed shapes, 400 with field messages for rule violations. Merging them into one handler produces vague errors clients cannot act on.

Body limits bound the worst case before buffering. DefaultBodyLimit::max(2 1024 1024) on JSON routers rejects 50MB accidents with 413 instead of allocating them, while RequestBodyLimitLayer covers multipart paths. Size from real payload distributions: p99 request size times four gives headroom without inviting abuse. Log rejected sizes with route tags so legitimate growth, like bulk import endpoints needing 10MB, surfaces as data rather than surprise 413 complaints.

Response serialization deserves the same care as parsing. Return Json<T> directly for 200s, (StatusCode::CREATED, Json<T>) for creations, and Stream or Body for payloads above a few megabytes that should never buffer. Large exports stream rows as newline-delimited JSON with constant memory instead of building a 400MB Vec first. Content-type stays application/json automatically, and clients parse one shape whether the payload is one row or one million.

Versioning JSON without breaking clients relies on additive evolution. Add optional fields with defaults, never rename required fields in place, and deprecate with documentation plus sunset headers before removal. Serde's default attribute makes old clients survive new fields and new clients survive missing old fields during rollout windows. Breaking renames ship behind /v2 routers sharing extractors but not response structs, so the old contract freezes while the new one evolves.

Error payloads for JSON failures should teach. Map missing-field 422s to messages naming the field plus an example body, and missing content-type 415s to the exact curl header to add. Integrators fix themselves from the message without opening tickets. One platform measured a 60% drop in integration Slack threads after adding example bodies to its three most common rejection messages.

Performance tuning for JSON is rarely the bottleneck teams fear. Simd-accelerated serde_json parses typical 2KB payloads in under 20us, dwarfed by database time. The wins that matter are structural: reject oversized bodies at the layer, stream large responses, and avoid double serialization through intermediate Value when a typed struct suffices. Profile before optimizing parsers; the pool wait queue is almost always longer than the parse time.

Streaming uploads bypass Json buffering entirely for memory safety. Accept Body streams with explicit byte caps and backpressure into object storage, reporting progress via channels rather than buffering 500MB in RAM. Pair stream handlers with tight timeouts plus auth-first ordering so unauthenticated giants fail before consuming bandwidth. Buffer what fits in limits; stream everything above them.

Idempotency-Key handling on POST creates turns retries safe: store key plus result hash for 24 hours and replay stored responses on duplicates. Clients retry freely, doubles vanish, and support threads about duplicate charges end. Record key plus fingerprint at creation time so replays stay exact across deploys.

use axum::{Json, Router, http::StatusCode, routing::post};
use serde::{Deserialize, Serialize};
use axum::extract::DefaultBodyLimit;

#[derive(Deserialize)]
struct CreateItem {
    name: String,
    #[serde(default = "default_qty")]
    qty: u32,
}

fn default_qty() -> u32 {
    1
}

#[derive(Serialize)]
struct Item {
    id: u64,
    name: String,
    qty: u32,
}

async fn create(Json(input): Json<CreateItem>) -> (StatusCode, Json<Item>) {
    let item = Item { id: 7, name: input.name, qty: input.qty };
    (StatusCode::CREATED, Json(item))
}

#[tokio::main]
async fn main() {
    let app = Router::new()
        .route("/items", post(create))
        .layer(DefaultBodyLimit::max(2 * 1024 * 1024));
    let listener = tokio::net::TcpListener::bind("127.0.0.1:3000")
        .await
        .expect("bind");
    axum::serve(listener, app).await.expect("serve");
}

Database Pools: sqlx Injection Without Handler Clutter

Sqlx pools inject through AppState exactly like any shared resource, with compile-time query checking as the bonus. Startup builds PgPoolOptions with max_connections near 20, min_connections 2, acquire_timeout 3 seconds, and idle_timeout 10 minutes, connects once with a 10-second overall deadline, runs migrations via sqlx::migrate!, then stores the pool in AppState behind Arc. Handlers receive State(pool): State<PgPool> through FromRef and call sqlx::query_as! with checked SQL. Connection failures happen at boot where they page clearly, never mid-request where they masquerade as handler bugs.

Checked queries shift a whole bug class left. query_as! validates SQL against the live schema at compile time with DATABASE_URL set, catching renamed columns before CI finishes. Runtime queries via query_as without the bang cover dynamic SQL with identical runtime shapes. One team renamed a users column and fixed 9 call sites from compiler errors in an hour; the unchecked equivalent would have shipped 9 runtime 500s discovered over a week. Keep DATABASE_URL pointed at a Curtis-fresh schema dump in CI so checks never go stale.

Transaction discipline keeps handlers honest. Single-statement reads use pool queries directly with per-query timeouts via sqlx's statement timeout. Multi-statement writes acquire one connection, begin, execute, and commit explicitly, rolling back on any Err through the ? operator into AppError::Db. Never hold transactions across vendor calls: the connection sits idle while the vendor stalls, starving the pool exactly like the missing-timeout incident. Fetch data, release the connection, then call vendors with owned values.

Pool observability predicts exhaustion before p99 moves. Export size, idle, and wait-queue depth, alert when wait time exceeds 50ms for 2 minutes, and graph pool waits beside handler latency so the correlation is obvious. At 3k rps against 20 connections, simple queries queue gracefully; complex analytics on the same pool starves interactive traffic. Split OLTP and OLAP pools by role when wait queues diverge, routing reports to read replicas through a second pool in AppState.

Migration strategy pairs with pool wiring for safe deploys. Run migrations as a Kubernetes init container or a pre-serve step that exits nonzero on failure, blocking the rollout before new code serves old schemas. Keep migrations backward-compatible across one release: add columns nullable first, backfill, then enforce constraints next release. Expand-migrate-contract sequencing lets rolling deploys mix old and new pods without 500s, which matters most during the checkout-path changes nobody wants to rollback.

Testing database handlers layers by cost. Pure SQL asserts run against a template-cloned test database per suite with serial_test or per-test schemas to avoid cross-talk. Handler tests use ServiceExt::oneshot against routers wired to the test pool, asserting status codes plus JSON shapes. Unit tests cover mapping logic with fake rows and no database at all. The pyramid keeps per-commit suites under 30 seconds while nightly runs the full migration matrix from empty to latest.

Failure mapping turns driver errors into client guidance. Unique violations become 409 with the conflicting field named, foreign-key violations become 400 with the dangling reference explained, connection timeouts become 503 with retry-after. Match on sqlx::Error variants plus database error codes rather than string-matching messages that change across versions. Clients handle 409 by prompting for new input instead of retrying blindly, which ends duplicate-submit support threads.

Queryable pool metrics decide scaling before users notice. Alert when acquire waits exceed 100ms p99 for 5 minutes, scale read replicas when replica lag plus pool waits correlate, and split pools by workload when analytics queries inflate interactive waits. One dashboard with pool size, idle, waits, and slow-query count per role replaces four separate war rooms with one glance.

use axum::{Json, Router, extract::State, routing::get};
use serde::Serialize;

#[derive(Clone)]
struct AppState {
    pool_label: &'static str,
}

impl axum::extract::FromRef<AppState> for &'static str {
    fn from_ref(s: &AppState) -> &'static str {
        s.pool_label
    }
}

#[derive(Serialize)]
struct Row {
    id: i64,
    name: String,
}

async fn list(State(label): State<&'static str>) -> Json<Vec<Row>> {
    // In production: sqlx::query_as!("select id, name from items").fetch_all(&pool).await
    Json(vec![Row { id: 1, name: format!("via {label}") }])
}

#[tokio::main]
async fn main() {
    // Production wiring: PgPoolOptions::new().max_connections(20).connect(&url).await
    let state = AppState { pool_label: "primary" };
    let app = Router::new().route("/rows", get(list)).with_state(state);
    let listener = tokio::net::TcpListener::bind("127.0.0.1:3000")
        .await
        .expect("bind");
    axum::serve(listener, app).await.expect("serve");
}

Graceful Shutdown: Drains That Make Deploys Invisible

Graceful shutdown drains in-flight requests before exit so rolling deploys never cut responses mid-write. The shape is axum::serve(listener, app).with_graceful_shutdown(shutdown_signal()).await where shutdown_signal awaits SIGTERM plus SIGINT via tokio::signal and optionally a Kubernetes preStop hook. In-flight handlers run to completion within a 30-second terminationGracePeriodSeconds window while the readiness probe fails fast, pulling the pod from rotation. New pods pass probes before old pods die, and users see zero failed requests across the rollout.

Signal handling needs both orchestrator and local paths. Tokio::signal::unix::signal catches SIGTERM from Kubernetes and SIGINT from Ctrl-C, combined with tokio::select! so either triggers drain. Local development sends SIGINT, CI sends SIGTERM to containers, and systemd sends SIGTERM to binaries: covering both signals handles all three without flags. Log shutdown start with in-flight counts so the deploy timeline shows drain duration per pod rather than a silent gap.

Readiness versus liveness separation makes drains safe. Readiness at /readyz checks pool connectivity plus migration currency and fails the moment shutdown starts, removing the pod from endpoints. Liveness at /health stays green during drain so the kubelet does not kill a draining pod mid-checkout. Merging both into one probe causes exactly the wrong behavior: either traffic keeps arriving during drain or the kubelet murders in-flight work. Two probes with distinct checks is the entire trick.

Connection and pool teardown order matters after the listener stops. Drop the watch shutdown broadcaster first so background tasks exit their loops, await their JoinHandles with a 10-second timeout, then close sqlx pools with a 5-second acquire grace so in-flight queries finish. Reverse the order and background tasks panic on closed pools during every deploy, logging errors that look like crashes but are just teardown races. Explicit teardown reads longer but deploys silently.

Timeout caps keep drains from hanging forever. Wrap the drain in tokio::time::timeout(Duration::from_secs(25)) inside a 30-second grace period, aborting stragglers that exceed it and logging their routes for follow-up. A vendor-hung handler without its own timeout becomes a drain-hung pod without the cap, delaying the whole rollout. Per-handler TimeoutLayer plus a global drain cap gives two chances to shed before the kubelet resorts to SIGKILL.

Deploy verification proves invisibility instead of assuming it. Run a constant 500-rps load with oha or k6 across kubectl rollout restart, asserting zero 502s and p99 within 10% of baseline throughout. Record drain durations per pod and alert when p95 drain exceeds 15 seconds: growing drains predict the next failed rollout. Teams that graph deploys like releases catch shutdown regressions in the deploy they ship, not three deploys later.

Local shutdown testing is one command away. Boot the service, curl a slow endpoint in the background, send SIGTERM to the PID, and confirm the slow request completes while new curls refuse. Repeat with a database write mid-flight and verify commit-or-rollback rather than partial rows. These two manual tests take ninety seconds and validate more than a page of theory about signal disposition.

Zero-downtime migration coordination uses health signals as the rollout gate. New pods run migrations in init containers, report ready only after schema checks pass, and old pods drain against the compatible schema window. Expand-migrate-contract discipline means no release ever requires simultaneous old-code plus new-schema downtime. The health endpoint is the contract both sides honor.

PreStop hooks sleeping 5 seconds before SIGTERM let endpoints propagate removal before connections close. That pause plus failing readiness first is what turns rolling restarts from 502 generators into nonevents at 500 rps. Verify the pause with load tests asserting zero refused connections per rollout.

use axum::{Router, routing::get};

async fn health() -> &'static str {
    "ok"
}

async fn readyz() -> &'static str {
    "ready"
}

async fn shutdown_signal() {
    let ctrl_c = async {
        tokio::signal::ctrl_c().await.expect("ctrl-c");
    };
    #[cfg(unix)]
    let term = async {
        tokio::signal::unix::signal(tokio::signal::unix::SignalKind::terminate())
            .expect("sigterm")
            .recv()
            .await;
    };
    #[cfg(not(unix))]
    let term = std::future::pending::<()>();
    tokio::select! {
        _ = ctrl_c => {},
        _ = term => {},
    }
    println!("draining");
}

#[tokio::main]
async fn main() {
    let app = Router::new()
        .route("/health", get(health))
        .route("/readyz", get(readyz));
    let listener = tokio::net::TcpListener::bind("127.0.0.1:3000")
        .await
        .expect("bind");
    axum::serve(listener, app)
        .with_graceful_shutdown(shutdown_signal())
        .await
        .expect("serve");
}

Request Lifecycle: Limits, Compression, and Request IDs

Every request traverses the same lifecycle and each stage needs one explicit control. Connection acceptance inherits Tokio's TCP backlog plus Hyper's HTTP/1 and HTTP/2 settings. Routing matches method plus path in microseconds. Middleware observes and transforms. Extractors parse and validate. Handlers run business logic. Responses serialize and compress. Shutdown drains. Leaving any stage unconfigured lets defaults decide production behavior, and Hyper defaults favor throughput over protection: unbounded patience, generous buffers, no IDs. Production config tightens each stage deliberately.

Body limits belong at two levels with distinct jobs. DefaultBodyLimit caps buffered extractors like Json at 2MB with a 413 on excess, while RequestBodyLimitLayer caps streaming bodies before they reach handlers. Set both from payload distributions rather than round numbers: bulk import routes get 10MB with auth, public JSON stays at 2MB. Oversized attempts log route plus size so growth requests arrive as data. Without limits one 50MB accident buffers fully and spikes RSS per request until the pod tips.

Compression trades CPU for bandwidth with clear break-evens. CompressionLayer with gzip or zstd shrinks JSON 60 to 80%, paying roughly 30us per 2KB response on modern cores. Enable it for browser and mobile traffic where bytes cost real money, predicate on content-type json plus a 1KB minimum so tiny health probes skip the work. Internal service-to-service JSON often skips compression on 10Gb networks where CPU matters more than bytes. Measure both sides per route class instead of toggling globally.

Request IDs thread observability through every stage. A from_fn middleware mints or propagates x-request-id, stores it in extensions, and the TraceLayer span plus AppError bodies include it. Logs, error responses, and upstream calls share one ID, so a user report reconstructs in a single query. Generate UUIDv7 for time-ordering when clients omit the header, honor client values under 64 chars when present, and reject absurd lengths at the edge. The ID costs nanoseconds and saves hours per incident.

Header hygiene prevents the slow weirdness that evades tests. Enforce content-type on JSON posts so 415s teach instead of confusing, normalize trailing slashes with a redirect or strict router so /users and /users/ never diverge, and cap header counts plus sizes via Hyper settings against slow-loris accumulation. CORS origins stay explicit per environment: permissive in dev, exact origins in prod. Each rule is one line; together they end the class of it-works-in-curl-but-not-the-browser tickets.

HTTP/2 and keep-alive tuning finish the lifecycle for high-scale APIs. Enable http2 behind TLS where clients support multiplexing, keep idle timeouts near 75 seconds to recycle stale connections, and set max frame sizes from payload profiles. Hyper handles protocol negotiation once configured, and Axum inherits it through serve. Load-test both protocols separately: multiplexed streams shift bottlenecks from connections to handlers, which changes optimal worker counts slightly.

Timeout layering nests deadlines like Russian dolls. Edge TimeoutLayer at 5 seconds bounds total handler time, per-vendor tokio timeouts at 250 to 800ms bound each dependency, and sqlx statement timeouts near 2 seconds bound each query. The innermost deadline fires first, fallbacks compose outward, and the edge layer catches whatever inner code forgot. Document the three tiers next to their constructors so future edits preserve the nesting instead of flattening it.

Access logging format deserves versioning like the API itself. Emit JSON lines with request ID, route template, status, latency, plus bytes in and out, sampled at 100% for 5xx and 1% for 2xx at high volume. Pipe them into the same trace backend as handler spans so edge and handler views join on one ID. Log formats changed without notice break more dashboards than API changes do.

Sampling tail traces at 100% for timeouts while keeping successes at 1% focuses storage where diagnosis happens. Retention policies keep full timeout traces 30 days and success samples 7, matching how often each gets queried.

use axum::{Router, http::HeaderMap, routing::get};
use tower_http::{compression::CompressionLayer, limit::RequestBodyLimitLayer};
use axum::extract::DefaultBodyLimit;

async fn headers(headers: HeaderMap) -> String {
    let id = headers
        .get("x-request-id")
        .and_then(|v| v.to_str().ok())
        .unwrap_or("none");
    format!("id={id}")
}

#[tokio::main]
async fn main() {
    let app = Router::new()
        .route("/headers", get(headers))
        .layer(DefaultBodyLimit::max(2 * 1024 * 1024))
        .layer(RequestBodyLimitLayer::new(10 * 1024 * 1024))
        .layer(CompressionLayer::new());
    let listener = tokio::net::TcpListener::bind("127.0.0.1:3000")
        .await
        .expect("bind");
    axum::serve(listener, app).await.expect("serve");
}

Testing APIs: ServiceExt oneshot Plus axum-test Notes

Tower's ServiceExt::oneshot is the fastest correct way to test Axum routers: it drives the stacked router with middleware and state in-process, no sockets required. Build the exact production router with test state, call router.oneshot(Request::builder().uri("/health").body(Body::from("")).unwrap()).await, and assert status plus body bytes. Each test runs in milliseconds, exercises layers including timeouts and body limits, and fails on wiring bugs that handler-only unit tests miss. Route coverage starts here, not at the handler.

Test organization follows the wiring pyramid. Router tests assert every path returns its handler's status: 200 for health, 201 for creates, 404 JSON for unknown routes, 405 for wrong verbs. Extractor tests post malformed JSON, missing headers, and bad UUIDs asserting 415, 400, and 422 shapes. State tests wire fake pools or stub substates proving handlers read shared resources. Layer tests fire slow and oversized requests proving timeouts shed and limits reject. Handler-logic unit tests cover business rules synchronously. Each layer has one job and one file.

State substitution keeps tests hermetic without mocking frameworks. Handlers depending on State<Config> receive test configs with tight budgets, handlers needing pools receive test-database pools migrated fresh per suite, and pure mapping functions receive plain values. Avoid global statics: construct fresh AppState per test so parallel runs never cross-talk. Testcontainers or template-cloned databases give real Postgres semantics at suite cost measured in seconds, worth it for the query that only fails against real SQL.

Timeout and limit tests assert the survival behaviors directly. A hung handler behind TimeoutLayer must return 503 within 6 seconds of virtual or real time, an oversized JSON post must return 413 without buffering, and a missing content-type post must return 415 with a teaching message. These three tests would have caught the pricing pileup, the 50MB RSS spike, and the midnight 415 thread before any of them shipped. They run in under a second combined and belong in every per-commit suite.

axum-test extends oneshot testing when convenience pays. Its TestServer boots the router on ephemeral ports and adds ergonomic get, post_json, and assert_json helpers that read better than manual request builders for large suites. Use it for integration-style tests asserting full JSON contracts across dozens of routes, and keep raw oneshot tests for layer-ordering and rejection specifics where explicit request construction matters. Either harness runs offline; pick per file by readability, not dogma.

CI splits keep feedback tight as suites grow. Per-commit jobs run router, extractor, and unit tests in under 30 seconds with no external services. Nightly jobs run database integration plus chaos-delay vendor tests that take minutes. Flaky-test quarantine tracks rerun rates per test with automatic filing above 1%: ordering-sensitive tests get fixed or marked single-threaded instead of haunting main. The split preserves fast merges while nightly guards the deep matrix.

Contract snapshots lock the API clients depend on. Serialize example 200 and error responses to golden JSON files, assert byte stability in tests, and review diffs like code. A renamed error code field fails the snapshot before it breaks mobile parsing. Combined with OpenAPI generation from the same structs, snapshots keep docs, tests, and wire format in triple agreement that no amount of careful editing sustains manually.

Fuzzing the router table finds shadowing no review catches. A property test generating random paths against the router asserts each matches at most one handler and that /health always wins over params. Run it nightly over the generated route list rather than per commit for speed. Deterministic unit tests lock known paths; fuzzing guards the unknown combinations growth introduces.

Mutation testing on router asserts proves the tests actually guard: flipping a route order or dropping a layer must fail at least one test. Quarterly mutation runs catch the green suite that asserts nothing load-bearing.

use axum::{Router, body::Body, http::{Request, StatusCode}, routing::get};
use tower::ServiceExt;

async fn health() -> &'static str {
    "ok"
}

fn app() -> Router {
    Router::new().route("/health", get(health))
}

#[tokio::main]
async fn main() {
    let res = app()
        .oneshot(Request::builder().uri("/health").body(Body::from("")).unwrap())
        .await
        .expect("oneshot");
    assert_eq!(res.status(), StatusCode::OK);
    println!("smoke ok");
}

#[cfg(test)]
mod tests {
    use super::*;

    #[tokio::test]
    async fn health_returns_ok() {
        let res = app()
            .oneshot(Request::builder().uri("/health").body(Body::from("")).unwrap())
            .await
            .expect("oneshot");
        assert_eq!(res.status(), StatusCode::OK);
    }

    #[tokio::test]
    async fn unknown_route_returns_404() {
        let res = app()
            .oneshot(Request::builder().uri("/nope").body(Body::from("")).unwrap())
            .await
            .expect("oneshot");
        assert_eq!(res.status(), StatusCode::NOT_FOUND);
    }
}