Rust Ollama Capstone: Ship a Streaming Local-LLM Service

You've finished the track. You can model data with serde, serve HTTP with Axum, juggle tasks with Tokio, and call APIs with reqwest. What you haven't done yet is combine all four into one system that behaves like production software instead of four separate tutorials.

That's what this capstone builds: a local-LLM service where a Rust CLI client and a Rust Axum server both talk to Ollama over plain HTTP on localhost:11434. No cloud keys, no per-token bills, no prompts leaving your machine. You'll post a prompt, stream tokens back word by word, handle the failures that real models produce, and prove quality with an eval harness before you claim anything works.

You're going to hit the same walls every team hits. Streamed bodies arrive as newline-delimited JSON rather than one tidy object, so naive parsing dies on the second token line. Model reloads cost gigabytes and tens of seconds, so a fresh client per request turns 6 concurrent users into an out-of-memory kill. Raw model text breaks strict parsers, so trusting output without schemas fails 200 queued jobs on a Friday evening.

You'll solve all three here with patterns that transfer anywhere. A single shared reqwest Client with explicit timeouts, a bounded queue that turns stalls into degraded-but-alive responses, serde schemas with two retries before escalation, and config entirely from env so the same binary runs on your laptop and in Docker without recompiling. You've completed the earlier track pieces on async, APIs, serialization, and error handling — this project knots them together.

Don't expect frontier-model quality from an 8B local model, and don't let that stop you. For support triage, summarization, and retrieval over private docs, a pinned llama3.1 answering in 2-8 seconds on CPU wins on privacy and cost where it counts. Let's build the whole loop.

Capstone Blueprint: Tying Axum, Tokio, Serde, and Ollama Into One Service

Every earlier piece of the track was a rehearsal for this project, so start by naming how the four crates share the work. Ollama is the model plane: it holds weights resident, serves POST /api/generate and POST /api/embeddings on localhost:11434, and knows nothing about your users. reqwest is the transport: one shared async Client with connection pooling carries JSON prompts down and token replies back. serde plus serde_json is the contract: typed structs shape requests and validate replies so malformed model text fails at a named boundary. Tokio is the runtime: it multiplexes concurrent prompts, enforces timeouts, streams chunks, and bounds the queue so one slow generation cannot take the service down. Axum sits on top as the product surface: routes, extractors, and shared state turn model capability into a chat endpoint your frontend or CLI can rely on.

The build order matters because each layer gives the next something to test against. First prove Ollama answers curl with stream:false, so daemon problems never masquerade as Rust bugs. Then build the CLI client variant that posts one prompt and prints one answer, which locks the request shape and the parsing code in under a hundred lines. Then add streaming over NDJSON mapped to server-sent events, which is where perceived latency drops from seconds to 300ms for interactive users. Then wrap both paths in the Axum server with AppState holding the shared client, semaphore, and cache, so concurrency policy lives in one place. Then add thiserror types, env config, the eval harness, and Docker — in that order — because each assumes the previous layer is green.

Keep the file layout boring so reviewers navigate by habit. A src/main.rs binary that can run as CLI or server behind a subcommand, src/ollama.rs holding the typed client with generate plus generate_stream plus embed, src/server.rs with routes and handlers, src/config.rs reading env, src/error.rs with the thiserror enum, and tests/eval.rs with fixtures plus golden prompts. Shared logic stays in library functions both binaries call; no handler builds JSON by hand and no CLI duplicates the server's parsing. One team enforced this with a ten-line rule: any JSON literal appearing twice becomes a serde struct by Friday. Duplicated shapes drift within a sprint and produce the classic 422-on-valid-payload incident that wastes an afternoon.

Measure the baseline before adding sophistication, because capstone numbers anchor every later claim. A single warm prompt to llama3.1 8B answers in 2-8 seconds on CPU with first streamed token near 300ms, resident memory near 5GB, and per-prompt overhead near 9ms on a shared client versus 180ms constructing clients per call. Write these four numbers on the design doc's first page. Every optimization below — pooling, batching, caching, bounding — moves one of them, and any change that moves none of them is complexity without a customer.

Tracing ties the layers together from the first request. Instrument the client with a span carrying model tag plus prompt hash, propagate the request id from Axum middleware into Ollama calls, and record elapsed plus token counts as span fields rather than scattered log lines. A slow chat request then reads as one trace: admission wait 40ms, generation 4.2s, parsing 3ms, with the bottleneck named instead of guessed. Teams that add tracing during the capstone keep it forever; teams that postpone it reconstruct their first incident from disconnected logs across two binaries.

Start the repo tonight with the smallest possible proof: curl against /api/tags, then the CLI asking one question, then streaming, then the server. Each milestone delivers a demoable artifact — a printed answer, progressive tokens, a curl-able endpoint — which keeps momentum through the slower hardening phases. Developers who sequence milestones this way finish capstones; developers who scaffold everything upfront debug five layers simultaneously and stall. Green milestones in order keep capstones shippable at every step.

[package]
name = "ollama-capstone"
version = "0.1.0"
edition = "2021"

[dependencies]
axum = { version = "0.7", features = ["macros"] }
tokio = { version = "1", features = ["full"] }
reqwest = { version = "0.12", features = ["json", "stream"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
thiserror = "2"
clap = { version = "4", features = ["derive"] }
tower = "0.4"
tracing = "0.1"
tracing-subscriber = "0.3"

[dev-dependencies]
wiremock = "0.6"

Talking to Ollama REST From Rust: reqwest Client, Models, and First Prompts

Ollama's REST surface is deliberately small, and learning it with curl before writing Rust saves hours of misattributed blame. The daemon listens on port 11434 once ollama serve runs, lists local weights at GET /api/tags, and generates text at POST /api/generate accepting model, prompt, stream, plus options like num_ctx and temperature. A first proof with stream:false returns one JSON object carrying a response string and a done boolean, which is the shape your serde structs will mirror. Pull llama3.1 once with ollama pull, confirm ollama ps shows it resident, and keep that terminal open while you build — every later Rust error is either a client bug or a daemon state change, and curl tells you which in seconds.

The Rust client centers on one shared reqwest::Client built at startup with a 120-second default timeout, json support, and connection pooling across thousands of prompts. Construct it once, wrap it in an Arc or Axum state, and never call Client::new inside a handler — per-request construction adds roughly 180ms of TLS plus pool warmup per call versus 9ms amortized on the shared instance, a 20x tax invisible at one prompt per minute and decisive at fifty. Define request structs with serde Serialize: model as String, prompt as String, stream as bool defaulting false, options carrying num_ctx caps that keep one 8B model near 5GB instead of ballooning past host RAM. Define reply structs with Deserialize on response plus done, tolerating extra fields so minor daemon upgrades never break parsing.

Post with .json(&body), check status before parsing, and convert failures into typed errors rather than stringly panics. A 404 means the model tag is wrong — list /api/tags and compare spelling including the :latest suffix behavior. A 500 mid-stream usually means the model was evicted under memory pressure, which your retry path handles by re-pinning and replaying once. Timeouts convert hangs into retryable errors your queue can reason about instead of workers lost for nine minutes. Log token counts per call from the prompt_eval_count plus eval_count fields so cost and latency stay visible in dashboards long after the demo. Teams that log tokens from day one spot prompt-bloat regressions in review. Teams that add logging after the incident reconstruct usage from vibes.

Prove the client with tests that never need a GPU. Unit tests serialize request structs and deserialize golden replies from tests/fixtures, including malformed variants that must fail loudly. Integration tests gate on an OLLAMA_URL env and skip gracefully when no daemon runs, so CI stays green on machines without models. A wiremock-based test asserts timeout plus retry behavior deterministically by serving a delayed stub. This three-layer suite — golden fixtures, gated integration, mocked resilience — is what lets you refactor prompts fearlessly later, because the contract is pinned in four places instead of one developer's memory.

Model selection belongs in the same conversation as client code because tags change behavior more than versions suggest. Pin exact tags like llama3.1:8b in config and lockfiles rather than floating latest, record which tag served each eval run, and re-run tier two whenever the tag changes — a minor Ollama image bump once shifted extraction formatting enough to break 4% of parses. Keep a fallback tag configured for degraded mode: a 3B model answering classification in a second beats a queued 8B answering nothing for minutes when the host saturates.

Record client decisions in a short DECISIONS log beside the code: chosen model tags with dates, num_ctx caps with memory math, timeout values with incident references. Six months later, when someone proposes doubling context windows, the log shows exactly which memory budget constrained the original choice. Teams with decision logs tune from reasons; teams without tune from archaeology and repeat settled debates quarterly. Logged reasons today prevent repeated debates next quarter when tuning resurfaces.

use reqwest::Client;
use serde::{Deserialize, Serialize};
use std::time::Duration;

#[derive(Debug, Serialize)]
pub struct GenerateRequest {
    pub model: String,
    pub prompt: String,
    #[serde(default)]
    pub stream: bool,
}

#[derive(Debug, Deserialize)]
pub struct GenerateReply {
    #[serde(default)]
    pub response: String,
    #[serde(default)]
    pub done: bool,
}

#[derive(Clone)]
pub struct Ollama {
    client: Client,
    base: String,
}

impl Ollama {
    pub fn new(base: String) -> Self {
        let client = Client::builder()
            .timeout(Duration::from_secs(120))
            .build()
            .expect("client builds");
        Self { client, base }
    }

    pub async fn generate(&self, model: &str, prompt: &str) -> anyhow::Result<String> {
        let body = GenerateRequest {
            model: model.to_string(),
            prompt: prompt.to_string(),
            stream: false,
        };
        let reply: GenerateReply = self
            .client
            .post(format!("{}/api/generate", self.base))
            .json(&body)
            .timeout(Duration::from_secs(60))
            .send()
            .await?
            .error_for_status()?
            .json()
            .await?;
        Ok(reply.response)
    }
}

Building the CLI Client Variant: Args, Env Config, and Single-Shot Generation

The CLI variant is the cheapest possible proving ground for the whole contract, and senior teams build it before any server code exists. Its job description fits in one sentence: parse args, load env config, post one prompt with stream:false, print the answer or a typed error. Clap derives the interface — a prompt positional, --model defaulting to llama3.1, --no-stream versus --stream flags, and a --json flag emitting machine-readable output for pipelines. Env supplies OLLAMA_URL defaulting to http://localhost:11434 plus OLLAMA_MODEL and a REQUEST_TIMEOUT_SECS default of 60, so the same binary runs against a laptop daemon and a Docker sidecar without recompiling. Single-shot generation keeps the first milestone deterministic: one request, one JSON object, one exit code.

Structure the binary so the server reuses everything it proves. Put argument parsing in main, but generation, error mapping, and output formatting in library functions the Axum handlers call too — format_answer(reply, json_flag) and map_error(err) shared by both surfaces. Exit codes carry meaning scripts can rely on: zero for success, two for model errors with the message on stderr, three for timeouts that suggest retry with a longer budget. Print timings to stderr in verbose mode: time to first byte is meaningless for single-shot, but total elapsed plus eval_count tokens per second tells you immediately whether slowness is model compute or client overhead. A CLI that reports its own latency teaches its operator to read the system.

Handle the three CLI-specific failures before they reach users. Missing daemon produces connection refused, which your error path translates into a one-line remedy naming ollama serve and the exact OLLAMA_URL in effect — never a bare reqwest stack trace. Unknown model tag produces 404, translated into the three closest local tags from /api/tags so a typo costs seconds instead of a docs hunt. Slow generations hit the 60-second timeout, which prints elapsed plus a hint to raise REQUEST_TIMEOUT_SECS or shrink num_ctx rather than hanging until Ctrl-C. Each translation is a five-line match arm written once and inherited by the server, which is exactly why CLI-first pays: error empathy gets designed under no load and deployed under full load.

Ship the CLI as a pipeline citizen, not a demo toy. The --json flag emits {answer, model, elapsed_ms, tokens} so nightly jobs and eval harnesses consume it without scraping prose. Stdin support reading the prompt when no positional is given enables echo plus pipe workflows and file-driven batch runs. A --system flag prepending instructions keeps prompt templates versioned in scripts rather than tribal knowledge. Teams that treat the CLI as the batch path keep it maintained for years; teams that treat it as scaffolding delete it right before the first bulk backfill needs it.

Output handling completes the CLI contract: plain mode prints the answer with a trailing newline for terminal use, --json emits answer plus model plus elapsed_ms for pipelines, and verbose mode adds tokens-per-second to stderr without polluting stdout parsers. Pipe discipline matters — never mix diagnostics into stdout when --json is set, because downstream jq breaks on the first stray line. A CLI whose output modes are tested with golden comparisons becomes infrastructure other teams build on; one whose modes drift becomes a demo people reimplement.

Polish the CLI surface with generated shell completions plus a man page derived from the clap definition, so operators discover flags without reading source. Thirty minutes of build-script work pays back for years across every engineer who types the binary name followed by tab. Small courtesies like consistent exit codes and machine-readable output modes are what promote a demo binary into infrastructure other teams willingly depend on. Stable CLI output modes plus documented exit codes turn a demo binary into infrastructure other teams willingly consume daily without reading its source.

use clap::{Parser, Subcommand};
use std::time::Instant;

#[derive(Parser)]
#[command(name = "ollama-capstone")]
struct Cli {
    #[command(subcommand)]
    cmd: Cmd,
}

#[derive(Subcommand)]
enum Cmd {
    Ask {
        prompt: String,
        #[arg(long, default_value = "llama3.1")]
        model: String,
        #[arg(long)]
        json: bool,
    },
    Serve {
        #[arg(long, default_value = "3000")]
        port: u16,
    },
}

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let cli = Cli::parse();
    let base = std::env::var("OLLAMA_URL")
        .unwrap_or_else(|_| "http://localhost:11434".to_string());
    match cli.cmd {
        Cmd::Ask { prompt, model, json } => {
            let t0 = Instant::now();
            let ollama = ollama_capstone::Ollama::new(base);
            match ollama.generate(&model, &prompt).await {
                Ok(answer) => {
                    if json {
                        println!(
                            "{}",
                            serde_json::json!({
                                "answer": answer,
                                "model": model,
                                "elapsed_ms": t0.elapsed().as_millis()
                            })
                        );
                    } else {
                        println!("{}", answer);
                    }
                }
                Err(e) => {
                    eprintln!("error: {:#}", e);
                    std::process::exit(2);
                }
            }
        }
        Cmd::Serve { port } => ollama_capstone::serve(port).await?,
    }
    Ok(())
}

Streaming Tokens With Tokio: NDJSON Chunks, SSE Responses, and Backpressure

Streaming is where local LLMs start feeling alive, and the mechanics are simpler than the mystique suggests. Sending stream:true to /api/generate changes the reply from one JSON object into newline-delimited JSON: each line is a complete object with a response fragment plus a done flag on the final line. Your Rust code reads the reqwest bytes_stream, splits on newlines, deserializes each chunk with serde_json, and forwards fragments immediately — to stdout for the CLI or as server-sent events for the browser. First tokens reach users near 300ms while the full answer still takes seconds, which cuts perceived latency roughly 20x without changing model speed at all. Single-shot fits jobs that consume whole answers; streams fit humans watching text appear.

Tokio provides the plumbing that keeps streams honest under load. The bytes_stream yields Result<Bytes> items your loop maps into typed chunk structs, skipping empty heartbeat lines that keep intermediaries from closing idle connections. A tokio::time::timeout wraps the whole stream at 60 seconds so a stuck generation becomes a typed timeout your fallback handles, not a worker held for nine minutes. Backpressure arrives through bounded mpsc channels of 32 between the Ollama reader task and the SSE writer: when the browser pauses, the channel fills, the reader awaits send, and memory stays flat instead of buffering an unbounded answer. Unwrap this design and you get either lost chunks, leaked tasks, or RAM graphs that climb until the OOM killer arrives.

The SSE mapping is a small, exact protocol worth implementing carefully. Each fragment becomes an event: data line carrying JSON with the token text, flushed immediately so intermediaries forward without buffering. A final done event closes the stream, and an error event carries typed failure JSON when the upstream fails mid-stream — never an abrupt TCP close the frontend reads as success. Set Content-Type to text/event-stream, disable proxy buffering with X-Accel-Buffering: no, and flush after every event; missing any of the three produces the classic works-locally-buffers-in-staging symptom. The CLI mirrors this by printing fragments as they arrive and flushing stdout, giving operators the same progressive feedback developers demo in browsers.

Test streams deterministically because live models are the opposite of deterministic. A fixture file with five NDJSON lines plus a done line feeds the parser unit test asserting exact fragment sequence and final assembly. A wiremock stub serves the fixture with 50ms line delays while the test asserts first-token arrival under a second and full assembly equality. A chaos test injects a malformed line mid-stream and asserts the parser skips plus logs rather than aborting the whole answer. These three tests pin chunking, timing, and resilience independently, so a later change to buffering policy fails exactly one of them instead of manifesting as flaky end-to-end timeouts nobody trusts.

Client-side rendering of streams needs equal care on the frontend half. Open the EventSource with explicit reconnect plus last-event-id handling, render tokens incrementally into a buffer that preserves markdown boundaries, and display a typing indicator from connection open until first token rather than a blank box. Timeouts on the frontend mirror the backend: show the cached-or-degraded notice if no token arrives in 10 seconds instead of hanging. The 300ms first-token win only lands when every layer flushes — Rust handler, proxy buffering flags, and DOM updates alike.

Prove streaming backpressure with a slow-consumer harness that pauses mid-read for seconds while the model keeps producing, then assert memory stays flat and no chunks are lost. Fast-consumer tests pass even with unbounded buffers, hiding the leak until a real user on a slow connection triggers it. The pause-and-resume test is the only honest verification that your bounded channel actually bounds anything under production-shaped pressure. Flush everywhere.

use futures::StreamExt;
use serde::Deserialize;
use std::time::Duration;

#[derive(Debug, Deserialize)]
struct Chunk {
    #[serde(default)]
    response: String,
    #[serde(default)]
    done: bool,
}

pub async fn stream_prompt(
    client: &reqwest::Client,
    base: &str,
    model: &str,
    prompt: &str,
    on_token: impl Fn(String),
) -> anyhow::Result<String> {
    let body = serde_json::json!({
        "model": model, "prompt": prompt, "stream": true,
    });
    let resp = client
        .post(format!("{}/api/generate", base))
        .json(&body)
        .send()
        .await?
        .error_for_status()?;
    let mut full = String::new();
    let mut stream = resp.bytes_stream();
    let mut buf: Vec<u8> = Vec::new();
    let fut = async {
        while let Some(item) = stream.next().await {
            let bytes = item?;
            buf.extend_from_slice(&bytes);
            while let Some(pos) = buf.iter().position(|&b| b == b'\n') {
                let line: Vec<u8> = buf.drain(..=pos).collect();
                let text = String::from_utf8_lossy(&line);
                let text = text.trim();
                if text.is_empty() {
                    continue;
                }
                let chunk: Chunk = serde_json::from_str(text)?;
                if !chunk.response.is_empty() {
                    on_token(chunk.response.clone());
                    full.push_str(&chunk.response);
                }
                if chunk.done {
                    return Ok::<_, anyhow::Error>(());
                }
            }
        }
        Ok::<_, anyhow::Error>(())
    };
    tokio::time::timeout(Duration::from_secs(60), fut).await??;
    Ok(full)
}

Serving Chat Over Axum: State, Routes, and the Chat Endpoint

The Axum server turns the proven client into a product surface, and its design centers on one decision: everything handlers need lives in a Clone AppState built once at startup. That state holds the shared reqwest Client, the Ollama base URL, the model default, a tokio Semaphore bounding concurrent generations, an mpsc-bounded queue depth of 32, and a small response cache for repeat prompts. Handlers take State<AppState> plus a Json<ChatRequest> extractor and return typed Json<ChatReply> or SSE streams, which keeps validation at the framework boundary where Axum's rejection messages are precise. Building the client per handler would reintroduce the 180ms tax the shared design eliminated; state makes the fast path the only path.

Routes stay minimal by design: POST /v1/chat for single-shot answers, POST /v1/chat/stream for SSE token flow, and GET /healthz reporting daemon reachability plus queue depth for load balancers. The chat handler acquires a semaphore permit with a 5-second queue timeout — shedding load fast with 503 plus Retry-After instead of queueing for nine minutes — then calls generate under the 60-second model timeout, records token counts, and caches successes keyed by model plus prompt hash. The stream handler follows the same admission control, then maps NDJSON chunks to SSE events with per-event flushes. Middleware adds tracing spans per request, a 30-second tower timeout on the089089 outer layer, and JSON size limits rejecting 10MB prompt bombs before they reach the model.

Extractor discipline prevents the 422-on-valid-payload class of incidents. Derive Deserialize on ChatRequest with explicit rename_all and defaults so frontend camelCase never mismatches Rust snake_case silently, and return Axum's rejection body verbatim in development so the missing-field name reaches the developer instead of a generic bad request. Version the path prefix as /v1 from day one so prompt-schema changes ship as /v2 without breaking existing callers mid-migration. Document the two shapes — single-shot reply versus SSE event sequence — with curl examples in the README, because every frontend developer integrates faster from a copy-pasteable curl than from a prose description of streaming semantics.

Admission control is the difference between degraded and dead. The semaphore of 2 for heavy generations plus queue of 32 means the 33rd concurrent caller gets an instant cached-or-503 answer while 32 proceed; without it, all 50 pile onto the model host and everyone times out together. Health reporting exposes queue_depth so autoscalers and alerts react to saturation before users do. One triage service held p99 at 4 seconds through a 3x spike purely because the queue shed the excess in milliseconds while competitors without admission control cascaded into nines of minutes. Concurrency policy is a product feature users feel on the worst day, not an optimization for the best day.

State growth stays disciplined through a review rule: anything added to AppState must justify why every handler needs it, with per-request values flowing through extractors instead. A state struct accumulating caches, metrics handles, and feature flags without ownership review becomes a global in disguise within two quarters. Document each field's sharing semantics — Arc for concurrent readers, Mutex only where mutation is genuinely shared — so contributors extend state without introducing contention on the hot path.

Extend /healthz beyond a static ok to report model tag, queue depth, and daemon reachability, giving load balancers, dashboards, and incident reviews one shared picture of saturation. A health endpoint that hides queue state forces every consumer to infer load from latency, which lags minutes behind reality. Rich health output turns capacity conversations from forensic reconstruction into reading the current line. Shared health output aligns balancers, dashboards, and reviewers on one truth instead of three inferred pictures that lag minutes behind reality.

use axum::{extract::State, http::StatusCode, routing::{get, post}, Json, Router};
use serde::{Deserialize, Serialize};
use std::{sync::Arc, time::Duration};
use tokio::sync::Semaphore;

#[derive(Clone)]
pub struct AppState {
    pub client: reqwest::Client,
    pub ollama_base: String,
    pub model: String,
    pub slots: Arc<Semaphore>,
}

#[derive(Debug, Deserialize)]
pub struct ChatRequest {
    pub message: String,
    #[serde(default = "default_model")]
    pub model: String,
}

fn default_model() -> String {
    "llama3.1".to_string()
}

#[derive(Debug, Serialize)]
pub struct ChatReply {
    pub answer: String,
    pub model: String,
}

async fn chat(
    State(st): State<AppState>,
    Json(req): Json<ChatRequest>,
) -> Result<Json<ChatReply>, StatusCode> {
    let _permit = st.slots.try_acquire().map_err(|_| StatusCode::TOO_MANY_REQUESTS)?;
    let model = if req.model.is_empty() { st.model.clone() } else { req.model };
    let body = serde_json::json!({
        "model": model, "prompt": req.message, "stream": false,
    });
    let reply: serde_json::Value = st
        .client
        .post(format!("{}/api/generate", st.ollama_base))
        .json(&body)
        .timeout(Duration::from_secs(60))
        .send()
        .await
        .map_err(|_| StatusCode::BAD_GATEWAY)?
        .error_for_status()
        .map_err(|_| StatusCode::BAD_GATEWAY)?
        .json()
        .await
        .map_err(|_| StatusCode::BAD_GATEWAY)?;
    Ok(Json(ChatReply {
        answer: reply["response"].as_str().unwrap_or("").to_string(),
        model,
    }))
}

pub async fn serve(port: u16) -> anyhow::Result<()> {
    let state = AppState {
        client: reqwest::Client::builder()
            .timeout(Duration::from_secs(120))
            .build()?,
        ollama_base: std::env::var("OLLAMA_URL")
            .unwrap_or_else(|_| "http://localhost:11434".to_string()),
        model: std::env::var("OLLAMA_MODEL")
            .unwrap_or_else(|_| "llama3.1".to_string()),
        slots: Arc::new(Semaphore::new(32)),
    };
    let app = Router::new()
        .route("/v1/chat", post(chat))
        .route("/healthz", get(|| async { "ok" }))
        .with_state(state);
    let listener = tokio::net::TcpListener::bind(("0.0.0.0", port)).await?;
    axum::serve(listener, app).await?;
    Ok(())
}

Error Handling With thiserror: Typed Failures, Retries, and Fallbacks

Model output is the least trustworthy input your service handles, so error taxonomy is load-bearing design rather than polish. thiserror gives the vocabulary: an AppError enum with variants for Transport(reqwest failures), ModelStatus(u16 plus body excerpt), Parse(shape mismatches with the offending excerpt), Timeout(elapsed budget), QueueFull(current depth), and Config(env problems at startup). Each variant carries the fields dashboards need — endpoint, model tag, elapsed_ms, retryable bool — so logs answer which class failed without grepping prose. Retry policy keys off the type, not the message: 500s and timeouts retry once after re-pinning, 404s and 422s never retry because the request itself is wrong, and parse failures trigger the stricter-reminder retry before human escalation. Stringly errors make every failure look identical; typed errors make policy programmable.

The retry ladder stays short by rule: one immediate retry for transport 500s, one stricter-prompt retry for parse failures, then fallback or escalate — never unbounded loops. Unbounded retries turned one Next.js team's 3-hour incident into a $2,400 token bill when a poisoned prompt retried 2.1M tokens against a failing endpoint; the capstone equivalent is a queue that replays forever while depth climbs past 4,000. Encode the ladder in a single async helper with explicit attempts counted in the span, log each attempt with its number, and emit a metric distinguishing first-try success from retry success. When retries succeed you learn resilience worked; when they exhaust you learn exactly which class needs attention.

Fallbacks turn failures into degraded answers users can use. The cache from the server layer serves the last good answer for repeat prompts with a stale flag, so a timeout on a known question still returns value in milliseconds. For novel prompts with no cached answer, a smaller faster model tag or a templated safe-completion responds honestly about unavailability instead of failing open with invented content. Every fallback response carries degraded:true plus reason so frontends render appropriately and operators measure degradation rate as a first-class SLO. A service that degrades visibly earns trust during incidents; a service that fails silently or invents confidently loses it for quarters.

Map errors at every boundary so humans get remedies instead of traces. CLI prints connection refused as the daemon remedy with the effective OLLAMA_URL, Axum maps QueueFull to 503 with Retry-After: 2, and parse failures surface the expected schema plus excerpt rather than a serde line number. anyhow carries context through internal layers with .context() naming the stage — admission, generation, parsing, streaming — while thiserror types the public surface both binaries share. Test the mapping like features: unit tests asserting each variant's status code, message shape, and retryable flag, plus a resilience test proving a 90-second stub produces a fallback in 61 seconds. Error handling you cannot demonstrate in tests is documentation, not engineering.

Structured logging turns the taxonomy into searchable operations data. Emit one JSON line per request with class, model, elapsed_ms, tokens, retry count, and degraded flag, so dashboards group by failure class without regex. Reserve stack traces for unexpected variants and log expected fallbacks at info with their reason — alert routing keys off the class field, paging only on unknown classes plus queue saturation. Log volume stays flat because per-token logging never leaves the debug level; per-request lines carry the signal.

Tune alerting around the error taxonomy: page immediately on unknown failure classes that indicate novel breakage, while known classes like transient 500s alert only on rate thresholds reviewed quarterly. Alert routing keyed to typed classes stays meaningful as traffic grows; routing keyed to raw log volume becomes scenery engineers mute. The taxonomy investment pays its final dividend at 3 AM when the page names the class instead of dumping a trace.

use thiserror::Error;

#[derive(Debug, Error)]
pub enum AppError {
    #[error("transport failed: {0}")]
    Transport(#[from] reqwest::Error),
    #[error("model status {status}: {excerpt}")]
    ModelStatus { status: u16, excerpt: String },
    #[error("parse failed: {what}")]
    Parse { what: String },
    #[error("timed out after {secs}s")]
    Timeout { secs: u64 },
    #[error("queue full at depth {depth}")]
    QueueFull { depth: usize },
    #[error("config invalid: {0}")]
    Config(String),
}

impl AppError {
    pub fn retryable(&self) -> bool {
        matches!(
            self,
            AppError::Transport(_) | AppError::Timeout { .. }
                | AppError::ModelStatus { status: 500..=599, .. }
        )
    }

    pub fn status_code(&self) -> axum::http::StatusCode {
        use axum::http::StatusCode;
        match self {
            AppError::QueueFull { .. } => StatusCode::TOO_MANY_REQUESTS,
            AppError::Timeout { .. } => StatusCode::GATEWAY_TIMEOUT,
            AppError::Transport(_) => StatusCode::BAD_GATEWAY,
            AppError::ModelStatus { .. } => StatusCode::BAD_GATEWAY,
            AppError::Parse { .. } => StatusCode::UNPROCESSABLE_ENTITY,
            AppError::Config(_) => StatusCode::INTERNAL_SERVER_ERROR,
        }
    }
}

pub async fn with_one_retry<T, F, Fut>(mut f: F) -> Result<T, AppError>
where
    F: FnMut() -> Fut,
    Fut: std::future::Future<Output = Result<T, AppError>>,
{
    match f().await {
        Ok(v) => Ok(v),
        Err(e) if e.retryable() => f().await,
        Err(e) => Err(e),
    }
}

Config via Env: Layered Settings, Secrets, and Twelve-Factor Discipline

The same binary must run on a laptop, in CI, and in Docker without recompiling, which makes env-sourced config a deployment requirement rather than a style choice. The layering is strict and documented in one place: compile-time defaults for safe values like ports and timeouts, environment overrides for everything that varies — OLLAMA_URL, OLLAMA_MODEL, REQUEST_TIMEOUT_SECS, MAX_QUEUE, RUST_LOG — and explicit failures at startup for anything missing with no safe default. A Config::from_env constructor reads each variable, parses with context naming the variable plus the offending value, validates ranges such as timeouts between 5 and 300 seconds, and returns a typed struct the rest of the program consumes. No handler reads env directly, so behavior is fully determined by a struct constructed once and easily faked in tests.

Secrets follow the same path with stricter handling: API-adjacent tokens arrive via env or mounted files referenced by _FILE-suffixed variables, never baked into images or committed to version control, and Debug impls redact them so logs stay safe. Document every variable in a .env.example with defaults plus valid ranges, and fail fast on unknown strict-mode variables in CI to catch typos like OLLAMA_URl before they become silent-default mysteries. One team's staging pointed at production Ollama for a week because a misspelled variable fell back silently; strict validation would have crashed the deploy in seconds instead of leaking cross-environment traffic for days.

Per-request overrides stay narrow and audited: callers may select model tags from an allowlist and tune temperature within bounds, but never override base URLs, timeouts, or queue depths — those are operator concerns, not caller preferences. Log the effective config at startup with secrets redacted, including the resolved Ollama URL, model default, timeout seconds, and queue depth, so every incident review starts from what the binary believed rather than what the dashboard assumed. Pair this with a /healthz payload echoing model plus queue depth at runtime. Config disputes then resolve in one glance at startup logs instead of a forensic tour through compose files, shell histories, and tribal memory.

Test config like the contract it is. Unit tests set temp env vars asserting parsing, defaults, and range rejections, including the misspelled-variable strict failure. A CI step diffs .env.example against Config::from_env fields so new settings cannot ship undocumented. Docker tests boot the compose stack with alternate env proving the same image serves a different model without rebuild. When config is typed, validated, logged, and tested, environment differences stop being the prime suspect in every incident and become the first line you check and the fastest line you clear.

Secret rotation deserves a paragraph because local does not mean unguarded. Support reload-on-SIGHUP for non-listener config so token rotations apply without dropping connections, version the config struct so old files fail loudly instead of half-applying, and keep a config audit line in startup logs naming the source of each value — default, env, or file — with secrets redacted. Rotation that requires a rebuild is rotation that never happens before the leak.

Enforce config documentation with a CI check diffing .env.example against the fields Config::from_env actually reads, so new settings cannot ship undocumented. Reject unknown variables in strict CI mode to catch typos like OLLAMA_URl in seconds rather than discovering cross-environment leakage weeks later. Documentation enforced by builds stays current; documentation enforced by good intentions rots within a quarter. Enforced docs plus strict rejection keep config typos failing builds in seconds instead of leaking traffic silently for a week across environments nobody compared side by side. Good intentions rot within a quarter; builds stay honest for years when checks run on every single merge without any exception at all.

use crate::error::AppError;

#[derive(Debug, Clone)]
pub struct Config {
    pub ollama_url: String,
    pub model: String,
    pub timeout_secs: u64,
    pub max_queue: usize,
    pub port: u16,
}

impl Config {
    pub fn from_env() -> Result<Self, AppError> {
        let timeout_secs: u64 = std::env::var("REQUEST_TIMEOUT_SECS")
            .unwrap_or_else(|_| "60".to_string())
            .parse()
            .map_err(|_| AppError::Config("REQUEST_TIMEOUT_SECS must be a number".into()))?;
        if !(5..=300).contains(&timeout_secs) {
            return Err(AppError::Config("REQUEST_TIMEOUT_SECS must be 5..=300".into()));
        }
        Ok(Self {
            ollama_url: std::env::var("OLLAMA_URL")
                .unwrap_or_else(|_| "http://localhost:11434".to_string()),
            model: std::env::var("OLLAMA_MODEL")
                .unwrap_or_else(|_| "llama3.1".to_string()),
            timeout_secs,
            max_queue: std::env::var("MAX_QUEUE")
                .unwrap_or_else(|_| "32".to_string())
                .parse()
                .map_err(|_| AppError::Config("MAX_QUEUE must be a number".into()))?,
            port: std::env::var("PORT")
                .unwrap_or_else(|_| "3000".to_string())
                .parse()
                .map_err(|_| AppError::Config("PORT must be a number".into()))?,
        })
    }
}

Eval and Test Harness: Fixtures, Golden Prompts, and Regression Gates

Model behavior drifts with every prompt edit, so the eval harness is the CI gate that keeps quality measurable instead of vibes-based. Its anatomy is deliberately boring: tests/fixtures holds 50 versioned prompt-plus-context cases with golden expectations, tests/eval.rs runs each through the same generate path production uses, and assertions check structure plus key facts rather than exact prose. A summarization case asserts the reply mentions all three input entities and stays under 120 words; an extraction case deserializes into the business struct and asserts required fields parse. Exact-match assertions on model prose fail on every harmless rewording and teach developers to ignore red builds; property assertions fail only when behavior genuinely regresses.

The harness runs in three tiers with different budgets. Tier one is pure and fast: fixture parsing plus schema validation with no daemon, green in milliseconds on every save. Tier two is gated integration: against a real Ollama when OLLAMA_URL is set, asserting latency ceilings and parse-success ratios rather than prose equality — for example, 50 extraction prompts must parse at 99% with JSON mode plus retries, and p50 latency must stay under 10 seconds on CPU. Tier three is nightly and comparative: the full 50-prompt suite against candidate model tags or temperature changes, recording pass rates plus token counts into a JSON report diffed against main. A prompt-template pull request that drops extraction pass rate from 99% to 91% fails tier three with a named diff, which is exactly the conversation you want before users meet the regression.

Golden management prevents fixture rot. Each fixture carries an id, the prompt template version, the model tag it was calibrated against, and a last-reviewed date; anything older than 90 days gets flagged for re-review rather than silently asserting stale expectations. Flaky cases get quarantined with a reason and a re-enable date instead of being deleted, preserving the signal that nondeterminism exists at that prompt. Record token counts per case so efficiency regressions — a template change doubling output tokens for identical quality — show as numbers in review. One extraction workload caught a helpful-but-verbose instruction change adding 40% tokens across 50 fixtures before it shipped, saving an estimated doubling of nightly batch time.

Wire the harness into the workflow developers already use. The CLI's --json mode powers the runner so evals exercise the identical binary path as production, new prompt templates arrive as pull requests with fixture updates attached, and the merge checklist requires tier-one green plus tier-two numbers posted as a comment. Report format stays stable JSON: case id, pass boolean, latency_ms, tokens, and failure class, consumed by dashboards tracking quality over quarters. Teams with this harness change prompts confidently weekly. Teams without it freeze prompts in fear after the first Friday-evening parse failure takes down 200 queued jobs.

Flake triage keeps the harness trusted as model nondeterminism inevitably produces borderline cases. Retry each eval case three times and record pass-rate per case rather than binary pass-fail, promoting consistently green cases to merge gates while keeping flaky-but-valuable cases in nightly with their variance visible. A case passing 2 of 3 runs signals prompt brittleness worth fixing, not infrastructure worth ignoring — the harness distinguishes the two only when it records attempts instead of final outcomes.

Archive eval reports as stable JSON artifacts attached to every merge, recording pass rates, latencies, and token counts per case. Quarterly reviews then read quality as graphs — extraction climbing from 91% to 99%, tokens per summary falling 40% — instead of anecdotes in chat scrollback. Artifacts compound into institutional memory; chat messages evaporate and take the reasoning with them. Stored reports turn quarterly quality reviews into graphs rather than anecdotes lost in chat scrollback forever.

use serde::Deserialize;
use std::fs;

#[derive(Debug, Deserialize)]
struct Fixture {
    id: String,
    prompt: String,
    must_contain: Vec<String>,
    max_words: usize,
}

fn load_fixtures() -> Vec<Fixture> {
    let data = fs::read_to_string("tests/fixtures/cases.json")
        .expect("fixtures exist");
    serde_json::from_str(&data).expect("fixtures parse")
}

#[test]
fn fixtures_are_wellformed() {
    let cases = load_fixtures();
    assert!(cases.len() >= 50, "keep at least 50 fixtures");
    for c in &cases {
        assert!(!c.prompt.is_empty(), "{} needs a prompt", c.id);
        assert!(!c.must_contain.is_empty(), "{} needs expectations", c.id);
    }
}

#[test]
fn golden_reply_properties_hold() {
    let reply = fs::read_to_string("tests/fixtures/golden_reply.txt").unwrap();
    let words = reply.split_whitespace().count();
    assert!(words <= 120, "golden reply must stay concise");
    for entity in ["ollama", "queue", "timeout"] {
        assert!(
            reply.to_lowercase().contains(entity),
            "golden reply must mention {}",
            entity
        );
    }
}

#[tokio::test]
#[ignore]
async fn live_extraction_parses() {
    if std::env::var("OLLAMA_URL").is_err() {
        return;
    }
    let ollama = ollama_capstone::Ollama::new(
        std::env::var("OLLAMA_URL").unwrap(),
    );
    let out = ollama.generate("llama3.1", "Return JSON {\"ok\": true}").await.unwrap();
    assert!(out.contains("ok"), "model should echo the key");
}

Hardening for Production: Timeouts, Queues, and Load Budgets

Production hardening is the discipline of deciding in advance what breaks first, and the answers fit on an index card: timeouts bound every wait, queues bound every backlog, budgets bound every resource. Every model call carries a 60-second tokio timeout inside a 120-second client default, so a stuck generation becomes a typed error your fallback handles within a minute instead of a worker lost for nine. The semaphore admits 2 heavy generations concurrently with a 32-deep queue behind it; the 33rd caller gets an instant 503 with Retry-After while admitted work proceeds. Memory budget pins one resident 8B model near 5GB with modest num_ctx, and the load test proves 50 concurrent prompts stay under 80% of host RAM. Each bound converts a cascading failure into an isolated, measurable, degradable event.

Observability makes the bounds operable rather than decorative. Log token counts per call from prompt_eval_count plus eval_count, queue depth sampled every second, semaphore wait time per request, and fallback-serve rate as a first-class metric. Alert on queue depth over 100 sustained for two minutes and on memory over 80%, paging the model host runbook instead of the app on-call — the incident review showed 40 minutes lost paging Postgres for an OOM the model host caused. Dashboards display p50, p99, and fallback share side by side so a latency win bought with silent degradation reads honestly. Numbers the team watches weekly get managed; numbers nobody graphs get discovered during outages.

Validate with load before users do. The k6 or custom Tokio harness fires 50 concurrent prompts mixing single-shot and streaming, asserts p99 under 10 seconds for single-shot plus first-token under a second for streams, and fails the build on regression. Chaos cases belong in the suite: kill Ollama mid-run and assert fallbacks serve within the timeout budget, replay a poisoned prompt and assert exactly one retry before escalation, and flood 200 concurrent callers asserting instant 503s past depth 32 rather than growing queues. One triage service held p99 at 4 seconds through a real 3x spike because the suite had already rehearsed that exact shape monthly. Load tests are rehearsals; incidents are opening nights.

Document the runbook beside the code so 3 AM decisions are pre-made. Queue-full means scale hosts or shed lower-priority batch traffic, never raise the queue bound past 64 without a memory review. Timeout spikes mean check ollama ps for evicted models and free -g for pressure before touching app config. Parse-failure spikes mean freeze prompt-template deploys and run the eval suite against the candidate before rollback. Each entry names the metric, the threshold, the command, and the owner. A runbook with commands gets followed; a runbook with philosophy gets skimmed while the queue grows.

Capacity math turns budgets into purchase orders before the spike. Model resident memory plus per-connection buffers times 32 queue slots plus headroom for eval jobs yields the host size, and the load suite validates the arithmetic monthly as prompts lengthen. When prompts grow 20% year over year, num_ctx caps and queue depths get revisited on schedule instead of during the outage they would otherwise cause. Budget reviews are boring; OOM reviews are worse.

Schedule the spike rehearsal monthly with the full mixed load: single-shot plus streaming callers, a mid-run daemon kill proving fallbacks engage, poisoned prompts proving retry caps hold, and 200-way floods proving 503s stay instant. Monthly rehearsal keeps runbooks honest as code evolves, because procedures untested for two quarters fail exactly when the real spike rediscovers their staleness. Monthly rehearsals with mixed load plus chaos kills and poisoned prompts keep runbooks honest as code evolves past what stale procedures describe from two quarters ago. Untested procedures fail exactly when real spikes rediscover their staleness at the worst possible hour on a Friday evening rush hour traffic.

use std::time::Duration;
use tokio::sync::{mpsc, Semaphore};
use std::sync::Arc;

pub const QUEUE_DEPTH: usize = 32;
pub const MODEL_TIMEOUT: Duration = Duration::from_secs(60);
pub const QUEUE_WAIT: Duration = Duration::from_secs(5);

#[derive(Clone)]
pub struct Gate {
    slots: Arc<Semaphore>,
}

impl Gate {
    pub fn new() -> Self {
        Self { slots: Arc::new(Semaphore::new(QUEUE_DEPTH)) }
    }

    pub async fn admit(&self) -> Result<tokio::sync::OwnedSemaphorePermit, GateReject> {
        tokio::time::timeout(QUEUE_WAIT, self.slots.clone().acquire_owned())
            .await
            .map(|r| r.expect("semaphore open"))
            .map_err(|_| GateReject { retry_after_secs: 2 })
    }
}

pub struct GateReject {
    pub retry_after_secs: u64,
}

pub fn bounded_channel<T>() -> (mpsc::Sender<T>, mpsc::Receiver<T>) {
    mpsc::channel(QUEUE_DEPTH)
}

Shipping With Docker: Images, GPUs, and the Ollama Sidecar Note

Docker turns works-on-my-laptop into works-on-every-host by pinning the runtime the model path assumes. The Rust service ships as a multi-stage image: a rust:1.80-bookworm builder running cargo build --release, then a debian:bookworm-slim runtime carrying only the binary plus CA certificates, with both tags pinned so rebuilds stay reproducible. The binary reads all config from env, so the same image serves laptop Ollama and production sidecars with different OLLAMA_URL values and no rebuild. Keep the image lean — no toolchain, no target directory, no fixtures — because every extra hundred megabytes slows the deploys you most need to be fast during incidents.

Ollama rides beside the service as a named compose service, never as localhost assumptions baked into code. The compose file declares an ollama service from its pinned image with a persistent volume for pulled weights, a GPU reservation when hosts provide NVIDIA runtimes, and an explicit port mapping; the Rust service sets OLLAMA_URL to http://ollama:11434 and depends on healthy daemon state. This is the fix for the classic works-locally-refused-in-Docker failure: inside a container, localhost is the container itself, so only the compose service name reaches the daemon. Health ordering matters — gate the Rust service start on the daemon's /api/tags returning 200, and document the manual fallback of pointing OLLAMA_URL at host.docker.internal for single-container debugging.

GPU handling stays honest about environments. Request the NVIDIA runtime with a device reservation in compose, but keep the service fully functional on CPU so CI and laptops without GPUs still run the suite — slower answers beat no answers during development. Document VRAM expectations beside the model tag: 8B weights want roughly 5GB resident, 70B wants an order of magnitude more, and num_ctx caps translate directly into memory ceilings operators can plan. One team annotated each supported model with RAM plus expected CPU latency in the README, turning capacity planning from folklore into a table anyone could read before choosing a tag.

Operate the stack with the same budgets the code enforces. Cap container memory so the model host cannot silently consume the database sidecar's RAM again, persist weights volumes so restarts skip multi-gigabyte re-pulls, and ship the eval fixtures plus load harness in CI rather than in the runtime image. Tag releases with both the service version and the Ollama image digest, because model-server upgrades change behavior as surely as prompt edits. The capstone's final lesson is environmental: software you can build but cannot deploy repeatably is a demo, and this compose stack with pinned tags, named sidecars, and env-driven config is what promotes the demo to a service.

Supply-chain hygiene closes the shipping story: lock Cargo dependencies with committed lockfiles, scan images for CVEs in CI, and sign release tags so rollbacks restore exact binaries rather than rebuilds that drift. Rehearse rollback quarterly by redeploying the previous digest and confirming /healthz plus one eval prompt, because a rollback path tested only during real incidents fails exactly when it matters. Repeatable deploys plus rehearsed rollbacks are what separate services from demos.

Maintain the model-capacity table in the README with VRAM, expected CPU latency, and first-token times per supported tag, updated whenever tags or caps change. Capacity planning then reads as a table anyone can consult before choosing a model, instead of folklore held by the one engineer who ran the original benchmarks. Tables scale knowledge; memories leave with their owners. A living capacity table with VRAM plus latency per tag scales knowledge beyond the engineer who ran the original benchmarks months ago and may have moved teams since. Tables scale across teams; memories leave with their owners on the very last day of their tenure. Update the table with every single tag or cap change made each quarter sharp.

FROM rust:1.80-bookworm AS builder
WORKDIR /app
COPY Cargo.toml Cargo.lock ./
COPY src ./src
RUN cargo build --release

FROM debian:bookworm-slim
RUN apt-get update && apt-get install -y ca-certificates && rm -rf /var/lib/apt/lists/*
COPY --from=builder /app/target/release/ollama-capstone /usr/local/bin/ollama-capstone
ENV PORT=3000
EXPOSE 3000
CMD ["ollama-capstone", "serve"]