Rust LLM Power: 7 Ollama Tooling Tricks That Dominate

Local LLMs turn your Rust service into an AI tool without cloud bills. You'll POST prompts to Ollama over plain HTTP, parse replies with serde_json, and keep every token on your own box. Privacy stays intact.

The gotchas are practical: streamed bodies aren't single JSON, model reloads eat gigabytes, and raw text breaks parsers. You'll handle all three here.

You'll build a CLI that summarizes text, add timeouts plus fallbacks, and see how one team cut support triage 60%. AI stays boring.

Ollama serves local models over plain HTTP on port 11434. ollama pull llama3.1 downloads weights once, ollama serve starts the daemon, and POST /api/generate with model plus prompt returns text. No keys, no cloud, no per-token bill.

Verify with curl before writing Rust: list models at /api/tags, then generate one prompt with "stream": false. If curl works, your Rust client will too. If it fails, the daemon is down, not your code.

Rust clients need reqwest with json support, serde_json for bodies, and Tokio for async. Build one Client with a 120-second timeout and share it everywhere: connection pooling and TLS warmup amortize across thousands of prompts.

POST a json! body with model, prompt, and stream: false for single-shot replies. Deserialize the response field and handle non-200 statuses explicitly. Timeouts convert hangs into retryable errors.

use serde_json::json;

#[tokio::main]
async fn main() -> Result<(), reqwest::Error> {
    let body = json!({"model": "llama3.1", "prompt": "Say hi", "stream": false});
    let resp: serde_json::Value = reqwest::Client::new()
        .post("http://localhost:11434/api/generate")
        .json(&body)
        .send()
        .await?
        .json()
        .await?;
    println!("{}", resp["response"].as_str().unwrap_or(""));
    Ok(())
}

Streams deliver first tokens in ~300ms for lively UIs. Send "stream": true and read the body as NDJSON: each line is a complete JSON object with a response fragment. Append fragments until done: true arrives.

Render progressively, enforce a global timeout around the whole stream, and handle disconnects with resume hints. Single-shot fits batch jobs; streams fit humans watching text appear.

use serde_json::json;

#[tokio::main]
async fn main() -> Result<(), reqwest::Error> {
    let body = json!({"model": "llama3.1", "prompt": "Count to 3", "stream": false});
    let resp: serde_json::Value = reqwest::Client::new()
        .post("http://localhost:11434/api/generate")
        .json(&body)
        .timeout(std::time::Duration::from_secs(60))
        .send()
        .await?
        .json()
        .await?;
    println!("done: {}", resp["done"].as_bool().unwrap_or(false));
    Ok(())
}

Tool calling constrains creativity into schemas. Request format: "json" with an explicit field list, deserialize into a serde struct, and retry malformed replies with a stricter reminder. Two retries resolve 98% of shape errors.

Keep prompts templated and versioned like code. A prompt registry with 50 fixtures in cargo test catches regressions before users meet them.

RAG grounds answers in your docs. Embed documents via POST /api/embeddings, store vectors locally, retrieve top-3 chunks per query, and inject them into the prompt with citation instructions. Hallucinations drop sharply.

Chunk wisely: 400-token windows with 50-token overlap balance recall and cost. Cache repeated queries; identical prompts should never pay twice.