Prompt Caching Cuts LLM Bills 90% — Production Setup

LLM bills scale with tokens, and most production prompts resend the same system instructions, documents, and examples on every call. Recomputing identical prefixes thousands of times a day is pure waste. You'll see it the moment you price a RAG feature at scale.

Prompt caching fixes exactly that. Providers store the processed prefix and reuse it, cutting cached-portion costs by up to 90% and trimming time-to-first-token. It's the single biggest cost lever in most LLM APIs.

But caching is prefix-sensitive in ways that surprise teams. One timestamp in the wrong block zeroes your hit rate. A silent TTL default change once inflated bills 17-32% across the industry.

This guide shows the prompt layouts, TTL math, and monitoring that keep caching saving money instead of quietly spending it.

How Prompt Caching Works — KV Reuse, Not Magic

Transformer inference splits into prefilling (processing your prompt into KV tensors) and decoding (generating tokens). For long prompts, prefilling dominates cost and time-to-first-token. Resending a 10k-token system prompt a thousand times a day means paying for the same prefilling a thousand times.

Caching stores those KV tensors keyed by prompt prefix. A matching prefix skips prefilling and starts decoding immediately — cheaper (0.1x reads) and faster (lower TTFT). All major providers offer it: Anthropic via explicit breakpoints, OpenAI and Google automatically for eligible requests.

Minimum thresholds gate everything: typically 1024 tokens (Anthropic, OpenAI) to 4096 (Google). Below that, requests process normally with no error and no benefit — the silent case teams misread as broken config.

Set Breakpoints Right — Stable Early, Volatile Late

Anthropic's pattern is explicit: attach cache_control to the stable block (usually system), keep the volatile user message after it. The response usage fields tell you exactly what happened — creation tokens on first call, read tokens on subsequent ones.

Order is contractual: caching covers tools, system, and messages in that order up to the breakpoint. Put the long stable content early, the per-request content late, and never split stable content across volatile insertions.

Up to four breakpoints are allowed, so large RAG prompts can cache system plus document blocks separately. Each breakpoint is a prefix boundary — everything before it must be byte-identical for a hit.

import anthropic
client = anthropic.Anthropic()

SYSTEM = open("system_prompt.txt").read()  # long, stable
resp = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=1024,
    system=[{
        "type": "text",
        "text": SYSTEM,
        "cache_control": {"type": "ephemeral"},          # 5-min TTL
        # "cache_control": {"type": "ephemeral", "ttl": "1h"}  # 1-hour
    }],
    messages=[{"role": "user", "content": user_query}],  # volatile LAST
)
print(resp.usage.cache_creation_input_tokens,  # written this call
      resp.usage.cache_read_input_tokens)       # served from cache

TTL Math — 5 Minutes vs 1 Hour Break-Even

The TTL math decides the tier. Five-minute writes cost 1.25x base and refresh free on every hit — an active conversation stays cached cheaply forever. One-hour writes cost 2x and need roughly two reads per write to break even.

Chatty sequential traffic belongs on 5-minute. Cross-session stable artifacts (knowledge packs, few-shot libraries) with proven reuse justify 1-hour. Everything else on 1-hour is paying double for expiry.

The March 2026 incident is the cautionary tale: a default shift from 1h to 5min turned normal work pauses into full recomputes, inflating costs ~26% with no code change. Explicit TTLs plus ratio monitoring would have caught it in a day.

# Break-even rule: 5-min writes (1.25x) need ~1 reuse; 1h writes (2x) need ~2
python3 - <<'EOF'
import json
rows = [json.loads(l) for l in open('usage.jsonl')]
created = sum(r['cache_creation_input_tokens'] for r in rows)
read = sum(r['cache_read_input_tokens'] for r in rows)
print(f'created={created} read={read} hit_ratio={read / max(created, 1):.2f}')
assert read >= created, 'ALERT: writes dominate — fix prefix stability or TTL tier'
EOF

Prompt Layouts That Maximize Hit Rates

Design prompts cache-first: system instructions, retrieved docs, and few-shot examples as frozen early blocks; user query, timestamps, and session IDs late or outside cached regions. Template engines should render stable sections from versioned files, not inline string building that drifts.

Keep tool definitions stable too — reordering tools changes the prefix hash. Pin tool order, descriptions, and schemas; version them deliberately.

For agents with long horizons, research shows naive caching still helps but ordered, compact context helps more. Keep working memory small and stable rather than appending endless history before the breakpoint.

OpenAI, Google, and Gateway Caches Compared

OpenAI and Google cache eligible prefixes automatically — no breakpoints, but the same stability rules apply. That convenience hides the same volatility traps, so monitor cached-token counters (cached_tokens > 0) the same way.

Google adds storage billing for cache entries (hourly per million tokens), which changes the math for huge rarely-reused prefixes. LiteLLM-style gateways add their own response caching (Redis, S3) for identical requests — a different layer that complements prefix caching rather than replacing it.

Multi-provider teams should abstract TTL and breakpoint config per provider behind one interface. The concepts transfer; the parameters don't.

Monitor Like Production — Dashboards, Alerts, Realistic Tests

Production caching needs a dashboard: creation versus read tokens, hit rate by prompt template, cost per request before and after. Alert when creation spikes without matching reads — that's volatility or a TTL shift, and it costs money every hour it persists.

Review provider changelogs on a schedule and re-verify minimums and TTL behavior quarterly. Parameters move: minimums, lifetimes, and multipliers have all changed within the last year.

Load-test with realistic pauses, not back-to-back blasts. Back-to-back tests show perfect hit rates that real traffic with coffee breaks never achieves — the TTL interacts with human rhythms, not benchmark loops.

# Nightly cache health probe: same prompt twice, second call must read
for i in 1 2; do
  curl -s https://api.anthropic.com/v1/messages \
    -H "x-api-key: $ANTHROPIC_API_KEY" \
    -H 'anthropic-version: 2023-06-01' \
    -d @cached_request.json | python3 -c "import json,sys; print(json.load(sys.stdin)['usage'])";
done
# Expect: call 1 shows cache_creation_input_tokens, call 2 shows cache_read_input_tokens