Token Budgeting for LLMs — How a Missing Token Count Cost $4,000 in Overnight API Bills

We got paged at 2am because our chatbot started returning gibberish. The on-call engineer saw a 400 Bad Request on every third call. After 45 minutes of digging through logs, we found the culprit: a system prompt that had grown to 132,000 tokens overnight because a developer appended a full knowledge base dump without counting tokens. The GPT-4o API silently truncated the prompt at 128,000, dropping the user's question entirely. The model then responded to the truncated system prompt alone — and we were billed for every token of the oversized prompt.

Most tutorials on token budgeting stop at 'count your tokens with tiktoken' and call it a day. They don't tell you that different models use different tokenizers, that the context window is shared between input and output, or that a single runaway loop in a conversation can double your costs in minutes. They also skip the real production problem: how to enforce budgets programmatically before the API rejects your request.

This article covers exactly what you need to stop that 2am page. We'll walk through how tokenization actually works under the hood, how to build a reusable budget planner that splits your context window across system prompts, examples, history, and user input, and how to monitor and alert on token usage in production. We'll also show you the exact code that caused our $4,000 overnight bill — and the one-liner fix that prevents it.

How Tokenization Actually Works Under the Hood

Tokenization is not character counting. It's a model-specific encoding that maps text to integers via a learned vocabulary. GPT-4o uses the cl100k_base tokenizer, which has about 100,000 tokens in its vocabulary. Each token represents a common substring: 'hello' might be one token, but 'hello world' is two. This means a single word can be multiple tokens, and punctuation counts.

The critical production implication: different models use different tokenizers. GPT-4o and GPT-4o-mini both use cl100k_base, but Claude 3.5 Sonnet uses its own tokenizer. Counting tokens with the wrong encoder gives you incorrect budgets. We learned this when a developer used a character-count heuristic and ended up with prompts that were 30% larger than expected.

Another hidden detail: the tokenizer is bidirectional in practice. The same text tokenizes identically whether it's at the start or end of the prompt. But the model's attention mechanism is not — it can lose information in the middle. So token count alone isn't enough; you also need to structure your prompt to keep critical content at the edges.

import tiktoken

# GPT-4o uses cl100k_base. Always use the model-specific encoder.
enc = tiktoken.get_encoding("cl100k_base")

text = "Hello, world! This is a test prompt."
tokens = enc.encode(text)

print(f"Text length: {len(text)} characters")
print(f"Token count: {len(tokens)}")
print(f"Tokens: {tokens}")

# Decode back to verify no information loss
decoded = enc.decode(tokens)
print(f"Decoded: {decoded}")

# Common pitfall: newlines and whitespace count
# A single newline is one token!
multi_line = "Line 1\nLine 2"
print(f"Multi-line tokens: {len(enc.encode(multi_line))}")

Building a Reusable Token Budget Planner

A token budget planner splits the context window into fixed and variable components. Fixed components include the system prompt and few-shot examples. Variable components include conversation history and user input. The output budget must be reserved from the total.

Here's the formula: budget_input = context_window - reserved_output. Then you allocate budget_input across system prompt, examples, history, and user input. If the total exceeds budget_input, you must truncate history or reject the request.

A common mistake is to forget that the output counts toward the total token limit. GPT-4o has a 128,000 token context window, but max output is 16,384. If you set max_tokens=16384, the input can only be 111,616 tokens. But if you don't set max_tokens, the model might use all 128,000 for output and leave no room for input.

We enforce budgets with a simple class that tracks usage per component and raises an exception when the budget is exceeded. This prevents the silent truncation that cost us $4,000.

import tiktoken
from typing import List, Dict

class TokenBudgetPlanner:
    def __init__(self, model: str = "gpt-4o", reserved_output: int = 4096):
        self.enc = tiktoken.encoding_for_model(model)
        # GPT-4o context window is 128,000 tokens
        self.context_window = 128000
        self.reserved_output = reserved_output
        self.input_budget = self.context_window - self.reserved_output
        self.current_usage = 0

    def count_tokens(self, text: str) -> int:
        return len(self.enc.encode(text))

    def add_message(self, role: str, content: str) -> None:
        tokens = self.count_tokens(content)
        # Account for message format overhead (~4 tokens per message)
        tokens += 4
        if self.current_usage + tokens > self.input_budget:
            raise BudgetExceededError(
                f"Adding {role} message would exceed budget. "
                f"Current: {self.current_usage}, Adding: {tokens}, Budget: {self.input_budget}"
            )
        self.current_usage += tokens

    def get_available_input(self) -> int:
        return self.input_budget - self.current_usage

class BudgetExceededError(Exception):
    pass

# Usage
planner = TokenBudgetPlanner(reserved_output=4096)
planner.add_message("system", "You are a helpful assistant.")
planner.add_message("user", "What is the capital of France?")
print(f"Available input tokens: {planner.get_available_input()}")

When NOT to Use Token Budgeting

Token budgeting is essential for cost control, but it's not always the right tool. If you're doing batch inference with short prompts, the overhead of counting tokens per request can outweigh the savings. We saw a team add token counting to a batch job that processed 10,000 short prompts per minute. The tiktoken calls added 30% latency. They were better off estimating a fixed budget per prompt and only counting on a sample.

Another case: if your model has a very large context window (like Gemini 2.0 Flash at 1M tokens), budgeting becomes less about overflow and more about cost. At $0.10 per 1M input tokens, the cost of a full-window request is negligible. But the latency of processing 1M tokens is not — it can take 30+ seconds. In that case, budget for latency, not cost.

Finally, don't use token budgeting for tasks where the model needs the full context to be accurate. Legal document analysis or medical record summarization requires the entire input. Truncating history could introduce errors. In those cases, pay for the larger model or use a model with a bigger context window.

import time
import tiktoken

# Batch inference: 10,000 short prompts
prompts = ["What is 2+2?"] * 10000
enc = tiktoken.get_encoding("cl100k_base")

# Bad: count tokens for every prompt
start = time.time()
for p in prompts:
    _ = len(enc.encode(p))
print(f"Counting all: {time.time() - start:.2f}s")

# Good: count once and reuse
start = time.time()
first_count = len(enc.encode(prompts[0]))
print(f"Counting once: {time.time() - start:.2f}s")
# For uniform prompts, the variance is negligible

Production Patterns for Token Budgeting at Scale

At scale, token budgeting becomes a systems design problem. You need to track usage per request, per user, and per model. We use a middleware pattern that intercepts every API call, counts tokens, and enforces budgets before the request hits the model.

Three patterns we use in production:

1. Sliding Window: Keep only the last N messages in conversation history. N depends on the model's context window and your output budget. For GPT-4o with 128K window and 16K output, we keep the last 10 messages plus the system prompt.
2. Token Budget Header: Add a custom header to every API call that includes the current token usage. Log this in your monitoring system. This lets you trace cost spikes back to specific users or features.
3. Budget Exceeded Webhook: When a request exceeds the budget, instead of failing silently, send a webhook to the developer with the exact token counts. This turns a silent failure into an actionable alert.

import openai
from functools import wraps
import tiktoken

class TokenBudgetMiddleware:
    def __init__(self, max_input_tokens: int = 100000):
        self.max_input_tokens = max_input_tokens
        self.enc = tiktoken.get_encoding("cl100k_base")

    def __call__(self, func):
        @wraps(func)
        def wrapper(messages, **kwargs):
            # Count tokens for all messages
            total_tokens = sum(len(self.enc.encode(m["content"])) + 4 for m in messages)
            if total_tokens > self.max_input_tokens:
                raise BudgetExceededError(
                    f"Input tokens {total_tokens} exceed budget {self.max_input_tokens}"
                )
            # Add token count to kwargs for logging
            kwargs["extra_headers"] = {"X-Token-Count": str(total_tokens)}
            return func(messages, **kwargs)
        return wrapper

# Apply to OpenAI client
client = openai.OpenAI()
original_create = client.chat.completions.create
client.chat.completions.create = TokenBudgetMiddleware()(original_create)

Common Token Budgeting Mistakes (With Specific Examples)

We've seen the same mistakes across multiple teams. Here are the top three:

Mistake 1: Using character count instead of token count. A developer wrote len(prompt) > 100000 thinking 100K characters equals 100K tokens. In reality, 100K characters is about 25K tokens for English text. They were undercounting by 4x. The fix: always use tiktoken.

Mistake 2: Forgetting to reserve output tokens. Another team set max_tokens=16384 but didn't subtract it from the input budget. They sent 120K tokens of input, which with 16K output totaled 136K — exceeding the 128K limit. The API returned a 400 error. The fix: subtract max_tokens from the context window before allocating input.

Mistake 3: Not accounting for tokenizer differences between models. A team switched from GPT-4o to Claude 3.5 Sonnet but kept using the cl100k_base tokenizer. Claude's tokenizer produces different counts. They ended up with prompts that were 20% larger than expected, causing context overflows. The fix: use the model-specific tokenizer.

import tiktoken

# Mistake 1: Character count
prompt = "Hello, world!" * 1000
char_count = len(prompt)
token_count = len(tiktoken.get_encoding("cl100k_base").encode(prompt))
print(f"Characters: {char_count}, Tokens: {token_count}")
# Characters: 13000, Tokens: ~3250

# Mistake 2: Not reserving output
context_window = 128000
max_tokens = 16384
input_tokens = 120000
total = input_tokens + max_tokens  # 136384 > 128000
print(f"Total exceeds limit: {total} > {context_window}")

# Mistake 3: Wrong tokenizer
# Claude uses its own tokenizer, not cl100k_base
# Always check the model's documentation

Token Budgeting vs. Other Cost Control Strategies

Token budgeting is one of several cost control strategies. Here's how it compares:

Token Budgeting: Proactive. You count tokens before the call and reject or truncate if over budget. Best for real-time applications where you control the input.

Cost Alerts: Reactive. You monitor API costs and alert when they exceed a threshold. Useful as a safety net, but by the time you get the alert, the money is already spent.

Model Selection: Proactive. Use a cheaper model for simple tasks and a more expensive one for complex tasks. GPT-4o-mini costs 1/17th of GPT-4o per input token. But it also has lower accuracy.

Caching: Proactive. Cache responses for identical prompts. This works well for system prompts and few-shot examples, but not for user-specific queries.

We use all four in combination. Token budgeting is the first line of defense. Cost alerts catch anything that slips through. Model selection reduces baseline costs. Caching eliminates redundant calls.

import openai
from functools import lru_cache

# Model selection: route based on task complexity
def route_to_model(task: str) -> str:
    if task == "simple_qa":
        return "gpt-4o-mini"  # $0.15/M input
    elif task == "complex_reasoning":
        return "gpt-4o"  # $2.50/M input
    else:
        return "gpt-4o-mini"

# Caching: cache system prompt responses
@lru_cache(maxsize=1000)
def get_system_response(system_prompt: str, user_prompt: str) -> str:
    client = openai.OpenAI()
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": user_prompt}
        ],
        max_tokens=100
    )
    return response.choices[0].message.content

Debugging and Monitoring Token Budgets in Production

Monitoring token budgets requires logging token counts per request and setting alerts on anomalies. We use structured logging with a 'token_usage' field that includes prompt_tokens, completion_tokens, and total_tokens. This lets us query for outliers: 'find all requests where total_tokens > 100000'.

We also track token usage per user and per feature. If a single user's token usage spikes, it's likely a runaway loop in their conversation. If a feature's token usage spikes, it's likely a bug in the prompt construction.

import json
import logging
from datetime import datetime

# Structured logging for token usage
logger = logging.getLogger("token_budget")
handler = logging.FileHandler("token_usage.log")
formatter = logging.Formatter('%(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)
logger.setLevel(logging.INFO)

def log_token_usage(user_id: str, feature: str, prompt_tokens: int, completion_tokens: int):
    log_entry = {
        "timestamp": datetime.utcnow().isoformat(),
        "user_id": user_id,
        "feature": feature,
        "prompt_tokens": prompt_tokens,
        "completion_tokens": completion_tokens,
        "total_tokens": prompt_tokens + completion_tokens
    }
    logger.info(json.dumps(log_entry))

# Query for outliers (shell command)
# grep 'total_tokens' token_usage.log | python -c "import sys,json; logs=[json.loads(l) for l in sys.stdin]; print(max(l['total_tokens'] for l in logs))"