LLM Observability Tools — The $4k/month Token Leak We Caught at 3am

Three weeks ago, our recommendation engine started burning through $4,000/month in OpenAI API costs with no change in traffic. The p99 latency jumped from 2s to 8s. Users complained of timeouts. Our Grafana dashboard showed a flat line — no spikes, no errors. The system was 'working', just slower and more expensive. That's the lie LLM observability is supposed to catch.

Most tutorials hand you a tracing library and call it a day. They show you a pretty waterfall chart of one LLM call and tell you to instrument your app. They don't tell you that the real problems live in the gaps: retry storms from rate limits, token waste from prompt caching, cost attribution to a specific user who's gaming your system. They don't tell you that OpenTelemetry spans are only useful if you tag them with the right metadata.

This guide covers what I wish I'd known before that 3am page. We'll walk through production-grade LLM observability using OpenTelemetry and OpenLIT, with real code for tracing, metrics, and cost tracking. I'll show you the exact config that caught our token leak, the debug steps when your dashboard shows nothing useful, and the one metric you must alert on before your next bill arrives.

How LLM Observability Actually Works Under the Hood

LLM observability is not just API monitoring. A single user request can trigger a chain: prompt serialization, embedding lookup in a vector DB, context window management, multiple LLM calls (for reasoning, tool use, or re-ranking), and post-processing. Each step has its own latency, cost, and failure modes.

The core abstraction is the span. OpenTelemetry defines a span as a single operation with a start and end time, plus attributes. For LLM apps, you need spans at multiple levels: the user request, each LLM call, each vector DB query, and each tool invocation. The tricky part is linking them — you need a trace ID that propagates across service boundaries.

OpenLIT auto-instruments popular LLM libraries (OpenAI, LangChain, Anthropic) by monkey-patching the client's __call__ method. It creates a span for each LLM request, adds attributes like model, prompt_tokens, completion_tokens, and total_tokens, and exports them via OTLP. But auto-instrumentation only gets you so far. You still need to manually instrument your business logic: the prompt assembly, the retry logic, the caching layer.

What the docs don't tell you: span attributes are the difference between a useless waterfall chart and a cost-attribution dashboard. Tag every span with user_id, prompt_template, feature_name, and tenant_id. Without those, you can't answer 'which user is costing me $500/day?'

import openai
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

# Set up OpenTelemetry tracer
provider = TracerProvider()
processor = BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4317", insecure=True))
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)
tracer = trace.get_tracer(__name__)

client = openai.OpenAI()

def get_llm_response(prompt: str, user_id: str, template_name: str) -> str:
    with tracer.start_as_current_span("llm_call") as span:
        # Manual instrumentation: add business-logic attributes
        span.set_attribute("user_id", user_id)
        span.set_attribute("prompt_template", template_name)
        span.set_attribute("prompt_length", len(prompt))

        try:
            response = client.chat.completions.create(
                model="gpt-4",
                messages=[{"role": "user", "content": prompt}],
                max_tokens=500
            )
            # Auto-instrumentation by OpenLIT would add token counts here
            # But we add them manually for safety
            span.set_attribute("token_count", response.usage.total_tokens)
            span.set_attribute("model", response.model)
            return response.choices[0].message.content
        except openai.RateLimitError as e:
            span.set_attribute("error", True)
            span.set_attribute("error_type", "rate_limit")
            span.record_exception(e)
            raise

Practical Implementation: Setting Up OpenTelemetry + OpenLIT for Production

Let's walk through a production-ready setup. We'll use OpenLIT for auto-instrumentation of OpenAI and LangChain, then add manual instrumentation for business logic. We'll export traces and metrics to Grafana Cloud via OTLP.

First, install dependencies. Use pinned versions to avoid breakage. We learned this the hard way when OpenLIT 0.4.0 broke our LangChain integration.

Next, configure the OpenTelemetry SDK. The key decision is the exporter endpoint. For Grafana Cloud, you need the OTLP endpoint and a token. Never hardcode credentials — use environment variables.

Then, initialize OpenLIT. It will automatically patch openai.ChatCompletion.create and LangChain's LLMChain.run. But you still need to wrap your main request handler in a trace to link all spans together.

Finally, add custom metrics. The auto-instrumentation gives you latency histograms and token counts, but you need business metrics: requests per user, cost per template, error rate by model.

import os
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.resources import Resource
from opentelemetry.metrics import get_meter_provider, set_meter_provider
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
import openlit

# 1. Configure OpenTelemetry SDK with resource attributes
resource = Resource.create({
    "service.name": "llm-recommendation-engine",
    "service.version": "1.2.3",
    "deployment.environment": "production"
})

# 2. Set up trace exporter
# Grafana Cloud OTLP endpoint: https://otlp-gateway-prod-us-east-0.grafana.net/otlp
trace_exporter = OTLPSpanExporter(
    endpoint=os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT"),
    headers={"Authorization": f"Basic {os.getenv('GRAFANA_CLOUD_TOKEN')}"}
)
trace_provider = TracerProvider(resource=resource)
trace_provider.add_span_processor(BatchSpanProcessor(trace_exporter))
trace.set_tracer_provider(trace_provider)

# 3. Set up metric exporter
metric_exporter = OTLPMetricExporter(
    endpoint=os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT"),
    headers={"Authorization": f"Basic {os.getenv('GRAFANA_CLOUD_TOKEN')}"}
)
metric_reader = PeriodicExportingMetricReader(metric_exporter, export_interval_millis=10000)
meter_provider = MeterProvider(resource=resource, metric_readers=[metric_reader])
set_meter_provider(meter_provider)

# 4. Initialize OpenLIT for auto-instrumentation
openlit.init(
    application_name="llm-recommendation-engine",
    otlp_endpoint=os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT"),
    headers={"Authorization": f"Basic {os.getenv('GRAFANA_CLOUD_TOKEN')}"}
)

# 5. Now all OpenAI and LangChain calls are automatically traced
# But we still need to wrap the request handler for trace linkage
from opentelemetry import trace
tracer = trace.get_tracer(__name__)

def handle_request(user_id: str, prompt: str):
    with tracer.start_as_current_span("handle_request") as span:
        span.set_attribute("user_id", user_id)
        # The LLM call inside will be a child span
        response = get_llm_response(prompt, user_id, "default_template")
        return response

When NOT to Use OpenTelemetry for LLM Observability

OpenTelemetry is the standard, but it's not always the right choice. Here are three scenarios where you should consider alternatives.

1. You need real-time cost tracking at the sub-second level. OpenTelemetry's batch span processor exports every 5 seconds by default. If you need to enforce a per-user token budget in real time, you need a streaming approach. Consider using a middleware that sends token counts to a Redis counter or a streaming platform like Kafka.

2. You're running LLMs on a massive scale (10k+ requests/second). The OpenTelemetry collector can become a bottleneck. We saw the collector's CPU spike to 80% at 5k req/s. Consider sampling: use the tail-based sampler to keep only traces with errors or high latency.

3. You need deep prompt-level debugging. OpenTelemetry spans are not designed to store full prompts and responses. If you need to replay a specific conversation for debugging, store the prompts and responses in a separate store (e.g., S3 or a database) and link them to the trace via a trace ID.

For most teams, OpenTelemetry is the right choice. But know its limits.

Production Patterns & Scale: Cost Attribution and Tenant Isolation

At scale, the biggest challenge is cost attribution. You need to know which team, feature, or user is driving costs. This requires a consistent tagging strategy across all spans.

Pattern 1: Tag everything with tenant_id. If you have multiple customers or internal teams, tag every span with their ID. This lets you answer 'how much did tenant X cost us this month?'

Pattern 2: Use a cost calculation pipeline. Token counts are not enough. You need to multiply by the model's per-token cost. Create a batch job that reads spans from your observability backend, calculates cost per span, and writes it to a cost dashboard.

Pattern 3: Set per-tenant budgets. Use the cost data to enforce budgets. If tenant X exceeds their budget, you can throttle their requests or switch them to a cheaper model.

Pattern 4: Monitor prompt template drift. The same prompt template can produce wildly different token counts after a model update. Track token usage per template version and alert on significant changes.

import pandas as pd
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import read_span_data

# This is a simplified example. In production, you'd query your observability backend.
# We'll simulate reading spans from a parquet file.

def calculate_cost_per_tenant(span_file: str):
    df = pd.read_parquet(span_file)
    # Filter to LLM call spans
    llm_spans = df[df['span_name'] == 'llm_call']
    # Cost per token varies by model
    model_costs = {
        'gpt-4': 0.03 / 1000,  # $0.03 per 1K tokens
        'gpt-3.5-turbo': 0.002 / 1000,
        'text-embedding-ada-002': 0.0001 / 1000
    }
    llm_spans['cost'] = llm_spans.apply(
        lambda row: row['token_count'] * model_costs.get(row['model'], 0),
        axis=1
    )
    # Group by tenant_id
    cost_by_tenant = llm_spans.groupby('tenant_id')['cost'].sum()
    print(cost_by_tenant)
    # In production, write this to a database or dashboard
    return cost_by_tenant

if __name__ == '__main__':
    calculate_cost_per_tenant('spans_2025-05-22.parquet')

Common Mistakes with Specific Examples

Here are the three most common mistakes we've seen (and made) in production.

Mistake 1: Not setting max_retries on the OpenAI client. The default is 0. If you get a 429, the client raises an exception immediately. If you catch that exception and retry in a loop without exponential backoff, you create a retry storm. We saw this cause a 60% cost increase.

Mistake 2: Only monitoring error rates. A system can be 'working' (no errors) while burning money. Monitor token rate, cost rate, and latency distribution. Alert on deviations from baseline.

Mistake 3: Not tagging spans with prompt template version. When you update a prompt template, you need to know if the new version is more expensive or slower. Without template version tags, you can't attribute a cost spike to a prompt change.

Comparison vs Alternatives: OpenTelemetry vs Datadog LLM Observability

The main alternatives are OpenTelemetry (with OpenLIT) and Datadog's LLM Observability SDK. Here's a production comparison.

OpenTelemetry + OpenLIT: - Open source, vendor-agnostic. You can use any backend (Grafana, Jaeger, Prometheus). - Auto-instruments OpenAI, LangChain, Anthropic, and more. - Requires manual setup of the collector and exporters. - No built-in cost calculation or alerting. You build those yourself.

Datadog LLM Observability: - Proprietary, vendor-locked to Datadog. - Provides a higher-level SDK that includes cost tracking, evaluations, and a pre-built dashboard. - Easier to set up: just install the SDK and set environment variables. - More expensive at scale: Datadog's pricing is per-host + per-span.

When to choose OpenTelemetry: You want vendor independence, you already have a Grafana/Prometheus stack, or you need to customize your observability pipeline.

When to choose Datadog: You're already on Datadog, you need the higher-level features (evaluations, cost tracking), and you don't mind the vendor lock-in.

Debugging and Monitoring: The 3am Playbook

When you get paged at 3am because costs are spiking or latency is high, you need a systematic approach.

Step 1: Check the token rate. If costs are spiking but traffic is normal, look at token count per request. A sudden increase in token count per request indicates a prompt change or a retry storm.

Step 2: Check the retry rate. If p99 latency is high but p50 is normal, you likely have a retry storm. Check the retry_count span attribute.

Step 3: Check the model distribution. Did a recent deployment change the default model from gpt-3.5-turbo to gpt-4? That would explain a cost spike.

Step 4: Check the error rate by model. Some models have higher error rates. If you switched to a new model, it might be returning more errors, causing retries.

Step 5: Check the prompt template distribution. Did a new prompt template get deployed? It might be generating more tokens than expected.

import pandas as pd
from datetime import datetime, timedelta

# Simulate querying your observability backend
# In production, you'd use the Grafana API or Prometheus

def debug_cost_spike(start_time: datetime, end_time: datetime):
    # Query spans from the last hour
    df = pd.read_parquet('spans_latest.parquet')
    # Filter to the time range
    df = df[(df['timestamp'] >= start_time) & (df['timestamp'] <= end_time)]
    
    print("=== Step 1: Token rate ===")
    token_rate = df['token_count'].sum() / (end_time - start_time).total_seconds()
    print(f"Token rate: {token_rate:.2f} tokens/second")
    
    print("\n=== Step 2: Retry rate ===")
    retry_spans = df[df['retry_count'] > 0]
    print(f"Spans with retries: {len(retry_spans)}")
    
    print("\n=== Step 3: Model distribution ===")
    print(df['model'].value_counts())
    
    print("\n=== Step 4: Error rate by model ===")
    error_df = df[df['error'] == True]
    print(error_df.groupby('model').size())
    
    print("\n=== Step 5: Prompt template distribution ===")
    print(df['prompt_template'].value_counts())

if __name__ == '__main__':
    now = datetime.utcnow()
    debug_cost_spike(now - timedelta(hours=1), now)