Jaeger Missing Spans — Async Context Propagation Fix

Missing spans in Jaeger are almost always caused by broken context propagation across async boundaries, not sampling or network failures. When trace IDs fail to cross thread pools, message queues, or background workers, traces fragment into orphaned spans that hide critical latency bottlenecks. This article shows how to diagnose and fix async context propagation using OpenTelemetry's propagator API.

How Distributed Tracing with Jaeger Works — and Why Spans Go Missing

Distributed tracing with Jaeger tracks a single request as it propagates through microservices by assigning a unique trace ID and attaching spans — each span representing a unit of work with start time, duration, and metadata. The core mechanic is context propagation: the trace ID must be passed across service boundaries via HTTP headers (or message queue metadata) so that spans from different services can be stitched into one trace. Without correct propagation, spans become orphaned and the trace is incomplete.

Jaeger stores traces in a backend (Cassandra, Elasticsearch, or Kafka) and exposes them via a UI. In practice, each service must extract the incoming trace context, create child spans, and inject the context into outgoing requests. This is typically done with OpenTelemetry SDKs, which handle serialization and deserialization of trace context. The key property that matters: if any service in the chain fails to propagate context — due to async boundaries, thread pool switches, or manual HTTP clients — the trace breaks at that point.

Use Jaeger when you need to debug latency spikes, identify service dependencies, or trace errors across more than three services. In production systems handling thousands of requests per second, a single missing span can hide a 500ms bottleneck in a downstream service. Without tracing, you're debugging blind — logs give you local state, but only traces show the full causal chain.

Instrumenting a FastAPI App with OpenTelemetry

# pip install opentelemetry-distro opentelemetry-exporter-otlp
# pip install opentelemetry-instrumentation-fastapi

from fastapi import FastAPI
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor

# Configure tracer
provider = TracerProvider()
exporter = OTLPSpanExporter(endpoint='http://jaeger:4317')  # Jaeger OTLP endpoint
provider.add_span_processor(BatchSpanProcessor(exporter))
trace.set_tracer_provider(provider)

app = FastAPI()
FastAPIInstrumentor.instrument_app(app)  # auto-instruments all routes

tracer = trace.get_tracer(__name__)

@app.get('/orders/{order_id}')
async def get_order(order_id: int):
    with tracer.start_as_current_span('fetch-order') as span:
        span.set_attribute('order.id', order_id)

        # Manual span for a specific operation
        with tracer.start_as_current_span('db-query'):
            order = await db.get_order(order_id)

        with tracer.start_as_current_span('enrich-order'):
            user = await user_service.get_user(order.user_id)  # cross-service call

        return {'order': order

Running Jaeger with Docker

# Run Jaeger all-in-one (development setup)
docker run -d \n  --name jaeger \n  -p 16686:16686 \n  -p 4317:4317 \n  -p 4318:4318 \n  jaegertracing/all-in-one:latest

# Ports:
# 16686 — Jaeger UI
# 4317  — OTLP gRPC receiver
# 4318  — OTLP HTTP receiver

# Open Jaeger UI: http://localhost:16686
# Search by service name → see all traces
# Click a trace → see full span timeline
# Click a span → see attributes, events, errors

Understanding Spans, Traces, and Context Propagation

A span is a named, timed operation that carries a span ID, trace ID, parent span ID, and attributes. The entire set of spans linked by a common trace ID forms a trace. Context propagation is what connects spans across service boundaries — it passes the trace ID and parent span ID via HTTP headers (W3C Trace Context: traceparent and tracestate). Without propagation, each service creates a separate trace, and you lose the end-to-end view. Propagation is automatic when using OpenTelemetry instrumentation libraries (they inject headers on outgoing requests). If you use raw HTTP clients or message queues, you must manually inject and extract the context.

import requests
from opentelemetry import propagators, trace
from opentelemetry.propagate import inject, extract

# Sending service
tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span('outgoing-call') as span:
    headers = {}
    inject(headers)  # injects traceparent from current span
    response = requests.get('http://payment-service/process', headers=headers)

# Receiving service
# In middleware or at route entry, extract context
from opentelemetry.propagate import extract
ctx = extract(request.headers)
with tracer.start_as_current_span('process-payment', context=ctx) as span:
    # this span is now child of the sending service's span
    pass

Sampling Strategies in Production

Recording every trace at production scale is expensive — both in storage and network bandwidth. Sampling decides which traces to keep. Head-based sampling makes the decision at the start of a request (e.g., keep 1% of all traces). It's simple but can miss rare high-latency events. Tail-based sampling buffers traces and decides after they complete, keeping those that exceed a latency threshold or contain errors. Jaeger supports both. A common hybrid approach: sample 1–5% of normal requests and always sample requests with HTTP 5xx or custom error attributes.

Setting the right sampling rate is a trade-off. Too low (0.1%) and you'll miss most issues; too high (100%) for high-throughput services will overwhelm storage. Start at 1% and adjust based on storage budget and trace usefulness.

# Remote sampling configuration for Jaeger
service_config:
  - service: "order-service"
    operation: "/orders/{id}"
    probabilistic_sampling:
      sampling_rate: 0.01  # 1% sample
  - service: "order-service"
    operation: "*"
    probabilistic_sampling:
      sampling_rate: 0.005  # 0.5% for other ops
  - service: "payment-service"
    operation: "*"
    probabilistic_sampling:
      sampling_rate: 0.05  # 5% for payment (higher risk)

Troubleshooting Missing Spans and Broken Traces

When a trace doesn't appear in Jaeger UI, or appears incomplete, the root cause is almost always one of: (1) context not propagated, (2) spans not exported, (3) sampling dropped the trace, (4) clock skew between service hosts. Use the following systematic checks. First, confirm you're hitting the Jaeger endpoint by looking at application logs for OTLP export errors. Second, check the trace ID uniformity — if each service generates its own trace ID, propagation is missing. Third, verify that the "Trace" view shows all expected spans — missing spans may indicate a failing exporter or network issue. Fourth, if spans from different services appear with wrong timing, check NTP synchronisation: Jaeger relies on span timestamps for ordering.

Integrating Traces with Logs and Metrics

Distributed tracing alone doesn't replace logs or metrics — it complements them. The true power emerges when you correlate trace IDs with log entries and metric events. OpenTelemetry enables this via trace_id injection into log records (MDC in Java, structlog in Python). Metric tools like Prometheus can use trace IDs in labels for targeted alerting. Jaeger's UI allows you to drill from a trace to related logs if you configure the log integration.

A common production pattern: when a latency alert fires, grab the trace ID from the affected request, open Jaeger to see the breakdown, then jump to the logs from that span ID to inspect the exact error message.

import structlog
from opentelemetry import trace

span = trace.get_current_span()
trace_id = format(span.get_span_context().trace_id, '032x')
span_id = format(span.get_span_context().span_id, '016x')

# Inject trace context into log
logger = structlog.get_logger()
logger.info("payment processed", trace_id=trace_id, span_id=span_id, order_id=123)

Why Your Traces Are Silent: The gRPC vs HTTP Exporter Trap

Most beginners copy-paste a Jaeger exporter configuration and wonder why their traces never show up. The culprit is almost always the gRPC endpoint. Jaeger all-in-one runs three separate ports: 14250 for gRPC, 14268 for HTTP Thrift, and 9411 for Zipkin. If you use JaegerExporter (gRPC) but your Jaeger container isn't listening on 14250, your spans vanish into the void. I've debugged this in three separate microservice migrations. The fix is brutally simple: match your exporter to Jaeger's open port. For HTTP, use ThriftExporter. For gRPC, ensure COLLECTOR_GRPC_PORT is set. Don't assume both endpoints are active—check your docker logs. This mismatch wastes hours for teams that could be shipping features.

from opentelemetry.exporter.jaeger.thrift import JaegerExporter as ThriftExporter
from opentelemetry.exporter.jaeger.proto.grpc import JaegerExporter as GrpcExporter

# HTTP exporter (port 14268)
http_exporter = ThriftExporter(
    agent_host_name="localhost",
    agent_port=6831,
)

# gRPC exporter (port 14250)
grpc_exporter = GrpcExporter(
    collector_endpoint="http://localhost:14250",
    insecure=True
)

print("HTTP exporter configured on port 6831")
print("gRPC exporter configured on port 14250")

Sampling in Production: Don't Bankrupt Your Storage on Every Request

In development, trace every request. In production, that costs real money—storage, network, and CPU. Smart teams use head-based sampling to keep the firehose manageable. Jaeger's probabilistic sampler with a 5-10% rate catches most anomalies without exploding your budget. But here's the trick: combine it with rate-limiting per endpoint. Your health-check endpoint doesn't need tracing at all. Your payment service deserves higher sampling. I've seen a startup burn $2,000/month on Jaeger storage because they sampled 100% on a high-traffic API. Set sampler.type=probabilistic and sampler.param=0.1 in your OpenTelemetry config. For critical flows, inject a custom sampler that always traces on errors. Your SRE team will thank you.

from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.sampling import Sampler, Decision

class HealthCheckAwareSampler(Sampler):
    def should_sample(self, parent_context, trace_id, name, kind, attributes, links):
        # Skip tracing for health checks
        if name == "/health":
            return Decision.DROP
        # Sample 20% for everything else
        return Decision.RECORD_AND_SAMPLE if trace_id % 10 < 2 else Decision.DROP

trace.set_tracer_provider(
    TracerProvider(sampler=HealthCheckAwareSampler())
)

print("Custom sampler applied: health checks dropped, 20% sampling rate for others")