OpenTelemetry in Spring Boot 3: Distributed Tracing for Production Observability

Here's the hard truth: most teams get observability wrong. They slap on basic logging and wonder why production incidents take hours to debug. In microservices, a single user request can fan out across 10+ services, databases, and queues. When that request fails—say a payment timeout in a SaaS billing pipeline—you need more than stack traces. You need the full story: which service dropped the ball, how long each hop took, and what data was in flight.

This is where OpenTelemetry (OTel) comes in. It's the industry standard for distributed tracing, born from the merger of OpenTracing and OpenCensus. Spring Boot 3.2+ ships with first-class OTel support via Micrometer Tracing, replacing the deprecated Spring Cloud Sleuth. With a few annotations and configuration properties, you get end-to-end trace IDs propagated across HTTP calls, messaging queues, and database queries.

In this guide, we'll instrument a real-time analytics service step-by-step. You'll learn to trace gRPC calls, handle async operations, and export to Jaeger. We'll also dissect a production incident where distributed tracing saved a $50K/hour outage. By the end, you'll have production-grade observability that your on-call team will thank you for.

Why OpenTelemetry Is the Only Observability Standard That Matters

If you're still wiring Micrometer or Jaeger agents directly into your Spring Boot 3 app, you're building technical debt. OpenTelemetry (OTel) is the CNCF-graduated standard that unifies traces, metrics, and logs under a single API. Spring Boot 3.2+ ships with io.micrometer:micrometer-tracing-bridge-otel and io.opentelemetry:opentelemetry-exporter-otlp as first-class citizens. Here's the hard truth: most teams configure this incorrectly by mixing manual Span creation with auto-instrumentation, leading to broken context propagation. Start with the spring-boot-starter-actuator and add management.tracing.enabled=true in your application.yml. Then, for production, always set spring.sleuth.enabled=false (yes, Sleuth is dead—migrate now). Below is the minimal auto-configuration that works with any OTLP-compatible backend like Grafana Tempo or Honeycomb.

import io.micrometer.tracing.Tracer;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class OpenTelemetryConfig {
    @Bean
    public Tracer tracer() {
        return Tracer.NOOP;
    }
}

// application.yml
// management:
//   tracing:
//     enabled: true
//   otlp:
//     tracing:
//       endpoint: http://localhost:4318/v1/traces

What the Official Docs Won't Tell You

The Spring Boot 3.2 reference guide says to add io.micrometer:micrometer-tracing-bridge-otel and io.opentelemetry:opentelemetry-exporter-otlp—but it doesn't tell you that spring.sleuth.enabled=false is mandatory if you ever had Sleuth on the classpath, even if you removed the dependency. The OTel SDK's BatchSpanProcessor has a default export timeout of 30 seconds, which will drop traces under high load in a real-time analytics system. You must set otel.bsp.schedule.delay=500 and otel.bsp.max.export.batch.size=512 in your application.properties to avoid data loss. Also, the @NewSpan annotation from Micrometer is broken with spring.jpa.open-in-view=true because Hibernate sessions close before the span ends. The fix: set spring.jpa.open-in-view=false or use @Transactional(readOnly=true) on your repositories.

# Critical overrides for production OTel
spring.sleuth.enabled=false
management.tracing.enabled=true
otel.bsp.schedule.delay=500
otel.bsp.max.export.batch.size=512
otel.bsp.max.queue.size=2048
spring.jpa.open-in-view=false

Custom Context Propagation for Async and Reactive Pipelines

Distributed tracing breaks the moment you cross thread boundaries—@Async, CompletableFuture, WebFlux, or Kafka listeners. Spring Boot 3's Micrometer Tracing uses ThreadLocal context, which won't propagate to worker threads unless you explicitly wrap your code. For @Async, annotate your configuration with @EnableAsync and add a Executor bean that uses ContextSnapshotFactory from Micrometer. For reactive stacks (Spring WebFlux + Reactor), you must install the reactor-core micrometer hook: Hooks.enableAutomaticContextPropagation(). Without this, every flatMap() call loses the trace context. For Kafka consumers, set spring.kafka.consumer.properties[spring.tracing.enabled]=true—but note that this only works with spring-kafka 3.0+. Here's the reactor setup that saved a real-time analytics pipeline from context loss.

import reactor.core.publisher.Hooks;
import jakarta.annotation.PostConstruct;
import org.springframework.context.annotation.Configuration;

@Configuration
public class ReactiveTracingConfig {
    @PostConstruct
    public void enableReactorContext() {
        Hooks.enableAutomaticContextPropagation();
    }
}

// For @Async support
// @Bean
// public Executor asyncExecutor() {
//     return ContextSnapshotFactory.builder().build()
//         .wrap(Executors.newVirtualThreadPerTaskExecutor());
// }

Sampling Strategies That Won't Bankrupt Your Observability Budget

In production, tracing every request is financially and operationally infeasible—a payment-processing system handling 10k req/s generates 5GB of trace data per hour. The default AlwaysOnSampler is a rookie mistake. Use TraceIdRatioBased at 1-5% for general traffic, but implement head-based sampling with a custom Sampler for critical paths. Spring Boot 3 allows you to define a Sampler bean: return Sampler.alwaysSample() for /api/payments and Sampler.neverSample() for /health. For tail-based sampling, you need an OTel collector configuration with tail_sampling processor, which samples based on latency or error status. The collector is mandatory for multi-service traces because head-based sampling can drop the root span. Below is a Spring Boot sampler that prioritizes high-value transactions without drowning your backend.

import io.opentelemetry.api.common.AttributeKey;
import io.opentelemetry.api.trace.Sampler;
import io.opentelemetry.api.trace.SpanKind;
import io.opentelemetry.context.Context;
import io.opentelemetry.sdk.trace.data.LinkData;
import io.opentelemetry.sdk.trace.samplers.SamplerResult;
import java.util.List;

public class CustomSampler implements Sampler {
    private static final Sampler ALWAYS = Sampler.alwaysOn();
    private static final Sampler RATIO = Sampler.traceIdRatioBased(0.05);

    @Override
    public SamplerResult shouldSample(Context ctx, String traceId, String name,
            SpanKind kind, Attributes attrs, List<LinkData> parentLinks) {
        if (name.contains("/api/payments") || attrs.get(AttributeKey.stringKey("error")) != null) {
            return ALWAYS.shouldSample(ctx, traceId, name, kind, attrs, parentLinks);
        }
        return RATIO.shouldSample(ctx, traceId, name, kind, attrs, parentLinks);
    }

    @Override
    public String getDescription() { return "CustomSampler"; }
}

Production Debugging with OpenTelemetry and Spring Boot 3

When your distributed system starts misbehaving in production, OpenTelemetry is your first responder. Here's the hard truth: most teams get this wrong by only instrumenting happy paths. You need to trace failures, timeouts, and retries. In Spring Boot 3.2+, use the @WithSpan annotation on service methods that interact with external systems. For example, annotate your payment gateway call with @WithSpan("processPayment") and add attributes like paymentId and amount. This lets you correlate a failed payment with a spike in latency. Use the OpenTelemetry Java agent (opentelemetry-javaagent.jar v1.32+) for zero-code instrumentation of JDBC, HTTP clients, and messaging. But don't stop there: manually add spans for critical business logic using Span.current(). In your configuration, ensure otel.traces.exporter=otlp and otel.exporter.otlp.endpoint=http://localhost:4318. For high-traffic services, sample at 10% using otel.traces.sampler=parentbased_traceidratio and otel.traces.sampler.arg=0.1. This reduces storage costs while keeping enough traces for anomaly detection. Use Grafana Tempo or Jaeger to visualize traces. A common production issue is missing context propagation across thread pools; always use @Async with a custom TaskDecorator that propagates the OpenTelemetry context. Otherwise, you'll orphan spans and lose the trace chain.

import io.opentelemetry.api.trace.Span;
import io.opentelemetry.instrumentation.annotations.WithSpan;
import org.springframework.stereotype.Service;

@Service
public class PaymentService {

    @WithSpan("processPayment")
    public PaymentResult processPayment(PaymentRequest request) {
        Span span = Span.current();
        span.setAttribute("payment.id", request.getId());
        span.setAttribute("payment.amount", request.getAmount());
        // Simulate external call
        PaymentGatewayResponse response = paymentGateway.charge(request);
        if (!response.isSuccess()) {
            span.setAttribute("payment.error", response.getErrorCode());
            span.setStatus(StatusCode.ERROR, "Payment declined");
        }
        return new PaymentResult(response);
    }
}

Troubleshooting Trace Gaps and Missing Spans

You've instrumented your Spring Boot 3.2 application, but traces are incomplete or missing entirely. This is the #1 complaint from teams adopting OpenTelemetry. First, verify the agent is attached: check JVM arguments for -javaagent:opentelemetry-javaagent.jar. Second, ensure the OTLP exporter endpoint is reachable. Use curl http://localhost:4318/v1/traces to test. Third, check for library conflicts. Spring Boot 3.2 ships with Micrometer Tracing 1.2.x, which can clash with the OpenTelemetry agent. Remove micrometer-tracing-bridge-otel from your dependencies if using the agent. Fourth, inspect your application.properties for missing configuration. Common mistake: setting otel.traces.exporter=none by accident. Fifth, enable debug logging: logging.level.io.opentelemetry=DEBUG. This will show you if spans are being created but not exported. Sixth, check for sampling decisions. If you set a low sampling rate, most traces will be dropped. Use otel.traces.sampler=always_on temporarily for debugging. Seventh, verify context propagation across HTTP boundaries. For REST APIs, ensure your RestTemplate or WebClient uses the OpenTelemetry instrumentation. Add the opentelemetry-instrumentation-spring-webmvc library. Eighth, test with a simple endpoint: create a controller that calls a traced service and verify the trace appears in your backend. Ninth, if using Kafka, add the opentelemetry-instrumentation-kafka-clients-2.6 library and configure the interceptor. Tenth, check for thread-local issues: never reuse threads without propagating context. Use a ThreadPoolTaskExecutor with a custom TaskDecorator that wraps the OpenTelemetry context.

# Ensure exporter is set
otel.traces.exporter=otlp
otel.exporter.otlp.endpoint=http://localhost:4318

# Sampling for debugging
otel.traces.sampler=always_on

# Enable debug logging
logging.level.io.opentelemetry=DEBUG

# Remove if using agent
# management.tracing.bridge=otel

Advanced OpenTelemetry Configuration for High-Volume Production

In high-volume production environments, naive OpenTelemetry configuration will kill your performance and blow your storage budget. Here's how to configure for scale. Use batch span processing: set otel.bsp.schedule.delay=100 (milliseconds) and otel.bsp.max.export.batch.size=512. This reduces export overhead. For sampling, use head-based sampling with a consistent probability. In Spring Boot 3.2, configure otel.traces.sampler=parentbased_traceidratio with otel.traces.sampler.arg=0.1 for 10% sampling. But for critical services (e.g., payment processing), use always_on and rely on tail-based sampling in your backend (e.g., Grafana Tempo). To reduce span attributes, avoid adding high-cardinality attributes like user emails or session IDs. Instead, use span events for detailed data. For example, log request payloads as events: span.addEvent("request.payload", Attributes.of(AttributeKey.stringKey("payload"), requestBody)). This keeps traces searchable without bloating indexes. Use attribute limits: set otel.attribute.count.limit=128 and otel.attribute.value.length.limit=4096. This prevents a single span from consuming too much memory. For multi-region deployments, use different exporter endpoints per region: otel.exporter.otlp.endpoint=http://otel-collector.region1:4318. Finally, integrate with your existing monitoring stack. Export metrics via otel.metrics.exporter=otlp and logs via OpenTelemetry Collector. In Spring Boot 3.2, use Micrometer to expose JVM metrics and correlate them with traces via trace_id in logs.

# Sampling: 10% for high volume
otel.traces.sampler=parentbased_traceidratio
otel.traces.sampler.arg=0.1

# Batch processing
otel.bsp.schedule.delay=100
otel.bsp.max.export.batch.size=512

# Attribute limits
otel.attribute.count.limit=128
otel.attribute.value.length.limit=4096

# Exporter
otel.exporter.otlp.endpoint=http://otel-collector:4318
otel.metrics.exporter=otlp

Integrating OpenTelemetry with Spring Boot Actuator and Micrometer

Spring Boot Actuator and Micrometer are your allies in building a unified observability pipeline. In Spring Boot 3.2+, Micrometer Tracing is integrated with OpenTelemetry via the micrometer-tracing-bridge-otel dependency. However, if you're using the OpenTelemetry Java agent, remove this bridge to avoid double-instrumentation. Here's the correct setup: include micrometer-registry-otlp for metrics export, and use the agent for traces. Configure Actuator to expose metrics: management.endpoints.web.exposure.include=health,metrics,prometheus. Then, use Micrometer's @Timed annotation on controller methods to capture latency histograms. For example: @Timed(value = "payment.processing", description = "Time taken to process payment", histogram = true). This generates metrics like payment.processing_seconds with percentiles. Correlate these metrics with traces by adding the trace_id as a metric tag. In your custom metrics, use MeterRegistry to create counters and timers, and add the current trace ID from Span.current().getSpanContext().getTraceId(). This enables drill-down from a metric spike to the specific trace. For logs, use Structured Logging (e.g., Logback with JSON layout) and include trace_id and span_id in each log entry. In Spring Boot 3.2, configure logging.pattern.console=%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n and add a custom converter to inject trace IDs. Finally, use the OpenTelemetry Collector to receive traces, metrics, and logs, and forward them to your observability backend (e.g., Grafana LGTM stack). This unified approach eliminates silos and provides a single pane of glass for debugging.

import io.micrometer.core.instrument.MeterRegistry;
import io.micrometer.core.instrument.Timer;
import io.opentelemetry.api.trace.Span;
import org.springframework.context.annotation.Configuration;

@Configuration
public class MetricsConfig {

    public MetricsConfig(MeterRegistry meterRegistry) {
        Timer.Sample sample = Timer.start(meterRegistry);
        // After business logic
        Span span = Span.current();
        String traceId = span.getSpanContext().getTraceId();
        Timer timer = Timer.builder("custom.operation")
                .tag("trace_id", traceId)
                .register(meterRegistry);
        sample.stop(timer);
    }
}