Docker Observability with OpenTelemetry: Replace Your Scattered Monitoring Stack

Your microservices are running in Docker containers. You have Prometheus scraping metrics, Jaeger for traces, and the ELK stack for logs — three separate systems, each with its own configuration, storage, and alerting. A slow database query shows up as a high P99 latency in Prometheus, but connecting that to the specific trace in Jaeger and the error log in Kibana requires manual correlation. This fragmented approach is the legacy of the 2010s observability landscape.

OpenTelemetry changes this by unifying all three signals — traces, metrics, and logs — into a single data model and pipeline. The OTel Collector becomes the single point of configuration, processing, and routing. You define receivers (where data comes from), processors (what to do with it — sample, filter, enrich, batch), and exporters (where to send it). The same Collector instance can forward traces to Jaeger, metrics to Prometheus via prometheusremotewrite, and logs to any OTLP-compatible backend.

The latest evolution: OTel eBPF Instrumentation (OBI), which uses eBPF to automatically instrument network calls, database queries, and messaging without any code changes. This is game-changing for legacy applications that cannot be modified to add OTel SDKs. The eBPF agent attaches to network syscalls and reconstructs HTTP/gRPC spans, then exports them via OTLP to the Collector.

By the end of this guide, you'll have a complete OTel observability stack running in Docker — Collector, eBPF instrumentation, Prometheus storage, and trace/visualization — configured end-to-end with examples you can adapt to any application stack.

OTel Collector Architecture: Receivers, Processors, and Exporters

The OTel Collector is a pipeline — data flows from receivers through processors to exporters. Understanding this pipeline is the key to configuring OTel for any Docker environment.

Receivers are the data entry points. otlp receiver accepts OTel Protocol (gRPC on 4317, HTTP on 4318). prometheus receiver scrapes Prometheus endpoints. filelog receiver tails log files. hostmetrics receiver collects host-level metrics (CPU, memory, disk, network). Each receiver produces one or more of the three signals: traces, metrics, or logs.

Processors transform the data. batch aggregates for efficient export. filter drops data based on conditions. attributes adds or modifies attributes. memory_limiter prevents OOM by dropping data when memory exceeds threshold. tail_sampling implements trace sampling. transform applies OTTL for complex transformations.

Exporters are the data destinations. otlp exports to another Collector or backend. prometheusremotewrite exports metrics to Prometheus-compatible storage. jaeger exports traces to Jaeger. debug prints to stdout.

Define pipelines under service.pipelines. A typical traces pipeline: receivers: [otlp] → processors: [batch, memory_limiter] → exporters: [otlp, debug].

# io.thecodeforge — DevOps tutorial

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

  prometheus:
    config:
      scrape_configs:
        - job_name: 'docker-containers'
          docker_sd_configs:
            - host: unix:///var/run/docker.sock
          relabel_configs:
            - source_labels: [__meta_docker_container_name]
              target_label: container

  hostmetrics:
    collection_interval: 30s
    scrapers:
      cpu: {}
      memory: {}
      disk: {}
      network: {}

  filelog:
    include: [/var/log/containers/*.log]
    operators:
      - type: json_parser
        parse_from: body

processors:
  batch:
    timeout: 2s
    send_batch_max_size: 1000

  memory_limiter:
    check_interval: 5s
    limit_mib: 800

  attributes:
    actions:
      - key: environment
        value: production
        action: upsert

  tail_sampling:
    policies:
      - name: sample-errors
        type: status_code
        config:
          status_code_source: span
          rules:
            - status_code: ERROR
              min_spans: 1

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch, tail_sampling]
      exporters: [otlp, debug]
    metrics:
      receivers: [otlp, prometheus, hostmetrics]
      processors: [memory_limiter, batch]
      exporters: [prometheusremotewrite]
    logs:
      receivers: [otlp, filelog]
      processors: [memory_limiter, batch]
      exporters: [otlp, debug]

OTel eBPF Instrumentation (OBI): Zero-Code Tracing for Docker Containers

Traditional OTel instrumentation requires adding SDKs to application code. For legacy applications, vendor-modified frameworks, or polyglot environments, this is impractical. OTel eBPF Instrumentation (OBI) solves this by automatically instrumenting network-level operations from outside the application.

OBI works by attaching eBPF programs to the kernel's network stack. When a container makes an HTTP request, the eBPF program intercepts the syscall, reads the HTTP headers, and reconstructs a span. It captures: method, URL path, status code, latency, and body size. It also propagates trace context via the traceparent header.

OBI supports: HTTP/1.1 and HTTP/2, gRPC, TLS connections (via uprobe attachment to OpenSSL/BoringSSL), MySQL and PostgreSQL queries, Redis commands, and Kafka produce/consume operations.

Running OBI: deploy as a privileged container sharing the host PID namespace and mounting debugfs. Configure with OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_SERVICE_NAME.

# io.thecodeforge — DevOps tutorial

version: '3.8'
services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.156.0
    command: ["--config=/etc/otel/config.yaml"]
    volumes:
      - ./otel-config.yaml:/etc/otel/config.yaml
    ports:
      - "4317:4317"
      - "4318:4318"

  otel-ebpf:
    image: ghcr.io/otel-ebpf/instrumentation:latest
    privileged: true
    pid: host
    network_mode: host
    volumes:
      - /sys/kernel/debug:/sys/kernel/debug:rw
      - /proc:/var/run/proc:ro
      - /sys/fs/cgroup:/sys/fs/cgroup:ro
    environment:
      OTEL_EXPORTER_OTLP_ENDPOINT: "http://otel-collector:4317"
      OTEL_SERVICE_NAME: "eBPF-auto-instrumentation"

  app:
    image: python:3.11-slim
    command: python -m http.server 8080
    ports:
      - "8080:8080"

  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"

Metrics Pipeline: OTel Collector to Prometheus via prometheusremotewrite

The OTel Collector can replace Prometheus server's scraping function entirely. Instead of having Prometheus scrape metrics from each container, the Collector receives metrics via OTLP and exports them using the prometheusremotewrite exporter.

This architecture ('OTel as the metrics pipeline frontend') is the recommended pattern since Collector 0.156.0+. Benefits: single configuration for all metrics, ability to process metrics before storage (filter, aggregate, enrich), and reduced load on Prometheus.

The prometheusremotewrite exporter sends metrics to Prometheus's remote write endpoint (/api/v1/write). Configure with endpoint, namespace, and resource_to_telemetry_conversion. The exporter uses a queue with configurable consumers and retry settings.

For long-term storage, use Prometheus in remote-write-only mode with Thanos or Grafana Mimir as the backend. The Collector sends metrics to the remote-write endpoint, and Thanos/Mimir handles storage and querying.

Metric naming: OTel uses dots (http.server.duration), Prometheus uses underscores (http_server_duration). The Collector automatically converts dots to underscores.

# io.thecodeforge — DevOps tutorial

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
  prometheus:
    config:
      scrape_configs:
        - job_name: 'node-exporter'
          static_configs:
            - targets: ['node-exporter:9100']

processors:
  batch:
    timeout: 2s
    send_batch_max_size: 10000
  memory_limiter:
    limit_mib: 800
    check_interval: 5s
  attributes:
    actions:
      - key: deployment_environment
        value: production
        action: upsert
  filter:
    metrics:
      include:
        match_type: regexp
        metric_names:
          - "http.server.*"
          - "container.*"
          - "process.*"

exporters:
  prometheusremotewrite:
    endpoint: "http://prometheus:9090/api/v1/write"
    remote_write_queue:
      enabled: true
      num_consumers: 5
      queue_size: 10000
    resource_to_telemetry_conversion:
      enabled: true
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 60s

service:
  pipelines:
    metrics:
      receivers: [otlp, prometheus]
      processors: [memory_limiter, batch, filter, attributes]
      exporters: [prometheusremotewrite]

Structured Logging and Trace Correlation with OpenTelemetry

OTel's log signal unifies logs with traces and metrics. The key benefit: log entries can be correlated with traces using trace_id and span_id attributes, enabling clicking from a log line to the trace that generated it.

Structured logging: applications emit logs in JSON format with trace context embedded. Use OTel SDK logging (auto-injecting trace_id/span_id) and export via OTLP to the Collector.

For existing applications that log to files, use the filelog receiver. It tails log files, parses them (JSON, regex), and converts entries to OTel log records.

Log enrichment: use the attributes processor to add service.name, container.id, environment to all log entries.

Trace-log correlation: when the application emits logs with trace_id and span_id, the Collector correlates them. In Grafana, click from a log entry to the trace span.

#!/usr/bin/env python3
import logging
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.exporter.otlp.proto.grpc.log_exporter import OTLPLogExporter
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk._logs.export import BatchLogProcessor

trace.set_tracer_provider(TracerProvider())
tracer = trace.get_tracer(__name__)

logger_provider = LoggerProvider()
logger_provider.add_log_processor(
    BatchLogProcessor(OTLPLogExporter(endpoint="http://otel-collector:4317", insecure=True))
)

handler = LoggingHandler(level=logging.INFO, logger_provider=logger_provider)
logging.getLogger().addHandler(handler)

with tracer.start_as_current_span("process-order") as span:
    logging.info("Processing order %s", "ORD-12345")
    # Log automatically includes trace_id and span_id

Trace Propagation Across Containers: Making Distributed Tracing Actually Work

Distributed tracing across containers fails when trace context isn't propagated between services. This is the most common OTel implementation issue. If Service A calls Service B and the trace_id doesn't continue, you get two disconnected traces.

Trace context propagation uses the traceparent header (W3C Trace Context). When Service A makes an HTTP request, the OTel SDK injects the traceparent header. Service B's SDK extracts it and continues the trace. Format: 00-<trace_id_32_hex>-<span_id_16_hex>-01.

For SDK-instrumented services: set OTEL_PROPAGATORS=tracecontext,baggage. This enables W3C Trace Context plus baggage.

For eBPF-instrumented services: OBI automatically propagates trace context by capturing the traceparent header from outgoing requests and injecting it into downstream requests.

Common failure: proxies and load balancers strip unknown headers. Ensure your reverse proxy (nginx, Envoy, HAProxy) is configured to pass through the traceparent header.

For non-HTTP protocols (gRPC, messaging), use the OTel propagator for that protocol. gRPC uses binary propagation in metadata. Kafka messages propagate via message headers.

// io.thecodeforge — DevOps tutorial

# Check if traceparent header is being sent
curl -v http://service-b:8080/api/endpoint 2>&1 | grep -i traceparent
# If empty, propagation is broken

# Verify propagator configuration in running container
docker exec <container-id> env | grep OTEL_PROPAGATORS
# Should output: OTEL_PROPAGATORS=tracecontext,baggage

# Check proxy config (nginx example) to ensure header passthrough
# In nginx config:
# underscores_in_headers on;
# proxy_set_header traceparent $http_traceparent;

# Using curl to simulate trace propagation
export TRACEPARENT="00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01"
curl -H "traceparent: $TRACEPARENT" http://service-b:8080/api/endpoint