Monitoring Node.js with OpenTelemetry and Prometheus

Your API was running fine until last Thursday at 3 PM, when latency spiked from 50ms to 2 seconds. Nobody noticed for 45 minutes because you had no monitoring. By the time the alert fired, 10,000 users were affected. Observability is not optional for production services — it is the difference between a 5-minute incident and a 45-minute outage. This article covers setting up OpenTelemetry in a Node.js application, exposing Prometheus metrics, building Grafana dashboards, and configuring alerts that catch problems before users do.

Why OpenTelemetry + Prometheus for Node.js?

Monitoring Node.js in production requires more than just checking CPU and memory. You need distributed tracing, metrics, and logs to debug performance issues and failures. OpenTelemetry provides a unified standard for collecting telemetry data, while Prometheus excels at storing and querying metrics. Together, they give you end-to-end observability without vendor lock-in. In this article, we'll instrument a Node.js application with OpenTelemetry, export metrics to Prometheus, and set up dashboards. We'll cover common pitfalls like metric cardinality explosion and trace sampling decisions.

npm install @opentelemetry/api @opentelemetry/sdk-node @opentelemetry/instrumentation-http @opentelemetry/instrumentation-express @opentelemetry/exporter-prometheus

Setting Up the OpenTelemetry SDK

The OpenTelemetry SDK must be initialized before any application code runs. We'll create a tracing.ts file that configures the SDK with a Prometheus exporter. The SDK automatically instruments HTTP and Express via plugins. We'll also set up a metric reader that exports metrics every 10 seconds. Important: ensure the SDK is imported first in your entry point, otherwise some spans may be missed.

import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { PrometheusExporter } from '@opentelemetry/exporter-prometheus';

const prometheusExporter = new PrometheusExporter({
  port: 9464,
  endpoint: '/metrics',
  prefix: 'myapp_',
});

const sdk = new NodeSDK({
  metricReader: prometheusExporter,
  instrumentations: [getNodeAutoInstrumentations()],
});

sdk.start();

process.on('SIGTERM', () => {
  sdk.shutdown().then(() => console.log('SDK shut down'));
});

Instrumenting HTTP and Express Routes

With auto-instrumentation, HTTP requests and Express route handlers are automatically traced. Each incoming request gets a span with attributes like method, route, and status code. You can also create custom spans for business logic. We'll add a custom span for a database query to demonstrate manual instrumentation. The span will be a child of the incoming request span, preserving the trace context.

import { trace, Span } from '@opentelemetry/api';
import express from 'express';

const app = express();
const tracer = trace.getTracer('myapp');

app.get('/users/:id', async (req, res) => {
  const span = tracer.startSpan('getUser', {
    attributes: { 'user.id': req.params.id },
  });
  try {
    // Simulate DB query
    const user = await fakeDB.findUser(req.params.id);
    span.setAttribute('user.found', !!user);
    res.json(user);
  } finally {
    span.end();
  }
});

app.listen(3000);

Exporting Metrics to Prometheus

The PrometheusExporter exposes a /metrics endpoint that Prometheus scrapes. By default, it runs on port 9464. You can configure the prefix to avoid metric name collisions. The exporter automatically creates histogram metrics for HTTP request duration and counter metrics for request count. We'll also create custom metrics: a gauge for active users and a counter for errors. These metrics will be visible in Prometheus alongside the auto-generated ones.

import { metrics } from '@opentelemetry/api';

const meter = metrics.getMeter('myapp');

const activeUsers = meter.createUpDownCounter('myapp_active_users', {
  description: 'Number of active users',
});

const errorCounter = meter.createCounter('myapp_errors_total', {
  description: 'Total number of errors',
});

// Usage:
activeUsers.add(1, { 'user.type': 'premium' });
errorCounter.add(1, { 'error.type': 'timeout' });

Configuring Prometheus to Scrape Your App

Prometheus needs a scrape configuration to pull metrics from your Node.js app. We'll add a job that scrapes the /metrics endpoint every 15 seconds. In production, you'd run Prometheus as a separate service, but for local testing, we'll use docker-compose. We'll also set up a simple alerting rule for high error rate.

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'nodejs-app'
    static_configs:
      - targets: ['host.docker.internal:9464']

alerting:
  alertmanagers:
    - static_configs:
        - targets: []

rule_files:
  - 'alerts.yml'

Visualizing Metrics with Grafana Dashboards

Grafana connects to Prometheus as a data source and lets you build dashboards. We'll create a dashboard showing request rate, error rate, latency percentiles (p50, p95, p99), and active users. Use PromQL queries like rate(myapp_http_server_duration_ms_count[5m]) for request rate. We'll also set up a heatmap for latency distribution. Export the dashboard as JSON for reproducibility.

{
  "title": "Node.js App Overview",
  "panels": [
    {
      "title": "Request Rate",
      "type": "graph",
      "targets": [
        {
          "expr": "rate(myapp_http_server_duration_ms_count[5m])",
          "legendFormat": "{{method}} {{route}}"
        }
      ]
    },
    {
      "title": "Error Rate",
      "type": "graph",
      "targets": [
        {
          "expr": "rate(myapp_errors_total[5m])",
          "legendFormat": "{{error.type}}"
        }
      ]
    },
    {
      "title": "Latency (p99)",
      "type": "graph",
      "targets": [
        {
          "expr": "histogram_quantile(0.99, sum(rate(myapp_http_server_duration_ms_bucket[5m])) by (le))"
        }
      ]
    }
  ]
}

Handling High Cardinality and Performance

High cardinality metrics (e.g., unique user IDs as labels) can overwhelm Prometheus. OpenTelemetry allows you to configure metric views to drop or aggregate high-cardinality attributes. We'll create a view that drops the user.id attribute from the custom span metric. Also, we'll set up trace sampling to reduce volume: only 10% of traces are exported. This balances observability with cost.

import { View } from '@opentelemetry/sdk-metrics';

const sdk = new NodeSDK({
  metricReader: prometheusExporter,
  instrumentations: [getNodeAutoInstrumentations()],
  views: [
    new View({
      // Drop high-cardinality attribute
      attributeKeys: ['http.method', 'http.route', 'http.status_code'],
      instrumentName: 'myapp_http_server_duration',
    }),
  ],
  sampler: new ParentBasedSampler({
    root: new TraceIdRatioBasedSampler(0.1),
  }),
});

Alerting on Anomalies with Prometheus and Alertmanager

Alerting is critical for production. We'll define alert rules for high error rate, high latency, and low request rate (potential downtime). Alertmanager handles deduplication and routing. We'll send alerts to Slack. The alert rules use PromQL expressions that evaluate periodically. For example, a rule that fires if error rate > 5% for 5 minutes.

groups:
  - name: nodejs
    rules:
      - alert: HighErrorRate
        expr: rate(myapp_errors_total[5m]) / rate(myapp_http_server_duration_ms_count[5m]) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Error rate > 5% for 5 minutes"
      - alert: HighLatency
        expr: histogram_quantile(0.99, sum(rate(myapp_http_server_duration_ms_bucket[5m])) by (le)) > 1000
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "p99 latency > 1s"

Tracing in Production: Sampling and Context Propagation

In production, you can't trace every request. Use head-based sampling (e.g., 10%) or tail-based sampling for more intelligent decisions. OpenTelemetry supports context propagation via W3C Trace Context headers. This allows traces to span across microservices. We'll show how to propagate context to an external HTTP call using the OpenTelemetry HTTP instrumentation. Also, we'll discuss how to handle async contexts with Node.js AsyncLocalStorage.

import { context, propagation, trace } from '@opentelemetry/api';
import http from 'http';

// Inside a request handler
const currentSpan = trace.getSpan(context.active());
const headers = {};
propagation.inject(context.active(), headers);

http.get('http://external-service/api', { headers }, (res) => {
  // The external service will see the trace context
});

Common Pitfalls and How to Avoid Them

We've seen teams struggle with: (1) Not initializing the SDK early enough, causing missing spans. (2) Using too many unique label values, causing Prometheus OOM. (3) Forgetting to set up proper sampling, leading to high costs. (4) Not testing alert rules, resulting in missed incidents. We'll provide a checklist for production readiness: verify SDK initialization, monitor metric cardinality, set up sampling, and test alerts with a chaos experiment.

# 1. Check SDK initialization
curl http://localhost:9464/metrics | head -20

# 2. Check cardinality
curl 'http://localhost:9090/api/v1/query?query=count({__name__=~"myapp_.*"})'

# 3. Test alert
# Simulate high error rate
for i in {1..100}; do curl -X POST http://localhost:3000/error; done

Next Steps: Extending to Other Services and Logs

OpenTelemetry is not limited to Node.js. You can instrument Python, Go, Java services and have them all report to the same Prometheus. For logs, OpenTelemetry's Logs API is still experimental, but you can use the existing logging libraries with OpenTelemetry's log appender. We'll show how to correlate logs with traces by injecting trace ID into log entries. This gives you a single pane of glass for debugging.

import { trace } from '@opentelemetry/api';
import pino from 'pino';

const logger = pino({
  mixin() {
    const span = trace.getSpan(trace.getActiveContext());
    if (span) {
      return {
        traceId: span.spanContext().traceId,
        spanId: span.spanContext().spanId,
      };
    }
    return {};
  },
});

logger.info('User fetched', { userId: 123 });
// Output includes traceId and spanId

Conclusion: Observability as a Culture

Monitoring with OpenTelemetry and Prometheus is not a one-time setup. It requires ongoing maintenance: updating dashboards, tuning alerts, and reviewing cardinality. Treat observability as a first-class feature of your application. Invest in good instrumentation from the start, and you'll save countless hours debugging production issues. The code and configurations in this article are a starting point; adapt them to your specific needs.

# Start the app
npm run start

# Start Prometheus and Grafana
docker-compose up -d prometheus grafana

# Verify
curl http://localhost:9464/metrics
curl http://localhost:9090/api/v1/query?query=up

Docker Compose for Jaeger, Prometheus, and OTel Collector

Running a full observability stack locally is essential for testing before production. A single Docker Compose file can spin up Jaeger for traces, Prometheus for metrics, and the OpenTelemetry Collector as a central pipeline. The Collector receives OTLP data from your Node.js app, processes it (batch, filter, sample), and forwards traces to Jaeger and metrics to Prometheus. This setup avoids vendor lock-in and lets you swap backends later. Use the official OpenTelemetry Collector contrib image for advanced processors. Mount a config file for the Collector and Prometheus scrape configs. Jaeger UI runs on port 16686, Prometheus on 9090, and Grafana on 3000. Your app sends OTLP to the Collector on port 4318 (HTTP) or 4317 (gRPC). This local environment mirrors production patterns and catches misconfigurations early.

version: '3.8'
services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.102.0
    command: ['--config=/etc/otel-collector-config.yaml']
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - '4317:4317'   # gRPC
      - '4318:4318'   # HTTP
    depends_on:
      - jaeger
      - prometheus

  jaeger:
    image: jaegertracing/all-in-one:1.57
    ports:
      - '16686:16686'
      - '14250:14250'

  prometheus:
    image: prom/prometheus:v2.52.0
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - '9090:9090'

  grafana:
    image: grafana/grafana:10.4.2
    ports:
      - '3000:3000'
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin

OTel Collector Configuration: Batch, Filter, and Prometheus Exporter

The OpenTelemetry Collector is the brain of your observability pipeline. A well-tuned config ensures data is processed efficiently. Start with the batch processor to group spans and metrics before exporting, reducing network calls. Set a timeout of 200ms and a max batch size of 1000. Use the filter processor to drop high-cardinality attributes like user IDs or request parameters that aren't needed for metrics. For Prometheus, use the prometheusexporter to expose metrics on a dedicated port (e.g., 8889) that Prometheus scrapes. The exporter automatically converts OTel metrics to Prometheus format. Add a memory limiter processor to prevent the Collector from OOMing. Always enable the batch processor and set send_batch_max_size to a reasonable value. The filter processor can also drop debug spans or metrics you don't need. This config reduces load on both the Collector and downstream backends.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 200ms
    send_batch_max_size: 1000
  filter:
    metrics:
      include:
        match_type: strict
        metric_names:
          - http.server.duration
          - process.runtime.nodejs.eventloop.lag
          - process.runtime.nodejs.gc.duration
  memory_limiter:
    check_interval: 1s
    limit_mib: 512

exporters:
  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true
  prometheus:
    endpoint: 0.0.0.0:8889
    namespace: nodeapp
    const_labels:
      env: production

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/jaeger]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, filter, batch]
      exporters: [prometheus]

SLO-Based Alerting with AlertManager

Service Level Objectives (SLOs) define the reliability targets for your service. Instead of alerting on every spike, alert when the error budget is burning too fast. Use Prometheus recording rules to compute SLO metrics like the ratio of successful requests over a sliding window. For example, define a rule that calculates the 30-day error budget remaining. Then create an AlertManager rule that fires when the error budget consumption rate exceeds a threshold (e.g., 10% in 1 hour). This prevents alert fatigue and focuses on business impact. Configure AlertManager to route alerts to PagerDuty, Slack, or email. Use inhibition rules to suppress less critical alerts when a high-severity one is firing. Always include runbooks in alert annotations. SLO-based alerting requires accurate SLI definitions—choose the right metrics (latency, error rate, throughput) and set realistic targets.

groups:
  - name: slo
    rules:
      - record: job:slo_errors_total:rate30d
        expr: sum(rate(http_requests_total{status=~"5.."}[30d]))
      - record: job:slo_total:rate30d
        expr: sum(rate(http_requests_total[30d]))
      - record: job:slo_error_budget_remaining
        expr: 1 - (job:slo_errors_total:rate30d / job:slo_total:rate30d / 0.01)
      - alert: ErrorBudgetBurn
        expr: job:slo_error_budget_remaining < 0.5
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Error budget burned > 50%"
          runbook: "https://runbook.example.com/error-budget"

Tail-Based Sampling: 100% Errors, Sampled Success

In production, you can't store every trace. Head-based sampling (deciding at the start) is simple but misses rare errors. Tail-based sampling waits until the span is complete to decide. Use the OpenTelemetry Collector's tail_sampling processor to keep 100% of error traces and sample successful ones at a lower rate (e.g., 10%). Configure policies: 'status_code' policy for errors (e.g., HTTP 5xx), and 'rate_limiting' for success. This ensures you have full visibility into failures without drowning in success traces. The processor requires a decision wait time (e.g., 30 seconds) to collect all spans of a trace. Use a consistent hash to ensure all spans of the same trace go to the same Collector instance. Tail-based sampling increases memory usage but is worth it for error-rich observability.

processors:
  tail_sampling:
    decision_wait: 30s
    num_traces: 10000
    expected_new_traces_per_sec: 100
    policies:
      - name: errors-policy
        type: status_code
        config:
          status_code: ERROR
      - name: sampled-policy
        type: rate_limiting
        config:
          spans_per_second: 10

Log Correlation: Injecting trace_id and span_id into Pino

Correlating logs with traces is critical for debugging. When using OpenTelemetry, the SDK automatically generates trace_id and span_id for each request. Inject these into your Pino logger to connect logs with traces. Use the @opentelemetry/api to access the current span and extract its context. Create a custom Pino serialiser or mixin that adds trace_id and span_id to every log entry. This works with any logging framework that supports custom serialisers. In production, your logs will include these IDs, allowing you to jump from a log line to the corresponding trace in Jaeger. Ensure your log aggregation system (e.g., Loki, Elasticsearch) indexes these fields for fast lookup. This pattern is a cornerstone of distributed observability.

const pino = require('pino');
const { context, trace, SpanStatusCode } = require('@opentelemetry/api');

const logger = pino({
  mixin() {
    const span = trace.getSpan(context.active());
    if (!span) return {};
    const spanContext = span.spanContext();
    return {
      trace_id: spanContext.traceId,
      span_id: spanContext.spanId,
    };
  },
  level: process.env.LOG_LEVEL || 'info',
});

module.exports = logger;

Known Limitations: ESM Partial Support, Winston Duplicates, Cardinality

OpenTelemetry Node.js has known rough edges. ESM (ECMAScript Modules) support is partial—some auto-instrumentations don't work with ESM. Use CommonJS or the experimental ESM loader. Winston users often see duplicate log entries because the OpenTelemetry Winston instrumentation and the logger itself both write to the same stream. Disable the instrumentation's log sending or configure Winston to not duplicate. Cardinality is a persistent issue: high-cardinality attributes (user IDs, session IDs) can explode Prometheus memory. Use the OTel Collector's filter processor to drop them before they reach Prometheus. Also, the Node.js SDK's default metrics (eventloop lag, GC duration, heap space) are experimental and may change. Always pin the SDK version and test after upgrades. These limitations are documented in the OpenTelemetry GitHub issues—check before deploying.

const winston = require('winston');
const { WinstonInstrumentation } = require('@opentelemetry/instrumentation-winston');

// Disable log sending to avoid duplicates
const instrumentation = new WinstonInstrumentation({
  logHook: (span, record) => {
    // Only add trace context, don't send log as span event
  },
});

const logger = winston.createLogger({
  transports: [new winston.transports.Console()],
});