Monitoring vs Observability — P99 Blind Spot Disaster

Your production system crashes at 2 AM on Black Friday. Orders are failing, users are screaming on Twitter, and your on-call engineer is staring at a wall of dashboards wondering where to even begin. This scenario plays out every day at companies around the world — and the difference between a 10-minute fix and a 4-hour outage almost always comes down to one thing: how well-instrumented your system was before the incident.

Monitoring and observability are not luxury features you bolt on after launch. They are the engineering discipline that separates teams who fix incidents in minutes from teams who spend hours in Slack threads and frantic Zoom calls pointing fingers at each other's services. While monitoring is the act of watching known-knowns through dashboards and alerts, observability is the property of a system that allows you to understand its internal state from the external data it emits — logs, metrics, and traces. The operative word is property. Observability is not a tool you buy. It is a quality you engineer into your system from the start.

The critical misconception I see constantly at staff level: treating monitoring and observability as synonyms. They are not. Monitoring is a subset of observability. You can monitor without observability — you get dashboards with no context, alerts that fire with no actionable detail, and on-call engineers who solve incidents through gut instinct and luck. You cannot have meaningful observability without some form of monitoring — you still need alerts to know when to look at anything. The two are complementary, not competing. Get both right and your MTTR drops from hours to minutes. Get only one right and you are still flying partially blind.

Why Observability Is Not Just Fancy Monitoring

Monitoring tells you a system is down. Observability tells you why — and, critically, what you didn't know to ask about. Monitoring is built on predefined dashboards and static thresholds (e.g., CPU > 90%). Observability is a property of a system that lets you infer its internal state from its external outputs, without shipping new code. The core mechanic: high-cardinality, structured event data (traces, logs, metrics) stored in a way that supports ad-hoc exploration, not just fixed charts.

In practice, observability requires three pillars — traces, logs, and metrics — but the real power comes from correlation. A p99 latency spike in a metric is a symptom; the trace shows you which database query caused it; the log reveals the exact parameter that triggered a slow path. Without this linkage, you're guessing. Modern observability platforms (e.g., OpenTelemetry, Honeycomb, Datadog) enforce structured context propagation so every event carries the request's journey.

Use observability when your system has unpredictable failure modes — microservices, event-driven architectures, or any distributed system where a single request touches 10+ services. Monitoring catches the known unknowns; observability catches the unknown unknowns. In production, this is the difference between a 5-minute fix and a 3-day war room. Teams that only monitor hit the P99 blind spot: the 1% of slow requests that kill user experience but never trip a threshold.

The Three Pillars: Metrics, Logs, and Traces

True observability requires all three types of telemetry working together, not independently. Metrics provide a high-level, aggregated view of system health — request rate, error rate, CPU saturation. Logs provide the per-event forensic record of exactly what happened at a specific moment in time. Distributed traces connect everything by showing the complete path of a single request across every service it touched, with timing for each hop.

Each pillar serves a distinct cognitive role in an incident, and understanding those roles prevents the common mistake of trying to use one pillar for a job another pillar does better. Metrics are your early warning system — they tell you THAT something changed, and they tell you fast. A metric alert can fire within 30 seconds of a threshold being crossed. Logs are your forensic evidence — they tell you exactly WHAT happened at a specific request or event level, but querying them at scale is expensive and slow compared to a metric query. Traces are your request GPS — they tell you WHERE a request traveled and HOW LONG each individual service spent processing it. The power of the three-pillar model is not in any single pillar; it is in the correlation between them.

The correlation ID is the connective tissue. When a P99 latency alert fires (metric), you need to jump directly to the traces for the slow requests and then to the logs for the specific error messages those traces contain. If the trace ID is not present in your logs, that jump requires manual timestamp correlation — which is slow, error-prone, and genuinely miserable to do at 2 AM during an active incident. Instrument for correlation from day one, not as a retrofit after the first major outage.

One more thing that gets underweighted: the 'unknown unknowns' problem. Metrics cover what you thought to instrument. Logs cover what you thought to log. Traces cover what you thought to trace. Observability is the property that lets you ask questions you did not anticipate when you wrote the code — because the raw telemetry is rich enough to answer novel queries. A system where you can only ask questions you pre-configured dashboards for is a monitored system. A system where you can compose a new query during an incident and get a meaningful answer is an observable system.

package io.thecodeforge.monitoring;

import io.micrometer.core.instrument.Counter;
import io.micrometer.core.instrument.MeterRegistry;
import io.micrometer.core.instrument.Timer;
import io.opentelemetry.api.trace.Span;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.slf4j.MDC;

/**
 * io.thecodeforge — Three-Pillar Instrumentation Pattern
 *
 * Demonstrates all three observability pillars in a single operation:
 *   - Metric: forge.orders.success (counter) and forge.payment.latency (histogram timer)
 *   - Log:    Structured log line with traceId injected from active OpenTelemetry span
 *   - Trace:  Active span context is propagated by the OTel agent — no manual span creation needed
 *
 * Key design decisions:
 *   1. traceId comes from the active OTel Span — it is the same ID visible in Jaeger/Tempo
 *   2. MDC injection makes traceId appear in EVERY log line within this thread scope
 *   3. The Timer wraps the entire business operation — it measures wall clock time including I/O
 *   4. Counter uses .tag("status", "success") so failures can be counted separately:
 *      forge.orders.processed{status="failure"} without creating a separate metric
 */
public class OrderMonitor {

    private static final Logger log = LoggerFactory.getLogger(OrderMonitor.class);

    private final Counter orderSuccessCounter;
    private final Counter orderFailureCounter;
    private final Timer paymentTimer;

    public OrderMonitor(MeterRegistry registry) {
        // Use tagged counters so success and failure are queryable separately in PromQL
        // rate(forge.orders.processed_total{status="failure"}[5m]) gives failure rate
        this.orderSuccessCounter = registry.counter("forge.orders.processed", "status", "success");
        this.orderFailureCounter = registry.counter("forge.orders.processed", "status", "failure");

        // Timer automatically produces _count, _sum, and _bucket metrics
        // Enables histogram_quantile(0.99, rate(forge_payment_latency_seconds_bucket[5m]))
        this.paymentTimer = registry.timer("forge.payment.latency");
    }

    public void processOrder(String orderId) {
        // Inject the active OpenTelemetry trace ID into MDC before any log statements
        // This ensures EVERY log line within this method includes the traceId automatically
        String traceId = Span.current().getSpanContext().getTraceId();
        MDC.put("traceId", traceId);
        MDC.put("orderId", orderId);

        try {
            paymentTimer.record(() -> {
                // log.info output will include traceId and orderId from MDC automatically
                // Logback/Log4j2 pattern: %X{traceId} %X{orderId} %msg
                log.info("Order processing started");

                // ... business logic: validate, reserve inventory, charge payment ...

                orderSuccessCounter.increment();
                log.info("Order processing completed successfully");
            });
        } catch (Exception e) {
            orderFailureCounter.increment();
            // ERROR logs include traceId from MDC — jump directly to the trace in Jaeger
            log.error("Order processing failed — check trace for upstream dependency state",
                Map.of("error", e.getMessage(), "errorType", e.getClass().getSimpleName()));
            throw e;
        } finally {
            // Always clear MDC — thread pool reuse will leak context if you skip this
            MDC.clear();
        }
    }
}

/*
 * Example log output (JSON format via logstash-logback-encoder):
 * {
 *   "level":   "INFO",
 *   "logger":  "io.thecodeforge.monitoring.OrderMonitor",
 *   "message": "Order processing completed successfully",
 *   "traceId": "4bf92f3577b34da6a3ce929d0e0e4736",   <-- same ID in Jaeger/Tempo
 *   "orderId": "ORD-101"
 * }
 *
 * Metrics emitted (visible in Prometheus):
 *   forge_orders_processed_total{status="success"} 1.0
 *   forge_payment_latency_seconds_count 1.0
 *   forge_payment_latency_seconds_sum 0.043
 *   forge_payment_latency_seconds_bucket{le="0.05"} 1.0
 */

Observability in Practice: Prometheus and Grafana

Modern observability stacks typically center on a pull-based collection model, and Prometheus is the reference implementation of that model in the open-source world. Instead of your application pushing metrics to a central store on a schedule it controls, Prometheus scrapes a /metrics endpoint on your service at an interval Prometheus controls. Your application is passive — it just maintains counters and histograms in memory and serves them when asked.

The pull model has a meaningful architectural advantage: a misbehaving application cannot overwhelm the monitoring system with a flood of metric writes. If your application starts logging every debug event as a metric push, a push-based system absorbs the explosion and potentially falls over. Prometheus, by contrast, simply scrapes at its configured interval regardless of what the application is doing. The trade-off is the opposite failure mode: if your application crashes between scrapes, you lose the data from that window. For short-lived jobs and batch processes that complete in under one scrape interval, the Pushgateway solves this by holding pushed metrics until the next Prometheus scrape cycle.

Prometheus stores metrics as time series: a metric name, a set of key-value labels, and a sequence of timestamped float64 values. The label set is everything — it is how you slice a metric by service, region, endpoint, or status code. But the label set is also where most production Prometheus problems originate. Every unique combination of label values creates a separate time series in the TSDB. A metric with three labels, each with ten possible values, creates one thousand time series. Add a fourth label with a thousand possible values and you have a million time series for a single metric name. This is the cardinality problem, and it is not theoretical — I have watched it take down a Prometheus instance in production within 20 minutes of a bad deployment that added a user_id label to a high-traffic metric.

PromQL is where the real diagnostic power lives. PromQL is not a query language for fetching raw data — it is a functional language for computing derived signals from time series. The most important function in incident response is histogram_quantile(), which computes a true percentile from a histogram metric. histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) gives you the true P99 latency over a 5-minute window — the slowest experience that 1% of your users received. No dashboard based on averages can show you this. Building this query and putting it on your primary service dashboard is a one-time 10-minute investment that has caught more production incidents for me than any other single piece of instrumentation.

# io.thecodeforge — Production Prometheus Scrape Configuration
#
# Design decisions explained inline:
#   - scrape_interval: 15s is the standard for most services
#     Reduce to 5s for latency-sensitive paths, but watch Prometheus CPU/memory
#   - honor_labels: false (default) — Prometheus overwrites conflicting labels
#     from the target with its own. Set true only if you trust the target's labels.
#   - relabel_configs allow normalizing instance labels before storage
#     This prevents cardinality drift from inconsistent hostname formats

global:
  scrape_interval: 15s
  evaluation_interval: 15s  # How often alerting rules are evaluated
  # Global labels added to every time series scraped by this instance
  external_labels:
    datacenter: 'us-east-1'
    environment: 'production'

rule_files:
  - 'rules/forge_alerts.yml'   # Alert rules — evaluated every evaluation_interval
  - 'rules/forge_records.yml'  # Recording rules — pre-aggregate expensive PromQL queries

scrape_configs:
  - job_name: 'forge-api-service'
    # Spring Boot Actuator exposes Prometheus metrics at /actuator/prometheus
    # Not /metrics — this is a common misconfiguration that produces DOWN targets
    metrics_path: '/actuator/prometheus'
    scrape_interval: 5s      # Override global for latency-sensitive checkout service
    scrape_timeout: 4s       # Must be less than scrape_interval to avoid overlap
    static_configs:
      - targets:
          - 'forge-prod-app-01:8080'
          - 'forge-prod-app-02:8080'
        labels:
          service: 'order-service'  # Added to every time series from these targets
    relabel_configs:
      # Normalize the instance label to a stable name instead of host:port
      # Prevents cardinality drift if the port changes or hostname format varies
      - source_labels: [__address__]
        regex: '(forge-prod-app-\d+):.*'
        target_label: instance
        replacement: '$1'

  - job_name: 'forge-payment-service'
    metrics_path: '/actuator/prometheus'
    static_configs:
      - targets: ['forge-payment:8080']
        labels:
          service: 'payment-service'

---
# rules/forge_alerts.yml
# Alert on P99 latency — not average. Average latency alerts miss tail latency problems.
groups:
  - name: forge.latency
    rules:
      - alert: ForgeCheckoutP99LatencyHigh
        expr: |
          histogram_quantile(
            0.99,
            rate(http_server_requests_seconds_bucket{
              job="forge-api-service",
              uri="/api/checkout"
            }[5m])
          ) > 2.0
        for: 3m   # Must be elevated for 3 minutes before firing — reduces noise
        labels:
          severity: critical
          team: platform
        annotations:
          summary: 'Checkout P99 latency exceeds 2s'
          description: |
            P99 latency for /api/checkout is {{ $value | humanizeDuration }}.
            Check Jaeger for slow traces on this endpoint.
            Dashboard: https://grafana.thecodeforge.io/d/checkout-slo

---
# rules/forge_records.yml
# Recording rules pre-aggregate expensive queries so dashboards load instantly
# Without this, a 7-day P99 query runs at render time and takes 30+ seconds
groups:
  - name: forge.recordings
    rules:
      - record: job:http_request_duration_p99:rate5m
        expr: |
          histogram_quantile(
            0.99,
            sum by (job, le) (
              rate(http_server_requests_seconds_bucket[5m])
            )
          )