Spring Boot Observability: Micrometer, Prometheus, Grafana Guide for Production

Observability isn't just buzzword bingo — it's the difference between knowing your app is dying at 3 AM and being woken up by a pager. I've been burned too many times by apps that looked healthy in logs but were silently leaking connections or burning CPU. Spring Boot 3.2.x with Micrometer 1.12.x gives you a first-class observability pipeline, but the official docs often skip the hard parts: what to actually measure, how to avoid metric explosion, and how to debug when Prometheus goes silent. In this guide, you'll move beyond the hello-world dashboards. We'll instrument a real payment-processing service, expose custom business metrics (like 'failed refunds per second'), and set up Prometheus rules that catch anomalies before they hit users. We'll also cover the production incident I had where a missing Maven dependency caused a silent metrics outage — and how we fixed it with a simple health check. By the end, you'll have a production-grade observability stack that actually helps you sleep at night.

Setting Up the Micrometer-Prometheus Pipeline

Let's start by instrumenting a Spring Boot 3.2.5 payment-processing service. First, add the required dependencies to your pom.xml. The key is spring-boot-starter-actuator plus the Micrometer Prometheus registry. Without the registry, /actuator/prometheus returns 404 — a silent failure we'll discuss later. The Micrometer core is auto-configured by Spring Boot, so you don't need explicit beans for basic JVM metrics. But for custom business metrics, you'll use MeterRegistry. Here's the minimal setup:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
    <version>1.12.0</version>
</dependency>
<!-- For custom metrics if needed -->
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-core</artifactId>
</dependency>

What the Official Docs Won't Tell You

The official Spring Boot docs show you how to enable Actuator and add the Prometheus dependency, but they skip the hard lessons. First, Micrometer uses dimensional metrics — every label combination creates a new time series. If you add a label like 'userId' to a counter, you'll create millions of series and blow up Prometheus storage. Second, the default http_server_requests_seconds metric includes a 'uri' label with path variables expanded (e.g., /api/orders/123). This is fine for low-cardinality paths, but if you have /api/orders/{id} with thousands of IDs, you'll get metric explosion. The fix is to use Spring's @Timed annotation with a custom tag that buckets the URI. Third, Prometheus scrape intervals matter — if your app has bursts of high-latency requests that last under 15 seconds, you'll miss them with a default 15s scrape. Set scrape_interval to 5s for critical services. Finally, the /actuator/prometheus endpoint returns metrics in Prometheus exposition format, which is text-based and easy to debug with curl. But the format is strict — any malformed metric name (e.g., starting with a digit) causes Prometheus to reject the entire scrape. I've debugged a production issue where a custom metric named '2xx_count' caused the whole endpoint to fail silently.

scrape_configs:
  - job_name: 'payment-service'
    scrape_interval: 5s
    metrics_path: '/actuator/prometheus'
    static_configs:
      - targets: ['localhost:8080']
    # Add basic auth if needed
    basic_auth:
      username: 'monitoring'
      password: 'secret'

Instrumenting Business Metrics with Micrometer

Now let's add custom business metrics to a payment service. We'll track refund failures, payment processing time, and active transactions. Use MeterRegistry to create counters and timers. For the refund failure counter, we need to track by payment method (credit card, PayPal) and failure reason (insufficient funds, expired card). But remember the cardinality lesson — keep labels limited to 3-4 values. Here's a service class that records metrics:

@Service
public class PaymentService {
    private final MeterRegistry meterRegistry;
    private final Counter refundFailureCounter;
    private final Timer paymentTimer;

    public PaymentService(MeterRegistry meterRegistry) {
        this.meterRegistry = meterRegistry;
        this.refundFailureCounter = Counter.builder("payment.refund.failures")
            .description("Total refund failures")
            .tag("paymentMethod", "unknown")
            .register(meterRegistry);
        this.paymentTimer = Timer.builder("payment.processing.time")
            .description("Time to process payment")
            .publishPercentiles(0.5, 0.95, 0.99)
            .register(meterRegistry);
    }

    public void processRefund(String paymentMethod, String reason) {
        refundFailureCounter.increment();
        // Use tags for specific dimensions
        meterRegistry.counter("payment.refund.failures",
            "paymentMethod", paymentMethod,
            "reason", reason).increment();
    }

    public PaymentResult processPayment(PaymentRequest request) {
        return paymentTimer.record(() -> {
            // actual payment logic
            return new PaymentResult(true, "txn_123");
        });
    }
}

Configuring Prometheus for Spring Boot Metrics

Prometheus needs a proper scrape configuration to discover your Spring Boot instances. In production with multiple replicas, use service discovery (e.g., Kubernetes pod annotations) instead of static targets. For local development, use docker-compose with a prometheus.yml file. Here's a production-grade configuration that includes relabeling for environment tags and rate limiting:

version: '3.8'
services:
  prometheus:
    image: prom/prometheus:v2.52.0
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.retention.time=30d'
  grafana:
    image: grafana/grafana:10.4.0
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin

Building Grafana Dashboards for Real-Time Insights

Grafana dashboards are where observability becomes actionable. Start with a JVM dashboard using the Micrometer metrics. Use PromQL queries like rate(jvm_memory_used_bytes[5m]) for memory trends. For business metrics, create panels for refund failure rate per second and payment latency percentiles. Here's a key panel configuration for payment latency:

{
  "targets": [
    {
      "expr": "histogram_quantile(0.99, rate(payment_processing_time_seconds_bucket[5m]))",
      "legendFormat": "p99",
      "refId": "A"
    },
    {
      "expr": "histogram_quantile(0.95, rate(payment_processing_time_seconds_bucket[5m]))",
      "legendFormat": "p95",
      "refId": "B"
    }
  ],
  "title": "Payment Processing Latency",
  "type": "timeseries"
}

Alerting Rules for Production Readiness

Prometheus alerting rules turn metrics into actionable notifications. Define rules in YAML and load them via the rule_files directive. For the payment service, we need alerts for: high refund failure rate, elevated p99 latency, and JVM heap usage > 80%. Here's a rule file that triggers alerts for these conditions:

groups:
  - name: payment_service_alerts
    rules:
      - alert: HighRefundFailureRate
        expr: rate(payment_refund_failures_total[5m]) > 10
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Refund failure rate is high"
      - alert: HighLatencyP99
        expr: histogram_quantile(0.99, rate(payment_processing_time_seconds_bucket[5m])) > 2
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "P99 latency above 2 seconds"
      - alert: HighHeapUsage
        expr: jvm_memory_used_bytes{area="heap"} / jvm_memory_max_bytes{area="heap"} > 0.8
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Heap usage above 80%"

Advanced: Custom Actuator Endpoint for Metrics Health

To prevent the 'silent metrics outage' scenario, create a custom Actuator health indicator that verifies Micrometer registries are active and metrics are being published. This runs alongside the default health check and gives you a separate signal. Here's an implementation:

@Component
public class MetricsHealthIndicator implements HealthIndicator {
    private final MeterRegistry meterRegistry;

    public MetricsHealthIndicator(MeterRegistry meterRegistry) {
        this.meterRegistry = meterRegistry;
    }

    @Override
    public Health health() {
        try {
            // Check if Prometheus registry is present
            PrometheusMeterRegistry promRegistry = meterRegistry
                .getRegistries()
                .stream()
                .filter(r -> r instanceof PrometheusMeterRegistry)
                .map(r -> (PrometheusMeterRegistry) r)
                .findFirst()
                .orElse(null);

            if (promRegistry == null) {
                return Health.down()
                    .withDetail("error", "Prometheus registry not found")
                    .build();
            }

            // Check if any metrics are registered
            long metricCount = promRegistry.getMeters().size();
            if (metricCount == 0) {
                return Health.down()
                    .withDetail("error", "No metrics registered")
                    .build();
            }

            return Health.up()
                .withDetail("registries", promRegistry.getClass().getSimpleName())
                .withDetail("metricCount", metricCount)
                .build();
        } catch (Exception e) {
            return Health.down(e).build();
        }
    }
}

Debugging Common Observability Issues

Even with proper setup, things go wrong. Here are the most common issues I've seen: (1) Prometheus returns 'invalid metric type' — check that your custom metrics use correct types (Counter, Gauge, Timer). A common mistake is using a Counter for values that can decrease (use Gauge instead). (2) Grafana shows 'No data' — verify Prometheus is scraping by checking the target page at /targets. If it shows 'DOWN', check network connectivity and authentication. (3) Metrics appear but with wrong labels — use Prometheus's /api/v1/labels endpoint to debug label values. For example, curl http://localhost:9090/api/v1/labels shows all label names. (4) High cardinality causing Prometheus OOM — use metric_relabel_configs to drop high-cardinality labels at scrape time. Here's a relabel config that drops 'userId' labels:

scrape_configs:
  - job_name: 'payment-service'
    scrape_interval: 5s
    metrics_path: '/actuator/prometheus'
    static_configs:
      - targets: ['localhost:8080']
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: '.*'
        action: keep
      - source_labels: [userId]
        regex: '.*'
        action: drop
        # Drops any metric with userId label to prevent explosion