Jenkins Monitoring with Prometheus: Stop Reacting, Start Predicting Failures

Your Jenkins master is a single point of failure for your entire delivery pipeline. I've seen a 200-node cluster grind to a halt because nobody noticed the build queue hit 5000 pending jobs at 2 AM. Monitoring isn't optional — it's survival. This article gives you the exact Prometheus configs, alert rules, and dashboards to catch executor exhaustion, queue backlogs, and JVM memory leaks before they take down your deployments. You'll walk away with production-ready scrape configs, Grafana dashboard JSON, and alert rules that have kept my Jenkins masters alive through Black Friday traffic.

Why Default Jenkins Monitoring Fails at Scale

The built-in Jenkins monitoring page shows you the last 10 builds and a memory graph that updates every 30 seconds. That's fine for a hobby project. In production, you need historical trends, correlation with deployments, and alerts that wake you up at 3 AM. Without Prometheus, you're flying blind. The classic mistake is relying on Jenkins' own health check — it reports green even when the queue is 10,000 deep because the master process is still alive. Prometheus gives you the real picture: executor utilization, queue depth, build duration percentiles, and JVM internals.

// io.thecodeforge — DevOps tutorial

scrape_configs:
  - job_name: 'jenkins'
    metrics_path: '/prometheus'
    # Use a dedicated port to avoid CSRF issues
    # Add ?include=[] to filter metrics if needed
    params:
      include: ['jenkins_executor_*', 'jenkins_queue_*', 'jvm_*']
    static_configs:
      - targets: ['jenkins-master:8080']
        labels:
          env: 'production'
    scrape_interval: 60s
    scrape_timeout: 30s
    # Relabel to add instance name
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance
        replacement: 'jenkins-prod'

Installing and Configuring the Prometheus Plugin

The Prometheus plugin exposes metrics at /prometheus. Install it via Jenkins Plugin Manager. Then configure which metrics to expose. By default, it exposes everything — including job-level metrics that can balloon the response. In production, you want to filter aggressively. Use the system property jenkins.metrics.prometheus.exclude to drop high-cardinality metrics like job build durations per branch. I've seen a 1000-job folder generate a 200MB metrics page. Filter it down to aggregate metrics only.

// io.thecodeforge — DevOps tutorial

// Set this in Jenkins script console or as system property
// Exclude job-level metrics to reduce cardinality
System.setProperty('jenkins.metrics.prometheus.exclude', '.*job.*')

// Or include only essential metrics via query parameter
// /prometheus?include[]=jenkins_executor_*&include[]=jenkins_queue_*&include[]=jvm_*

// Verify the endpoint
import jenkins.metrics.impl.PrometheusMetrics
println PrometheusMetrics.instance.getSampleCount()

Essential Metrics: What to Watch and Why

Not all metrics are equal. Focus on the ones that predict failure. Queue depth (jenkins_queue_size_value) tells you if builds are piling up. Executor utilization (jenkins_executor_in_use) tells you if you need more capacity. Build duration (jenkins_runs_duration_seconds) helps detect performance regressions. JVM metrics (jvm_memory_bytes_used, jvm_gc_collection_seconds) catch memory leaks. Ignore job-level metrics like per-branch counters — they create high cardinality and slow down Prometheus.

// io.thecodeforge — DevOps tutorial

# Alert when queue exceeds 50 for 5 minutes
alert: JenkinsQueueHigh
expr: jenkins_queue_size_value > 50
for: 5m
labels:
  severity: critical
annotations:
  summary: "Jenkins queue depth is {{ $value }}"

# Alert when executor utilization > 90%
alert: JenkinsExecutorsExhausted
expr: (jenkins_executor_in_use / jenkins_executor_total) > 0.9
for: 2m
labels:
  severity: warning
annotations:
  summary: "Executor utilization at {{ $value | humanizePercentage }}"

# Alert on long GC pauses
alert: JenkinsGCPauseHigh
expr: rate(jvm_gc_collection_seconds_sum[5m]) > 1
for: 1m
labels:
  severity: critical
annotations:
  summary: "GC pause > 1s in last 5 minutes"

Building a Grafana Dashboard That Tells a Story

A good dashboard shows the system's health at a glance. Start with a row for 'Build Pipeline Health': queue depth, executor utilization, build duration (p50/p95/p99). Second row for 'JVM Health': heap usage, GC pause time, thread count. Third row for 'Throughput': builds completed per minute, success rate. Use Grafana's time series panels with thresholds. Color-code: green (<70%), yellow (70-90%), red (>90%). Add a log panel for recent build failures.

// io.thecodeforge — DevOps tutorial

{
  "title": "Jenkins Production Overview",
  "panels": [
    {
      "title": "Queue Depth",
      "type": "graph",
      "targets": [{"expr": "jenkins_queue_size_value", "legendFormat": "queue"}],
      "thresholds": [{"value": 50, "color": "red"}]
    },
    {
      "title": "Executor Utilization",
      "type": "graph",
      "targets": [{"expr": "jenkins_executor_in_use / jenkins_executor_total", "legendFormat": "utilization"}],
      "thresholds": [{"value": 0.9, "color": "red"}]
    },
    {
      "title": "Build Duration (p99)",
      "type": "graph",
      "targets": [{"expr": "histogram_quantile(0.99, rate(jenkins_runs_duration_seconds_bucket[5m]))", "legendFormat": "p99"}]
    }
  ]
}

Alerting Rules That Don't Wake You Up for Nothing

Bad alerts are worse than no alerts. You need to tune thresholds and durations. A queue depth of 100 for 30 seconds is noise. A queue depth of 100 for 10 minutes is a problem. Use 'for' clauses to avoid flapping. Also set up 'no data' alerts — if Prometheus stops scraping Jenkins, you won't know. Alert on absent(jenkins_queue_size_value) for 5m. Finally, route alerts to the right channel: critical to PagerDuty, warnings to Slack.

// io.thecodeforge — DevOps tutorial

route:
  receiver: 'pagerduty-critical'
  routes:
    - match:
        severity: critical
      receiver: 'pagerduty-critical'
    - match:
        severity: warning
      receiver: 'slack-warnings'

receivers:
  - name: 'pagerduty-critical'
    pagerduty_configs:
      - routing_key: 'your-pagerduty-key'
  - name: 'slack-warnings'
    slack_configs:
      - api_url: 'https://hooks.slack.com/...'
        channel: '#jenkins-alerts'

When Not to Use Prometheus for Jenkins Monitoring

Prometheus is overkill if you have a single Jenkins master with <10 executors and <50 builds per day. In that case, the built-in monitoring page and email alerts suffice. Also, if your Jenkins is ephemeral (spun up per branch), Prometheus scraping becomes complex — consider using a push gateway or centralized logging instead. Finally, if your team doesn't have the bandwidth to maintain a Prometheus stack, use a SaaS solution like Datadog or New Relic with their Jenkins integrations.

Advanced: Custom Metrics and Plugin Development

Sometimes the built-in metrics aren't enough. You may need to expose custom metrics from your pipelines or plugins. The Prometheus plugin provides a Java API to register custom metrics. For pipelines, you can use the Prometheus Pipeline Utility plugin to push metrics during builds. For example, track the number of tests passed/failed, or the size of a deployment artifact. Be careful with cardinality: always use a fixed set of label values.

// io.thecodeforge — DevOps tutorial

// In a Jenkins pipeline, use the Prometheus Pipeline Utility
@Library('prometheus-pipeline-utility') _

node {
    stage('Build') {
        // ... build steps
        prometheus().counter('my_custom_builds_total', 'Total builds', ['status']).inc('success')
    }
}

Scaling Prometheus for Large Jenkins Deployments

When you have hundreds of Jenkins masters or thousands of agents, a single Prometheus server may not suffice. Use federation: have one Prometheus per Jenkins master, then a global Prometheus that scrapes the per-master Prometheus servers. For agents, use service discovery (e.g., Consul) to scrape node_exporter metrics. Consider using Thanos or Cortex for long-term storage and global view. Also, use recording rules to aggregate metrics before alerting to reduce query load.

// io.thecodeforge — DevOps tutorial

# Global Prometheus scrapes per-master Prometheus servers
scrape_configs:
  - job_name: 'federation'
    scrape_interval: 60s
    honor_labels: true
    metrics_path: '/federate'
    params:
      'match[]':
        - '{job="jenkins"}'
    static_configs:
      - targets:
        - 'jenkins-master-1:9090'
        - 'jenkins-master-2:9090'

Integrating with Incident Management and Runbooks

Alerts are useless without a response plan. Integrate Alertmanager with PagerDuty, Opsgenie, or Slack. For each alert, create a runbook with exact steps to diagnose and fix. For example, a 'JenkinsQueueHigh' alert runbook should include: 1) Check which jobs are queued (curl /queue/api/json). 2) Check executor utilization. 3) Check for stuck builds. 4) Increase executors or kill stuck builds. Use alert annotations to link to runbooks.

// io.thecodeforge — DevOps tutorial

# Alert rule with runbook annotation
alert: JenkinsQueueHigh
expr: jenkins_queue_size_value > 50
for: 5m
annotations:
  summary: "Jenkins queue depth is {{ $value }}"
  runbook: "https://runbooks.example.com/jenkins-queue-high"
  severity: critical

Security Considerations for Exposing Metrics

The /prometheus endpoint can leak sensitive information like job names, branch names, and build parameters. In production, you should protect this endpoint. Options: 1) Use a separate port that is not exposed to the internet. 2) Use firewall rules to allow only Prometheus server IPs. 3) Add basic authentication via a reverse proxy (e.g., Nginx). 4) Use the Prometheus plugin's built-in authentication (if available). Never expose /prometheus on the public internet without authentication.

// io.thecodeforge — DevOps tutorial

# Nginx reverse proxy with basic auth for /prometheus
location /prometheus {
    auth_basic "Restricted";
    auth_basic_user_file /etc/nginx/.htpasswd;
    proxy_pass http://jenkins:8080/prometheus;
}

Troubleshooting Common Prometheus Plugin Issues

Even with proper configuration, things go wrong. Common issues: 1) Metrics not appearing: check plugin version, restart Jenkins, verify endpoint. 2) High memory usage: filter metrics, increase heap. 3) Scrape timeouts: reduce metrics, increase timeout. 4) Missing JVM metrics: ensure Jenkins is running with JMX enabled (but Prometheus plugin uses its own JVM metrics, so JMX not required). 5) Duplicate metrics: check if multiple Prometheus instances scrape the same Jenkins.

// io.thecodeforge — DevOps tutorial

# Check if /prometheus is accessible
curl -s -o /dev/null -w "%{http_code}" http://jenkins:8080/prometheus

# Check response size
curl -s -o /dev/null -w "%{size_download}" http://jenkins:8080/prometheus

# Check Jenkins logs for Prometheus errors
journalctl -u jenkins --no-pager | grep -i prometheus

# Check Prometheus target status
curl -s http://prometheus:9090/api/v1/targets | jq '.data.activeTargets[] | select(.labels.job=="jenkins")'

Future-Proofing: What's Next for Jenkins Monitoring

Jenkins monitoring is evolving. OpenTelemetry is gaining traction as a unified standard for metrics, traces, and logs. The Jenkins OpenTelemetry plugin is in active development. In the future, you may replace Prometheus with OpenTelemetry for a single-pane-of-glass observability. However, Prometheus is still the most mature and widely adopted solution. Keep an eye on the Jenkins community for updates. Also, consider using machine learning for anomaly detection on metrics like queue depth and build duration.