Jenkins Performance Tuning: JVM, Threads, and Backlog Hell

I remember the day our Jenkins master started gasping. Build queue length hit 500, executors were stuck at 100% usage, and developers were screaming. We had just migrated to a new 32-core, 64GB RAM server, thinking it would solve all our problems. Instead, it got worse. The JVM was constantly GCing, causing 10-second pauses. Our thread pool was set to 50 executors on master, believing 'more is better'. SCM polling was set to H/5 for every job—thousands of jobs hitting GitHub every 5 minutes. We had 200 plugins installed, many we never used. JENKINS_HOME was 500GB with years of build artifacts. That incident taught me that Jenkins performance tuning is not just about throwing hardware at it; it's about understanding JVM internals, thread pool behavior, and the subtle interplay of plugins. After weeks of debugging, we cut queue time by 90%, reduced GC pauses to under 200ms, and halved our infrastructure cost. This article shares those hard-won lessons.

1. JVM Heap Tuning: -Xms, -Xmx, and GC Selection

JVM heap tuning is the single most impactful change for Jenkins performance. Start by setting -Xms and -Xmx to the same value to avoid heap resizing overhead. For most production Jenkins masters with 16-64GB RAM, set -Xms8g -Xmx8g. The choice of garbage collector depends on heap size: for heaps >4GB, use G1GC (-XX:+UseG1GC) with a target pause time (-XX:MaxGCPauseMillis=200). For smaller heaps, ParallelGC (-XX:+UseParallelGC) offers better throughput. Add -XX:+UseStringDeduplication to reduce memory from duplicate strings (common in Jenkins logs). Enable -XX:+AlwaysPreTouch to pre-touch all pages at startup, reducing runtime page faults. Example JVM args: -Xms8g -Xmx8g -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:+UseStringDeduplication -XX:+AlwaysPreTouch -XX:+DisableExplicitGC. Disable explicit GC calls to avoid system.gc() from plugins. Monitor GC with -Xlog:gc*:file=gc.log:time,uptime,level,tags -Xlog:safepoint:file=safepoint.log. Use GCeasy or gcviewer to analyze logs; look for Full GC events, long pause times, and promotion failures. In one incident, we saw Full GCs every 10 minutes due to a plugin caching large objects; increasing heap to 12GB and switching to G1GC reduced pauses from 5s to 200ms.

2. Thread Pool Sizing: Master Executor Count and Agent Tuning

Jenkins master executor count is a common misconfiguration. The master should not execute many builds; it's a controller. Set master.executor.count to a low number, typically (CPU cores * 2) or less. For a 32-core master, 8-16 executors is sufficient. Too many executors cause thread contention and GC overhead. Agent executors should be set to CPU cores of the agent machine (e.g., 8 executors on an 8-core agent). Use agent connection pooling: set -masterExecutors to 0 on agents that only run builds, and use 'Jenkins agents via SSH' with connection retries. For queue management, monitor Jenkins queue length via /queue/api/json. If queue grows, increase agent executors or add agents, not master executors. Use 'Pipeline Durability Settings' (e.g., PERFORMANCE_OPTIMIZED) to reduce I/O for pipeline stages. In production, we reduced master executors from 50 to 16, and queue time dropped from 30 minutes to 2 minutes.

3. Build Queue Backlog Analysis

A growing build queue indicates a bottleneck. First, check queue length via /queue/api/json?pretty=true. Look for blocked builds: a build is 'blocked' if waiting for an executor or another build. Use the 'Build Queue' plugin to visualize. Common causes: insufficient agents, too many concurrent pipeline stages, or a slow SCM checkout. Use the 'Pipeline Stage View' to see stage durations. If many builds are blocked on the same stage, consider parallelizing or increasing agent capacity. Use the 'Queue Item' API to get details: curl -s http://jenkins:8080/queue/api/json | jq '.items[] | {id, why, params}'. In one case, we found all builds blocked on 'Checkout' stage because SCM polling was triggering too many builds simultaneously. We added a 'quiet period' of 5 seconds and reduced polling frequency.

4. SCM Polling Optimization: H Syntax and Poll Interval

SCM polling is a major source of unnecessary load. Jenkins uses a hash (H) to distribute polls evenly. Default H/5 means every 5 minutes, but with thousands of jobs, this creates a thundering herd. Change to H/15 or H/30. Better yet, use webhooks (GitHub/GitLab/Bitbucket) to trigger builds only on changes. For jobs that must poll, use 'Polling ignores commits from certain users' to skip internal commits. Use the 'SCM Skip' plugin to avoid triggering builds if only documentation changed. Monitor polling activity via /computer/api/json?tree=computer[executors[currentExecutable]] to see which jobs are polling. In production, we changed 500 jobs from H/5 to H/30 and saw CPU drop 40%.

5. Plugin Bloat Identification and Removal

Plugins are the biggest source of memory leaks and performance issues. List all plugins via /pluginManager/api/json?depth=1. Check which are actually used by scanning job configs. Use a groovy script to find unused plugins: Jenkins.instance.pluginManager.plugins.each { plugin -> if (!plugin.has dependents) println plugin.shortName }. Remove unused plugins via Plugin Manager. Common bloat: build-monitor-plugin, greenballs, extra-columns. In production, we removed 80 unused plugins and saw heap usage drop 2GB. Also, update plugins regularly; older versions have memory leaks. Use the 'Plugin Usage' plugin to see which jobs use which plugins.

6. JENKINS_HOME Size Management

JENKINS_HOME grows fast with build artifacts, logs, and workspace files. Set 'Discard Old Builds' globally with max 30 days retention. For workspace cleanup, use 'Workspace Cleanup Plugin' or 'Job Configuration History Plugin' to limit history. Archive only necessary artifacts. Use symlinks to move large artifact directories to separate storage. Set log rotation: Jenkins system log rotation via logrotate or within Jenkins. In production, we had a 500GB JENKINS_HOME; after cleanup, it dropped to 50GB. Use du -sh JENKINS_HOME/jobs//builds//archive | sort -rh | head -20 to find large builds.

7. GC Logging and Analysis with GCeasy and gcviewer

GC logging is essential for diagnosing heap issues. Enable with -Xlog:gc*:file=gc.log:time,uptime,level,tags -Xlog:safepoint:file=safepoint.log. Use GCeasy (gceasy.io) to upload logs for analysis. Key metrics: throughput, max pause time, number of Full GCs, and allocation rate. For Jenkins, look for frequent Full GCs (indicates memory leak or insufficient heap). Use gcviewer for local analysis. In production, we found a plugin causing 100% allocation rate; GCeasy suggested increasing heap and switching to G1GC. After changes, throughput improved from 95% to 99.9%.

8. Jenkins Monitoring Plugin and Prometheus Metrics

The 'Monitoring' plugin provides graphs of CPU, memory, and GC activity. Better: use 'Prometheus Metrics Plugin' to expose metrics at /prometheus. Key metrics: jenkins_queue_size_value, jenkins_executor_in_use, jenkins_node_builds_duration_seconds. Set up Prometheus and Grafana dashboards. Alert on queue length >10 for 5 minutes, executor usage >90%, or GC pause >1s. In production, we use Prometheus to track build duration trends and detect regressions.

9. Database Optimization for External DB

If using external database (PostgreSQL/MySQL), tune connection pool. Jenkins uses HikariCP; set maximumPoolSize to 10-20. Set connectionTimeout to 30000ms. Use read-only replicas for reporting. Optimize queries: ensure indexes on build_queue, build_run. In production, we moved from embedded DB to PostgreSQL and saw 50% improvement in job scheduling.

10. Nginx Reverse Proxy Caching

Nginx as reverse proxy can cache static assets and reduce load on Jenkins. Configure proxy_cache for /static/ and /pluginManager/. Set expires 7d. Use proxy_buffering to reduce I/O. Example config: location /static/ { proxy_cache jenkins_cache; proxy_cache_valid 200 7d; }. In production, nginx caching reduced Jenkins CPU by 30%.

11. Agent Connection Pooling and Pipeline Durability

For agents, use SSH with connection pooling (set -masterExecutors 0 on agent). Use 'Pipeline Durability Settings' to set PERFORMANCE_OPTIMIZED for pipelines that don't need full durability. This reduces I/O by 50%. In production, we set durability to 'PERFORMANCE_OPTIMIZED' and saw pipeline execution time drop 20%.

12. Large Instance Performance Checklist

For large instances (1000+ jobs, 50+ agents): use external DB, Prometheus monitoring, G1GC heap 16GB+, master executors 8, agent executors = CPU cores, webhooks instead of polling, plugin audit monthly, artifact retention 30 days, nginx caching, and regular thread dump analysis. Use jmap -dump:live,format=b,file=heap.hprof <pid> for heap dump analysis. Use jstack to find deadlocks. In production, we follow this checklist and achieve 99.9% uptime.