Upgrading Jenkins: Safe Strategy, Rollback, Plugin Compat Gotchas

I once upgraded Jenkins from 2.249.3 to 2.277.1 on a Friday afternoon, confident in the changelog. By Monday, every pipeline using Groovy sandbox failed with SecurityExceptions. The plugin compatibility matrix had silently changed – the sandbox was stricter. We spent two days rolling back, restoring JENKINS_HOME from last week's backup, and pinning plugin versions. That incident taught me that Jenkins upgrades are not just about core; they're about the entire ecosystem. Since then, I've adopted a rigorous strategy: pre-upgrade checklist, staging test with identical plugin set, and a proven rollback drill. This article covers that strategy, including blue-green for HA, plugin compatibility tools, and a real production incident with the Groovy sandbox breaking change.

1. Understanding Jenkins Release Cadence

Jenkins follows a predictable release cadence: LTS releases every 12 weeks (e.g., 2.361.1, 2.375.1) and weekly security releases (e.g., 2.375.2). LTS releases are stable and recommended for production. Weekly releases include security fixes and bug fixes but may introduce regressions. Production teams should always use LTS and only apply weekly security releases after testing. The versioning scheme is: MAJOR.MINOR.PATCH where MAJOR is the weekly release base, MINOR is the LTS baseline, and PATCH is the security patch level. For example, 2.361.1 means LTS based on weekly 2.361 with one patch. Upgrading across LTS lines (e.g., 2.361.x to 2.375.x) requires careful plugin compatibility checks.

2. Pre-Upgrade Checklist

Before any upgrade, execute this checklist: 1) Backup JENKINS_HOME: tar czf jenkins_backup_$(date +%Y%m%d).tar.gz /var/lib/jenkins. 2) Export plugin list: curl -s -u admin:token http://localhost:8080/pluginManager/api/json?depth=1 | jq -r '.plugins[] | "\(.shortName):\(.version)"' > plugins_current.txt. 3) Record current version: curl -s -u admin:token http://localhost:8080/api/json | jq -r '.version' > version_current.txt. 4) Check plugin compatibility: use plugin-versions-compat tool. 5) Review changelog for breaking changes. 6) Schedule maintenance window. 7) Notify team.

3. Upgrade Methods: apt, WAR, Docker, Helm

Jenkins can be upgraded via multiple methods. For Debian/Ubuntu: sudo apt update && sudo apt upgrade jenkins. This upgrades the package and restarts the service. For WAR file replacement: download new WAR, stop Jenkins, replace war, start. Example: sudo systemctl stop jenkins; sudo mv /usr/share/jenkins/jenkins.war /usr/share/jenkins/jenkins.war.old; sudo wget -O /usr/share/jenkins/jenkins.war https://get.jenkins.io/war/2.361.1/jenkins.war; sudo systemctl start jenkins. For Docker: pull new image tag (e.g., jenkins/jenkins:lts-jdk11), stop container, run with same volumes. For Helm (Kubernetes): helm upgrade jenkins jenkins/jenkins --version 4.2.0 --set controller.imageTag=2.361.1. Each method has trade-offs: apt is simplest but may auto-update plugins; WAR gives control; Docker/Helm enable blue-green.

4. Plugin Compatibility Checking

The biggest risk in upgrading Jenkins is plugin incompatibility. Use the plugin-versions-compat tool (available on GitHub) to check compatibility. Run: java -jar plugin-versions-compat.jar --jenkins-version 2.361.1 --plugin-file plugins_current.txt. It outputs a report showing which plugins are compatible, require upgrade, or are incompatible. Another tool is jenkins-plugin-compat-checker which can be run as a Jenkins CLI command. Additionally, Jenkins Plugin Manager shows compatibility information for each plugin. Always check the plugin's documentation and changelog. For critical plugins (Pipeline, Git, Blue Ocean), ensure they support the target core version.

5. Blue-Green Upgrade for HA Setups

For high-availability Jenkins setups (e.g., active-passive with shared storage), use blue-green upgrade. Have two identical Jenkins instances (blue and green) with separate JENKINS_HOME directories but share the same storage for jobs and artifacts. Upgrade the inactive instance (green), test it, then switch traffic via load balancer. Steps: 1) Ensure both instances are running same version initially. 2) Stop green instance, upgrade its JENKINS_HOME (from backup) and core version. 3) Start green, run smoke tests. 4) Switch load balancer to green. 5) Keep blue as rollback target. This minimizes downtime. For Kubernetes, use two Helm releases or two deployments with different tags.

6. Rolling Back Jenkins

Rollback plan is essential. Steps: 1) Stop Jenkins. 2) Restore JENKINS_HOME from backup: tar xzf jenkins_backup_20230101.tar.gz -C /var/lib/jenkins. 3) Restore previous WAR file: sudo mv /usr/share/jenkins/jenkins.war /usr/share/jenkins/jenkins.war.new; sudo mv /usr/share/jenkins/jenkins.war.old /usr/share/jenkins/jenkins.war. 4) Pin plugin versions: restore plugins from backup or use versions.txt to install specific versions. 5) Start Jenkins. 6) Verify health. For apt upgrades, downgrade package: sudo apt install jenkins=2.249.3. For Docker, use previous image tag. For Helm, helm rollback jenkins 1. Test rollback in staging regularly.

7. Testing Upgrades in Staging

Always test upgrades in a staging environment that mirrors production. Use Docker Compose to create a test Jenkins instance: docker-compose.yml with Jenkins image, same plugins, and a sample pipeline. Steps: 1) Copy production plugins.txt and JENKINS_HOME (anonymized). 2) Build Docker image with those plugins. 3) Run container, trigger sample pipeline. 4) Check health endpoint. 5) Run a full pipeline smoke test that exercises all critical jobs. 6) Verify plugin list matches production expectations. Use tools like curl and jq to automate validation. Staging should have same core version, plugins, and job configurations as production.

8. Plugin Pinning Strategy

To avoid unexpected plugin upgrades, pin plugin versions in a versions.txt file. Jenkins supports a pinned plugins file: create JENKINS_HOME/pinnedPlugins.txt with format pluginId:version. Jenkins will not auto-upgrade pinned plugins. Alternatively, use plugins.txt in Docker image to install specific versions. Example: git:4.11.0 pipeline:2.7.4. Commit this file to version control. When upgrading core, update plugin versions in the file after compatibility check. This ensures reproducible builds and easy rollback.

9. Upgrade Validation Script

After upgrade, run a validation script to ensure everything is working. Example script: ``bash #!/bin/bash # Health check if curl -s -o /dev/null -w '%{http_code}' http://localhost:8080/login | grep -q 200; then echo "Health check passed" else echo "Health check failed" exit 1 fi # Run test job curl -X POST -u admin:token http://localhost:8080/job/test-pipeline/build?delay=0 sleep 30 # Verify job result curl -s -u admin:token http://localhost:8080/job/test-pipeline/lastBuild/api/json | jq -r '.result' | grep -q SUCCESS # Verify plugin list matches expected curl -s -u admin:token http://localhost:8080/pluginManager/api/json?depth=1 | jq -r '.plugins[] | "\(.shortName):\(.version)"' > plugins_after.txt diff plugins_expected.txt plugins_after.txt || { echo "Plugin mismatch"; exit 1; } `` Integrate this into CI/CD pipeline to automate post-upgrade checks.

10. Managing Plugin Compatibility Matrix

Jenkins core version and plugin versions have a compatibility matrix. For example, Pipeline plugin 2.7.4 requires Jenkins 2.361.x or later. Tools like plugin-versions-compat generate this matrix. Maintain a spreadsheet or document mapping core versions to compatible plugin versions. When upgrading core, update this matrix. Use jenkins-plugin-compat-checker CLI to query compatibility. Example: java -jar jenkins-plugin-compat-checker.jar --plugin git --core 2.361.1. Also check plugin dependencies: some plugins depend on other plugins with specific versions.

11. Upgrade Incident: The Groovy Sandbox Breaking Change

In January 2021, Jenkins 2.277.1 introduced a stricter Groovy sandbox. Our pipelines used sh with script parameters that were previously allowed. After upgrade, all such pipelines failed with 'Scripts not permitted to use method groovy.lang.GroovyShell evaluate'. The fix was to either add method signatures to script approval or rewrite pipelines to use sh with approved scripts. We rolled back and updated all pipelines. Lesson: always check Jenkins security advisories for Groovy sandbox changes. Since then, we test all pipeline scripts in staging with the new core version before production upgrade.

12. Weekly Upgrade Maintenance Checklist

For weekly security releases, follow this checklist: 1) Review security advisory. 2) If vulnerability affects your setup, proceed. 3) Backup JENKINS_HOME. 4) Export plugin list. 5) Upgrade in staging, run validation script. 6) If staging passes, upgrade production during maintenance window. 7) Run validation script in production. 8) Monitor for 24 hours. For LTS upgrades (every 12 weeks), add: full plugin compatibility check, pipeline smoke test, rollback drill. Schedule LTS upgrades with a longer maintenance window (2-4 hours). Keep a changelog of upgrades and any issues encountered.