Jenkins Backup and Disaster Recovery: The Only Guide That Won't Fail You at 3 AM

I remember the 3 AM call vividly. The Jenkins master had crashed due to a disk failure. I thought, 'No problem, we have backups.' But the backup script had silently failed for weeks because the backup directory ran out of space. The incremental backups were corrupt, and the last full backup was six months old. Restoring took two days, and we lost hundreds of build records. That night, I learned that backup is not a one-time setup; it's a living process that requires constant validation. Since then, I've built a disaster recovery strategy that has survived multiple real incidents. This guide compiles those hard-learned lessons.

1. Understanding Jenkins Home Directory

The Jenkins home directory ($JENKINS_HOME) is the heart of your Jenkins master. It contains everything: job configurations, build logs, plugins, credentials, and system configuration. A common mistake is backing up only part of it. For example, forgetting the .jenkins secret key files can render credentials unrecoverable. The default location is /var/lib/jenkins on Linux. To find yours, check the Jenkins system info page at http://yourjenkins:8080/systemInfo. The JENKINS_HOME variable is set in the init script or systemd service file. For example, in /etc/default/jenkins or /etc/sysconfig/jenkins. You should also verify the size: du -sh $JENKINS_HOME. A typical production Jenkins with 100 jobs and 1 year of builds can be 50-200 GB. Excluding large artifacts (e.g., lastSuccessful/, lastStable/) can reduce backup size significantly. Use find $JENKINS_HOME -type d -name 'lastSuccessful' -prune to identify them. The backup must include: config.xml, credentials.xml, secrets/, plugins/, jobs/, nodes/, users/, updates/, identity.key, secret.key, and master.key. Missing any of these can cause partial or failed restore.

2. Full vs Incremental Backup Strategy

A full backup captures everything at a point in time. An incremental backup captures changes since the last full. For Jenkins, a weekly full backup and daily incremental is standard. Use tar for full: tar -czf jenkins-full-$(date +%Y%m%d).tar.gz -C / $JENKINS_HOME. For incremental, use rsync with --link-dest to create hard links to unchanged files: rsync -a --delete --link-dest=/backup/full/current $JENKINS_HOME/ /backup/incremental/$(date +%Y%m%d). This saves space and speeds up backups. However, incremental restores require the full backup and all incrementals in order. A simpler approach is to use a tool like duplicity which handles encryption and incremental backups automatically. Example: duplicity --full-if-older-than 7D $JENKINS_HOME s3://bucket/jenkins. For large Jenkins instances, consider excluding build artifacts: --exclude 'builds//archive' --exclude 'builds//lastSuccessful'. Always test your restore process: spin up a fresh Jenkins, apply the full backup, then apply incrementals. If any incremental is missing, you lose that data. In production, we had a case where the incremental script failed due to a permissions change, and we didn't notice for a week. The full backup was two weeks old. We lost a week of job configurations and build records. Now we run a daily integrity check: compare file count and total size between backup and live.

3. Automating Backups with Jenkins Jobs

Using Jenkins itself to back up Jenkins is meta but effective. Create a freestyle job that runs a shell script. Set it to run weekly for full and daily for incremental. Use the Throttle Concurrent Builds plugin to prevent overlapping runs. Example script for full backup: #!/bin/bash BACKUP_DIR=/backup/full TIMESTAMP=$(date +%Y%m%d_%H%M%S) tar -czf $BACKUP_DIR/jenkins-full-$TIMESTAMP.tar.gz -C / $JENKINS_HOME # Upload to S3 aws s3 cp $BACKUP_DIR/jenkins-full-$TIMESTAMP.tar.gz s3://my-bucket/jenkins/ # Cleanup old backups (keep last 4 weeks) find $BACKUP_DIR -name '*.tar.gz' -mtime +28 -delete. For incremental, use rsync as above. Store the script in a version-controlled repository. Add error handling: if tar fails (e.g., disk full), send an email and trigger a PagerDuty alert. Use the Email Extension plugin: emailext body: 'Backup failed: ${BUILD_LOG}' subject: 'Jenkins Backup Failure' to: 'ops@company.com'. Also, log the backup status to a file: echo $? > /var/log/jenkins-backup-status.txt. Monitor this file with a separate job. Another best practice: run a restore test job monthly on a staging Jenkins. This job copies the latest backup, extracts it to a temporary directory, starts a new Jenkins instance on a different port, and runs a few smoke tests. If it fails, alert the team.

4. Credential Backup and Encryption

Credentials are the most sensitive part of Jenkins. They include API tokens, SSH keys, and passwords. Jenkins stores them in credentials.xml and encrypted with a master key in secrets/ directory. The master key is stored in master.key and secret.key. If you lose these, credentials cannot be decrypted. Therefore, backup credentials separately and encrypt them with GPG. First, extract credentials: cp $JENKINS_HOME/credentials.xml /tmp/. Then encrypt: gpg --encrypt --recipient 'ops@company.com' /tmp/credentials.xml. Store the encrypted file in a secure location like HashiCorp Vault or AWS KMS. Also, backup the master key files: cp $JENKINS_HOME/secrets/master.key $JENKINS_HOME/secrets/secret.key /backup/secure/. Encrypt those as well. For restore, you must restore the master key first before credentials.xml. Otherwise, Jenkins will generate a new key and fail to decrypt. In production, we had an incident where the backup script excluded secrets/ because it was hidden. Restore resulted in all credentials being corrupted. Now we explicitly include secrets/ and validate by running java -jar jenkins-cli.jar list-credentials after restore.

5. Plugin Backup and Compatibility

Plugins are stored in $JENKINS_HOME/plugins/. Each plugin has a .jpi file and a directory with configuration. When restoring, you must restore all plugins to the exact versions they were before. Otherwise, jobs may fail due to missing extensions or API changes. Use the Plugin Manager API to get a list: curl -s 'http://jenkins:8080/pluginManager/api/json?depth=1' | jq '.plugins[] | {shortName, version}'. Store this list alongside backups. During restore, install plugins from the backup directory: cp -r /backup/plugins/* $JENKINS_HOME/plugins/. Then restart Jenkins. Alternatively, use the Jenkins CLI: java -jar jenkins-cli.jar -s http://jenkins:8080 install-plugin <plugin>. However, this may download the latest version, not the backed-up one. To ensure version consistency, always restore from the backup copy. In production, we had a case where the backup script copied only .jpi files but not the plugin directories (which contain configuration). After restore, plugins worked but lost their settings. Now we use rsync -a to preserve directories.

6. Job Configuration Backup

Each job is stored in $JENKINS_HOME/jobs/<jobname>/config.xml. Additionally, build records are in builds/ subdirectory. For backup, include all job directories. However, you may want to exclude large build artifacts (e.g., builds/*/archive). Use rsync with --exclude patterns. For restore, simply copy the job directory back. If you only have config.xml (no builds), you can recreate the job but lose history. In production, we back up config.xml daily and full builds weekly. This balances space and recovery point objective. To restore a single job, you can use the Jenkins API: curl -X POST -H 'Content-Type: text/xml' -d @config.xml 'http://jenkins:8080/createItem?name=jobname'. But this creates a new job without history. For full restore, copy the entire jobs/ directory. Be careful with job names that contain special characters; they are URL-encoded in the filesystem. For example, a job named 'my job' becomes 'my%20job'. Use ls -b to see literal names.

7. Off-Site Storage and Versioning

Storing backups on the same server as Jenkins defeats the purpose. Use off-site storage like AWS S3, Google Cloud Storage, or a remote NFS mount. Enable versioning on S3 buckets to protect against accidental deletion or overwrite. Example S3 bucket policy: aws s3api put-bucket-versioning --bucket my-jenkins-backups --versioning-configuration Status=Enabled. Also, set lifecycle rules to expire old versions after 90 days. For encryption, use server-side encryption with AWS KMS. When uploading, use aws s3 cp --sse aws:kms --sse-kms-key-id <key-id>. For cross-region replication, set up a replication rule to another region for disaster recovery. In production, we had a scenario where a disgruntled employee deleted the backup bucket. Because versioning was enabled, we recovered all objects. Without versioning, we would have lost everything. Also, consider using a tool like restic which supports multiple backends and encryption natively. Example: restic -r s3:s3.amazonaws.com/bucket/backup backup $JENKINS_HOME.

8. Restore Order and Procedure

Restoring Jenkins is not just copying files back. The order matters: 1) Install the same Jenkins version. 2) Stop Jenkins. 3) Restore secrets/ (master.key, secret.key). 4) Restore plugins/ directory. 5) Restore config.xml (system config). 6) Restore jobs/, nodes/, users/ etc. 7) Restore credentials.xml. 8) Start Jenkins. 9) Verify plugins load correctly. 10) Run a test job. If you restore credentials before plugins, Jenkins may fail to decrypt because plugin versions changed. Always restore secrets first because they are needed to decrypt credentials. Document this order in a runbook. Use a script to automate: #!/bin/bash JENKINS_HOME=/var/lib/jenkins BACKUP=/backup/latest systemctl stop jenkins cp -a $BACKUP/secrets/ $JENKINS_HOME/secrets/ cp -a $BACKUP/plugins/ $JENKINS_HOME/plugins/ cp $BACKUP/config.xml $JENKINS_HOME/ cp -a $BACKUP/jobs $JENKINS_HOME/ cp $BACKUP/credentials.xml $JENKINS_HOME/ systemctl start jenkins. Test this script on a staging environment first. In production, we had a failed restore because we forgot to restore identity.key, which caused SSH agent connections to fail. Now we include all files from $JENKINS_HOME except builds/ artifacts.

9. Disaster Recovery for Distributed Jenkins (Master/Agent)

If you use Jenkins agents (nodes), you need to back up the master configuration for agents (stored in nodes/ directory) and the agent's own workspace if needed. However, agents are often ephemeral; you can rebuild them with configuration management (Ansible, Terraform). The master's config.xml contains agent definitions. For restore, after master is up, agents will reconnect automatically if their credentials are intact. For agent-specific data (e.g., workspaces), back up only if required. In production, we use a warm standby master that syncs backups from the primary. We use rsync to replicate $JENKINS_HOME to a standby server every hour. If the primary fails, we promote the standby by updating DNS. The standby runs with the same Jenkins version and plugins. We test failover quarterly: stop the primary, point DNS to standby, verify jobs run. The key is to have identical plugin sets. We use a version file: cat plugins/*.jpi | md5sum > plugin_checksum.txt and compare between primary and standby.

10. Monitoring Backup Health

Backups are useless if they fail silently. Monitor backup jobs with alerts. Use a monitoring tool like Prometheus with the Jenkins Exporter to track backup job status. Create a custom metric: jenkins_job_last_result{job='backup-full'} = 0 for success. Set up alerts for failure. Also, monitor disk space on the backup destination: df -h /backup | awk 'NR==2 {print $5}' | sed 's/%//'. Alert if above 80%. For S3, monitor bucket size and version count. Use CloudWatch metrics: aws cloudwatch get-metric-statistics --namespace AWS/S3 --metric-name BucketSizeBytes --dimensions Name=BucketName,Value=my-bucket --statistics Average. In production, we had a case where the backup script ran but the S3 upload failed due to permissions. The script didn't check the exit code of aws s3 cp. Now we use set -o pipefail and check $? after each command. Also, we generate a backup report email daily with file count, size, and checksum.

11. Backup for Jenkins Configuration as Code (JCasC)

If you use Jenkins Configuration as Code (JCasC), your configuration is in YAML files. Backup these files separately. They are typically stored in a Git repository. However, JCasC does not cover everything (e.g., build history). You still need to backup jobs and builds. The advantage is that you can quickly recreate the master from scratch by applying the YAML. The backup strategy becomes: backup the JCasC repo (already in Git), backup jobs and builds, backup credentials (encrypted). For restore, you can spin up a new Jenkins, apply JCasC, then restore jobs and credentials. This reduces restore time because you don't need to backup the entire $JENKINS_HOME. In production, we use JCasC for the base configuration and backup only jobs/, credentials.xml, and secrets/. This cut backup size by 80%. However, we had an incident where a JCasC change broke the master. We rolled back by reverting the Git commit and reapplying. No data loss.

12. Testing Your Disaster Recovery Plan

A disaster recovery plan is only as good as its last test. Schedule quarterly restore drills. Use a staging environment with the same OS and Jenkins version. Steps: 1) Simulate failure by deleting $JENKINS_HOME. 2) Run your restore script. 3) Verify all jobs exist and build successfully. 4) Check credentials are accessible. 5) Run a sample pipeline. 6) Measure time to restore. Document the results and improve. In production, we found that our restore script took 4 hours because we were copying large artifact directories. We optimized by excluding them. Also, we discovered that the backup of users/ directory caused permission issues on the new server. Now we use chown -R jenkins:jenkins after restore. Another test: restore to a different AWS region to verify cross-region recovery. Use tools like Chaos Monkey to randomly fail Jenkins master and trigger automatic failover. After each drill, update the runbook. We had a drill where the standby master couldn't start because of a missing plugin dependency. We added a pre-flight check script that validates plugin compatibility.