Jenkins Backup & Restore: The ThinBackup Plugin + S3 Playbook

I remember the call at 2 AM: 'Jenkins is down, and the last backup was from three months ago.' Our CI/CD pipeline was dead, and we had no idea which plugin versions we were running or what job configurations we'd changed. That night, I learned the hard way that a backup strategy isn't just about having a copy—it's about having the right copy, tested and restorable. Before that, we had a simple cron job that tarred the entire /var/lib/jenkins directory. It worked, but it was bloated, slow, and when we tried to restore on a new server, we discovered that the secrets/ folder didn't match, and all credentials were corrupted.

1. Understanding JENKINS_HOME Backup Strategy: Files vs Directories

The fundamental decision in Jenkins backup is whether to back up individual files or entire directories. The default location is /var/lib/jenkins (or $JENKINS_HOME). This directory contains: - config.xml (global Jenkins config) - jobs/ (job configs and builds) - plugins/ (installed plugins) - secrets/ (encryption keys) - credentials.xml (encrypted credentials) - users/ (user configs) - nodes/ (agent configs) - logs/ (Jenkins logs) - workspace/ (job workspaces) - builds/ (build records) - caches/ (plugin caches)

File-level backup (ThinBackup) copies only specific files and directories, allowing exclusions. This is faster, smaller, and safer because you exclude transient data like workspace/, builds/, logs/, and caches/. These can be regenerated and don't need restoration for Jenkins to function.

Directory-level backup (tar/rsync) copies everything. It's simpler but bloated. For example, a 2 GB JENKINS_HOME can balloon to 20 GB if workspaces are included. Worse, restoring a full tar can overwrite necessary files with stale data (e.g., old master.key).

Production pattern: Use ThinBackup for daily incremental backups (file-level) and a weekly full tar (excluding workspaces) as a safety net. Example tar command: ``bash sudo tar --exclude='workspace' --exclude='builds' --exclude='logs' --exclude='caches' -czf /backup/jenkins-weekly-$(date +%Y%m%d).tar.gz /var/lib/jenkins ``

2. Installing and Configuring ThinBackup Plugin

ThinBackup plugin (version 1.16) is the de facto standard for Jenkins backup. Install it via UI: Manage Jenkins -> Manage Plugins -> Available tab -> search 'ThinBackup' -> Install without restart. Or via CLI: ``bash java -jar jenkins-cli.jar -s http://localhost:8080/ install-plugin thinbackup -restart ` After installation, configure it under Manage Jenkins -> ThinBackup. Key settings: - Backup directory: /backup/jenkins (must exist and be writable by Jenkins user) - Full backup every: 7 days - Diff backup every: 1 day (incremental) - Max number of backup sets: 10 (retention) - Excluded files pattern: workspace/, builds/, logs/, caches/ - Include in backup: check 'Jenkins home directory' and 'Plugin archives' - Cron expression: 0 2 *` (daily at 2 AM)

Example XML configuration file (org.jenkinsci.plugins.thinbackup.ThinBackupPlugin.xml): ``xml <?xml version='1.0' encoding='UTF-8'?> <org.jenkinsci.plugins.thinbackup.ThinBackupPlugin_-DescriptorImpl> <backupDir>/backup/jenkins</backupDir> <fullBackupSchedule>7</fullBackupSchedule> <diffBackupSchedule>1</diffBackupSchedule> <maxNumberOfBackups>10</maxNumberOfBackups> <excludedFilesRegex>workspace/.|builds/.|logs/.|caches/.</excludedFilesRegex> <cron>0 2 *</cron> <includePluginArchives>true</includePluginArchives> </org.jenkinsci.plugins.thinbackup.ThinBackupPlugin_-DescriptorImpl> `` This file can be manually edited, but changes take effect only after Jenkins restart.

3. Cron-Based Backup Scheduling and Automation

ThinBackup handles scheduling internally via its cron configuration, but you can also use system cron for additional automation (e.g., syncing to S3). The plugin's cron expression uses standard cron syntax (e.g., 0 2 * for 2 AM daily). To verify the cron is running, check Jenkins logs: ``bash sudo grep -i 'thinbackup' /var/log/jenkins/jenkins.log | tail -5 `` You should see entries like 'Starting backup' and 'Backup finished'.

For external automation, create a shell script that runs before/after the ThinBackup backup. Example: a cron job that syncs to S3 after backup: ``bash #!/bin/bash # /usr/local/bin/jenkins-backup-s3.sh BACKUP_DIR='/backup/jenkins' S3_BUCKET='s3://my-jenkins-backups' aws s3 sync "$BACKUP_DIR" "$S3_BUCKET" --delete --exclude '.tmp' ` Add to crontab: `bash 0 3 /usr/local/bin/jenkins-backup-s3.sh `` This runs at 3 AM, one hour after ThinBackup completes.

Production tip: Use flock to prevent overlapping backups if backup takes longer than interval: ``bash 0 2 * /usr/bin/flock -n /tmp/jenkins-backup.lock /usr/local/bin/jenkins-backup.sh ``

4. Restore Procedure Walkthrough: Full and Partial Restore

Full restore (complete failure): 1. Stop Jenkins: sudo systemctl stop jenkins 2. Backup current broken state (optional): sudo mv /var/lib/jenkins /var/lib/jenkins.broken 3. Restore from ThinBackup: sudo cp -a /backup/jenkins/2024-01-15_02-00-00/. /var/lib/jenkins/ (replace with actual backup directory) 4. Ensure permissions: sudo chown -R jenkins:jenkins /var/lib/jenkins 5. Start Jenkins: sudo systemctl start jenkins 6. Verify: curl -sI http://localhost:8080/login should return HTTP 200.

Important: For partial restore of credentials, you must restore the entire secrets/ directory, not just credentials.xml, because the encryption key is in secrets/master.key.

5. Credential Backup and Restore Considerations (Encrypted keys.xml)

Credentials in Jenkins are stored in credentials.xml (system-level) and jobs/*/credentials.xml (folder-level). These files are encrypted using a secret key stored in secrets/master.key and secrets/hudson.util.Secret. The encryption is symmetric: the same key must be used to decrypt.

Backup: Include the entire secrets/ directory and credentials.xml files. Do not exclude them. Example backup command for credentials only: ``bash sudo tar -czf /backup/jenkins-credentials-$(date +%Y%m%d).tar.gz /var/lib/jenkins/secrets /var/lib/jenkins/credentials.xml ``

Restore: 1. Stop Jenkins. 2. Copy the secrets/ directory from backup: sudo cp -a /backup/jenkins/secrets /var/lib/jenkins/ 3. Copy credentials.xml: sudo cp /backup/jenkins/credentials.xml /var/lib/jenkins/ 4. Set permissions: sudo chown -R jenkins:jenkins /var/lib/jenkins/secrets /var/lib/jenkins/credentials.xml 5. Start Jenkins.

Migration: If moving to a new server, you must copy secrets/master.key and secrets/hudson.util.Secret from the old server to the new one before starting Jenkins. If you start Jenkins without these files, it generates new keys and all credentials become undecryptable.

Verification: After restore, go to Credentials -> System -> Global credentials and check that decryption works. You can also use the Jenkins CLI: ``bash java -jar jenkins-cli.jar -s http://localhost:8080/ list-credentials-as-xml system::system::jenkins `` If it returns XML without errors, credentials are intact.

6. Migration Guide: Moving Jenkins to a New Server

Migrating Jenkins to a new server requires careful planning to avoid credential loss and plugin version mismatches. Steps:

1. On old server: Create a full backup using ThinBackup (or tar with exclusions). Also copy plugin list:
2. ```bash
3. java -jar jenkins-cli.jar -s http://localhost:8080/ list-plugins > /tmp/plugins.txt
4. ```
5. On new server: Install same Java version (e.g., OpenJDK 11) and Jenkins version (e.g., 2.440.1). Install same plugins via UI or CLI:
6. ```bash
7. cat /tmp/plugins.txt | awk '{print $1}' | xargs -I {} java -jar jenkins-cli.jar -s http://localhost:8080/ install-plugin {}
8. ```
9. Stop Jenkins on both servers: sudo systemctl stop jenkins
10. Copy JENKINS_HOME: Use rsync to transfer files, excluding transient dirs:
11. ```bash
12. sudo rsync -avz --exclude='workspace' --exclude='builds' --exclude='logs' --exclude='caches' /var/lib/jenkins/ user@new-server:/var/lib/jenkins/
13. ```
14. Handle secrets: Ensure secrets/master.key and secrets/hudson.util.Secret are identical. If not, copy them from old server.
15. Fix permissions: sudo chown -R jenkins:jenkins /var/lib/jenkins
16. Start Jenkins: sudo systemctl start jenkins
17. Test: Run a few jobs to verify everything works.

Rollback plan: Keep old server running until new one is fully verified. If migration fails, switch DNS back to old server.

7. S3 and Cloud Storage for Backups

Storing backups in S3 provides off-site redundancy. Use AWS CLI to sync ThinBackup directory to S3. Example script: ``bash #!/bin/bash # /usr/local/bin/sync-to-s3.sh BACKUP_DIR='/backup/jenkins' S3_BUCKET='s3://my-jenkins-backups' aws s3 sync "$BACKUP_DIR" "$S3_BUCKET" --delete --exclude '.tmp' --no-follow-symlinks ` Add to crontab (runs after ThinBackup): `bash 30 2 /usr/local/bin/sync-to-s3.sh ``

S3 lifecycle policy to manage retention: ``json { "Rules": [ { "Id": "jenkins-backup-lifecycle", "Status": "Enabled", "Filter": { "Prefix": "" }, "Transitions": [ { "Days": 30, "StorageClass": "GLACIER" } ], "Expiration": { "Days": 90 } } ] } `` This archives backups to Glacier after 30 days and deletes after 90.

Encryption: Enable S3 server-side encryption (SSE-S3) or use KMS. Add --sse aws:kms to sync command.

Restore from S3: First download to local backup directory: ``bash aws s3 sync s3://my-jenkins-backups /backup/jenkins-restored `` Then follow restore procedure.

8. Backup Verification and Testing

A backup is only as good as its restore. Regularly test backups in a staging environment. Automate verification with a script:

```bash #!/bin/bash # /usr/local/bin/test-restore.sh BACKUP_DIR='/backup/jenkins' TEST_DIR='/tmp/jenkins-restore-test' JENKINS_HOME='/var/lib/jenkins'

# Clean test dir rm -rf "$TEST_DIR" mkdir -p "$TEST_DIR"

# Copy latest backup (assumes ThinBackup format) LATEST=$(ls -td "$BACKUP_DIR"/*/ | head -1) cp -a "$LATEST" "$TEST_DIR"

# Check for essential files echo 'Checking essential files...' for f in 'config.xml' 'secrets/master.key' 'credentials.xml'; do if [ -f "$TEST_DIR/$f" ]; then echo "OK: $f" else echo "MISSING: $f" exit 1 fi done

# Simulate restore by starting Jenkins in Docker (optional) echo 'Backup verification passed.' ```

Schedule this script weekly via cron: ``bash 0 4 0 /usr/local/bin/test-restore.sh >> /var/log/jenkins-backup-test.log 2>&1 ``

Manual verification: Periodically perform a full restore in a staging environment and run a few build jobs to ensure everything works.

Alerting: Set up monitoring to alert if backup size is below threshold (e.g., < 100 MB) or if backup fails. Use Jenkins' own monitoring or external tools like Prometheus.

9. ThinBackup Plugin Internals and Customization

ThinBackup stores backups in a directory structure like /backup/jenkins/YYYY-MM-DD_HH-MM-SS/. Each backup set contains a full backup (every N days) and incremental diffs (daily). The plugin uses Java's ZipOutputStream to compress files. You can customize the backup by editing the XML config file directly.

Manual backup trigger: You can trigger a backup via Jenkins CLI: ``bash java -jar jenkins-cli.jar -s http://localhost:8080/ thinBackupBackup ` Or via HTTP POST with authentication: `bash curl -X POST -u admin:token http://localhost:8080/thinBackup/backup ``

Restore from CLI: ``bash java -jar jenkins-cli.jar -s http://localhost:8080/ thinBackupRestore /backup/jenkins/2024-01-15_02-00-00 `` This restores the entire backup set.

Custom script to list backups: ``bash ls -d /backup/jenkins/*/ ``

10. Handling Large-Scale Jenkins Instances (1000+ Jobs)

For large Jenkins instances, backup performance matters. ThinBackup can be slow with thousands of jobs. Optimizations: - Exclude builds/ and workspace/ as always. - Use incremental backups (diff) instead of full daily backups. - Increase JVM heap for Jenkins if backup times out: add -Xmx4g to JAVA_ARGS in /etc/default/jenkins. - Distribute backup load: Schedule full backup weekly, incremental daily. - Use parallel compression: ThinBackup doesn't support parallel, but you can compress backup directory externally with pigz: ``bash tar --use-compress-program=pigz -cf /backup/jenkins-$(date +%Y%m%d).tar.gz /backup/jenkins/2024-01-15_02-00-00 ``

Backup time benchmark: For 500 jobs, a full backup takes ~10 minutes with exclusions. Without exclusions, it can take over an hour.

Monitoring: Use Jenkins' own 'Backup' page to view last backup status. Integrate with monitoring tools like Datadog or Prometheus by parsing logs.

11. Backup Security and Compliance

Backups contain sensitive data (credentials, job configs with passwords). Protect them: - Encrypt at rest: Use S3 SSE-S3 or KMS. For local backups, use gpg: ``bash tar czf - /backup/jenkins | gpg --encrypt --recipient admin@example.com > /backup/jenkins-encrypted.tar.gz.gpg ` - Access control: Restrict backup directory permissions to jenkins user only: chmod 700 /backup/jenkins`. - Audit logs: Enable Jenkins audit trail plugin to log backup/restore actions. - Compliance: For SOC2/PCI, ensure backups are stored in a separate region and access is logged. Use S3 bucket policies that enforce encryption and deny public access. - Retention policy: Define how long backups are kept. Example: 30 days hot, 90 days cold, then delete. - Incident response: Have a documented procedure for restoring backups in case of ransomware. Store an offline backup (e.g., tape or air-gapped).

12. Advanced: Scripting Your Own Backup Solution

If ThinBackup doesn't fit your needs (e.g., custom compression, multi-cloud), you can script your own backup using Jenkins CLI and shell. Example script:

```bash #!/bin/bash # /usr/local/bin/custom-jenkins-backup.sh JENKINS_HOME='/var/lib/jenkins' BACKUP_DIR='/backup/jenkins-custom' DATE=$(date +%Y%m%d_%H%M%S)

mkdir -p "$BACKUP_DIR/$DATE"

# Backup configs and jobs (exclude builds, workspaces, logs, caches) rsync -av --exclude='workspace' --exclude='builds' --exclude='logs' --exclude='caches' "$JENKINS_HOME/" "$BACKUP_DIR/$DATE/"

# Backup plugin list java -jar jenkins-cli.jar -s http://localhost:8080/ list-plugins > "$BACKUP_DIR/$DATE/plugins.txt"

# Backup credentials separately (if needed) cp "$JENKINS_HOME/credentials.xml" "$BACKUP_DIR/$DATE/" cp -a "$JENKINS_HOME/secrets" "$BACKUP_DIR/$DATE/"

# Compress cd "$BACKUP_DIR" tar -czf "jenkins-backup-$DATE.tar.gz" "$DATE" rm -rf "$DATE"

# Sync to S3 aws s3 cp "jenkins-backup-$DATE.tar.gz" "s3://my-bucket/" ```

Restore script: ```bash #!/bin/bash # /usr/local/bin/custom-jenkins-restore.sh BACKUP_FILE='s3://my-bucket/jenkins-backup-20240115_020000.tar.gz' TEMP_DIR=$(mktemp -d)

aws s3 cp "$BACKUP_FILE" "$TEMP_DIR/" cd "$TEMP_DIR" tar -xzf "$BACKUP_FILE"

sudo systemctl stop jenkins sudo cp -a . /var/lib/jenkins/ sudo chown -R jenkins:jenkins /var/lib/jenkins sudo systemctl start jenkins ```

This gives full control but requires maintenance.