Jenkins High Availability: The NFS Locking Gotcha That Wrecked Our Builds

I'll never forget the 3 AM PagerDuty alert: 'Jenkins master unreachable'. Our single Jenkins controller had died, and with it, all running builds. The queue vanished, artifacts were lost, and we spent the next 8 hours restoring from backups. That incident taught me that Jenkins HA isn't just about spinning up another instance — it's about shared storage, database consistency, and locking. After moving to an active-passive setup with NFS v4 and PostgreSQL, we've survived multiple failures without losing a single build. But we also hit the NFS locking gotcha that caused silent corruption of job configurations. In this guide, I'll share the exact production setup that has kept our CI/CD pipeline running for over two years.

Active-Passive HA Architecture with Shared JENKINS_HOME via NFS

The foundation of Jenkins HA is an active-passive pair of controllers sharing a single JENKINS_HOME directory via NFS. The active node handles all user requests and build executions. The passive node mounts the same NFS share but remains idle, monitoring the active node's health. On failure detection, the passive node promotes itself to active and begins serving requests. This architecture is simple but has critical caveats around file locking. NFS v3 with nolock is a common pitfall that can lead to data corruption. We recommend NFS v4 with vers=4.0,minorversion=0 and enabling advisory locking. The /etc/fstab entry should include hard,intr,noatime for reliability. Also, ensure the NFS export has sync option to guarantee data integrity. The JENKINS_HOME directory must be writable by both nodes, but only one should write at a time. We use a simple lock file mechanism: the active node creates a lock file on the NFS share, and the passive node checks for its existence before taking over. This prevents split-brain scenarios where both nodes think they are active.

External Database Setup (PostgreSQL/MySQL for Jenkins)

Jenkins by default uses an embedded Derby database stored in JENKINS_HOME. This database is not designed for concurrent access and will corrupt if both controllers write to it. For HA, you must externalize the database to PostgreSQL or MySQL. We use PostgreSQL 14 with a dedicated database user jenkins and a database named jenkins. The connection is configured via the JENKINS_JDBC_URL environment variable or in jenkins.model.JenkinsLocationConfiguration.xml. Important settings: useSSL=true, sslmode=verify-full, and poolSize=10. The database should be highly available itself, using streaming replication or a managed service like RDS. During failover, the passive node connects to the same database with no data loss. We also recommend setting builds.keepDependencies=true and queue.keepItems=true to preserve build queue across restarts.

Shared /var/lib/jenkins Across Controllers

The entire JENKINS_HOME directory (/var/lib/jenkins) is shared via NFS. This includes job configurations, build records, plugin data, and the secrets directory. Sharing secrets is particularly tricky because both controllers need the same encryption keys. We copy the secrets/ directory manually during initial setup and ensure it's never overwritten. The identity.key and secret.key files must be identical on both nodes. Also, org.jenkinsci.main.modules.instance_identity.InstanceIdentity module's key must match. To avoid conflicts, we set the passive node's Jenkins to not start the web server until it's promoted. We use a systemd service that waits for a lock file. The shared volume must be mounted with hard,intr to prevent application hangs on NFS server failure. We also set retrans=2 and timeo=600 for faster recovery.

Load Balancer Configuration (nginx/HAProxy with Session Stickiness)

The load balancer routes traffic to the active Jenkins controller. Session stickiness ensures that a user's session remains on the same controller. We use HAProxy with cookie-based stickiness. The health check endpoint is /login which returns HTTP 200 when Jenkins is ready. For nginx, we use ip_hash or sticky module. Configuration must include active health checks to detect controller failure. We set rise=2 and fall=3 to avoid flapping. Also, we configure the load balancer to send a GET /login every 5 seconds. If the active node fails, the load balancer automatically routes traffic to the passive node (once it becomes active). We also set option httpchk GET /login in HAProxy. For nginx, we use health_check interval=5s fails=3 passes=2.

Health Check Endpoint (/login)

Jenkins provides a built-in health check endpoint at /login which returns HTTP 200 when the service is ready. This is the standard endpoint for load balancer health checks. However, it only indicates that the web server is running, not that Jenkins has fully initialized. For more robust checks, we use a custom script that queries the Jenkins API: curl -f http://localhost:8080/login. We also check that the passive node is not accidentally serving requests. We configure the passive node's Jenkins to listen only on localhost until promoted, using --httpListenAddress=127.0.0.1. The load balancer health check should target the public IP of the active node. We also implement a custom health check script that verifies the database connection and shared volume writability.

Failover Procedures

Failover can be automatic or manual. Automatic failover relies on the load balancer detecting the active node's failure and routing traffic to the passive node. However, the passive node must promote itself to active. We use a systemd service on the passive node that monitors the active node's health. When the active node fails, the passive node remounts the NFS share as read-write, starts Jenkins, and signals the load balancer. Manual failover is used for planned maintenance. Steps: 1) Disable health checks on active node. 2) Wait for passive node to detect failure (or manually promote). 3) Verify passive node is serving traffic. 4) Perform maintenance on old active. 5) Switch back. We also have a runbook that includes checking jenkins.model.Jenkins.instance.isAcceptingTasks() to ensure no builds are running before failover.

Blue-Green Upgrade Pattern

Blue-green deployment for Jenkins involves two identical environments: blue (current active) and green (new version). To upgrade, we spin up a new green controller with the new Jenkins version, sync JENKINS_HOME from the blue node, and then switch traffic. Steps: 1) Take a snapshot of JENKINS_HOME. 2) Create a new green node with same NFS mount. 3) Install new Jenkins version on green. 4) Copy secrets and plugins. 5) Start green node in passive mode (read-only NFS). 6) Perform smoke tests. 7) Switch load balancer to green. 8) Keep blue as fallback. This pattern minimizes downtime and allows easy rollback. We use configuration as code (JCasC) to ensure identical configurations. Plugin compatibility is critical; we test plugins on a staging environment first.

Plugin Compatibility with HA Setup

Not all Jenkins plugins work well with HA. Plugins that store state in local files or use the embedded database can cause issues. Known problematic plugins: Blue Ocean (stores UI state locally), Pipeline: Multibranch with SCM polling (duplicate triggers), and certain credential plugins. We maintain a list of HA-compatible plugins and test each new plugin in a staging HA environment. Plugins that use @Initializer or @PostConstruct may run on both nodes, causing conflicts. We recommend using the Lockable Resources plugin for distributed locking instead of relying on SCM polling. Also, avoid plugins that modify JENKINS_HOME directly without using Jenkins APIs. Always check the plugin documentation for HA support.

Cron Trigger Locking (Prevent Duplicate Runs)

Cron triggers are a common source of duplicate builds in HA. Both nodes may trigger the same job if they both have the same cron configuration. To prevent this, we use the Lockable Resources plugin to create a distributed lock. The cron trigger acquires the lock before starting the build. Alternatively, we externalize scheduling to a systemd timer on the active node only. Another approach is to use a shared database table for job scheduling. We also set hudson.triggers.TimerTrigger to true only on the active node via a configuration file. In our setup, we have a script that disables cron triggers on the passive node by modifying the job configs during failover.

Build Queue Persistence

The build queue is stored in the embedded database by default. When using an external database, the queue is persisted in PostgreSQL. However, during failover, the queue may be lost if the passive node restarts. To persist the queue, we set hudson.model.Queue.queuePersist=true in Jenkins system properties. This forces the queue to be saved to the database on every change. Additionally, we configure the load balancer to drain the queue before failover. We also use the build-pipeline-plugin to track queue items across restarts. In our setup, we have a script that backs up the queue every minute using the Jenkins API.

Artifact Storage on Shared Volumes (S3/NFS)

Artifacts must be stored on a shared volume to be accessible after failover. We use two options: NFS for small artifacts and S3 for large ones. The S3 artifact manager plugin stores artifacts directly to S3. For NFS, we use a separate mount point /var/lib/jenkins/artifacts with the same NFS export. This directory is not part of JENKINS_HOME to avoid locking issues. We set up a lifecycle policy in S3 to expire old artifacts. The artifact manager plugin must be configured on both nodes identically. We also use the Copy Artifact plugin to copy artifacts between jobs, which works across nodes if the storage is shared.

Disaster Recovery Testing

Disaster recovery (DR) testing is essential to validate HA setup. We perform quarterly DR tests where we simulate a complete failure of the active node and data center. Steps: 1) Take a snapshot of JENKINS_HOME and database. 2) Shut down active node. 3) Verify passive node takes over. 4) Run a test build. 5) Simulate NFS failure. 6) Restore from backup. We also test database failover by switching to a replica. We use a separate DR environment that mirrors production. We document all steps in a runbook and review after each test. We also test the recovery of artifacts from S3.