Jenkins Controller and Agent: Stop Running Everything on One Machine

I remember my first Jenkins setup: a single server running everything. It worked fine for a few weeks, then builds started queueing. The UI became sluggish, and I’d see 'Pending—Waiting for executor' messages. I'd restart Jenkins, it would be fine for a day, then the cycle repeated. One Friday, a memory leak in a test suite caused the entire Jenkins process to OOM. The controller died, all jobs were lost, and the team couldn't deploy for hours. That weekend, I migrated to a controller-agent architecture. The difference was night and day. The controller became responsive, builds ran in parallel on dedicated agents, and we could horizontally scale by adding more agents. This article teaches you how to do the same, with production-hardened configurations and real debugging techniques.

1. Why You Need a Controller-Agent Architecture

Running builds directly on the Jenkins controller is a common anti-pattern. The controller is a single point of failure: if it crashes, you lose the UI, job configurations, and all running builds. Moreover, the controller's resources (CPU, memory, disk) are shared between serving the UI, scheduling, and executing builds. A memory-intensive build can OOM the controller, taking down the entire CI system. In production, we learned this the hard way when a test suite with a memory leak killed our Jenkins master during peak hours. After migrating to a dedicated controller and multiple agents, we saw: 99.9% uptime, build times reduced by 60% due to parallelism, and the ability to scale horizontally by adding agents. The controller-agent architecture also enables running builds on different operating systems or with specialized hardware (e.g., GPU for ML workloads).

2. Controller vs Agent: Responsibilities and Boundaries

The controller owns: job configuration, credential storage, user authentication, build scheduling, artifact management, and the web UI. The agent owns: workspace management (checkout source, compile, test), executing build steps defined in Jenkinsfile, and reporting results back. The controller never executes sh or bat steps; it sends them to agents. To enforce this, set the 'Number of executors' on the controller to 0. This ensures no build runs on the controller. Additionally, restrict the controller's label to something like 'master' and never assign that label to any job. In Jenkins 2.x, the default controller has 2 executors; change it to 0 in Manage Jenkins → Configure System → # of executors. If you need to run maintenance tasks on the controller, create a dedicated agent for that.

3. Setting Up a Permanent Agent via SSH

Permanent agents are long-running machines (VMs or bare metal) that connect to the controller. The SSH method is common: the controller connects to the agent via SSH and launches the agent process. To set up: 1) On the agent, create a 'jenkins' user with SSH access. 2) Generate an SSH key pair on the controller: ssh-keygen -t rsa -b 4096 -f /var/lib/jenkins/.ssh/id_rsa. 3) Copy the public key to the agent's ~/.ssh/authorized_keys. 4) In Jenkins UI: Manage Jenkins → Manage Nodes → New Node → enter name, select 'Permanent Agent'. 5) Set Remote root directory (e.g., /home/jenkins/workspace). 6) Set Labels (e.g., 'linux docker'). 7) Launch method: 'Launch agents via SSH'. 8) Enter host, credentials (private key), port (22). 9) Set 'Host Key Verification Strategy' to 'Non verifying Verification Strategy' for simplicity, or use 'Known hosts file'. 10) Save. The controller will connect and start the agent. If you see 'Connection refused', check if SSH is running on the agent and firewall allows port 22 from controller.

4. Setting Up a Dynamic Agent on Kubernetes

Dynamic agents are created on demand, run a single build, then terminate. The Kubernetes plugin is the most popular for this. Setup: 1) Install the 'Kubernetes' plugin. 2) Configure cloud in Manage Jenkins → Manage Nodes and Clouds → Configure Clouds → Add a new cloud → Kubernetes. 3) Provide Kubernetes URL (e.g., https://kubernetes.default.svc), namespace (e.g., jenkins-agents), and credentials (service account token). 4) Set 'Jenkins URL' to the controller's address (e.g., http://jenkins-controller:8080). 5) Define a Pod Template: container image (e.g., jenkins/inbound-agent:latest), label (e.g., 'k8s'), and resource limits. 6) In job, use label 'k8s'. When a build triggers, Jenkins creates a pod with the agent container. The pod runs the build, then terminates. Common issues: pod stuck in 'Pending' due to insufficient resources; check kubectl describe pod <pod-name> for events. Also, if the Jenkins URL is incorrect, the agent cannot connect back. Use the internal service DNS name.

5. Agent Connection Protocols: SSH, JNLP, and WebSocket

Jenkins supports three agent connection protocols. SSH: controller initiates connection to agent via SSH, then runs the agent jar. Requires SSH server on agent and inbound port 22 from controller. JNLP (Java Web Start): agent initiates connection to controller. The agent downloads a .jnlp file from controller and connects. Requires inbound port 8080 (or whatever Jenkins HTTP port) from agent to controller. WebSocket: similar to JNLP but uses WebSocket protocol (ws:// or wss://). It's the modern replacement for JNLP. Advantages: works through firewalls and proxies without needing extra ports, automatically reconnects on network failure, and is more secure. To use WebSocket: in agent configuration, set 'Launch method' to 'Launch agent by connecting it to the controller' and check 'Use WebSocket'. On agent command line, use -webSocket. For SSH agents, you cannot use WebSocket; SSH agents use SSH protocol. Recommendation: Use WebSocket for all new agents where possible.

6. Securing Agent-Controller Communication

Agent-controller communication must be encrypted and authenticated. Use HTTPS for the controller's web interface, and use the agent secret token. For SSH agents, use SSH keys with passphrase (or use SSH agent forwarding). For WebSocket agents, the connection is over HTTP, so you must enable HTTPS on the controller to encrypt traffic. In the agent launch command, use -jnlpUrl https://controller:8443/... and -secret <secret>. The secret is a UUID that authenticates the agent. Never hardcode secrets in scripts; use Jenkins credentials or environment variables. Additionally, restrict agent permissions: in Manage Jenkins → Configure Global Security → Agent → set 'TCP port for inbound agents' to 'Fixed' and specify a port (e.g., 50000) and restrict firewall access to only known agent IPs. For Kubernetes agents, use a service account with minimal permissions (only create pods, get logs, etc.).

7. Monitoring Agent Health and Performance

Monitor agent availability, resource usage, and build queue depth. Use Jenkins monitoring plugins: 'Monitoring' plugin (shows graphs of executor usage, queue length), 'Resource Disposer' (cleans up unused agents), and 'Build Monitor View'. Set up alerts: if queue depth exceeds threshold for 5 minutes, send Slack notification. For agents, monitor CPU, memory, disk, and JVM heap. Use Prometheus and Grafana: the 'Prometheus' plugin exposes metrics like jenkins_node_online, jenkins_executor_count, jenkins_queue_length. Example alert: jenkins_node_online == 0 means agent offline. Also monitor agent logs for errors. In production, we had an agent with a faulty hard drive that caused frequent disconnections; disk I/O alerts caught it.

8. Scaling Agents: Static vs Dynamic Provisioning

Static provisioning: you pre-create a fixed number of agents (e.g., 10 VMs). This is simple but wasteful during low load and insufficient during peak. Dynamic provisioning: agents are created on demand and destroyed after use. Kubernetes plugin, EC2 plugin, and Azure VM agents are examples. Dynamic provisioning allows scaling to zero (no agents when idle) and scaling up to hundreds during high load. The trade-off: startup time (a new VM or container takes seconds to minutes). For dynamic agents, use a minimal base image with only Java and tools. Optimize startup by pre-pulling images. In production, we use a hybrid: a small pool of static agents for quick jobs, and Kubernetes agents for everything else. The Kubernetes plugin has a 'maxInstances' setting to limit concurrent agents. Also, set 'idle minutes' to terminate idle agents quickly.

9. Agent Labels and Job Placement Strategies

Labels are tags assigned to agents (e.g., 'linux', 'windows', 'gpu', 'docker'). Jobs specify a label expression in 'Restrict where this project can be run'. The scheduler matches jobs to agents whose labels satisfy the expression. Use labels to route jobs to appropriate environments. For example, a Windows build job uses label 'windows'. You can combine labels with boolean operators: 'linux && docker' means agent must have both labels. In production, we use labels 'x86_64' and 'arm64' for cross-compilation. We also use labels for capacity: 'high-mem' for memory-intensive jobs. Avoid using agent names directly because they change. Instead, use labels. To debug label mismatches, check 'Label' field on agent page and job configuration. If a job is stuck in queue, check 'Build Executor Status' to see which agents match.

10. Handling Agent Failures: Retry and Resilience

Agents can fail due to network, hardware, or software issues. Builds running on a failed agent are lost. Strategies: 1) Use the 'Rebuild' plugin to allow users to restart failed builds. 2) Use 'Pipeline: Stage' and 'Restart from Stage' feature to skip completed stages. 3) Implement retry logic in Jenkinsfile: retry(3) { ... } around critical steps. 4) Use the 'Node and Label parameter' plugin to let users choose a different agent. 5) For dynamic agents, the Kubernetes plugin automatically creates a new pod if the previous one fails during provisioning. 6) Set agent 'Retry interval' and 'Retry count' in agent configuration. In production, we had a network switch failure that took down 10 agents. Because we had retry configured, agents reconnected after the switch rebooted. We also use 'Build Blocker' plugin to prevent multiple builds on the same agent if it's unstable.

11. Best Practices for Workspace Management on Agents

Each agent has a 'Remote root directory' where workspaces are created. Workspaces can accumulate large files, leading to disk full. Use 'Workspace Cleanup Plugin' to delete old workspaces after build. Use 'Disk Usage Plugin' to monitor disk usage per agent. Set a cron job to clean workspaces older than 7 days: find /home/jenkins/workspace -maxdepth 1 -type d -atime +7 -exec rm -rf {} +. Use 'Custom Workspace' in job configuration to isolate workspaces per job. For shared libraries, use 'Pipeline: Shared Libraries' with a dedicated checkout. In production, we had an agent disk fill up because a job downloaded 10GB of dependencies. We added a step to clean up after build: post { always { cleanWs() } }.

12. Migrating from Single-Master to Controller-Agent: A Step-by-Step Plan

Step 1: Set up a new controller (or use existing) with executors set to 0. Step 2: Install plugins: SSH Agents, Kubernetes, etc. Step 3: Create a permanent agent on the same machine as the old master (or a separate VM) with label 'migration'. Step 4: Move all jobs to use label 'migration' by updating their 'Restrict where this project can be run'. Step 5: Test builds on the agent. Step 6: Gradually add more agents and adjust labels. Step 7: Once all builds run on agents, disable executors on the controller. Step 8: Monitor for any jobs that still run on controller (use 'Build Executor Status' view). Step 9: Remove the old master's build capability. In production, we did this incrementally over a week. We started with non-critical jobs, then moved critical ones. We also set up a backup controller for disaster recovery. Common pitfalls: forgetting to update job labels, and credential paths that assume local filesystem.