Jenkins Controller and Agent: Stop Running Everything on One Machine

Most Jenkins setups I've seen in production start the same way: someone installs Jenkins on a single VM, runs everything there, and it works fine — until it doesn't. The controller gets overloaded, builds queue up, and a single rogue job can take down the entire CI/CD pipeline. I've watched a payments team lose an entire release day because a memory leak in a test suite killed the Jenkins process. Don't be that team.

The controller-agent pattern solves this by separating the brain from the brawn. The controller handles lightweight orchestration — scheduling, authentication, UI — while agents do the heavy lifting: compiling, testing, packaging. This isn't just about scaling; it's about survival. Without it, you can't run builds on different platforms, you can't isolate untrusted jobs, and you can't recover from a single point of failure.

By the end of this article, you'll be able to set up a Jenkins controller with multiple agents, configure agent security, and diagnose the most common production failures — including the one that cost a fintech startup 6 hours of downtime last year.

Why You Need a Separate Controller and Agent

Running everything on the controller is like using your laptop as a production server. It works until you need to deploy at 3 AM and your laptop runs out of battery. The controller is the nervous system — it should be lean, responsive, and always available. Agents are the muscles — they do the heavy lifting and can be swapped out when they fail.

Without agents, every build competes for the same CPU, memory, and disk I/O. A single memory-intensive test suite can starve the controller's web UI, making it impossible to cancel the build or check logs. I've seen this bring down a CI pipeline for a 200-person engineering org because no one could access the Jenkins UI to kill the runaway job.

Agents also let you run builds on different platforms. Want to test on Windows, Linux, and macOS? Spin up an agent for each. Want to isolate untrusted pipeline code? Run it on a disposable agent in a Docker container. The controller never executes arbitrary code — it only orchestrates.

// io.thecodeforge — DevOps tutorial

# Step 1: On the controller, generate agent connection details
# Go to Manage Jenkins > Nodes > New Node
# Name: linux-agent-01
# Type: Permanent Agent
# Remote root directory: /home/jenkins/agent
# Labels: linux docker
# Launch method: Launch agent via SSH
# Host: 192.168.1.101
# Credentials: SSH key (jenkins-agent-key)
# Host Key Verification Strategy: Known hosts file verification

# Step 2: On the agent machine, install Java (required for agent.jar)
sudo apt-get install -y openjdk-11-jre-headless

# Step 3: Create the agent user and workspace directory
sudo useradd -m -s /bin/bash jenkins
sudo mkdir -p /home/jenkins/agent
sudo chown jenkins:jenkins /home/jenkins/agent

# Step 4: The controller will push agent.jar via SSH and launch it automatically
# Verify agent is connected:
# java -jar jenkins-cli.jar -s http://controller:8080/ list-nodes
# Output should show 'linux-agent-01' with status 'Online'

Agent Connection Methods: SSH vs JNLP vs Web Start

You have three ways to connect an agent to the controller. Each has trade-offs. SSH is the most common for permanent agents — the controller pushes the agent.jar and manages the connection. JNLP (Java Web Start) is for agents that can't accept inbound SSH connections, like Windows machines behind a firewall. Web Start is deprecated but still seen in legacy setups.

SSH is preferred because it's encrypted by default, supports key-based auth, and the controller handles reconnection automatically. JNLP requires the agent to initiate the connection, which is useful when the agent is in a different network segment. But JNLP agents need manual restart if the connection drops — they don't auto-reconnect without extra configuration.

I've seen teams use JNLP because 'it's easier' — then they wonder why agents go offline after a network blip. Use SSH unless you have a specific reason not to. If you must use JNLP, wrap the agent launch in a systemd service with Restart=always.

// io.thecodeforge — DevOps tutorial

# systemd service for JNLP agent with auto-reconnect
# Save as /etc/systemd/system/jenkins-agent.service

[Unit]
Description=Jenkins Agent (JNLP)
After=network.target

[Service]
User=jenkins
Group=jenkins
WorkingDirectory=/home/jenkins/agent
ExecStart=/usr/bin/java -jar /home/jenkins/agent/agent.jar -jnlpUrl http://controller:8080/computer/linux-agent-01/jenkins-agent.jnlp -secret @/home/jenkins/agent/secret.txt -workDir /home/jenkins/agent
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

# Enable and start:
# sudo systemctl daemon-reload
# sudo systemctl enable jenkins-agent
# sudo systemctl start jenkins-agent

Configuring Agent Labels for Targeted Builds

Labels are how you tell Jenkins which agent should run a specific job. Without labels, Jenkins picks any available agent — which is fine for simple setups, but dangerous when you need specific tools or environments. For example, a Docker build must run on an agent with Docker installed. A Windows build needs a Windows agent.

Labels are free-form strings. You can assign multiple labels to an agent (e.g., 'linux docker high-mem'). In your pipeline, use the label directive to constrain where the job runs. If no agent matches the label, the job waits indefinitely — which is why you should always have a fallback or timeout.

I've seen a team label their agents 'production' and 'testing', then accidentally run a destructive database migration on the production agent because the pipeline didn't specify a label. Always label explicitly, and never rely on the default 'any' agent for sensitive jobs.

// io.thecodeforge — DevOps tutorial

pipeline {
    agent none  // Don't run on any agent by default
    
    stages {
        stage('Build') {
            agent { label 'linux && docker' }  // Must match both labels
            steps {
                sh 'docker build -t myapp:latest .'
            }
        }
        stage('Test') {
            agent { label 'linux && high-mem' }  // Requires high memory agent
            steps {
                sh 'mvn test -Xmx4g'
            }
        }
        stage('Deploy') {
            agent { label 'production' }  // Only runs on production-tagged agent
            steps {
                sh 'ansible-playbook deploy.yml'
            }
        }
    }
}

Scaling Agents Horizontally with Cloud Plugins

When your build demand spikes — say, during a release day — you don't want to manually spin up agents. Cloud plugins (EC2, Kubernetes, Azure VM) let Jenkins provision agents on demand. The controller detects a queued job, launches a new agent, runs the job, then terminates the agent after a timeout.

This is the gold standard for CI/CD at scale. You pay only for what you use, and you never have idle agents wasting resources. The Kubernetes plugin is especially popular: each build runs in a pod with ephemeral storage, so no workspace cleanup needed.

But cloud agents introduce latency. Spinning up a VM takes 30-60 seconds. For short jobs, that overhead might exceed the build time. Use a hybrid approach: keep a pool of warm agents (e.g., 2-3 always-on) for quick jobs, and use cloud agents for spikes.

// io.thecodeforge — DevOps tutorial

# Jenkins Kubernetes plugin configuration (via JCasC)
# This defines a pod template for Maven builds

apiVersion: v1
kind: Pod
spec:
  containers:
  - name: jnlp
    image: jenkins/inbound-agent:latest
    args: ['$(JENKINS_SECRET)', '$(JENKINS_NAME)']
    resources:
      requests:
        memory: "256Mi"
        cpu: "200m"
      limits:
        memory: "512Mi"
        cpu: "500m"
  - name: maven
    image: maven:3.8-openjdk-11
    command:
    - cat
    tty: true
    resources:
      requests:
        memory: "1Gi"
        cpu: "500m"
      limits:
        memory: "2Gi"
        cpu: "1"
    env:
    - name: MAVEN_OPTS
      value: "-Xmx1536m"
  nodeSelector:
    kubernetes.io/os: linux
  serviceAccountName: jenkins-agent

Securing the Controller-Agent Connection

The controller-agent channel carries sensitive data: source code, credentials, deployment keys. If an attacker compromises an agent, they can exfiltrate secrets or inject malicious builds. You must secure the connection.

SSH agents use the controller's SSH key to authenticate. Protect that key with a passphrase and store it in Jenkins credentials with restricted scope. For JNLP agents, use a secret token that's unique per agent. Never reuse secrets across agents.

Beyond authentication, encrypt the traffic. SSH is encrypted by default. For JNLP, use HTTPS for the controller URL and enable TCP encryption if using the TCP agent port. Also, run agents in isolated environments — don't give them access to production networks unless necessary.

I've seen a company where an agent had access to the production database because it was on the same VLAN. A compromised build script dumped the entire user table. Isolate agents in a separate subnet with strict firewall rules.

// io.thecodeforge — DevOps tutorial

# Security checklist for Jenkins agents

# 1. Use SSH keys with passphrase (not password auth)
ssh-keygen -t ed25519 -f /var/lib/jenkins/.ssh/agent-key -N "your-passphrase"

# 2. Restrict agent user to minimal commands (in ~jenkins/.ssh/authorized_keys)
command="/usr/local/bin/jenkins-agent-wrapper.sh",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-ed25519 AAA...

# 3. Run agent in a container with read-only root filesystem
docker run --read-only --tmpfs /tmp --tmpfs /home/jenkins/agent ...

# 4. Use Jenkins credentials binding for secrets (never in plaintext)
withCredentials([string(credentialsId: 'deploy-key', variable: 'DEPLOY_KEY')]) {
    sh 'echo $DEPLOY_KEY | ssh-add -'
}

# 5. Enable agent-to-controller security: Manage Jenkins > Configure Global Security > Agents > TCP port for inbound agents: Fixed (50000)
# Use firewall to allow only agent IPs to port 50000

Monitoring Agent Health and Performance

Agents die. Networks blip. Disks fill up. You need to know when an agent goes offline before a developer complains. Jenkins provides monitoring plugins (Monitoring, Metrics) that expose agent status via API and UI.

Set up alerts for agent disconnection. Use the Jenkins CLI or API to check agent status periodically. For example, a cron job that runs java -jar jenkins-cli.jar list-nodes and alerts if any agent is offline for more than 5 minutes.

Also monitor agent resource usage. A build that consumes 100% CPU for an hour might indicate an infinite loop. Use the 'Monitoring' plugin to track CPU, memory, and disk on each agent. Set thresholds and trigger notifications.

I've seen a build that wrote gigabytes of logs to the workspace, filling the agent's disk and causing all subsequent builds to fail with 'No space left on device'. Set workspace cleanup policies and disk usage alerts.

// io.thecodeforge — DevOps tutorial

#!/bin/bash
# Cron job to check agent status and alert if offline

CONTROLLER_URL="http://jenkins-controller:8080"
CLI_JAR="/opt/jenkins-cli.jar"
ALERT_EMAIL="devops@company.com"

# Get list of agents and their status
java -jar $CLI_JAR -s $CONTROLLER_URL list-nodes | while read line; do
    agent=$(echo $line | awk '{print $1}')
    status=$(echo $line | awk '{print $2}')
    if [ "$status" != "Online" ]; then
        echo "Agent $agent is $status" | mail -s "Jenkins Agent Offline" $ALERT_EMAIL
    fi
done

# Check disk usage on each agent via SSH (requires key-based auth)
for agent in agent1 agent2; do
    usage=$(ssh jenkins@$agent 'df -h /home/jenkins/agent | tail -1 | awk "{print \$5}"' | sed 's/%//')
    if [ "$usage" -gt 90 ]; then
        echo "Agent $agent disk usage at ${usage}%" | mail -s "Jenkins Agent Disk Full" $ALERT_EMAIL
    fi
done

Troubleshooting Common Agent Failures

Agents fail in predictable ways. Here are the top three I've seen in production:

1. Connection refused: The agent machine is down, or the SSH port is blocked. Check network connectivity and firewall rules. Use telnet agent-ip 22 to test SSH.
2. Authentication failure: The SSH key changed or the agent secret expired. Regenerate the key/secret and update the agent configuration. For SSH, verify the public key is in ~jenkins/.ssh/authorized_keys.
3. Out of disk space: Builds accumulate workspace files. Set up a cron job to clean workspaces older than 7 days. Use the 'Workspace Cleanup Plugin' to delete workspace after each build.
4. Java version mismatch: The agent requires Java 8 or 11, but the controller expects a different version. Check the agent's Java version with java -version. Use the same major version as the controller.

I've debugged an agent that disconnected every 30 minutes. Turned out the agent's JVM was running out of memory because the -Xmx was set too low for the agent process itself. Increased it from 64m to 256m and the disconnects stopped.

// io.thecodeforge — DevOps tutorial

# Diagnostic script for agent issues

# 1. Check if agent process is running
ps aux | grep agent.jar

# 2. Check agent logs (usually in agent workspace)
tail -100 /home/jenkins/agent/remoting.log

# 3. Test connectivity from controller to agent
ssh -i /var/lib/jenkins/.ssh/agent-key jenkins@agent-ip 'echo OK'

# 4. Check Java version on agent
java -version 2>&1 | grep version

# 5. Check disk space on agent
ssh jenkins@agent-ip 'df -h /home/jenkins/agent'

# 6. Restart agent service (if using systemd)
sudo systemctl restart jenkins-agent
sudo systemctl status jenkins-agent

# 7. Force reconnect from controller UI
# Manage Jenkins > Nodes > [Agent] > Mark node as temporarily offline > Mark node online again