Jenkins Kubernetes Deployment: Build Agents That Survive Production Chaos

I was on call at 3 AM when the first page hit: 'Jenkins queue depth > 1000'. Our monolithic Jenkins server, running on a bare-metal box, was thrashing. Builds were timing out, agents were stuck in 'pending' state, and the ops team was manually restarting the service every two hours. The root cause? A single Java memory leak in a plugin that caused the controller to GC for 10 minutes every hour. We had no isolation—one rogue job could kill the entire CI/CD pipeline. That night, I decided to migrate our entire Jenkins infrastructure to Kubernetes. It wasn't just about containerization; it was about building a system where agent failures are expected, not exceptional. Over the next six months, we transitioned to a Kubernetes-native Jenkins setup, and the chaos became manageable. This article documents the exact patterns, commands, and incidents that shaped our production deployment.

1. Architecture Overview: Controller and Agent Separation

The foundation of a production Jenkins on Kubernetes setup is a clear separation between the controller and agents. The controller runs as a Deployment with a single replica (for simplicity) and uses a PersistentVolumeClaim for JENKINS_HOME. Agents are ephemeral pods managed by the Kubernetes plugin. The controller must have a ServiceAccount with permissions to create, list, and delete pods in the agent namespace. We use a dedicated namespace 'jenkins' for the controller and 'jenkins-agents' for agent pods to enforce RBAC boundaries. The controller exposes two ports: 8080 (HTTP) and 50000 (JNLP). The JNLP port is used for agent-controller communication and should not be exposed externally. In production, we use an internal LoadBalancer or ClusterIP service for the controller. The plugin configuration includes a Kubernetes Cloud definition with the cluster's API server URL, credentials (usually a token), and pod templates. Each pod template specifies a container image (e.g., 'jenkins/inbound-agent:latest'), resource requests/limits, and a service account for the agent. We also set 'Jenkins URL' to the internal service DNS name (e.g., 'http://jenkins-service.jenkins.svc.cluster.local:8080') so agents can reach the controller without external DNS.

2. Persistent Storage: JENKINS_HOME and Workspaces

JENKINS_HOME must survive pod restarts. We use a PersistentVolumeClaim (PVC) with ReadWriteOnce access mode, backed by an EBS volume (on AWS) or a GCE PD (on GCP). The PVC is mounted at '/var/jenkins_home' in the controller pod. For workspaces, we have two options: ephemeral (default) or persistent. Ephemeral workspaces are lost when an agent pod is deleted, which is fine for simple builds. However, for large builds that take hours, we use a shared PVC mounted on all agent pods. The PVC is created with ReadWriteMany (e.g., using NFS or EFS). In the Kubernetes plugin pod template, we add a volume mount for the workspace. We also configure Jenkins to use 'Custom Workspace' per job to point to a subdirectory on the shared volume. This avoids workspace collisions. Important: The shared volume must be fast enough to handle concurrent I/O. We learned this the hard way when NFS performance caused build times to double. We switched to a dedicated EFS with provisioned throughput.

3. Pod Templates: Crafting the Perfect Agent

The pod template defines the agent's container spec, including image, resource requirements, environment variables, and volumes. In production, we create multiple templates for different build types: Java (Maven), Node.js, Python, and Docker. Each template has a unique label (e.g., 'java-agent', 'node-agent') that jobs can match. We set resource requests to 500m CPU and 512Mi memory for small builds, and limits to 2 CPU and 4Gi for large builds. We also configure a 'jnlp' container that runs the agent process, and optionally a 'sidecar' container for Docker-in-Docker (DinD). For DinD, we use a privileged container with a hostPath volume for /var/run/docker.sock (careful with security). We also set environment variables like 'JENKINS_URL', 'JENKINS_SECRET', and 'JENKINS_AGENT_NAME'. The pod template can also include init containers to pre-warm the workspace or install tools. We use ConfigMaps to inject custom scripts. One gotcha: the agent pod must have a service account that can access the cluster API if the build needs to run kubectl commands.

4. Resource Management: Avoiding Noisy Neighbors

Without proper resource limits, one build can starve others. We set CPU and memory requests/limits on both the controller and agent pods. For the controller, we allocate 2 CPU and 4Gi memory with limits of 4 CPU and 8Gi. For agents, we use the resource configuration in the pod template. We also set a default node selector to schedule agents on specific node pools (e.g., 'ci-cd' pool with larger instances). To prevent agents from consuming all cluster resources, we configure a maximum number of agents per cloud (e.g., 50) and a timeout for idle agents (e.g., 10 minutes). We also use pod priority classes: high priority for controller and critical agents, low priority for test agents. This ensures that when nodes are under pressure, less important builds are evicted first. We monitor resource utilization with Grafana dashboards and set up alerts when CPU usage exceeds 80% on agent nodes.

5. Networking: JNLP, DNS, and Security Policies

Agent pods communicate with the controller via JNLP on port 50000 (TCP). The controller's service must be reachable from agent pods. We use a ClusterIP service for the controller, and agents use the internal DNS name (e.g., 'jenkins-controller.jenkins.svc.cluster.local:50000'). Network policies are crucial: we create a policy that allows ingress from agent namespace to controller on port 50000, and egress from controller to agent pods on ephemeral ports. We also restrict egress from agent pods to only necessary external services (e.g., Docker Hub, Git repositories). For external access to the Jenkins UI, we use an Ingress with TLS termination. We use cert-manager to auto-renew certificates. One issue we faced: agent pods could not resolve DNS names due to cluster DNS misconfiguration. We added a dnsConfig to the pod template with custom nameservers. Also, ensure that the Jenkins URL in the plugin configuration matches the internal service URL, not the external ingress URL.

6. Security: RBAC, Secrets, and Pod Security

The Jenkins controller's service account needs permissions to manage pods. We create a Role in the agent namespace that allows create, get, list, watch, delete on pods, and bind it to the controller's service account via a RoleBinding. We avoid using cluster-admin. For secrets (e.g., Git credentials, Docker registry passwords), we use Jenkins credentials plugin backed by Kubernetes secrets. The controller mounts a secret volume containing the credentials.xml file (or uses the Credentials Provider plugin). Agent pods should not have access to the controller's secrets. We use PodSecurityPolicies (or OPA/Gatekeeper) to restrict agent pods: no privileged containers, read-only root filesystem, and drop all capabilities. For Docker builds, we use kaniko instead of DinD to avoid privileged mode. We also scan agent images with Trivy in a pipeline and fail builds if critical vulnerabilities are found.

7. Configuration as Code: Reproducible Jenkins

Using Jenkins Configuration as Code (JCasC) plugin, we define the entire Jenkins configuration in a YAML file stored in Git. This includes system settings, security realms, authorization strategies, and cloud configurations (Kubernetes plugin). We mount the JCasC YAML as a ConfigMap in the controller pod. On startup, Jenkins applies the configuration. This ensures that every controller pod is identical and can be recreated from scratch. We also version the ConfigMap and use a checksum annotation to trigger a rolling update when the config changes. For plugin management, we use the plugin-installation-manager-tool in the Dockerfile to pre-install a list of plugins. We pin plugin versions to avoid incompatibilities. One challenge: some plugins require restart after installation, which we handle by setting the 'restart' option in JCasC.

8. Monitoring and Alerting: Observability for CI/CD

We deploy the Prometheus plugin on Jenkins and expose metrics at /prometheus. Key metrics: queue length, executor count, build duration, and job status. We also monitor agent pod lifecycle with kube-state-metrics and cAdvisor. We set up alerts in Prometheus: if queue depth > 100 for 5 minutes, alert; if agent provisioning latency > 2 minutes, alert; if controller JVM heap > 80%, alert. We use Grafana dashboards to visualize build trends and resource usage. For logs, we use Fluentd to send Jenkins logs to Elasticsearch. We also monitor the Kubernetes cluster itself: node CPU/memory, pod restarts, and persistent volume usage. One incident: a silent build failure due to a plugin bug was not caught because we only monitored build status, not build log errors. We now parse build logs for error patterns and alert on them.

9. Graceful Shutdowns: Surviving Node Drains and Rollouts

When a Kubernetes node is drained or a rollout update occurs, agent pods receive a SIGTERM signal. Without handling, builds are killed instantly. We add a preStop lifecycle hook to the agent container that runs a script to disconnect from Jenkins gracefully. The script sends a 'shutdown' command to the agent process (via jenkins-cli.jar or the agent's shutdown API). We also set a terminationGracePeriodSeconds of 60 seconds to allow builds to complete. For the controller, we handle SIGTERM by running a script that stops accepting new builds and waits for running builds to finish. This is configured in the Jenkins init script. We also use a readiness probe on the controller to take it out of service when shutting down. In production, we use a preStop hook that calls 'curl -X POST http://localhost:8080/exit' (with authentication) to shut down Jenkins cleanly.

10. Scaling: Handling Burst Builds and Cost Optimization

During peak hours (e.g., after a code freeze), the number of builds can spike 10x. The Kubernetes plugin automatically provisions new agent pods, but we must ensure the cluster can handle the load. We use cluster autoscaler to add nodes when pods are pending. We configure the autoscaler with a minimum and maximum node count, and use spot instances for cost savings (with a fallback to on-demand). For agent pods, we set a node affinity to prefer spot instances. We also use pod disruption budgets to prevent all agents from being evicted at once. To control costs, we set a maximum number of concurrent agents (e.g., 100) and use a queue-based throttle. We also implement a 'quiet hours' policy where non-critical builds are postponed during off-peak hours. We use a custom script that checks the time and delays builds with a low priority label.

11. Backup and Disaster Recovery: What If Everything Goes Down?

We back up JENKINS_HOME every hour using a CronJob that creates a snapshot of the PVC. We also back up the JCasC ConfigMap and plugin list. The backup is stored in an S3 bucket with versioning. For disaster recovery, we have a script that deploys a new Jenkins controller from scratch: it creates a new PVC from the latest snapshot, applies the ConfigMap, and starts the deployment. The recovery time is under 30 minutes. We test the recovery process quarterly. One issue: the backup script failed silently because the PVC was not mounted read-only during backup. We now use a sidecar container that mounts the PVC read-only and performs the backup. We also back up the Jenkins secrets (credentials.xml) encrypted with KMS. For agent pods, we don't back up anything since they are ephemeral.

12. Advanced Patterns: Multi-Cluster, Hybrid, and GitOps

For large organizations, a single Jenkins instance may not suffice. We run multiple Jenkins controllers in different clusters (e.g., one per region) and use a load balancer to distribute jobs. The Kubernetes plugin can be configured to provision agents from multiple clusters via the 'Kubernetes Cloud' configuration. We also use a hybrid approach: on-premises agents for sensitive builds (e.g., PCI compliance) and cloud agents for standard builds. For GitOps, we use Argo CD to manage the Jenkins deployment itself. The Jenkins configuration (JCasC) and plugin list are stored in a Git repository, and Argo CD syncs the cluster state to the repo. This enables automated rollbacks and audit trails. We also implement a self-service model where teams can create their own pod templates via a custom resource (CRD) that generates the JCasC configuration.