Jenkins on Kubernetes: Helm, Pod Templates & Kaniko Gotchas

It was 3 AM on a Tuesday when my pager went off. A production pipeline had been running for 8 hours — a Java microservice build that normally took 20 minutes. The Jenkins master was showing 50+ queued builds, and every agent pod was stuck in Pending state. I checked the Kubernetes cluster: 30 nodes, plenty of resources. But the pod templates were configured with requests: { cpu: 4, memory: 8Gi } and the cluster had only 4 CPU per node. The scheduler couldn't find a node with enough resources, so builds piled up. That night I learned the hard way that resource limits in pod templates are not just suggestions — they're hard constraints that can bring your CI/CD to a halt.

Jenkins and Kubernetes integration has evolved from a nice-to-have to a must-have for any organization running cloud-native CI/CD. The Kubernetes plugin (version 1.31.5 as of writing) allows Jenkins to dynamically spin up agent pods on-demand, giving you elastic build capacity. But with great power comes great complexity: RBAC misconfigurations, pod retention policies that leak resources, and the eternal debate of DinD vs Kaniko for container builds.

In this article, I'll walk you through the complete production-grade setup: deploying Jenkins on Kubernetes using the official Helm chart (v5.0.8), configuring dynamic pod templates with multiple containers, setting up RBAC so Jenkins can talk to the Kubernetes API, and choosing the right container build strategy. I'll cover PVC-backed caching to speed up Maven/Gradle builds, and share the pod retention and resource limit settings that saved my 3 AM sanity.

This isn't a beginner's guide. You already know what Jenkins and Kubernetes are. You've probably tried the basic integration and hit a wall. I'm here to show you the walls I've hit, the scars I've earned, and the exact YAML and commands that made it work.

Deploying Jenkins on Kubernetes with Helm Chart v5.0.8

The official Jenkins Helm chart (jenkins/jenkins) is the recommended way to deploy Jenkins on Kubernetes. As of writing, the latest stable version is 5.0.8, which uses Jenkins 2.440.3. The chart creates a Deployment, Service, ServiceAccount, and RBAC resources. For production, you must override many defaults.

Start by adding the repo and creating a values file:

``bash helm repo add jenkins https://charts.jenkins.io helm repo update helm show values jenkins/jenkins --version 5.0.8 > values.yaml ``

Edit values.yaml for production:

``yaml controller: image: jenkins/jenkins tag: 2.440.3-lts-jdk17 serviceType: ClusterIP # expose via ingress servicePort: 8080 targetPort: 8080 ingress: enabled: true hostName: jenkins.example.com tls: - secretName: jenkins-tls hosts: - jenkins.example.com resources: requests: cpu: 2 memory: 4Gi limits: cpu: 4 memory: 8Gi javaOpts: '-Xms2048m -Xmx4096m -Djenkins.install.runSetupWizard=false' installPlugins: - kubernetes:1.31.5 - workflow-job:1385.vb_58b_86ea_fff1 - workflow-aggregator:596.v8c21c963d92d - git:5.2.0 - configuration-as-code:1775.v810dc9b_2c43 adminUser: admin adminPassword: <generate-secure-password> persistence: enabled: true size: 50Gi storageClass: gp2 # adjust for your cloud rbac: create: true serviceAccountName: jenkins ``

One critical gotcha: the default serviceType: LoadBalancer exposes Jenkins on a public IP. In production, always use ClusterIP and an ingress controller with TLS termination. Also, set jenkins.install.runSetupWizard=false to avoid manual setup.

Install:

``bash kubectl create namespace jenkins helm install jenkins jenkins/jenkins --version 5.0.8 -f values.yaml -n jenkins ``

After install, get the admin password:

``bash kubectl exec -n jenkins svc/jenkins -c jenkins -- cat /run/secrets/additional/chart-admin-password ``

Kubernetes Plugin Installation and Configuration via Helm

The Kubernetes plugin (version 1.31.5) is included in the installPlugins list. After deployment, you need to configure it to connect to the Kubernetes API. The plugin supports two modes: using the service account token (easiest) or a kubeconfig file.

For Helm deployment, the plugin automatically uses the pod's service account (jenkins) which has cluster-admin role by default. To verify:

``bash kubectl get clusterrolebinding jenkins -o yaml ``

If you want to restrict permissions, create a custom ClusterRole with only necessary verbs:

``yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: jenkins-agent rules: - apiGroups: [''] resources: ['pods', 'pods/log', 'pods/exec'] verbs: ['get', 'list', 'watch', 'create', 'delete'] - apiGroups: [''] resources: ['pods/attach'] verbs: ['create'] ``

Then update values.yaml:

``yaml rbac: create: true serviceAccountName: jenkins serviceAccountAnnotations: {} roleRef: jenkins-agent # custom ClusterRole name ``

In the Jenkins UI, go to Manage Jenkins → Nodes and Clouds → Configure Clouds → Add a new cloud → Kubernetes. Set: - Kubernetes URL: https://kubernetes.default.svc.cluster.local - Kubernetes Namespace: jenkins-agents (create it first) - Credentials: use the global token from service account (auto-detected) - Jenkins URL: http://jenkins:8080 (internal service) - Jenkins tunnel: jenkins:50000 (for JNLP)

Test connection by clicking 'Test Connection'. If it fails, check pod logs: kubectl logs -l app=jenkins -c jenkins.

Dynamic Pod Templates with Multi-Container Agents

Dynamic pod templates are defined in Jenkins as YAML or via the UI. For production, I recommend defining them in a shared library or using the configuration-as-code plugin. A pod template can have multiple containers: the JNLP container (mandatory) and sidecar containers for tools like Docker, Maven, or Kaniko.

Example pod template YAML for a Maven build with Docker and Kaniko:

``yaml apiVersion: v1 kind: Pod metadata: labels: jenkins-agent: maven-docker spec: containers: - name: jnlp image: jenkins/inbound-agent:3206.vb_15dcf73f6a_9-1 args: ['$(JENKINS_SECRET)', '$(JENKINS_NAME)'] resources: requests: cpu: 500m memory: 512Mi limits: cpu: 1 memory: 1Gi - name: maven image: maven:3.9.6-eclipse-temurin-21 command: ['cat'] tty: true resources: requests: cpu: 1 memory: 2Gi limits: cpu: 2 memory: 4Gi volumeMounts: - name: maven-cache mountPath: /root/.m2 - name: kaniko image: gcr.io/kaniko-project/executor:v1.21.0 command: ['cat'] tty: true resources: requests: cpu: 500m memory: 1Gi limits: cpu: 1 memory: 2Gi volumeMounts: - name: docker-config mountPath: /kaniko/.docker volumes: - name: maven-cache persistentVolumeClaim: claimName: maven-cache - name: docker-config secret: secretName: regcred items: - key: .dockerconfigjson path: config.json ``

To use this template in a pipeline:

``groovy podTemplate(yaml: libraryResource('podtemplates/maven-docker.yaml')) { node(POD_LABEL) { stage('Build') { container('maven') { sh 'mvn clean package' } } stage('Docker Build') { container('kaniko') { sh ''' /kaniko/executor \ --context=dir:///workspace \ --destination=registry.example.com/myapp:${BUILD_NUMBER} \ --cache=true \ --cache-repo=registry.example.com/cache ''' } } } } ``

Note: the JNLP container must be first in the list. The command: ['cat'] with tty: true keeps sidecar containers alive so they don't exit prematurely.

RBAC Configuration for Jenkins in Kubernetes

Jenkins needs permissions to create, list, watch, and delete pods in the agent namespace. The default Helm chart creates a ClusterRoleBinding with cluster-admin, which is excessive. For production, follow least privilege.

Create a dedicated ServiceAccount and Role:

``yaml apiVersion: v1 kind: ServiceAccount metadata: name: jenkins-agent namespace: jenkins-agents --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: jenkins-agent namespace: jenkins-agents rules: - apiGroups: [''] resources: ['pods', 'pods/log', 'pods/exec', 'pods/attach'] verbs: ['get', 'list', 'watch', 'create', 'delete'] - apiGroups: [''] resources: ['events'] verbs: ['get', 'list', 'watch'] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: jenkins-agent namespace: jenkins-agents roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: jenkins-agent subjects: - kind: ServiceAccount name: jenkins-agent namespace: jenkins ``

Then in Helm values.yaml:

``yaml rbac: create: false # we create our own serviceAccount: create: true name: jenkins annotations: {} ``

But wait — the Jenkins master itself needs to read secrets (like registry credentials) in its own namespace. So you also need:

``yaml apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: jenkins-master namespace: jenkins rules: - apiGroups: [''] resources: ['secrets'] verbs: ['get', 'list'] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: jenkins-master namespace: jenkins roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: jenkins-master subjects: - kind: ServiceAccount name: jenkins namespace: jenkins ``

Apply both sets of RBAC resources before deploying Jenkins.

One more gotcha: the Kubernetes plugin uses the service account from the namespace where Jenkins runs. If you change the namespace, the plugin won't have permissions. Always test with kubectl auth can-i.

Docker-in-Docker vs Kaniko: Building Container Images

When your pipeline builds Docker images, you have two main options: Docker-in-Docker (DinD) or Kaniko. Both have trade-offs.

Docker-in-Docker (DinD): Run a Docker daemon as a sidecar container. The daemon needs privileged mode, which is a security risk. Also, the daemon consumes resources and may leave behind images that fill up the pod's disk.

Example DinD sidecar:

``yaml - name: dind image: docker:24.0.7-dind securityContext: privileged: true volumeMounts: - name: docker-socket mountPath: /var/run - name: docker-storage mountPath: /var/lib/docker ``

Then in pipeline:

``groovy container('docker') { sh 'docker build -t myapp .' } ``

Kaniko: Builds images without privileged mode. It runs in userspace and pushes directly to the registry. Kaniko is faster for cold builds because it caches base layers.

Kaniko sidecar (as shown earlier) and pipeline step:

``groovy container('kaniko') { sh ''' /kaniko/executor \ --context=dir:///workspace \ --destination=registry.example.com/myapp:${BUILD_NUMBER} \ --cache=true \ --cache-repo=registry.example.com/cache ''' } ``

Comparison: Kaniko is more secure (no privileged mode), integrates with GCR/ECR/ACR natively, and supports caching. DinD is more familiar but requires privileged mode and careful cleanup. In production, I always choose Kaniko unless there's a specific reason to use DinD (e.g., building Dockerfiles with special features like multi-stage that Kaniko doesn't support — but Kaniko supports it now).

One gotcha with Kaniko: it needs a .docker/config.json file for registry authentication. Mount it as a secret:

``bash kubectl create secret docker-registry regcred --docker-server=registry.example.com --docker-username=myuser --docker-password=mypass -n jenkins-agents ``

Then mount in pod template as shown earlier.

PVC for Caching Across Builds (Maven/Gradle)

Build tools like Maven and Gradle download dependencies on every build. Without caching, each build downloads the internet. A PersistentVolumeClaim (PVC) mounted in the agent pod can cache these dependencies across builds.

Create a PVC that supports ReadWriteMany (RWX) so multiple pods can mount it simultaneously. For example, using NFS or EFS:

``yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: maven-cache namespace: jenkins-agents spec: accessModes: - ReadWriteMany resources: requests: storage: 20Gi storageClassName: nfs-client # adjust for your provisioner ``

In the pod template, mount the PVC at the Maven local repository path:

``yaml volumes: - name: maven-cache persistentVolumeClaim: claimName: maven-cache containers: - name: maven volumeMounts: - name: maven-cache mountPath: /root/.m2 ``

For Gradle, mount at /home/gradle/.gradle or /root/.gradle depending on the image.

Gotcha: Concurrent writes to the same cache can cause corruption. To avoid this, use a tool like mvn dependency:resolve with -o (offline) after the first build, or use a shared cache with proper locking. Alternatively, use a read-only cache and copy it to a writable volume per pod. But for most teams, the risk is low.

Another approach: use a sidecar that runs a caching proxy (like Nexus or Artifactory) and configure Maven/Gradle to use it. This avoids PVC altogether and is more reliable.

Production insight: I once used a PVC with ReadWriteOnce (RWO) and builds started failing with 'volume is already used by another pod'. The scheduler couldn't place two builds on the same node. Always use RWX for build caches.

Helm Chart Deployment Patterns for Jenkins

Beyond the basic install, there are several deployment patterns for production:

1. Blue/Green deployment: Run two Helm releases (jenkins-blue, jenkins-green) with different tags. Use a load balancer to switch traffic. This allows zero-downtime upgrades.

2. Canary deployment: Use Helm's --set to modify a small subset of replicas with a new image. But Jenkins is stateful (build queue, config), so this is tricky. Better to use a canary of the agent configuration.

3. GitOps with ArgoCD: Manage Jenkins Helm chart as an ArgoCD Application. The values.yaml is stored in Git. ArgoCD syncs automatically. This ensures reproducibility.

Example ArgoCD Application:

``yaml apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: jenkins spec: project: default source: repoURL: https://charts.jenkins.io chart: jenkins targetRevision: 5.0.8 helm: values: | controller: imageTag: 2.440.3-lts-jdk17 serviceType: ClusterIP destination: server: https://kubernetes.default.svc namespace: jenkins syncPolicy: automated: prune: true selfHeal: true ``

4. Helmfile: For multi-environment deployments (dev, staging, prod), use helmfile to manage differences.

``yaml releases: - name: jenkins chart: jenkins/jenkins version: 5.0.8 values: - values/{{ .Environment.Name }}.yaml ``

Gotcha: Helm upgrades can break if you change the nameOverride or fullnameOverride between releases. Always keep them consistent.

Production insight: I once did a helm upgrade with a new values.yaml that accidentally changed the serviceType from ClusterIP to LoadBalancer. The upgrade succeeded but exposed Jenkins to the internet. Now I use ArgoCD with automated drift detection.

Kubernetes Deployment via Jenkins Pipeline

Once Jenkins is set up, you can deploy applications to Kubernetes from within pipelines. Use the kubectl CLI or the Kubernetes plugin's kubeconfig step.

Method 1: Using kubectl with service account

Mount the service account token and use it in the pipeline:

``groovy stage('Deploy to K8s') { container('kubectl') { withKubeConfig([credentialsId: 'k8s-sa-token', serverUrl: 'https://kubernetes.default.svc']) { sh ''' kubectl set image deployment/myapp myapp=registry.example.com/myapp:${BUILD_NUMBER} -n production kubectl rollout status deployment/myapp -n production ''' } } } ``

Method 2: Using the Kubernetes Continuous Deploy plugin

Install the plugin and use it:

``groovy step([$class: 'KubernetesDeploy', apiUrl: 'https://kubernetes.default.svc', credentialsId: 'k8s-sa-token', namespace: 'production', configs: 'deployment.yaml', enableConfigSubstitution: true]) ``

Method 3: Helm deploy

If you use Helm for the target app:

``groovy stage('Helm Deploy') { container('helm') { sh ''' helm upgrade --install myapp ./chart \ --set image.tag=${BUILD_NUMBER} \ --namespace production ''' } } ``

Important: Use a dedicated service account for deployment, not the Jenkins master's token. Create a ServiceAccount with limited permissions in the target namespace:

``yaml apiVersion: v1 kind: ServiceAccount metadata: name: jenkins-deployer namespace: production --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: deployer namespace: production rules: - apiGroups: ['apps', 'extensions'] resources: ['deployments'] verbs: ['get', 'list', 'watch', 'update', 'patch'] - apiGroups: [''] resources: ['services'] verbs: ['get', 'list', 'watch', 'update', 'patch'] ``

Then create a secret for the token and use it in Jenkins credentials.

Gotcha: The kubectl container image must have kubectl installed. Use an image like bitnami/kubectl:1.29.0.

Pod Retention and Resource Limits Configuration

By default, Jenkins keeps agent pods for a while after the build completes (pod retention). This can lead to resource leaks. Configure pod retention in the pod template YAML or in the cloud configuration.

Set it in pod template:

``yaml spec: podRetention: never() ``

Or in the cloud configuration (Manage Jenkins → Configure Clouds → Advanced): - Pod Retention: Never

Resource limits: Set both requests and limits for each container. Without limits, a single build can consume all cluster resources.

Example for JNLP container:

``yaml resources: requests: cpu: 200m memory: 256Mi limits: cpu: 500m memory: 512Mi ``

For sidecar containers, set appropriate limits. For Maven:

``yaml resources: requests: cpu: 1 memory: 2Gi limits: cpu: 2 memory: 4Gi ``

Gotcha: If you set limits too low, builds will be OOMKilled. Check actual usage with kubectl top pod after a successful build.

Another gotcha: Pod retention always() combined with activeDeadlineSeconds can cause pods to be killed mid-build. Set activeDeadlineSeconds to a value higher than your longest build (e.g., 3600 for 1 hour).

Production insight: I set pod retention to always() for debugging and forgot to change it back. After a week, we had 5000 completed pods in the namespace, causing etcd performance issues. Now I use never() and rely on build logs.

Debugging Pod Templates: Logs, Events, and Labels

When a pod template doesn't work, start with these steps:

1. Check pod events: kubectl describe pod <agent-pod> shows events like 'FailedScheduling' or 'BackOff'.
2. Check pod logs: kubectl logs <agent-pod> -c jnlp shows JNLP connection attempts.
3. Verify pod spec: kubectl get pod <agent-pod> -o yaml shows the actual pod spec created from the template.
4. Check Jenkins system log: Manage Jenkins → System Log → Kubernetes Plugin logs.

Useful labels: Add labels to pod templates for easy identification:

``yaml metadata: labels: jenkins-agent: maven-docker team: platform ``

Then you can filter: kubectl get pods -l team=platform.

Production insight: I once spent 3 hours debugging why a pod template wasn't being used. Turns out I had two templates with the same label, and Jenkins randomly picked one. The fix: use unique labels per template.

Advanced: Using Init Containers and Secrets in Pod Templates

Pod templates can include init containers for setup tasks (e.g., cloning a repo, setting up credentials). They run before the main containers start.

Example init container that clones a Git repo:

``yaml spec: initContainers: - name: git-clone image: alpine/git:2.43.0 command: ['git', 'clone', 'https://github.com/myorg/myrepo.git', '/workspace'] volumeMounts: - name: workspace mountPath: /workspace containers: - name: jnlp ... volumes: - name: workspace emptyDir: {} ``

Secrets: Mount Docker config or SSH keys as secrets:

``yaml volumes: - name: ssh-key secret: secretName: jenkins-ssh-key defaultMode: 0600 containers: - name: jnlp volumeMounts: - name: ssh-key mountPath: /home/jenkins/.ssh ``

ConfigMap: Mount configuration files (e.g., Maven settings.xml):

``yaml volumes: - name: maven-settings configMap: name: maven-settings containers: - name: maven volumeMounts: - name: maven-settings mountPath: /root/.m2/settings.xml subPath: settings.xml ``

Production insight: We used an init container to decrypt secrets using Vault before the build started. This kept secrets out of the pod template YAML.

Performance Tuning: JNLP Tunnels and WebSocket Mode

By default, the JNLP agent connects to Jenkins master via TCP on port 50000. This requires a stable network path. In Kubernetes, use the Jenkins tunnel parameter to specify the service: jenkins:50000.

WebSocket mode: Newer Jenkins agents support WebSocket connections, which can bypass load balancers that don't handle TCP. Enable it in the pod template:

``yaml env: - name: JENKINS_WEB_SOCKET value: 'true' ``

Or in the cloud configuration: 'WebSocket' checkbox.

Performance: For large builds, increase the JVM heap of the agent:

``yaml env: - name: JAVA_OPTS value: '-Xmx2048m -Xms1024m' ``

Connection timeout: Set connectionTimeout in the cloud configuration to 300 seconds (default 60) to avoid premature disconnections.

Gotcha: WebSocket mode requires Jenkins 2.217+ and inbound-agent 4.7+. If you use an older agent image, it will fail with 'WebSocket not supported'.

Production insight: We had a load balancer (AWS NLB) that timed out TCP connections after 350 seconds. Long builds would disconnect. Switching to WebSocket mode fixed it because WebSocket uses HTTP upgrade and the NLB kept the connection alive.