Jenkins Configuration as Code: Stop Clicking Buttons, Start Managing Jenkins as Code

I remember the first time I had to rebuild a Jenkins master from scratch. It was a Friday afternoon, and our production Jenkins had been running for two years. The UI was a labyrinth of configurations—dozens of jobs, hundreds of plugins, custom security realms, and tool installations. When the server crashed, I spent the weekend clicking through UI menus, trying to remember every setting. I failed. Monday morning, the team couldn't deploy. That's when I discovered Jenkins Configuration as Code (JCasC).

At first, I was skeptical. Another plugin? But JCasC wasn't just a plugin—it was a paradigm shift. Instead of manually configuring Jenkins, I could define everything in a YAML file. No more Groovy init scripts that were hard to debug. No more 'I'll fix it later' configuration drift. Just a clean, declarative file that could be version-controlled, reviewed, and applied automatically.

In this article, I'll walk you through JCasC from a production engineer's perspective. We'll cover real incidents, debugging techniques, and advanced patterns. By the end, you'll be ready to stop clicking buttons and start managing Jenkins as code.

1. What is Jenkins Configuration as Code?

Jenkins Configuration as Code (JCasC) is a plugin that lets you define the entire configuration of a Jenkins master in a YAML file. It replaces manual UI configuration and Groovy init scripts with a declarative, version-controlled approach. The plugin was introduced in 2018 and has since become the standard for managing Jenkins at scale.

With JCasC, you can define system settings (like Jenkins URL, executors, and quiet period), security realms (LDAP, Active Directory, or Jenkins own database), authorization strategies (matrix-based, role-based, or logged-in users can do anything), tools (JDK, Maven, Gradle), clouds (Kubernetes, Docker, AWS EC2), and plugin-specific configurations (like GitHub branch source or Pipeline Multibranch defaults).

The YAML file is read at Jenkins startup. You can also trigger a hot reload without restarting the master, though this is not recommended for critical changes like security realms. The plugin supports multiple YAML files, which are merged in order. Environment variables and external secret stores (like Vault) can be used to inject secrets.

JCasC is not a replacement for job configuration. Jobs are typically defined via Jenkinsfile (Pipeline as Code) or job DSL. JCasC focuses on the master's global configuration. Together, they enable a fully declarative Jenkins setup.

To get started, install the 'configuration-as-code' plugin. Then create a jenkins.yaml file in $JENKINS_HOME. Set the CASC_JENKINS_CONFIG environment variable to point to the file if it's elsewhere. On startup, Jenkins applies the configuration. You can verify the applied config via the /configuration-as-code/checkConfig endpoint.

2. Setting Up JCasC in a Production Environment

Setting up JCasC in production requires careful planning. First, install the plugin via the Jenkins plugin manager or by adding it to your plugins.txt file if you use Docker. For a Docker-based Jenkins, you can use the jenkins/jenkins:lts-jdk11 image and install plugins via jenkins-plugin-cli.

Create a jenkins.yaml file. Start with the minimal configuration: jenkins.model.Jenkins settings. For example: ``yaml jenkins: systemMessage: "Production Jenkins - Managed by JCasC" numExecutors: 2 scmCheckoutRetryCount: 3 ` Then add security. For LDAP: `yaml security: realm: ldap: configurations: - server: ldap.company.com rootDN: "dc=company,dc=com" managerDN: "cn=admin,dc=company,dc=com" managerPassword: ${LDAP_PASSWORD} `` Never hardcode passwords. Use environment variables or a secret store.

Next, add authorization. For matrix-based security: ``yaml authorization: loggedInUsersCanDoAnything: allowAnonymousRead: false ` Or for role-based: `yaml authorization: projectMatrix: permissions: - "Overall/Administer:admin" - "Job/Read:authenticated" ``

Set up tools (JDK, Maven) and clouds (Kubernetes). For Kubernetes cloud: ``yaml jenkins: clouds: - kubernetes: name: "kubernetes" serverUrl: "https://kubernetes.default.svc" jenkinsUrl: "http://jenkins:8080" maxRequestsPerHost: 100 templates: - name: "default" label: "jenkins-agent" nodeUsageMode: NORMAL containers: - name: "jnlp" image: "jenkins/inbound-agent:latest" workingDir: "/home/jenkins" ``

Finally, apply the configuration. Use a Docker entrypoint that starts Jenkins with the JCasC file. For hot reload, use curl -X POST http://localhost:8080/configuration-as-code/reload with an admin token. But in production, prefer a restart for critical changes.

3. Understanding the JCasC YAML Structure

The JCasC YAML file follows a hierarchical structure that mirrors Jenkins configuration objects. The root key is usually jenkins for core settings, but there are other top-level keys like security, credentials, tool, unclassified, and plugin. Each key maps to a Jenkins configuration class.

For example, jenkins.model.Jenkins is configured under jenkins. The security key maps to jenkins.security.SecurityRealm and jenkins.security.AuthorizationStrategy. credentials maps to com.cloudbees.plugins.credentials.SystemCredentialsProvider. tool maps to jenkins.tools.ToolConfiguration.

Here's a typical structure: ``yaml jenkins: systemMessage: "Managed by JCasC" numExecutors: 0 securityRealm: ldap: configurations: - server: ldap.example.com authorizationStrategy: globalMatrix: permissions: - "Overall/Administer:admin" - "Job/Read:authenticated" credentials: system: domainCredentials: - credentials: - string: scope: GLOBAL id: "my-secret" secret: ${MY_SECRET} tool: jdk: installations: - name: "jdk11" home: "/usr/lib/jvm/java-11" unclassified: location: url: "http://jenkins.example.com" ``

To discover available keys, use the /configuration-as-code/schema endpoint. It returns a JSON schema of all configurable elements. You can also use the JCasC plugin's 'View Configuration' button in the UI to see the current YAML representation.

One common pitfall: YAML indentation. Use spaces, not tabs. A missing space can cause the entire section to be ignored. Always use a linter like yamllint.

4. Managing Secrets in JCasC

Secrets management is critical in JCasC. Never hardcode passwords, tokens, or API keys in the YAML file. Instead, use environment variables or external secret stores. JCasC supports ${VARIABLE} syntax for environment variables. For example: ``yaml security: realm: ldap: configurations: - managerPassword: ${LDAP_PASSWORD} ` Set the environment variable LDAP_PASSWORD before starting Jenkins. In Docker, use -e LDAP_PASSWORD=secret. In Kubernetes, use envFrom or valueFrom` with secrets.

For more advanced secret management, JCasC integrates with HashiCorp Vault. You can use the vault plugin to fetch secrets. Configure Vault in the YAML: ``yaml unclassified: vaultConfiguration: vaultUrl: "https://vault.example.com" vaultNamespace: "jenkins" vaultCredentialId: "vault-token" ` Then reference secrets as ${vault:path:key}`.

Another option is to use Kubernetes secrets mounted as files. JCasC can read secrets from files using the ${file:/path/to/secret} syntax. However, this is less common.

Remember: if a secret is not resolved, JCasC will fail to apply the configuration. Always test secret resolution via the /configuration-as-code/resolve endpoint. It shows which variables are resolved and which are missing.

5. Combining JCasC with Docker and Kubernetes

JCasC shines in containerized environments. You can build a Docker image with Jenkins, plugins, and the JCasC YAML file baked in. This ensures every container starts with the same configuration. For Kubernetes, you can use a ConfigMap for the YAML file, allowing updates without rebuilding the image.

Here's a production Dockerfile: ``Dockerfile FROM jenkins/jenkins:lts-jdk11 COPY plugins.txt /usr/share/jenkins/ref/plugins.txt RUN jenkins-plugin-cli --plugin-file /usr/share/jenkins/ref/plugins.txt COPY jenkins.yaml /var/jenkins_home/jenkins.yaml ENV CASC_JENKINS_CONFIG=/var/jenkins_home/jenkins.yaml ``

In Kubernetes, use a ConfigMap: ``yaml apiVersion: v1 kind: ConfigMap metadata: name: jenkins-config data: jenkins.yaml: | jenkins: systemMessage: "Kubernetes Jenkins" ` Then mount it in the pod: `yaml volumeMounts: - name: config mountPath: /var/jenkins_home/jenkins.yaml subPath: jenkins.yaml ``

For secrets, use a Secret resource and mount as env vars.

One advantage of Kubernetes: you can use the Kubernetes plugin to dynamically provision agents. Define the cloud in JCasC: ``yaml jenkins: clouds: - kubernetes: name: "kubernetes" serverUrl: "https://kubernetes.default.svc" jenkinsUrl: "http://jenkins:8080" templates: - name: "default" label: "jenkins-agent" containers: - name: "jnlp" image: "jenkins/inbound-agent:latest" ``

When Jenkins starts, it automatically connects to the Kubernetes cluster and can launch pods for builds.

6. Validating and Testing JCasC Configurations

Before applying a JCasC configuration to production, you must validate it. Start with YAML syntax validation using yamllint. Then check the structure against the JCasC schema. You can use the /configuration-as-code/checkConfig endpoint to validate the YAML without applying it. This endpoint returns a JSON with errors and warnings.

For automated testing, write a script that: 1. Starts a Jenkins instance with the new YAML (e.g., using Docker). 2. Waits for Jenkins to be ready. 3. Calls the checkConfig endpoint. 4. Parses the response for errors. 5. If errors exist, fail the pipeline.

Example script: ``bash docker run -d --name test-jenkins -p 8080:8080 -e CASC_JENKINS_CONFIG=/tmp/jenkins.yaml my-jenkins-image sleep 30 curl -s http://localhost:8080/configuration-as-code/checkConfig | jq '.result' | grep -q 'success' if [ $? -ne 0 ]; then echo "Validation failed" exit 1 fi docker stop test-jenkins docker rm test-jenkins ``

You can also use the JCasC plugin's 'Test Configuration' button in the UI (available in newer versions). It simulates the configuration application and reports errors.

For production, always have a staging Jenkins that mirrors production. Apply changes there first, run integration tests, and then promote to production.

7. Hot Reload vs. Restart: When to Use Each

JCasC supports hot reload via the /configuration-as-code/reload endpoint. This applies the YAML configuration without restarting Jenkins. It's convenient for non-critical changes like system messages or tool installations. However, hot reload has limitations: - Some changes require a restart (e.g., security realm changes, plugin updates). - Reloading can cause brief inconsistencies if the configuration is partially applied. - If the reload fails (e.g., due to a syntax error), the previous configuration remains, but some parts may be in an undefined state.

In production, use hot reload only for low-risk changes. For security-related changes, always restart Jenkins. To perform a reload, you need admin privileges or a reload token. Configure a reload token in the YAML: ``yaml jenkins: security: reloadToken: ${RELOAD_TOKEN} ` Then call: `bash curl -X POST http://localhost:8080/configuration-as-code/reload?casc-reload-token=myToken ``

If you prefer a restart, you can use the Jenkins CLI or a simple docker restart command. In Kubernetes, you can delete the pod, and the StatefulSet will recreate it with the latest ConfigMap.

Our rule of thumb: if the change modifies security, credentials, or core system settings, restart. For plugin configs or tools, reload is safe.

8. Debugging Common JCasC Issues

Even with careful validation, issues arise. Here are common problems and how to debug them.

Issue 1: Configuration not applied Check Jenkins logs for casc messages. If the YAML file is not found, ensure CASC_JENKINS_CONFIG points to the correct path. Use docker logs or kubectl logs to see errors. Also check the /configuration-as-code/checkConfig endpoint.

Issue 2: Security realm reset If after a reload the security realm reverts to default, it means the YAML is missing the security section or has a typo. Use the schema endpoint to verify the correct key. For example, securityRealm vs security_realm. The correct key is securityRealm under jenkins.

Issue 3: Plugin configuration not applied JCasC applies plugin configs only if the plugin is installed. If the plugin is missing, the config is silently ignored. Always install plugins before starting Jenkins. In Docker, use jenkins-plugin-cli to pre-install plugins.

Issue 4: Secrets not resolving If you see ${VARIABLE} in the applied configuration, the environment variable is not set. Check the resolve endpoint. In Kubernetes, ensure the secret is mounted and the env var name matches.

Issue 5: Reload fails with 403 You need a valid admin token or reload token. Use curl -u admin:token or pass the reload token as a query parameter.

For all issues, enable JCasC debug logging: ``bash java -jar jenkins-cli.jar -s http://localhost:8080/ groovy = <<'EOF' import java.util.logging.Logger Logger.getLogger('io.jenkins.plugins.casc').setLevel(java.util.logging.Level.FINE) EOF `` This logs detailed information about configuration application.

9. Advanced JCasC Patterns: Multiple YAML Files and Inheritance

For complex environments, you can split the JCasC configuration into multiple YAML files. Set CASC_JENKINS_CONFIG to a directory containing multiple .yaml files. JCasC merges them in alphabetical order. This allows you to separate concerns: security.yaml, tools.yaml, clouds.yaml, etc.

Example: ``bash CASC_JENKINS_CONFIG=/var/jenkins_home/casc_configs ` Inside that directory: - 00-security.yaml - 01-tools.yaml - 02-clouds.yaml - 99-plugins.yaml`

The order matters if there are overlapping keys. Later files override earlier ones. Use this to have a base configuration and environment-specific overrides.

Another pattern is using YAML anchors and aliases to reduce duplication. For example: ```yaml x-ldap: &ldap server: ldap.example.com rootDN: "dc=example,dc=com"

jenkins: securityRealm: ldap: configurations: - *ldap managerPassword: ${LDAP_PASSWORD} ```

You can also use the !include directive (if you use a YAML preprocessor) but that's not native to JCasC.

For secret inheritance, you can define secrets in a separate YAML file that is not checked into Git, but that defeats version control. Better to use environment variables.

10. Integrating JCasC with Pipeline as Code and Job DSL

JCasC manages the master configuration, but jobs are typically defined via Jenkinsfile (Pipeline as Code) or Job DSL. The combination gives you full control. For example, you can define a shared library in JCasC: ``yaml unclassified: globalLibraries: libraries: - name: "shared-lib" defaultVersion: "master" implicit: true retriever: modernSCM: scm: git: remote: "https://github.com/myorg/shared-library.git" ` Then all pipelines can use @Library('shared-lib')`.

You can also configure Multibranch Pipeline defaults: ``yaml jenkins: clouds: - kubernetes: templates: - name: "default" label: "jenkins-agent" containers: - name: "jnlp" image: "jenkins/inbound-agent:latest" envVars: - envVar: key: "JAVA_HOME" value: "/usr/lib/jvm/java-11" ``

For Job DSL, you can define the seed job in JCasC: ``yaml jobs: - script: > job('seed-job') { steps { dsl { external('jobs/*.groovy') } } } ` But note: the jobs key is not part of core JCasC; it requires the job-dsl plugin and the job-dsl` JCasC extension.

For a fully declarative setup, store Jenkinsfiles in Git repositories. JCasC sets up the master, and pipelines define the build logic.

11. Performance and Security Considerations for JCasC

JCasC itself has minimal performance impact since it only runs at startup or reload. However, the configuration it applies can affect performance. For example, defining too many executors or too many agent templates can consume resources.

Security is paramount. Never expose the reload endpoint without authentication. Use a reload token and rotate it regularly. In production, disable the reload endpoint entirely and rely on restarts. You can do this by not setting a reload token.

Also, ensure that the YAML file is not world-readable. In Kubernetes, use restrictive RBAC and mount ConfigMaps with read-only permissions. In Docker, set file permissions to 600.

Another security concern: secrets in environment variables can be leaked via process listing. Use a secret store like Vault for sensitive data. If you must use environment variables, ensure they are not logged.

For performance, consider the size of the YAML file. A very large file can slow down startup. Split into multiple files and use lazy loading where possible.

Finally, always run Jenkins with a non-root user. The official Jenkins image runs as jenkins user. In Kubernetes, set securityContext.runAsUser: 1000.

12. Migrating from UI Configuration to JCasC: A Step-by-Step Guide

Migrating an existing Jenkins master to JCasC can be daunting. Here's a step-by-step approach we used.

Step 1: Install JCasC plugin on the existing Jenkins. It won't change anything yet.

Step 2: Generate the initial YAML. Use the JCasC plugin's 'Export' button in the UI. It creates a YAML file representing the current configuration. Save this as jenkins.yaml.

Step 3: Review and clean up. The exported YAML may include unnecessary defaults or sensitive data. Remove hardcoded secrets and replace with environment variables. Also remove any configurations that are not needed.

Step 4: Validate the YAML. Use yamllint and the checkConfig endpoint. Fix any errors.

Step 5: Test in staging. Deploy the YAML to a staging Jenkins that mirrors production. Verify that all settings are applied correctly. Test user login, job creation, and agent connections.

Step 6: Apply to production. Set CASC_JENKINS_CONFIG to the YAML file and restart Jenkins. Monitor logs for errors.

Step 7: Remove manual configuration. Once JCasC is in place, disable the ability to change settings via UI. You can do this by setting the jenkins.model.Jenkins.disableRememberMe or using a plugin that locks configuration.

Step 8: Version control. Commit the YAML file to Git. Set up a CI pipeline to validate and apply changes.

Common pitfalls during migration: forgetting to export all plugins' configurations, missing credentials, and breaking security. Take it slow and test each change.