# Jenkins Quick Reference
## Version: Tested on Jenkins 2.440+ | Java 17/21

### Installation
- `java -jar jenkins.war` (quick start)
- `apt install jenkins` (Ubuntu/Debian)
- `docker run -p 8080:8080 -p 50000:50000 jenkins/jenkins:lts`
- Home: `/var/lib/jenkins` or `~/.jenkins`
- Initial admin password: `sudo cat /var/lib/jenkins/secrets/initialAdminPassword`

### Architecture: Controller & Agent
- **Controller** (formerly Master): orchestrates jobs, serves UI, stores config
- **Agent** (formerly Slave/Slave): executes builds; connects via SSH, JNLP (Java Web Start), or inbound
- Agent launch methods: `ssh` (Java on agent), `jnlp` (agent.jar -H controller -secret), `inbound` (websocket), `dumb` (direct)
- `Jenkins URL → Manage Jenkins → Nodes → New Node`
- Agent JAR: `https://JENKINS_URL/jnlpJars/agent.jar`
- Secure agent → use `-webSocket` flag; avoids JNLP port 50000
- Check agent log: `journalctl -u jenkins-agent` or agent's console log in UI
- **Gotcha**: Agent must have Java same major version as controller
- Agent can run as Windows service: `javaw -jar agent.jar -jnlpUrl http://JENKINS/computer/AGENT/slave-agent.jnlp`

### Freestyle Jobs
- `New Item → Freestyle project`
- Build steps: Execute shell (`sh`), Windows batch (`cmd`), Invoke Maven/Ant/Gradle
- Post-build: Archive artifacts, publish JUnit, send emails, trigger downstream
- Build triggers: SCM poll (`H/5 * * * *`), upstream/downstream, webhook
- Restrict where project can run: label expressions (`linux && !arm64`)
- **Gotcha**: Freestyle shell steps fail on non-zero exit; wrap with `set +e` or use `#!` header

### Plugins
- Core plugins: Pipeline, Git, JUnit, Maven Integration, Docker Pipeline, Kubernetes, Blue Ocean
- Install: `Manage Jenkins → Plugins → Available` or CLI: `java -jar jenkins-cli.jar install-plugin PLUGIN_ID`
- List installed: `java -jar jenkins-cli.jar list-plugins`
- Plugin file location: `JENKINS_HOME/plugins/*.jpi`
- Safe restart: `java -jar jenkins-cli.jar safe-restart` or `JENKINS_URL/safeRestart`
- **Gotcha**: After major upgrade always check plugin compatibility matrix; incompatible plugins cause 500 errors

### Declarative Pipeline (Jenkinsfile)
```groovy
pipeline {
    agent any
    // agent { label 'linux' }
    // agent { docker { image 'node:20' } }
    // agent { kubernetes { yamlFile 'pod.yaml' } }

    environment {
        APP_VERSION = "${BUILD_NUMBER}"
    }

    parameters {
        string(name: 'BRANCH', defaultValue: 'main')
    }

    triggers {
        cron('H */4 * * *')
        pollSCM('H/5 * * * *')
    }

    options {
        timestamps()
        timeout(time: 1, unit: 'HOURS')
        buildDiscarder(logRotator(numToKeepStr: '10'))
        disableConcurrentBuilds()
    }

    tools {
        maven 'Maven-3.9'
        jdk 'JDK-17'
    }

    stages {
        stage('Checkout') {
            steps {
                checkout scm
            }
        }
        stage('Build') {
            steps {
                sh 'mvn clean package -DskipTests'
            }
            post {
                failure { mail to: 'team@co.com', subject: 'Build failed' }
            }
        }
        stage('Test') {
            steps {
                sh 'mvn test'
            }
            post {
                always { junit 'target/surefire-reports/*.xml' }
            }
        }
    }

    post {
        always { cleanWs() }
        success { slackSend(channel: '#builds', message: "Success: ${env.JOB_NAME} #${env.BUILD_NUMBER}") }
        failure { echo 'FAILED' }
    }
}
```
- Stages run sequentially unless `parallel` is used
- `agent` can be set per-stage for different executors
- `when { branch 'main' }` to conditionally skip stages
- `when { expression { return params.DEPLOY == true } }`
- **Gotcha**: `agent none` at top means every stage must declare its own `agent`

### Scripted Pipeline (Groovy)
```groovy
node('linux') {
    stage('Checkout') { checkout scm }
    stage('Build') {
        try {
            sh 'mvn clean package'
        } catch (Exception e) {
            currentBuild.result = 'FAILURE'
            error("Build failed: ${e.message}")
        }
    }
    stage('Archive') {
        archiveArtifacts artifacts: 'target/*.jar'
    }
}
```
- Full Groovy: `def` variables, `for` loops, `try/catch/finally`, closures
- `parallel` in scripted: `parallel first: { ... }, second: { ... }`
- `currentBuild.result` tracks status; set to `'FAILURE'` or `'UNSTABLE'`
- No `post {}` block → use `try/catch/finally` or `catchError { ... }`
- **Gotcha**: Serialization errors with `def closure` across nodes; use `@NonCPS` when traversing in-memory data

### Stages & Parallel Execution
```groovy
pipeline {
    agent any
    stages {
        stage('Unit Tests') { steps { sh 'mvn test' } }
        stage('Integration Tests') {
            parallel {
                stage('API') { agent { label 'linux' }; steps { sh 'make api-test' } }
                stage('UI')  { agent { label 'linux' }; steps { sh 'make ui-test' } }
            }
        }
        stage('Deploy') {
            steps { sh 'deploy.sh' }
            input { message 'Deploy to prod?' ok 'Deploy' }
        }
    }
}
```
- `parallel` stages run on **same** agent unless each declares own `agent`
- Use `failFast true` in `parallel` block to abort all on first failure
- `input` gate pauses pipeline; timeout with `input(time: 120, message: 'Approve?')`
- `matrix` directive: `matrix { axes { axis { name 'PLATFORM'; values 'linux', 'windows' } } }`

### Shared Libraries
- Repository structure: `vars/`, `src/`, `resources/`
- `vars/deploy.groovy`:
  ```groovy
  def call(String env) {
      sh "deploy.sh ${env}"
  }
  ```
- `src/com/company/PipelineUtils.groovy` (class):
  ```groovy
  package com.company
  class PipelineUtils {
      static String version() { return '1.0' }
  }
  ```
- Load in Jenkinsfile: `@Library('my-shared-lib@v1.0') _` and `@Library('my-shared-lib@v1.0') import com.company.PipelineUtils`
- Configure: `Manage Jenkins → System → Global Pipeline Libraries`
- Library can load other libs: `@Library('other-lib')`
- Tests: use `JenkinsPipelineUnit` library (GitHub: jenkinsci/JenkinsPipelineUnit)
- **Gotcha**: Library name with spaces → URL-encode in annotation; version is Git branch/tag/ref

### Multibranch Pipeline
- `New Item → Multibranch Pipeline`
- Sources: Git, GitHub, Bitbucket, GitLab
- Discovers branches with a `Jenkinsfile` at root
- Automatically creates jobs for each branch + PRs
- Property strategies: `BuildStrategy { buildBranches(branches: ['main', 'develop']) }`
- Orphaned branch cleanup: `OrphanedItemStrategy { discardOldItems(branchMultiplier: 2) }`
- Branch API triggers via webhook (GitHub/GitLab/Bitbucket plugin)
- **Gotcha**: Branch indexing runs on cron/SCM trigger; heavy repos → throttle with `suppressAutomaticTriggering()`

### GitHub Webhooks
- GitHub plugin + `Manage Jenkins → Configure System → GitHub → Add GitHub Server`
- Webhook URL: `http(s)://JENKINS_URL/github-webhook/`
- Payload URL in GitHub repo: `Settings → Webhooks → Add webhook`
- Content type: `application/json`
- Events: `Just the push event` or `Send me everything`
- For PR builds also check `Pull requests` in GitHub webhook
- Multibranch with GitHub Branch Source: webhook triggers branch indexing automatically
- **Gotcha**: Jenkins behind NAT/ngrok needs public URL; webhook secret optional but recommended; use `GitHub Webhook Plugin` not generic `Git Plugin` for push-based triggers

### Maven Build
- `tools { maven 'Maven-3.9' }` — auto-downloads configured version
- `sh 'mvn clean install -DskipTests'`
- `sh 'mvn verify -Pintegration-test'`
- Post-build JUnit: `junit 'target/surefire-reports/*.xml'`
- JaCoCo coverage: `publishCoverage adapters: [jacocoAdapter('target/site/jacoco/jacoco.xml')]`
- Settings: `configFileProvider([configFile(fileId: 'maven-settings', variable: 'MAVEN_SETTINGS')]) { sh 'mvn -s $MAVEN_SETTINGS ...' }`
- **Gotcha**: Maven version must match auto-installer in Jenkins Global Tool Configuration; offline mode: `mvn -o`

### Docker Integration
- Agent from Docker: `agent { docker { image 'maven:3.9-eclipse-temurin-17' } }`
- Docker Pipeline Groovy block:
  ```groovy
  docker.image('node:20').inside('-v /data:/data') {
      sh 'npm test'
  }
  docker.build('myapp:latest', '.')
  docker.withRegistry('https://registry.hub.docker.com', 'docker-hub-creds') {
      docker.image('myapp:latest').push("${env.BUILD_NUMBER}")
      docker.image('myapp:latest').push('latest')
  }
  ```
- Custom Dockerfile: `agent { dockerfile { filename 'Dockerfile.ci' } }`
- Docker in Docker: mount `/var/run/docker.sock` or use `docker:dind` sidecar
- **Gotcha**: `inside()` mounts workspace from host; on macOS Docker Desktop this is slow; shared memory issues → `shmSize '2g'`

### Kubernetes Deployment
- Deploy via `kubectl`:
  ```groovy
  sh """
    kubectl set image deployment/myapp myapp=myregistry.io/myapp:\${BUILD_NUMBER}
    kubectl rollout status deployment/myapp --timeout=5m
  """
  ```
- With Helm:
  ```groovy
  sh "helm upgrade --install myapp ./charts/myapp --set image.tag=${BUILD_NUMBER} --namespace prod"
  ```
- Kubernetes plugin for dynamic agents:
  ```groovy
  agent {
      kubernetes {
          yaml """
  apiVersion: v1
  kind: Pod
  spec:
    containers:
    - name: maven
      image: maven:3.9-eclipse-temurin-17
      command: ['cat']
      tty: true
    - name: docker
      image: docker:20
      command: ['cat']
      tty: true
      volumeMounts:
      - name: docker-sock
        mountPath: /var/run/docker.sock
    volumes:
    - name: docker-sock
      hostPath:
        path: /var/run/docker.sock
  """
      }
  }
  ```
- **Gotcha**: Pod must have `command: ['cat']` and `tty: true` to stay alive; Jenkins namespace must have appropriate RBAC

### SonarQube & Quality Gates
- Pre-req: SonarQube Scanner plugin, SonarQube server configured in Jenkins
- Pipeline:
  ```groovy
  withSonarQubeEnv('SonarQube') {
      sh 'mvn sonar:sonar'
  }
  timeout(time: 5, unit: 'MINUTES') {
      waitForQualityGate()
  }
  ```
- `sonar-project.properties` in repo root or inline via `-Dsonar.*`
- Quality Gate webhook: `SONAR_URL/webhooks` → `JENKINS_URL/sonarqube-webhook/`
- **Gotcha**: `waitForQualityGate()` hangs without webhook; install SonarQube plugin and configure webhook in SonarQube admin

### Credentials & Secrets
- Types: Username+password, SSH key, secret text, file, certificate, Docker creds
- Pipeline:
  ```groovy
  withCredentials([string(credentialsId: 'API_KEY', variable: 'TOKEN')]) {
      sh 'curl -H "Authorization: Bearer $TOKEN" ...'
  }
  withCredentials([usernamePassword(credentialsId: 'GIT_CREDS', usernameVariable: 'U', passwordVariable: 'P')]) { ... }
  sshagent(['SSH_KEY_ID']) { sh 'git push' }
  ```
- Secret files: `withCredentials([file(credentialsId: 'SVC_ACCT', variable: 'KEY_FILE')]) { ... }`
- Global credentials: `Manage Jenkins → Credentials → System → Global → Add Credentials`
- Encrypted on disk at `JENKINS_HOME/credentials.xml` (master key at `JENKINS_HOME/secrets/master.key` + `hudson.util.Secret`)
- **Gotcha**: Never `echo` credential variables; mask in logs automatically but NOT if you print to console via `echo`

### Security & RBAC
- Authentication: Jenkins own user DB, LDAP, SAML, OIDC (GitHub/Azure AD via plugins)
- Authorization: `Logged-in users can do anything`, `Matrix-based security`, `Role-based` (Role Strategy Plugin)
- Role Strategy: `Manage Jenkins → Manage Roles → Global Roles / Project Roles`
  - Assign `Job/Read + Agent/Connect` per project group via regex: `my-team-.*`
- Agent → Controller Access Control (since 2.445):
  - `Manage Jenkins → Security → Agent → Controller Security`
  - Restrict what agents can do: e.g., disallow `Jenkins.getInstance().doSomething()`
- **Gotcha**: Role Strategy Plugin must be installed separately; after install restart required; misconfiguration can lock out admin

### Distributed Agents (Advanced)
- Cloud providers: AWS (EC2 Plugin), Azure (VM Agents), GCP, Kubernetes (Kubernetes Plugin), Docker (Docker Plugin)
- Kubernetes plugin spins pods per build; label selector pins agent types
- Agent template in `Manage Jenkins → Nodes → Configure Clouds`
- Ephemeral agents: Docker/kubernetes agents created per-job, destroyed after
- Agent isolation: run each build in clean container/VM
- **Gotcha**: Docker cloud agents must pull images on each spin; pre-pull or use `neverPull()`; label matching must be exact — no regex

### Configuration as Code (JCasC)
- `jenkins.yaml` in `JENKINS_HOME/jenkins.yaml` or `CASC_JENKINS_CONFIG` env var
- Example:
  ```yaml
  jenkins:
    systemMessage: "Managed by JCasC"
    numExecutors: 0
    scm:
      git:
        globalConfigName: "jenkins"
        globalConfigEmail: "jenkins@co.com"
  credentials:
    system:
      domainCredentials:
        - credentials:
            - string:
                scope: GLOBAL
                id: "my-secret"
                secret: "${MY_SECRET}"
  unclassified:
    location:
      url: "https://jenkins.example.com"
  jobs:
    - script: >
        pipelineJob('example-pipeline') {
          definition { cpsScm {
            scm { git('https://github.com/org/repo.git') }
            scriptPath('Jenkinsfile')
          }}
        }
  ```
- Install `Configuration as Code` plugin (also `Job DSL` for job definitions)
- Reload: `Manage Jenkins → Configuration as Code → Apply new config` or POST `JENKINS_URL/configuration-as-code/reload`
- Validate: `curl -X POST -u admin:token JENKINS_URL/configuration-as-code/checkSchema`
- **Gotcha**: JCasC runs on startup only unless triggered; secrets reference env vars at apply time; Job DSL uses Groovy DSL syntax

### Monitoring (Prometheus)
- Plugin: `Prometheus metrics` → exposes `JENKINS_URL/prometheus/`
- Key metrics:
  - `default_jenkins_node_count{node_type="controller"} 1`
  - `default_jenkins_job_count{type="pipeline"} 42`
  - `default_jenkins_queue_size` (queue depth)
  - `default_jenkins_node_build_count{node_name="controller"}`
  - `default_jenkins_executor_count{labels="",node_name="controller"}`
  - `default_jenkins_executor_in_use{labels="",node_name="controller"}`
- Grafana dashboard: use `grafana.net` ID `11103` (Jenkins Performance Monitoring)
- Health check: `JENKINS_URL/plugin/metrics/api/` → `/healthcheck`
- Log check: `journalctl -u jenkins -n 50 --no-pager`
- **Gotcha**: Prometheus plugin adds overhead in large controllers; configure `Metrics` plugin access restriction

### Backup & Recovery
- **ThinBackup** plugin: schedule full + differential backups of `JENKINS_HOME`
- Manual backup: `tar czf jenkins-backup-$(date +%Y%m%d).tgz /var/lib/jenkins`
- Exclude: `workspace/`, `builds/`, `caches/`, `logs/`, `war/`, `tools/`
- Minimum backup set:
  - `config.xml`, `*.xml`, `jobs/`, `users/`, `secrets/`, `plugins/`, `nodes/`, `credentials.xml`, `identity.key`
- Restore: install same Jenkins version, restore backup, restart
- Job-specific backup: `jobs/JOB_NAME/config.xml`
- Pipeline script backup: jobs with `Jenkinsfile` defined inside job config (not SCM) store script in `jobs/JOB_NAME/config.xml`
- **Gotcha**: `identity.key` is critical for agent auth; lose it = agents cannot re-connect; back it up separately

### Groovy Scripting
- Script Console: `JENKINS_URL/script` (admin only)
- Examples:
  ```groovy
  // List all jobs
  Jenkins.instance.getAllItems(Job.class).each { println it.fullName }

  // Disable all jobs matching pattern
  Jenkins.instance.getAllItems(Job.class).findAll { it.name =~ /legacy-*/ }.each { it.setDisabled(true) }

  // Clean old builds
  Jenkins.instance.getAllItems(Job.class).each { job ->
    job.getBuilds().findAll { it.number < 100 }.each { it.delete() }
  }

  // View queue
  Jenkins.instance.queue.items.each { println "${it.name} - ${it.why}" }

  // Reload config from disk
  Jenkins.instance.reload()

  // Reset executors
  Jenkins.instance.nodes.each { node ->
    node.getComputer().setAcceptingTasks(true)
    node.getComputer().getExecutors().each { it.interrupt() }
  }
  ```
- Groovy Hook Scripts: place `.groovy` files in `JENKINS_HOME/init.groovy.d/` — runs at startup
- **Gotcha**: Script Console has no undo; test in dev first; Groovy scripts in Pipeline run in CPS (continuation-passing) mode → `@NonCPS` annotation required for non-serializable operations

### Email Notification
- Configure SMTP: `Manage Jenkins → Configure System → Extended E-mail Notification`
- SMTP defaults: `localhost:25`, STARTTLS: `smtp.gmail.com:587`
- Pipeline `emailext` (Email Extension Plugin):
  ```groovy
  emailext(
    to: 'team@co.com',
    subject: "Build ${env.JOB_NAME} #${env.BUILD_NUMBER} - ${currentBuild.currentResult}",
    body: "Check console at ${env.BUILD_URL}console",
    attachLog: true,
    compressLog: true
  )
  ```
- Attach artifacts: `attachmentsPattern: 'target/*.jar'`
- Pre-send scripts: Groovy script runs before send for conditional delivery
- **Gotcha**: Gmail requires App Password (not regular password) with 2FA; SMTP timeout default 30s may need increase

### Environment Variables
- Built-in: `BUILD_NUMBER`, `BUILD_ID`, `JOB_NAME`, `BUILD_URL`, `WORKSPACE`, `JENKINS_HOME`, `NODE_NAME`, `BRANCH_NAME`, `CHANGE_ID`, `GIT_COMMIT`, `GIT_URL`
- Set in Jenkinsfile:
  ```groovy
  environment {
      MY_VAR = 'value'
      COMPUTED_VAR = sh(script: 'echo v1.0.${BUILD_NUMBER}', returnStdout: true).trim()
  }
  ```
- `sh 'printenv'` to debug all vars
- `withEnv(['PATH+MAVEN=/usr/local/maven/bin']) { ... }` — prepend to `PATH`
- `env.getEnvironment()` returns mutable map of env vars
- **Gotcha**: `WORKSPACE` changes per agent; do not hardcode paths; env vars in declarative are lazy (set before `stages` runs)

### Build Triggers
| Trigger | Config |
|---|---|
| Timer/cron | `triggers { cron('H */6 * * *') }` — `H` for hash spreading |
| SCM poll | `triggers { pollSCM('H/5 * * * *') }` — queries Git remote |
| Upstream | `trigger(job: 'downstream-pipeline', wait: false)` |
| Webhook | GitHub/GitLab/Bitbucket plugin POST to `/JENKINS_URL/github-webhook/` |
| Generic Webhook | `Generic Webhook Trigger` plugin — `token` + JSONPath filters |
| Manual | `input { message 'Proceed?' }` in pipeline; or `Build Now` button |
| Trigger from CLI | `curl -X POST JENKINS_URL/job/JOB_NAME/build --user user:token` |
| Parameterized | `curl -X POST JENKINS_URL/job/JOB_NAME/buildWithParameters --data-urlended json='{"parameter": [{"name":"PARAM", "value":"val"}]}' --user user:token` |

### Pipeline Best Practices
- Keep Jenkinsfile in SCM (not job config)
- Use Shared Libraries for reusable logic
- Prefer declarative over scripted for standard CI/CD
- Use `when { branch 'main' }` instead of `if` in scripted
- Pin tool versions in `tools {}` block
- `options { buildDiscarder(logRotator(numToKeepStr: '10')) }` prevents disk bloat
- Always `cleanWs()` in `post { always {} }` to reclaim workspace
- Use `retry(3) { ... }` for flaky network steps
- `timeout` wraps long-running steps
- Set `Build Name` with `setBuildName("${env.BRANCH_NAME}-${env.BUILD_NUMBER}")`

### Security Best Practices
- Always run Jenkins as non-root user
- Use HTTPS (reverse proxy with nginx/Caddy + Let's Encrypt)
- Enable CSRF Protection (`Manage Jenkins → Security → CSRF Protection`)
- Enable Agent → Controller Access Control (2.445+)
- Use credentials binding over plain `sh 'export PASS=...'`
- Restrict Script Console to admin only
- Disable JNLP agent port (use webSocket instead)
- Rotate Jenkins admin token; use personal tokens for automation
- Keep plugins updated; subscribe to jenkinsci-security@googlegroups.com
- Pin Jenkins LTS versions; test upgrades on non-prod first
- Audit `JENKINS_HOME/config.xml` permissions (`chmod 600`)

### Docker Pipeline Reference
```groovy
docker.image('image').inside()          // run inside container
docker.build('name', 'dir')              // build from Dockerfile
docker.image('name').withRun() { c ->    // run container, handle lifecycle
    sh "docker exec ${c.id} command"
}
docker.image('name').push('tag')         // push to registry
```
- `docker.build()` returns an Image object
- `inside()` mounts workspace, sets working dir; use `-v` for extra mounts
- Registry auth: `docker.withRegistry('https://index.docker.io/v1/', 'creds-id')`
- **Gotcha**: `inside()` on Windows uses Docker Desktop path translation; avoid symlinks

### Kubernetes Reference (kubectl)
```groovy
sh 'kubectl get pods -n jenkins'
sh 'kubectl logs deployment/myapp -n prod --tail=50'
sh 'kubectl apply -f k8s/deployment.yaml'
sh 'kubectl rollout undo deployment/myapp'
```
- Kubeconfig: `withKubeConfig([credentialsId: 'kube-config']) { sh 'kubectl ...' }`
- Helm: `sh 'helm dependency update && helm upgrade --install --atomic ...'`
- Skaffold: `sh 'skaffold run -p ci'`
- **Gotcha**: kubectl context must be set or use `--kubeconfig` flag

### Useful CLI Commands (`jenkins-cli.jar`)
```bash
java -jar jenkins-cli.jar -s http://localhost:8080 -auth @/tmp/token help
java -jar jenkins-cli.jar -s http://localhost:8080 build JOB_NAME -p PARAM=val -s -v
java -jar jenkins-cli.jar -s http://localhost:8080 list-jobs --tree name,url
java -jar jenkins-cli.jar -s http://localhost:8080 create-job NEW_JOB < config.xml
java -jar jenkins-cli.jar -s http://localhost:8080 reload-configuration
java -jar jenkins-cli.jar -s http://localhost:8080 restart
java -jar jenkins-cli.jar -s http://localhost:8080 groovy console.groovy
```
- Auth: `-auth user:token` (preferred) or `-auth user:password`
- Token from: `JENKINS_URL/user/USER/configure` → API Token → Add New Token

### Jenkinsfile Snippets — Quick Copy

**Conditional stage**:
```groovy
stage('Deploy') {
    when { branch 'main' }
    steps { sh 'deploy.sh' }
}
```

**Input gate**:
```groovy
stage('Approval') {
    input { message 'Deploy to production?'; ok 'Yes' }
    steps { sh 'deploy.sh' }
}
```

**Lock resources**:
```groovy
lock('deploy-lock') { sh 'deploy.sh' }
```

**Retry & timeout**:
```groovy
timeout(time: 10, unit: 'MINUTES') {
    retry(3) { sh 'flaky-command.sh' }
}
```

**Parallel matrix**:
```groovy
matrix {
    axes {
        axis { name 'OS'; values 'linux', 'windows' }
        axis { name 'JDK'; values '17', '21' }
    }
    stages {
        stage('Test') { steps { sh "make test OS=${OS} JDK=${JDK}" } }
    }
}
```

**Slack notification**:
```groovy
slackSend(channel: '#builds', color: '#36a64f', message: "Build ${env.BUILD_NUMBER} complete")
```

**Archive & JUnit**:
```groovy
post {
    always {
        archiveArtifacts artifacts: 'target/*.jar'
        junit 'target/surefire-reports/*.xml'
    }
}
```

### Troubleshooting & Debug
- `Manage Jenkins → System Log` — add custom loggers at `FINE`/`FINEST` for specific plugins
- Pipeline steps view: `JOB_URL/BUILD_NUMBER/pipeline-steps` — inspect each step's output
- Replay: `JOB_URL/BUILD_NUMBER/replay` — re-run with modified pipeline without commit
- `println` in scripted pipeline outputs to build log; in declarative use `echo`
- Check `JENKINS_HOME/logs/` — `tasks.log`, `controller.log` and `agent.log`
- Thread dump: `JENKINS_URL/threadDump` (admin)
- GC: `JENKINS_URL/gc` forces garbage collection
- Check loaded plugins: `JENKINS_URL/pluginManager/api/json?depth=1`
- Jenkins health: `JENKINS_URL/overall/health` or `/administrativeMonitor/`

### Gotchas Summary
- **Workspace cleanup**: `cleanWs()` in `post` of every pipeline — jobs leave GBs of temp files
- **Restart**: `safeRestart` waits for jobs; `restart` kills immediately
- **Job names**: Avoid spaces/special chars; causes URL encoding and script issues
- **Script Security**: Default deny for `new`, `Class.forName`, method calls; approve in `Manage Jenkins → In-process Script Approval` or use `@Grab` carefully
- **Concurrent builds**: `disableConcurrentBuilds()` blocks parallel runs of same branch
- **Sensitive env**: `set +x` before `sh` with secrets, or use `sh(label: '', script: '...', mask: true)`

### REST API & CLI (jenkins-rest-api-cli)
- `curl -u user:token JENKINS_URL/api/json` (list jobs in JSON)
- `curl -u user:token JENKINS_URL/job/JOB_NAME/{BUILD}/consoleText` (get console log)
- `curl -X POST -u user:token -H "Jenkins-Crumb:CRUMB" JENKINS_URL/job/JOB_NAME/build` (trigger build)
- `curl -X POST -u user:token JENKINS_URL/job/JOB_NAME/buildWithParameters?VERSION=x` (param build)
- `curl -X POST -u user:token JENKINS_URL/scriptText --data-urlencode "script=println(Jenkins.instance.pluginManager.plugins.size())"` (run Groovy via REST)
- `java -jar jenkins-cli.jar -s JENKINS_URL -auth @creds list-jobs` (CLI: list jobs)
- `java -jar jenkins-cli.jar -s JENKINS_URL -ssh -user admin -i ~/.ssh/id_rsa list-plugins` (SSH auth)
- `jenkins-cli.jar build JOB_NAME -p VERSION=1.0 -s` (CLI: build with params, `-s` waits)
- `curl -u user:token JENKINS_URL/computer/api/json` (list nodes via API)
- API token: `JENKINS_URL/user/USER/configure` → API Token → Add new token
- CSRF crumb: `curl -u user:token JENKINS_URL/crumbIssuer/api/json` (get `crumb` field)
- **Gotcha**: `403 No valid crumb` → missing `Jenkins-Crumb` header; disable crumb only with `-Dhudson.security.csrf.GlobalCrumbIssuerConfiguration.DISABLE_CSRF_PROTECTION=true`
- **Gotcha**: `curl -X POST` requires both auth header AND crumb header
- Rate limit: default 100 actions/min; override with `-Dhudson.model.ParametersAction.keepUndefinedParameters=true`

### Performance Tuning (jenkins-performance-tuning)
- JVM: `-Xmx4096m -Xms2048m -XX:+UseG1GC -XX:MaxGCPauseMillis=100 -XX:+ParallelRefProcEnabled`
- Thread pool: `master.executor.count=2` (never > CPUs/2); agent executors = CPUs - 1
- GC logging: `JAVA_OPTS="$JAVA_OPTS -Xloggc:/var/log/jenkins/gc.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=10M"`
- Analyze GC: `gcee -file gc.log` (GCease), or `gcviewer gc.log`
- Queue backlog: `JENKINS_URL/queue/api/json` → `why` field shows why jobs block
- SCM polling: use `H/5 * * * *` (not `*/5 * * * *`) — H spreads load, no stampede
- Plugin bloat: `curl -u user:token JENKINS_URL/pluginManager/api/json?depth=1 | python3 -c "import sys,json; d=json.load(sys.stdin); [print(p['shortName'],p['version']) for p in d.get('plugins',[]) if not p['active']]"`
- JENKINS_HOME: `du -sh /var/lib/jenkins/{jobs,plugins,builds,workspace}` find bloat
- Build artifact cleanup: logRotator `numToKeepStr: '30', artifactNumToKeepStr: '10'`
- Pipeline durability: `pipeline { options { durabilityHint 'PERFORMANCE_OPTIMIZED' } }` (skips write to disk on every step)
- nginx proxy cache: `proxy_cache_path /var/cache/nginx/jenkins levels=1:2 keys_zone=jenkins:10m max_size=1g;`
- **Gotcha**: Too many executors → GC thrashing. 2 executors per 4GB heap max.
- **Gotcha**: Pipeline `PERFORMANCE_OPTIMIZED` loses steps on crash. Use `SURVIVABLE_NON_SERIALIZABLE` for critical jobs.

### High Availability (jenkins-high-availability)
- Active-passive: one controller serves, standbys mount same JENKINS_HOME via NFS
- NFS: `mount -t nfs4 -o hard,intr,noatime,rsize=1048576,wsize=1048576 nfs-server:/export/jenkins /var/lib/jenkins`
- External DB: `JAVA_OPTS="$JAVA_OPTS -Djenkins.install.runSetupWizard=false"` + PostgreSQL config in `jenkins.model.JenkinsLocationConfiguration.xml`
- PostgreSQL setup: `CREATE DATABASE jenkins; CREATE USER jenkins WITH PASSWORD '...'; GRANT ALL PRIVILEGES ON DATABASE jenkins TO jenkins;`
- Load balancer: nginx `upstream jenkins { server 10.0.1.10:8080; server 10.0.1.11:8080; }` + `ip_hash` for stickiness
- Health check: `curl -I http://localhost:8080/login` (expect 200, not 503)
- Failover: `systemctl stop jenkins` on primary → DNS/LB points to standby → `systemctl start jenkins` on standby
- Blue-green upgrade: deploy new version to standby, test, switch LB, upgrade original
- Cron locking: use `@daily` instead of `0 0 * * *` to leverage Jenkins H hash (cron distributes across instances)
- Build queue: persisted to `$JENKINS_HOME/queue.xml`; active-passive shares this via NFS
- **Gotcha**: `git` plugin's `pollSCM` triggers on ALL controllers simultaneously — use `H` syntax to distribute
- **Gotcha**: Workspace not shared — agents connect to the active controller; passive controllers have no executor running
- **Plugins that break HA**: `external-monitor-job`, `build-pipeline-plugin`, `view-job-filters`, any plugin using `hudson.model.AbstractBuild.getBuiltOnStr()`
- Weekly HA smoke test: failover → health check → build → rollback → verify artifacts

### Supply Chain Security (jenkins-supply-chain-security)
- SBOM generation: `mvn org.cyclonedx:cyclonedx-maven-plugin:makeBom` → `target/bom.json`
- Cosign sign: `cosign sign-blob --key cosign.key target/app.jar > target/app.jar.sig`
- Cosign verify: `cosign verify-blob --key cosign.pub --signature app.jar.sig app.jar`
- Cosign keyless: `cosign sign-blob target/app.jar --fulcio-url=https://fulcio.sigstore.dev` (OIDC token from env)
- Generate SPDX: `mvn org.spdx:spdx-maven-plugin:createSPDX`
- SLSA provenance: `slsa-generator-generic --artifact target/app.jar --source github.com/org/repo --builder-id https://jenkins.example.com`
- Attest in pipeline: `pipeline { agent any; stages { stage('Sign') { steps { sh 'cosign sign-blob --key $KEY target/app.jar' } } } }`
- Verification gate: `stage('Verify') { steps { sh 'cosign verify-blob --key /keys/pub.pem --signature app.jar.sig app.jar && echo ARTIFACT_VERIFIED > verified.txt' } }`
- Rekor transparency log: `rekor-cli search --artifact target/app.jar` (find all entries)
- Docker content trust: `export DOCKER_CONTENT_TRUST=1 && docker push myorg/app:tag`
- **Gotcha**: Cosign keyless requires OIDC from Jenkins — `export COSIGN_OIDC_ISSUER=https://token.actions.githubusercontent.com` + `COSIGN_OIDC_CLIENT_ID=sigstore`
- **Gotcha**: SLSA Level 3 requires hermetic builds (no network, pinned deps) — use `docker build --network=none`

### Pipeline Unit Testing (jenkins-pipeline-unit-testing)
- Dependency: `testCompile group: 'com.lesfurets', name: 'jenkins-pipeline-unit', version: '1.0+'`
- Basic Spock test: `class PipelineTest extends BasePipelineTest { def 'run pipeline'() { when: def script = loadPipelineScript('Jenkinsfile'); script.execute(); then: assertJobStatus(script, 'SUCCESS') } }`
- Mock `sh` step: `helper.registerAllowedMethod('sh', [String]) { String cmd -> println "mock sh: $cmd"; return '' }`
- Mock credentials: `helper.registerAllowedMethod('withCredentials', [List, Closure]) { List creds, Closure body -> body.call() }`
- Test branch logic: `binding.setVariable('BRANCH_NAME', 'feature/foo'); script = loadPipelineScript('Jenkinsfile')`
- Test `input` step: `helper.registerAllowedMethod('input', [Map]) { Map params -> return true }`
- Test post conditions: `script.execute() → assert script.env.currentBuild.result == 'SUCCESS'`
- JenkinsPipelineUnit + Gradle: add to `build.gradle`; `./gradlew test` runs pipeline tests in CI
- Mock shared library: `helper.registerSharedLibrary('my-lib') { it.registerMethod('deployToK8s', { ... }) }`
- **Gotcha**: `NonCPS` methods cannot be mocked — refactor into separate helper class
- **Gotcha**: `Error: groovy.lang.MissingMethodException` → missing `registerAllowedMethod` call for that step
- Pipeline test CI: add `./gradlew test` to Jenkinsfile that runs BEFORE the pipeline stage

### Artifact Management (jenkins-artifact-management)
- Maven deploy: `mvn deploy -DaltDeploymentRepository=nexus::default::https://nexus.example.com/repository/maven-releases/`
- Gradle publish: `publishing { repositories { maven { url 'https://nexus.example.com/repository/maven-releases/' } } }`
- Artifactory upload: `jf rt u "target/*.jar" "libs-release-local/" --build-name=my-service --build-number=$BUILD_NUMBER`
- Nexus upload via REST: `curl -u user:pass -X PUT "https://nexus.example.com/repository/maven-releases/org/my/service/1.0/service-1.0.jar" -T target/app.jar`
- S3 plugin: `s3Upload(file:'target/app.jar', bucket:'my-build-artifacts', path:"${BUILD_NUMBER}/app.jar")`
- Promote via REST: `curl -X POST -u user:pass "https://nexus.example.com/service/rest/v1/staging/promote/{stagingRepoId}"`
- Artifactory promotion: `jf rt build-promote my-service $BUILD_NUMBER libs-release-local`
- Build cache: Maven settings `<mirror><url>https://nexus.example.com/repository/maven-public/</url></mirror>`
- Artifact cleanup: Nexus `-DscheduledTasks.cleanupService.deleteBlob=true`; retention `artifactNumToKeepStr: '5'`
- **Gotcha**: `401 Unauthorized` on deploy → verify `settings.xml` credentialsId matches Jenkins credential ID
- **Gotcha**: Checksum mismatch → Nexus/Artifactory stores md5/sha1; rebuild with `-Dchecksum.algorithms=SHA-256`

### Job DSL (jenkins-job-dsl)
- Freestyle seed: `job('example-job') { scm { github('org/repo', 'main') } triggers { scm('H/5 * * * *') } steps { shell('mvn clean test') } }`
- Pipeline seed: `pipelineJob('example-pipeline') { definition { cps { script('pipeline { agent any; stages { stage("Build") { steps { sh "mvn clean package" } } } }') } } }`
- Multibranch seed: `multibranchPipelineJob('example-multibranch') { branchSources { git { remote('https://github.com/org/repo.git') credentialsId('github-creds') } } triggers { periodic(10) } }`
- Folder: `folder('team-a') { displayName('Team A Projects') }`
- View: `listView('team-a-view') { jobs { regex('team-a-.*') } columns { status(), weather(), lastSuccess(), lastFailure() } }`
- Parameter: `parameters { choiceParam('ENV', ['dev', 'staging', 'prod']) stringParam('VERSION', '1.0') }`
- Trigger: `triggers { cron('H/15 * * * *') upstream('parent-job', 'SUCCESS') }`
- Process DSL: `curl -X POST -u admin:token -H "Jenkins-Crumb:..." JENKINS_URL/job/seed-job/buildWithParameters?dryRun=true`
- Test DSL: `def jobParent = new JobDslHelper().getDslFactory(this).job('test') { ... }` (in Spock)
- **Gotcha**: Job DSL sandbox blocks `new`, `System.getenv`, file reads — use `@Grab` or script approval
- **Gotcha**: `job('job-name')` creates jenkins job with `-` in name → SSH/SCM URL encoding issue; use `job { name 'job_name' }`
- **Gotcha**: If DSL script > 1000 jobs → split into folder-level scripts with `multibranchPipelineJob` + `branchSources`

### Upgrading Jenkins (jenkins-upgrading)
- LTS cadence: every 12 weeks (Feb/May/Aug/Nov); weekly security patches
- Check current version: `curl -sI http://localhost:8080/login | grep X-Jenkins` or `java -jar jenkins.war --version`
- Pre-upgrade: `systemctl stop jenkins; cp -a /var/lib/jenkins /var/lib/jenkins-backup-$(date +%Y%m%d)`
- Plugin list export: `curl -u admin:token JENKINS_URL/pluginManager/api/json?depth=1 | jq '.plugins[] | {shortName, version}' > plugins-before.json`
- Upgrade apt: `sudo apt update && sudo apt install --only-upgrade jenkins`
- Upgrade WAR: `systemctl stop jenkins; mv jenkins.war jenkins.war.old; wget http://updates.jenkins-ci.org/latest/jenkins.war; systemctl start jenkins`
- Upgrade Docker: `docker pull jenkins/jenkins:lts && docker compose up -d`
- Plugin compat check: `java -jar jenkins-plugin-compat-checker.jar --jenkins-version 2.440 --plugins plugins.txt`
- Pin plugins: `echo "plugin-name:1.2.3" >> pinned-plugins.txt`; JCasC: `tool: { pinnedPlugins: { fromFile: "pinned-plugins.txt" } }`
- Blue-green upgrade: deploy new to passive → test → switch LB → decommission old
- Rollback: `systemctl stop jenkins; rm -rf /var/lib/jenkins; cp -a /var/lib/jenkins-backup-DATE /var/lib/jenkins; mv jenkins.war.old jenkins.war; systemctl start jenkins`
- Test upgrade: `docker run -v jenkins_home_backup:/var/jenkins_home -p 8081:8080 jenkins/jenkins:NEW --httpPort=8080`
- **Gotcha**: Groovy sandbox breaking changes between 2.x and 2.y — always test Groovy scripts in staging
- **Gotcha**: Removed plugins on upgrade → disable before upgrade: `curl -X POST -u admin:token JENKINS_URL/plugin/PLUGIN_NAME/disable`
- Upgrade checklist: backup → export plugins → test in staging → drain pipeline → upgrade → smoke test → monitor GC/logs → remove old backup after 1 week

### File Locations
| Item | Path |
|---|---|
| Jenkins home | `/var/lib/jenkins` or `JENKINS_HOME` |
| Plugin `.jpi` | `JENKINS_HOME/plugins/` |
| Job config | `JENKINS_HOME/jobs/JOB_NAME/config.xml` |
| Agent config | `JENKINS_HOME/nodes/AGENT_NAME/config.xml` |
| Secrets/keys | `JENKINS_HOME/secrets/` |
| Master key | `JENKINS_HOME/secrets/master.key` |
| Credentials | `JENKINS_HOME/credentials.xml` |
| Identity | `JENKINS_HOME/identity.key` |
| Logs | `JENKINS_HOME/logs/` |
| Groovy init | `JENKINS_HOME/init.groovy.d/*.groovy` |
| JCasC | `JENKINS_HOME/jenkins.yaml` or `$CASC_JENKINS_CONFIG` |
| Build artifacts | `JENKINS_HOME/jobs/JOB_NAME/builds/BUILD_ID/archive/` |
| Workspace | `JENKINS_HOME/workspace/JOB_NAME/` |
| CLI JAR | `http://JENKINS_URL/jnlpJars/jenkins-cli.jar` |
| Agent JAR | `http://JENKINS_URL/jnlpJars/agent.jar` |