Jenkins Multibranch Pipeline: Automate Branch Builds Without the Pain

You've got 47 branches, and someone just pushed to 'feature/payment-v2'. Without Multibranch Pipeline, you're manually creating a job, configuring SCM, and praying you didn't miss a step. That's not DevOps—that's busywork. Multibranch Pipeline automates this: it scans your repo, finds branches, and creates pipelines on the fly. By the end of this, you'll know how to set it up, avoid the thread-pool exhaustion that took down our CI at 3 AM, and recognize when it's overkill.

Why Multibranch Pipeline Exists: The Manual Job Hell

Before Multibranch Pipeline, every new branch meant a new Jenkins job. Click 'New Item', copy an existing job, change the branch name, update the SCM config, and hope you didn't miss the 'Build when a change is pushed' trigger. Multiply that by 20 branches and you've got a full-time job. Worse: when the build logic changed (e.g., new test step), you had to update every job manually. Inevitably, someone forgot a branch, and the broken build went unnoticed until production caught fire. Multibranch Pipeline solves this by deriving the pipeline from the Jenkinsfile in each branch. The branch itself defines its build. No manual job creation, no drift. The Jenkinsfile is the single source of truth.

// io.thecodeforge — DevOps tutorial

pipeline {
    agent any
    stages {
        stage('Build') {
            steps {
                echo 'Building branch: ' + env.BRANCH_NAME
                sh 'mvn clean compile'
            }
        }
        stage('Test') {
            steps {
                sh 'mvn test'
            }
        }
        stage('Deploy') {
            when {
                branch 'main'
            }
            steps {
                echo 'Deploying main branch'
                sh 'deploy.sh'
            }
        }
    }
}

Setting Up Your First Multibranch Pipeline

Creating a Multibranch Pipeline is straightforward: in Jenkins, click 'New Item', enter a name, select 'Multibranch Pipeline'. Under 'Branch Sources', add your Git repo. Choose 'Git' and enter the repository URL. Under 'Discover branches', select 'All branches' (or filter with regex if you only want specific patterns). Save. Jenkins will scan the repo and create a pipeline for each branch containing a Jenkinsfile. The first scan may take a minute—be patient. After that, any push to a branch triggers a scan and builds the branch if it has a Jenkinsfile. Pro tip: use 'Filter by name (with wildcards)' to exclude branches like 'release/*' if you want separate handling.

// io.thecodeforge — DevOps tutorial

pipeline {
    agent any
    triggers {
        // Poll every 5 minutes as fallback
        pollSCM('H/5 * * * *')
    }
    stages {
        stage('Checkout') {
            steps {
                checkout scm
            }
        }
        stage('Build') {
            steps {
                sh 'make build'
            }
        }
    }
}

Branch-Specific Logic: When and How

Not all branches are equal. You want to run unit tests on feature branches, integration tests on develop, and deploy to production only from main. Multibranch Pipeline supports this via the 'when' directive. Use 'branch' condition to run stages only on specific branches. For more complex logic, use 'expression' with Groovy. Common pattern: run a lightweight build on all branches, but skip expensive integration tests unless on main or a release branch. This saves time and resources. Also, use 'tools' to set JDK or Maven versions per branch if needed.

// io.thecodeforge — DevOps tutorial

pipeline {
    agent any
    stages {
        stage('Build') {
            steps {
                sh 'mvn compile'
            }
        }
        stage('Integration Tests') {
            when {
                anyOf {
                    branch 'main'
                    branch 'develop'
                    branch pattern: 'release/.*', comparator: 'REGEXP'
                }
            }
            steps {
                sh 'mvn verify -Pintegration'
            }
        }
        stage('Deploy to Staging') {
            when {
                branch 'develop'
            }
            steps {
                sh 'deploy-staging.sh'
            }
        }
        stage('Deploy to Production') {
            when {
                branch 'main'
            }
            steps {
                sh 'deploy-production.sh'
            }
        }
    }
}

Handling Pull Requests: The CI/CD Sweet Spot

Multibranch Pipeline natively supports pull requests (PRs) from GitHub, Bitbucket, or GitLab. When you add a PR source, Jenkins creates a pipeline for each PR, using the Jenkinsfile from the source branch. This is perfect for CI: run tests on every PR before merge. The pipeline status is reported back to the PR, giving immediate feedback. To set up, in the Multibranch Pipeline config, under 'Branch Sources', add the PR source (e.g., 'GitHub Pull Requests'). Configure triggers: 'Build on pull request creation', 'Build on pull request update'. You can also filter by target branch (e.g., only PRs targeting main).

// io.thecodeforge — DevOps tutorial

pipeline {
    agent any
    stages {
        stage('Checkout PR') {
            steps {
                checkout scm
            }
        }
        stage('Run Tests') {
            steps {
                sh 'npm test'
            }
        }
        stage('Lint') {
            steps {
                sh 'npm run lint'
            }
        }
    }
    post {
        success {
            // Update PR status via GitHub API
            script {
                pullRequest.labels.add('ci-passed')
            }
        }
        failure {
            script {
                pullRequest.labels.add('ci-failed')
            }
        }
    }
}

When Multibranch Pipeline Breaks: The Gotchas

Multibranch Pipeline isn't a silver bullet. Here are the pain points I've seen: 1) Scan overload: As mentioned, too many branches or frequent scans exhaust executors. 2) Jenkinsfile not found: If a branch lacks a Jenkinsfile, no pipeline is created. This is silent—no error. 3) Branch deletion: When a branch is deleted, the pipeline stays unless you configure 'Orphaned Item Strategy' to remove it. 4) Concurrent builds on same branch: By default, Jenkins allows multiple concurrent builds of the same branch. This can cause race conditions. Use 'Disable concurrent builds' in the pipeline options. 5) Shared libraries: Loading shared libraries from a different branch can cause version mismatch. Pin the library version.

// io.thecodeforge — DevOps tutorial

pipeline {
    agent any
    options {
        // Prevent concurrent builds on same branch
        disableConcurrentBuilds()
        // Keep only last 10 builds
        buildDiscarder(logRotator(numToKeepStr: '10'))
    }
    stages {
        stage('Build') {
            steps {
                sh 'make'
            }
        }
    }
}

When NOT to Use Multibranch Pipeline

Multibranch Pipeline is overkill for: 1) Single-branch repos: If you only have a main branch, use a simple Pipeline job. 2) Monorepos with multiple projects: Each project might need its own pipeline. Use a single Pipeline job with parameterized builds instead. 3) When you need fine-grained control over job configuration: Multibranch Pipeline abstracts away job config. If you need custom triggers, advanced permissions, or specific agent labels per branch, you might hit limitations. 4) When your Jenkinsfile is complex and shared across repos: Consider Shared Libraries instead of duplicating Jenkinsfiles. Multibranch Pipeline works best when each branch has its own Jenkinsfile.