Jenkins GitHub Webhooks: The Production Setup That Won't Fail at 3 AM

You set up Jenkins GitHub webhooks, everything works in testing, then at 3 AM your production pipeline goes silent. No builds triggered. The team is paged. The root cause? A thread pool exhausted by a flood of webhooks from a misconfigured GitHub app. I've seen this bring down a payments service when the thread pool was exhausted at 3am because a developer pushed 500 commits in one push. The webhook storm killed Jenkins, and no builds ran for 20 minutes.

The problem is that most tutorials show you the happy path: install plugin, add webhook, done. They don't tell you what happens when GitHub sends 50 webhooks per second, or when your Jenkins URL changes, or when SSL certificates expire. This article covers the real production setup: securing webhooks, scaling the receiver, handling retries, and debugging when things go silent.

By the end, you'll be able to set up Jenkins GitHub webhooks that survive production traffic, diagnose failures without guessing, and know exactly when to use webhooks vs polling vs chatops triggers.

Why Webhooks Beat Polling — And When They Don't

Polling SCM every minute wastes resources and introduces latency. Webhooks give you instant triggers. But webhooks introduce a failure mode: if Jenkins is down, the webhook is lost. GitHub retries for 24 hours, but if you're down longer, you miss builds. Polling is more reliable for critical pipelines. Use webhooks for speed, but have a fallback polling job for missed triggers.

In production, I run both: webhooks for instant feedback on PR checks, and a nightly polling job that catches any missed pushes. This hybrid approach has saved us multiple times when Jenkins restarts or network blips occur.

// io.thecodeforge — DevOps tutorial
// Jenkinsfile: hybrid webhook + polling
pipeline {
    agent any
    triggers {
        // Webhook trigger (GitHub Push)
        githubPush()
        // Fallback polling every 5 minutes
        pollSCM('H/5 * * * *')
    }
    stages {
        stage('Build') {
            steps {
                echo 'Building...'
            }
        }
    }
}

Securing Webhooks: Don't Let Anyone Trigger Your Builds

Without a secret, anyone who knows your Jenkins URL can send a fake webhook and trigger builds. GitHub supports a secret token that Jenkins can verify. The GitHub plugin does this automatically if you set the secret in both GitHub and Jenkins. But there's a catch: the secret must be configured in the webhook itself, not in the Jenkins job.

To set it up: in GitHub webhook settings, add a secret. In Jenkins, go to Manage Jenkins > Configure System > GitHub > Advanced > Shared secret. This secret is used to validate all incoming webhooks. If the HMAC doesn't match, Jenkins rejects the request with 403. I've seen teams skip this because 'it's internal' — until a disgruntled employee triggers 10,000 builds and exhausts the executor pool.

// io.thecodeforge — DevOps tutorial
// No code needed — configuration steps:
// 1. In GitHub repo: Settings > Webhooks > Add webhook
//    Payload URL: https://jenkins.example.com/github-webhook/
//    Content type: application/json
//    Secret: my-secret-token
// 2. In Jenkins: Manage Jenkins > Configure System > GitHub > Advanced
//    Shared secret: my-secret-token
// 3. Jenkins verifies HMAC SHA1 signature automatically.

Handling Webhook Payloads: What Jenkins Actually Receives

When GitHub sends a push event, the payload contains the entire commit history, branch ref, and repository info. Jenkins parses this to determine which branches to build. The GitHub plugin maps the 'ref' field to branch names. If your job uses a branch specifier like '*/main', it will only trigger for pushes to main.

But here's the gotcha: if you use a multibranch pipeline, Jenkins automatically creates jobs for each branch. That's great for feature branches, but it can create hundreds of jobs if you have many branches. Each job consumes memory. I've seen Jenkins OOM because a team had 2000 branches from automated testing. Set a branch limit in the multibranch pipeline configuration.

// io.thecodeforge — DevOps tutorial
// Jenkinsfile for multibranch pipeline
pipeline {
    agent any
    triggers {
        // Only trigger on push events
        githubPush()
    }
    stages {
        stage('Checkout') {
            steps {
                checkout scm
            }
        }
        stage('Build') {
            steps {
                echo "Building branch ${env.BRANCH_NAME}"
            }
        }
    }
}
// In multibranch pipeline config:
// Branch Sources > GitHub > Repository URL
// Property strategy: Limit branches to 50
// Discover branches: only those with PRs

Scaling Webhook Reception: Thread Pools and Queues

Jenkins handles webhooks using a thread pool. By default, the pool size is small (around 10 threads). If you get a burst of webhooks — say from a force push that rewrites history — the pool can be exhausted. New webhooks are rejected, and you see 'RejectedExecutionException' in logs. The fix is to increase the pool size.

Set the system property -Dhudson.triggers.SCMTrigger.itemThreadPoolSize=50 in Jenkins startup. Or use the script console: System.setProperty('hudson.triggers.SCMTrigger.itemThreadPoolSize', '50'). But don't go too high — each thread consumes memory. Monitor thread count and queue depth via Jenkins metrics plugin.

I once set it to 200 on a large instance. Jenkins became unresponsive because 200 threads were all trying to parse webhook payloads simultaneously. The sweet spot is 20-50 for most teams.

// io.thecodeforge — DevOps tutorial
// Jenkins script console: increase webhook thread pool
import hudson.triggers.SCMTrigger

// Set pool size to 50
System.setProperty('hudson.triggers.SCMTrigger.itemThreadPoolSize', '50')

// Verify
println "Pool size: " + System.getProperty('hudson.triggers.SCMTrigger.itemThreadPoolSize')

// Also increase queue capacity
System.setProperty('hudson.triggers.SCMTrigger.itemQueueCapacity', '200')

Debugging Silent Failures: When Webhooks Don't Trigger

The most common issue: webhook is sent, Jenkins returns 200, but no build runs. This happens when the job's trigger configuration doesn't match the event. For example, the job expects a push to 'main' but the push is to 'develop'. GitHub sends the webhook, Jenkins acknowledges it, but the SCMTrigger checks the branch specifier and decides not to build.

To debug: check GitHub webhook delivery log for the request and response. Look at the 'ref' in the payload. Then check your job configuration. Also check Jenkins system log for 'SCMTrigger' messages. If you see 'Ignoring push event for branch refs/heads/develop', that's your answer.

Another silent failure: the webhook URL is wrong. GitHub will show a 404 or 301 in the delivery log. If you changed Jenkins URL, update the webhook URL in GitHub. Don't forget to update all repos.

// io.thecodeforge — DevOps tutorial
// Bash script to check webhook delivery from GitHub API
# Get recent webhook deliveries for a repo
curl -H "Authorization: token YOUR_GITHUB_TOKEN" \
  https://api.github.com/repos/owner/repo/hooks/123456789/deliveries

# Check the response status and payload
curl -H "Authorization: token YOUR_GITHUB_TOKEN" \
  https://api.github.com/repos/owner/repo/hooks/123456789/deliveries/987654321

When Not to Use Webhooks: The Case for Polling

Webhooks are great for instant feedback, but they're not reliable for critical pipelines. If Jenkins is down for maintenance, webhooks are lost (GitHub retries for 24 hours, but if you're down longer, they're gone). For production deployments, use polling with a short interval (1-2 minutes) instead of webhooks. Or use a hybrid approach.

Another case: if you have a monorepo with many teams, webhooks can trigger builds for every change, even unrelated ones. This wastes resources. Use path-based triggers or polling with file filtering.

Finally, if your Jenkins is behind a firewall and can't receive incoming connections from GitHub, webhooks won't work. Use polling or a reverse proxy with a public endpoint.

// io.thecodeforge — DevOps tutorial
// Jenkinsfile: polling only, no webhook
pipeline {
    agent any
    triggers {
        // Poll every 2 minutes
        pollSCM('H/2 * * * *')
    }
    stages {
        stage('Build') {
            steps {
                echo 'Polling triggered build'
            }
        }
    }
}

Advanced: Custom Webhook Processing with Shared Libraries

Sometimes the built-in GitHub plugin isn't enough. You need to parse custom webhook events (like release, issue comment) or validate payloads before triggering builds. Use a shared library to process webhooks via a generic webhook endpoint.

Create a pipeline job that listens to a generic webhook (HTTP endpoint plugin). The job receives the raw payload, validates it, and triggers the appropriate pipeline. This gives you full control. I've used this to trigger different pipelines based on the event type (push, PR, release) from a single webhook.

// io.thecodeforge — DevOps tutorial
// vars/genericWebhookProcessor.groovy (shared library)
def call(Map config) {
    pipeline {
        agent any
        triggers {
            GenericTrigger(
                genericVariables: [
                    [key: 'ref', value: '$.ref'],
                    [key: 'action', value: '$.action']
                ],
                token: config.token,
                causeString: 'Triggered by $ref'
            )
        }
        stages {
            stage('Route') {
                steps {
                    script {
                        if (env.ref.startsWith('refs/heads/')) {
                            build job: 'build-branch', parameters: [
                                string(name: 'BRANCH', value: env.ref.replace('refs/heads/', ''))
                            ]
                        } else if (env.action == 'opened') {
                            build job: 'pr-check', parameters: [
                                string(name: 'PR_NUMBER', value: env.action)
                            ]
                        }
                    }
                }
            }
        }
    }
}