Trunk-Based Development — 40 Feature Branches Caused a 3-Day Merge Hell
A team with 40 long-lived feature branches spent 3 days resolving merge conflicts.
20+ years shipping production infrastructure and CI/CD at scale. Everything here is grounded in real deployments.
- ✓Git, GitHub/GitLab, CI/CD pipeline (e.g., GitHub Actions, Jenkins), Docker, Kubernetes (optional), feature flag system (e.g., LaunchDarkly), monitoring stack (e.g., Prometheus, Grafana), basic understanding of DevOps and continuous integration.
The rules: 1. Branch from main. 2. Make small changes. 3. Merge back to main within 2 days. 4. Use feature flags to hide incomplete work. 5. Never have a branch older than 2 days without merging.
Imagine a highway where everyone merges into the same lane (main) every few minutes. Traffic flows smoothly because each merge is small. Now imagine a highway where 40 cars drive in their own lanes for weeks, then all try to merge at once. That's feature branches. Trunk-based development is the 'merge early, merge often' approach that prevents merge hell.
| Chrome | Firefox | Safari | Edge |
|---|---|---|---|
| ✓ | ✓ | ✓ | ✓ |
Long-lived feature branches are the #1 cause of merge pain in Git. Two developers working on separate branches for 3 weeks will have conflicts when they merge — guaranteed. The codebases diverged too far. Trunk-based development solves this by merging to main multiple times daily. Incomplete features are hidden behind feature flags (if/else blocks that toggle features on/off). The result: no merge hell, continuous integration catches conflicts immediately, and deployment can happen at any time. This guide covers the trunk-based workflow, the incident that triggered a 3-day merge catastrophe, and how to migrate from feature branches to trunk-based development.
What Trunk-Based Development Actually Means
Trunk-based development (TBD) is a version control practice where developers merge small, frequent changes into a single shared branch (the trunk, often main or master) multiple times per day. The goal is to avoid long-lived feature branches that diverge from the mainline, which cause merge hell and integration delays. In TBD, every commit is a candidate for release. This requires a culture of small, incremental changes, extensive automated testing, and feature flags to hide incomplete work. TBD is not just about merging often; it's about reducing the batch size of changes to minimize risk and accelerate feedback. Teams practicing TBD typically have a main branch that is always in a deployable state. This contrasts with GitFlow or other branch-heavy workflows where features are developed in isolation for days or weeks. The core metric: time from code commit to production. In TBD, that should be minutes, not days.
Setting Up Your CI Pipeline for High-Frequency Merges
To support trunk-based development, your CI pipeline must be fast and reliable. The pipeline should run on every push to main, executing unit tests, integration tests, static analysis, and build steps. Target a total pipeline duration under 10 minutes. If tests take longer, parallelize them. Use build caching (e.g., Docker layer caching, dependency caching) to avoid redundant work. The pipeline must also enforce quality gates: if any step fails, the commit is blocked from merging. This requires a robust test suite with high coverage (at least 80% for critical paths). Additionally, the pipeline should produce deployable artifacts (e.g., Docker images, compiled binaries) that are ready for production. Avoid manual approvals in the pipeline; they slow down the flow. Instead, rely on automated checks and feature flags for risky changes. The CI system should also run a subset of tests on feature branches (if any) but the main pipeline is the source of truth.
Feature Flags: The Safety Net for Incomplete Work
Feature flags (toggles) allow you to merge incomplete code to main without exposing it to users. This is essential for trunk-based development because you can't wait for a feature to be fully done before merging. Instead, you wrap new functionality behind a flag that is off by default. As you develop, you merge small increments, each time ensuring the flag remains off. Once the feature is complete and tested, you flip the flag on for a subset of users (canary) and eventually for all. Feature flags also enable instant rollback: if a feature causes issues, you toggle it off without a code revert. This reduces the risk of merging. However, feature flags add complexity: you must manage flag lifecycle, clean up old flags, and avoid flag debt. Use a mature feature flag system (e.g., LaunchDarkly, Flagsmith, or a simple config file) with proper access controls. Never use flags for long-term configuration; they are temporary.
Short-Lived Branches: The Only Branches You Need
In trunk-based development, branches are ephemeral. Developers create a branch from main, make a small change, commit, push, and open a pull request (PR). The PR is reviewed quickly (within hours) and merged. The branch is then deleted. The key is that branches live for less than a day. This minimizes divergence. If a branch lives longer, it accumulates changes that conflict with other merges. To enforce this, use branch protection rules: require linear history, require PR reviews, and require CI to pass. But keep the review process lightweight: focus on logic and design, not style (use linters). Encourage pair programming or mob programming to reduce review cycles. Some teams even bypass PRs for trivial changes (e.g., typo fixes) and commit directly to main. The goal is to reduce friction. Remember: the branch is not the unit of work; the commit is.
Code Reviews at Trunk Speed
Code reviews must be fast to support trunk-based development. Aim for a review turnaround time of under 2 hours. This requires a culture of prioritizing reviews over new work. Use small PRs (under 200 lines) to make reviews quick and focused. Larger changes should be broken into multiple PRs. Automate what you can: linting, formatting, and simple logic checks via CI. The human reviewer should focus on correctness, security, and design. Use tools like GitHub's pull request templates to guide reviewers. If a review is taking too long, consider pair programming instead. Some teams adopt 'mob programming' where the whole team reviews a change in real-time. The key is to avoid blocking the developer. If a reviewer is unavailable, have a backup reviewer. Remember: a delayed review is a tax on the entire team's velocity.
Automated Testing Strategies for Continuous Merging
With multiple merges per day, your test suite must be comprehensive and fast. Use the test pyramid: many unit tests (fast), fewer integration tests (slower), and a few end-to-end tests (slowest). Unit tests should run on every push and complete in under a minute. Integration tests run on every merge to main. End-to-end tests run on a schedule or on demand. Use test parallelization to speed up execution. For critical paths, use contract testing to ensure services interact correctly. Avoid flaky tests at all costs; they erode trust. If a test fails intermittently, quarantine it and fix it immediately. Use test impact analysis to run only tests affected by the change. This reduces pipeline time. Also, consider using feature flags to test in production with real traffic (canary testing). The goal is to have confidence that every merge is safe.
Deployment Automation: From Merge to Production in Minutes
Trunk-based development demands continuous deployment: every merge to main should trigger an automated deployment to production (or at least to a staging environment). This requires a mature deployment pipeline with canary releases, blue-green deployments, or rolling updates. The pipeline should include smoke tests after deployment to verify the service is healthy. Use infrastructure as code (e.g., Terraform, CloudFormation) to manage environments. Automate rollback: if health checks fail, automatically revert to the previous version. The deployment process should be fully automated, with no manual steps. This reduces the time from commit to production to minutes. However, not all teams can do full continuous deployment; some may use continuous delivery where deployments are manual but automated. The key is that the artifact is always ready to deploy.
Handling Merge Conflicts in a High-Frequency World
Merge conflicts are inevitable when multiple developers merge to main frequently. However, with trunk-based development, conflicts are smaller and easier to resolve because changes are small. The key is to resolve conflicts immediately when they occur. Use rebase instead of merge to keep a linear history. When a conflict arises, the developer should pull the latest main, rebase their branch, and resolve conflicts locally. This keeps the main branch clean. Avoid merge commits; they clutter history. Use tools like git rerere to remember conflict resolutions. If conflicts become frequent, it's a sign that teams are stepping on each other's toes. Consider breaking the monolith into microservices or using feature flags to isolate work. Also, communicate: use team chat to announce when you're about to merge a large change. The goal is to keep the main branch always green.
git pull --rebase to keep a linear history. Merge commits make it harder to understand the history and can cause issues with bisecting.Monitoring and Observability for Continuous Deployment
With multiple deployments per day, you need real-time monitoring to detect issues immediately. Use metrics (latency, error rate, throughput), logs, and traces. Set up alerts for anomalies: if error rate spikes above 1% or latency increases by 20%, alert the on-call engineer. Use dashboards to visualize deployment frequency and success rate. Implement canary analysis: compare metrics between the old and new versions. If the new version shows degradation, automatically roll back. This requires a robust observability stack (e.g., Prometheus, Grafana, ELK, Jaeger). Also, track deployment metadata: which commit, who deployed, what changed. This helps in post-mortems. The goal is to detect and revert bad deployments within minutes. Without observability, trunk-based development is risky because you're deploying frequently without visibility.
Scaling Trunk-Based Development to Large Teams
Trunk-based development scales to large teams, but requires discipline. Use feature flags to isolate work. Break the codebase into modules with clear ownership. Use monorepos or polyrepos depending on your tooling. For monorepos, use build systems like Bazel or Nx to only build and test affected code. For polyrepos, ensure each service has its own CI/CD pipeline. Communication is key: use a shared calendar for major merges. Consider using 'merge trains' where merges are queued and automatically merged one at a time. This prevents conflicts from multiple simultaneous merges. Also, invest in developer tooling: pre-commit hooks, automated refactoring, and code generation. The larger the team, the more important it is to have a culture of small changes. Enforce branch protection rules and use bots to remind developers of best practices.
Common Pitfalls and How to Avoid Them
Trunk-based development is not a silver bullet. Common pitfalls include: (1) Incomplete testing: if tests are slow or flaky, developers will bypass them. Fix the test suite first. (2) Feature flag debt: flags left in code forever increase complexity. Schedule regular cleanup. (3) Too many concurrent changes: if too many developers are touching the same code, conflicts increase. Break the codebase into smaller services. (4) Lack of discipline: some developers will still create long-lived branches. Enforce branch lifetime limits with automation. (5) Insufficient monitoring: without observability, you can't detect issues from frequent deployments. Invest in monitoring before adopting TBD. (6) Cultural resistance: developers used to long-lived branches may resist. Start with a pilot team and show metrics. The key is to address these pitfalls proactively. TBD requires a mature engineering culture.
Measuring Success: Metrics That Matter
To know if trunk-based development is working, track key metrics: deployment frequency (should be multiple times per day), lead time for changes (time from commit to production, should be minutes), mean time to recovery (MTTR, should be under 1 hour), and change failure rate (should be under 5%). Use DORA metrics. Also track merge conflict frequency and branch lifetime. If deployment frequency is low, investigate bottlenecks: slow CI, long reviews, or cultural resistance. If MTTR is high, improve monitoring and rollback automation. If change failure rate is high, improve testing and feature flag usage. Share these metrics with the team in a dashboard. Celebrate improvements. The goal is not just to do TBD, but to achieve high performance. Remember: TBD is a means to an end, not the end itself.
40 Feature Branches, 3 Days of Merge Hell, 1 Deleted Feature
git log --all --full-history -- <deleted-file> to find the last commit that had the feature. 3. Cherry-picked the missing feature commits back. 4. Initiated trunk-based development: max branch lifetime 2 days, feature flags for incomplete work, mandatory daily merges. 5. Set up CI that flags branches older than 2 days without merging to main.- Long-lived branches cause merge hell.
- Trunk-based development: merge to main multiple times daily.
- Use feature flags for incomplete work.
- Max branch lifetime: 2 days.
- CI should flag stale branches.
| File | Command / Code | Purpose |
|---|---|---|
| git-workflow.sh | git checkout main | What Trunk-Based Development Actually Means |
| .github | name: CI | Setting Up Your CI Pipeline for High-Frequency Merges |
| feature-flag.js | const flags = { | Feature Flags |
| short-lived-branch.sh | git checkout -b fix/login-typo | Short-Lived Branches |
| CODEOWNERS | * @team-leads | Code Reviews at Trunk Speed |
| jest.config.js | module.exports = { | Automated Testing Strategies for Continuous Merging |
| .github | name: Deploy to Production | Deployment Automation |
| resolve-conflict.sh | git checkout feature-branch | Handling Merge Conflicts in a High-Frequency World |
| prometheus-alerts.yml | groups: | Monitoring and Observability for Continuous Deployment |
| merge-queue.yml | merge_queue: | Scaling Trunk-Based Development to Large Teams |
| check-stale-branches.sh | for branch in $(git branch -r --merged main | grep -v main); do | Common Pitfalls and How to Avoid Them |
| metrics.py | def calculate_lead_time(commit_time, deploy_time): | Measuring Success |
Key takeaways
Frequently Asked Questions
20+ years shipping production infrastructure and CI/CD at scale. Everything here is grounded in real deployments.
That's Git. Mark it forged?
6 min read · try the examples if you haven't