Feature Flags: Stale Flag Causes 15-Minute Outage

A stale feature flag left at 100% for three months caused a NullPointerException that took down checkout for 10% of users for 15 minutes. This incident illustrates why feature flags are ephemeral infrastructure, not permanent configuration. The problem isn't the technology—it's the discipline of removing flags once they outlive their purpose.

Why Feature Flags Are a Double-Edged Sword

Feature flags (also called toggles) are conditional branches in code that allow you to turn functionality on or off without deploying new code. The core mechanic is simple: a boolean check at runtime reads a flag value from a configuration source — environment variable, database, or dedicated service — and gates execution accordingly. This decouples deployment from release, letting you ship incomplete features to production safely.

In practice, flags introduce a persistent state dependency into your application. Every flag evaluation adds latency (typically <1ms for local config, 5-50ms for remote evaluation) and, more critically, creates combinatorial complexity. With N flags, you have 2^N possible system states. Teams often forget to remove flags after a rollout completes, leaving dead code paths that accumulate technical debt and obscure the actual control flow.

Use feature flags for canary releases, A/B testing, and kill switches — not for permanent configuration. The real value is in reducing deployment risk: you can roll back a bad feature by flipping a flag instead of reverting a deploy. But every flag you add is a liability. The industry rule of thumb: if a flag has been in production for more than two release cycles without being removed, it's already stale.

Basic Flag Implementation

The simplest feature flag is just an if statement controlled by an environment variable or a config value. This pattern works for small teams and simple rollouts. For a production grade approach, you need consistent user bucketing — the same user must always see the same experience. A common way is to hash the flag name with the user ID and take modulo 100 to assign a bucket.

Here's the minimal pattern in Python, using the io.thecodeforge namespace for all production packages.

# Package: io.thecodeforge.python.devops

# Simplest possible feature flag — environment variable
import os

def get_recommendations(user_id: int):
    if os.getenv('ENABLE_ML_RECOMMENDATIONS', 'false') == 'true':
        return ml_recommendations(user_id)   # new ML-based system
    else:
        return rule_based_recommendations(user_id)  # old system

# Better: percentage rollout — test on a fraction of users
import hashlib

def is_flag_enabled(flag_name: str, user_id: int, percentage: float) -> bool:
    """Consistently assign users to buckets using hash — same user always gets same result."""
    hash_input = f'{flag_name}:{user_id}'.encode()
    hash_val   = int(hashlib.md5(hash_input).hexdigest(), 16)
    bucket     = (hash_val % 100) + 1  # 1-100
    return bucket <= percentage

# Roll out to 5% of users
def get_checkout_flow(user_id: int):
    if is_flag_enabled('new_checkout', user_id, 5.0):
        return new_checkout_flow(user_id)
    return old_checkout_flow(user_id)

Feature Flag Service — LaunchDarkly SDK Pattern

When your team needs targeting by user attributes (plan, country, beta group), a dedicated flag service is the way to go. The SDK handles evaluation, caching, and streaming updates. This example shows how to use the LaunchDarkly SDK in Python, evaluating a flag with a rich user context.

# Package: io.thecodeforge.python.devops

# Using a feature flag service (LaunchDarkly, Unleash, Flagsmith)
import ldclient
from ldclient.config import Config

ldclient.set_config(Config(sdk_key='your-sdk-key'))
client = ldclient.get()

# Evaluate a flag for a specific user
def get_dashboard(user):
    context = {
        'key': str(user.id),
        'name': user.name,
        'email': user.email,
        'plan': user.subscription_plan,   # target premium users
        'country': user.country           # GDPR rollout by country
    }

    # Flag evaluated with user context — targeting rules in dashboard
    if client.variation('new-dashboard-v2', context, default=False):
        return render_new_dashboard(user)
    return render_old_dashboard(user)

Types of Feature Flags

Not all feature flags are the same. Pete Hodgson's taxonomy (from Martin Fowler's article) defines four types: release toggles, experiment toggles, ops toggles, and permission toggles. Release toggles are short-lived — they control rollout of a new feature. Experiment toggles are for A/B tests and should be removed after the experiment ends. Ops toggles are kill switches and circuit breakers — they must be fast and reliable. Permission toggles (entitlement flags) enable features for specific user segments (e.g., premium plan users) and can live long-term.

Mixing these types leads to confusion. Use naming conventions to distinguish: release_, exp_, ops_, perm_.

# Package: io.thecodeforge.python.devops

# Naming convention for flag types
release_flag_variation = client.variation('release_new_checkout_v3', context, default=False)
experiment_flag_variation = client.variation('exp_checkout_button_color', context, default='blue')
ops_flag_variation = client.variation('ops_disable_payment_gateway', context, default=False)
perm_flag_variation = client.variation('perm_premium_dashboard', context, default=False)

Canary Releases and Gradual Rollout with Flags

Canary releases are about routing a percentage of traffic to a new version of the service at the infrastructure level (e.g., Kubernetes canary deployments). But feature flags can enhance canaries by allowing you to target specific user segments within the canary pod. For example, you deploy the new version to 5% of pods, then use a feature flag to only enable the new feature for 10% of users hitting those pods. This gives you fine-grained control.

This pattern is common at large scale: you canary the deployment at the pod level, and inside the pod, use a flag to limit exposure further. This reduces blast radius if the new version has a bug — only a subset of the canary group sees the broken code.

# Package: io.thecodeforge.python.devops

# Canary with feature flag: even if the pod receives traffic, only a fraction of users get the new feature
import hashlib

def compute_bucket(user_id, flag_name, total_percent):
    hash_val = int(hashlib.md5(f'{flag_name}:{user_id}'.encode()).hexdigest(), 16)
    return (hash_val % 100) + 1 <= total_percent

# Canary: 5% of pods run new code, but only 20% of users on those pods get the feature
# That's effectively 1% of total users
if compute_bucket(user_id, 'new_recommendation_v2', 20):
    # This code only runs in the canary pods
    return new_recommendation_system(user_id)
else:
    return old_system(user_id)

Managing Flag Debt and Cleanup

Flag debt is the accumulation of stale conditionals in your code. Every flag that is no longer needed but still present forces your team to maintain two paths. Over time, the old path can break silently because it's rarely tested. The solution is to make flags ephemeral: set a removal date when you create the flag, automate reminders, and schedule cleanup as part of your sprint cycle.

A good rule: if a release flag has been at 100% for more than two weeks, it must be removed. For experiment flags, remove after the experiment analysis is complete — don't keep them 'just in case'. Ops flags and permission flags are exceptions, but they should be reviewed quarterly.

# Package: io.thecodeforge.python.devops

# Example: automate flag cleanup detection in CI
# This would be a script that checks git for old flag references

import subprocess
import re

FLAG_PATTERN = r'client\.variation\([\'"]([\w-]+)[\'"]'

def find_old_flags(months: int = 3):
    # Get all flags used in codebase
    result = subprocess.run(['grep', '-roPh', FLAG_PATTERN, 'src/'], capture_output=True, text=True)
    flags = set(re.findall(FLAG_PATTERN, result.stdout))
    # Check each flag's metadata (would use API in real life)
    # For now, just list them
    return flags

# In CI, warn if a release flag is older than 2 weeks
# This helps reduce flag debt

Flag-Driven Development Is a Testing Trap Without Kill Switches

Most teams add feature flags for gradual rollouts but forget the most critical flag: the kill switch. A kill switch is a flag that disables an entire feature category instantly — no dashboard login, no targeting rule tweak, just off. Without one, a broken feature that passes canary at 5% might kill your p99 latency at 25%. I've seen teams scramble to redeploy because their feature flags only controlled visibility, not execution. Kill switches live at the infrastructure level — environment variables or static configs loaded at startup — not in your feature management SDK. They should be toggleable from your CI/CD pipeline or a simple file change, not a slow API call. Netflix calls this the 'circuit breaker for features.' You need it. Because when your new checkout flow accidentally charges customers twice, you don't want to debug targeting rules — you want that code path dead in 10 seconds.

# io.thecodeforge.feature-flags.kill-switch
import os

KILL_SWITCH_CHECKOUT = os.getenv("KILL_SWITCH_CHECKOUT", "false").lower() == "true"

def process_checkout(user, cart):
    if KILL_SWITCH_CHECKOUT:
        logger.warning("Checkout kill switch active — falling back to legacy path")
        return legacy_checkout(user, cart)
    
    if feature_flags.is_enabled("new-checkout"):
        return new_checkout(user, cart)
    return legacy_checkout(user, cart)

Feature Flags Don't Replace Contract Tests — They Expose Missing Ones

Teams often think feature flags let them skip contract testing because they can 'just turn it off' if something breaks. That's dangerously wrong. A flag hides the UI or the code path, but your services still need to handle the new data shapes, the new API responses, the new database schema. I've watched a team spend two hours rolling back a flag-enabled feature because the old checkout service started receiving new payloads from a misconfigured flag that targeted the wrong user segment. The fix isn't more flags — it's consumer-driven contract tests (CDCTs) that validate both branches of every flag. Write Pact tests that verify the old path behaves correctly when the flag is off AND the new path when the flag is on. Every time you add a flag, you double your testing surface. Cover it with contracts, not hope.

# io.thecodeforge.feature-flags.contract-tests
from pact import Consumer, Provider

consumer = Consumer('CheckoutFrontend').has_pact_with(Provider('CheckoutService'))

# Test both flag states
@consumer.given('new-checkout flag is off')
def test_legacy_contract():
    expected = {'total': 29.99, 'items': [...]}
    pact = consumer.upon_receiving('legacy checkout request').with_request('POST', '/checkout', body={'cart_id': 'abc'})
    pact.will_respond_with(200, body=expected)
    with pact:
        result = request_checkout({'cart_id': 'abc', 'flag_override': False})
        assert result == expected

@consumer.given('new-checkout flag is on')
def test_new_contract():
    expected = {'total': 29.99, 'items': [...], 'promo_applied': True}
    pact = consumer.upon_receiving('new checkout request').with_request('POST', '/checkout', body={'cart_id': 'abc'})
    pact.will_respond_with(200, body=expected)
    with pact:
        result = request_checkout({'cart_id': 'abc', 'flag_override': True})
        assert result == expected