Canary Releases — Why 2% Traffic Broke 30% of Users

Every engineer has lived through it: a deployment goes out on a Friday afternoon, the monitoring dashboards start lighting up red at 5:03 PM, and the on-call rotation becomes everyone's nightmare. The root cause is almost always the same — code that looked perfect in staging hit a production edge case nobody anticipated. The bigger your user base, the bigger the blast radius. Netflix, Google, and Amazon all independently arrived at the same antidote: never trust staging completely, and never ship to everyone at once.

Canary releases solve the confidence gap between 'it works in CI' and 'it works for your users.' The core idea is surgical: you route a controlled percentage of live traffic — say 1% — to the new version of your service while the other 99% of users hit the stable version. You instrument that 1% slice with the same production observability you'd use for a full rollout, measure error rates, latency percentiles, and business metrics, and only widen the traffic gate when the numbers stay green. If they don't, you pull the canary back without most users ever knowing something was wrong.

By the end of this article you'll understand exactly how traffic splitting works at the infrastructure level (Nginx, Kubernetes, and service mesh layers), how to write automated promotion and rollback logic tied to real SLO signals, and the subtle production gotchas that sink canary strategies at scale — things like session stickiness breaking A/B consistency, database schema drift between canary and stable, and metric lag causing premature promotion. Let's build this from the ground up.

What is Canary Releases Explained?

Canary Releases Explained is a core concept in DevOps. Rather than starting with a dry definition, let's see it in action and understand why it exists.

The name comes from the old coal mining practice: miners would bring a canary into the mine. If toxic gases accumulated, the canary would die first, warning the miners to escape. Your software canary does the same — if the new version has a critical bug, only a small slice of users experiences it, giving you the signal before the whole user base is affected.

At its core, a canary release is about blast radius containment. You don't trust staging to simulate real traffic patterns, user behaviors, or data volumes. So you use production itself as the testbed — but with a controlled, reversible exposure. The key difference from a simple rollout is that you have a decision gate at each traffic percentage: if metrics go red, you stop and revert before more users are hit.

The term has stuck for decades because the analogy holds — your canary is a small indicator of system health. In production, the canary isn't just a passive passenger; it actively sends metrics back. If anything looks off, you pull it before the whole mine collapses.

One thing engineers often overlook: the canary itself can become a single point of failure if it shares the same config as stable. Always run the canary with its own configuration to avoid mode confusion.

Another nuance: the canary must be able to talk to the same downstream services as stable. If the canary uses a different service discovery endpoint or a different database, the test is invalid. Keep everything identical except the code version.

Deepening the concept: Canary releases aren't just for services handling HTTP traffic. They apply to batch jobs, data pipelines, and even infrastructure changes. For example, if you're rolling out a new Spark job version, you can route a subset of partitions to the new job while the rest process on the old one. The same principles apply: compare output quality, latency, and resource usage before switching fully. The blast radius is smaller, but the need for automated rollback is just as critical.

Real failure story: I once saw a team deploy a canary that changed the log format. The stable logs were parsed by a monitoring pipeline that expected the old format. The canary's logs broke the pipeline, leading to a 45-minute observability blind spot. The canary looked healthy because no errors were logged — but the pipeline had silently died. We now validate log format compatibility as part of the canary preparation.

Performance impact: Running two versions side by side increases resource usage. Expect ~2x CPU/memory during the canary window. Plan cluster capacity accordingly.

Trade-off: Canary releases add complexity: you need observability, automated rollback, and careful traffic management. For low-traffic services, they may not provide statistically significant signal.

package io.thecodeforge.canary;

import java.util.Map;

/**
 * Production-grade canary configuration validator.
 * Ensures the canary and stable are compatible before traffic split.
 */
public class CanaryConfig {

    public static boolean validateCanaryConfig(Map<String, String> stableConfig, Map<String, String> canaryConfig) {\n        // Critical: canary must not share DB write paths without compatibility\n        if (canaryConfig.getOrDefault(\"db.migration.phase\", \"none\").equals(\"breaking\")) {\n            System.err.println(\"Canary skipped: DB migration is backward-incompatible.\");\n            return false;\n        }\n        // Ensure same service discovery endpoint\n        if (!stableConfig.get(\"service.discovery.url\").equals(canaryConfig.get(\"service.discovery.url\"))) {\n            System.err.println(\"Canary has different service discovery — test invalid.\");\n            return false;\n        }\n        // Ensure log format compatibility\n        if (!stableConfig.get(\"log.format\").equals(canaryConfig.get(\"log.format\"))) {\n            System.err.println(\"Log format changed — monitoring pipeline may break.\");\n            return false;\n        }\n        return true;\n    }\n}",
        "output": "Config validation logs warnings if incompatibilities are found. Prevents silent production failures."
      }

How Traffic Splitting Works at Each Infrastructure Layer

Traffic splitting is the core mechanism behind canary releases. At the infrastructure level, you have three common layers to implement it:

1. Load balancer layer (e.g., Nginx, HAProxy): Use upstream weights. Nginx example: server backend-v1 weight=99; server backend-v2 weight=1;. Simple but requires manual updates or a reload.
2. Orchestration layer (e.g., Kubernetes with Services): Use multiple Deployments and a single Service with label selectors. You can't do fractional traffic with plain Services — you need a service mesh or ingress controller that supports weighted routing (e.g., Istio VirtualService, Nginx Ingress with canary annotation).
3. Service mesh layer (e.g., Istio, Linkerd): Fine-grained traffic splitting with headers, cookies, or percentage-based weights. Istio VirtualService example: route 99% to stable, 1% to canary via weight field. Also supports A/B testing by header.

The choice depends on your infrastructure maturity. If you already have a service mesh, use it — it gives you the richest control (session stickiness, retry budgets, fault injection). Without mesh, use Nginx or your ingress controller's canary support (like Nginx Ingress's canary-weight annotation).

The more control you have over routing, the more you have to understand. Istio gives you header-based routing but adds sidecar overhead and debugging complexity. Nginx is simpler but lacks session stickiness without additional configuration. Choose the layer that matches your team's ability to debug at 2 AM.

A practical tip: if you're using cloud load balancers (AWS ALB, GCP HTTP LB), their weighted target groups are easy to set up but lack header-based routing. Use them as a starting point and move to mesh when you need more granularity.

If you're on Nginx, you can also use split clients module for cookie-based routing, but that requires custom Lua scripting. Keep it simple unless you need complex rules.

Traffic splitting at DNS is dangerous: Weighted round-robin via Route53 or similar gives no fine-grained control. DNS caching can cause traffic to stick to the canary for hours even after you roll back. Always use layer 7 routing for canaries.

Real example: A junior team once used Route53 weighted record sets for canaries. They pushed 5% traffic to the new version, saw no errors, and promoted to 100%. What they didn't realize: DNS resolvers cached the canary's IP for TTL hours, so users kept hitting the old version long after the switch. The canary had been handling 0% of real traffic. They learned the hard way: never trust DNS for canary traffic control.

Failure scenario: If the load balancer is misconfigured and routes 100% to canary, the stable version sits idle and its cluster autoscaler scales down. When rollback triggers, there are no stable pods ready. Always set a minimum replica count for stable during canary. Debugging: Use kubectl get virtualservice -o yaml to verify the current traffic weights. For Nginx, check the upstream status with curl localhost/status.

Additional nuance: When using Istio, remember that each sidecar proxy adds latency. For very high-throughput services, consider using a dedicated ingress gateway that handles canary routing without per-pod proxies. Also, test with realistic traffic patterns before relying on header-based routing in production.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: myapp
spec:
  hosts:
  - myapp
  http:
  - match:
    - headers:
        canary:
          exact: "true"
    route:
    - destination:
        host: myapp
        subset: v2
        weight: 100
  - route:
    - destination:
        host: myapp
        subset: v1
        weight: 90
    - destination:
        host: myapp
        subset: v2
        weight: 10

Metric-Based Promotion: SLOs That Actually Work

Promoting a canary to production isn't a manual thumbs-up. It should be gated on a set of SLOs that reflect both technical and business health. The classic anti-pattern is to promote based on CPU and memory alone — your code can be efficient but break user flows.

Define a **Canary SLO Window**: a sliding time window (e.g., 10 minutes) where all metrics must be within thresholds. Popular choices: - Error rate: < 0.1% 5xx over 1 minute - Latency p99: < 200ms increase over baseline - Request rate: within ±5% of expected traffic (detects silent drops) - Business metric: checkout conversion rate >= 99% of stable

Use a pipeline that automatically promotes through traffic steps: 1% → 5% → 20% → 50% → 100%. Each step waits for the SLO window to pass. If at any step the SLOs are breached, the rollback is triggered.

For Kubernetes + Prometheus, you can implement this with an operator (like Flagger or Argo Rollouts) that watches metric thresholds and adjusts traffic automatically.

The most common failure mode in canary promotions is technical health vs. business health mismatch. You can have 0 errors and 200ms p99 but lose 10% of signups because a button moved. Always include at least one business SLO per critical flow.

Here's a hard truth: business SLOs are hard to define because they often require cross-team agreement. Start with one that matters most, like checkout completion or signup rate, and add more as you gain confidence.

Another practical issue: business metrics often have lower resolution (e.g., hourly). In that case, use the canary as a long-running test before full promotion. Run at 20% for an hour, measure conversion, then promote.

Sliding windows vs cumulative windows: Sliding windows over short intervals (1-2 minutes) catch spikes fast, but can be noisy. Cumulative windows smooth noise but delay detection. For canary promotion, use sliding windows for error rate and peak latency, and a longer cumulative window (10 minutes) for business metrics to avoid false positives from transient dips.

Production scenario: A team once had a canary running at 5% for 10 minutes — error rate 0%, latency p99 150ms, everything green. They promoted to 100%. Ten minutes later, the revenue tracking dashboard showed a 15% drop. Turns out the canary had a CSS bug that hid the 'Buy Now' button on mobile. No technical metric caught it. They now run a business SLO for 'click-through rate on purchase flow' before promoting beyond 20%.

Failure scenario: If you set the error rate threshold too tight (e.g., 0.01%), a single 5xx from a transient glitch will roll back a healthy canary. Use for: 2m in Prometheus to require sustained breach. Debugging: Use Grafana panels comparing canary vs stable side-by-side. A step change in latency that appears only on canary is a clear signal.

Additional insight: Consider using a "burn rate" approach: if the error budget is being consumed faster than expected, that's a signal to abort the canary even if the absolute error rate is still within threshold. This catches issues that gradually worsen.

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: myapp
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp
  service:
    port: 80
  canaryAnalysis:
    interval: 1m
    threshold: 5
    stepWeight: 10
    maxWeight: 50
    metrics:
    - name: error-rate
      thresholdRange:
        max: 1
      interval: 1m
    - name: latency-p99
      thresholdRange:
        max: 0.5
      interval: 1m
    - name: request-success-rate
      threshold: 99.9
      interval: 1m

Rollback Strategies: Fast, Gradual, and Safe

A canary release without an automated rollback is just a slow rollout. The whole point is to minimise blast radius, so when metrics go red, you need to revert fast.

Three rollback strategies, each with trade-offs:

1. Zero-Kill Rollback: Immediately set canary traffic to 0%. This is the fastest. Works if the canary hasn't mutated any shared state (e.g., database writes). Use when you're confident the canary's state can be discarded.
2. Gradual Rollback: Reverse the traffic steps in order: 50% → 20% → 5% → 1% → 0%, waiting at each step to ensure no cascading effects. This is safer if the canary might have created data that needs to be reconciled (e.g., incomplete write-backs).
3. Full Redeploy of Previous Version: If the canary changed configuration or data, you may need to redeploy the old version with the old config. This is a nuclear option — use only if gradual rollback fails.

Engineers often hesitate to roll back because it feels like admitting failure. That hesitation costs users. Build the habit: roll back first, investigate later. Your users don't care why it broke — they care that it's fixed.

One more thing: if your rollback automation hasn't been tested, it doesn't exist. Schedule a chaos experiment where you inject a fault and verify the rollback fires within the expected window.

Also consider: What if the canary has been running for an hour and has processed orders? Zero-kill might be unsafe. Track state: use canary-state annotations to know if the canary touched any database. Only use zero-kill for stateless services.

Rollback for stateful canaries: If the canary wrote to a database, a gradual rollback gives time for compensating transactions. For example, if the canary created user records with a new schema, the rollback might need to revert those records. This requires careful design of write paths to be idempotent and backward-compatible.

Real incident: A team's canary had been running at 50% for three hours. A bug was discovered in the new pricing logic that had been updating prices in the shared database. When they executed a zero-kill rollback, the stable version immediately started reading the wrong prices — data corruption had already occurred. They had to run a full database restore. Lesson: track canary writes and use gradual rollback for stateful canaries.

Failure scenario: If stable is scaled down during canary (to save cost), a rollback may find no replicas ready. Always keep at least the original stable replica count during canary. Debugging: Use kubectl get pods -l version=stable to verify stable availability. For gradual rollback, watch Flagger logs to ensure each step passes.

Additional depth: Use a "rollback guard" that prevents zero-kill if the canary has written to any shared storage. You can implement this with a canary-sidecar that tracks writes and flips a readiness flag. Also, consider using feature flags alongside canaries: if the canary's feature is toggled off, the code is deployed but inactive — making rollback trivial.

# io.thecodeforge.rollback_handler
import time

def gradual_rollback(current_weight, step=5):
    """Gradual rollback from current weight to 0%."""
    for weight in range(current_weight, -1, -step):
        set_traffic_weight('stable', weight)
        set_traffic_weight('canary', 100 - weight)
        print(f'Rollback step: stable={weight}%, canary={100-weight}%')
        # Wait for metrics to stabilise
        if not watch_slo_window(window=60, threshold=0.1):
            print('SLO breach during rollback — accelerating to 0%')
            set_traffic_weight('canary', 0)
            break
        time.sleep(30)
    print('Rollback complete. Stable handles 100% traffic.')

Production Gotchas That Sink Canary Releases at Scale

After implementing canary releases across several teams and platforms, I've seen the same handful of issues surface repeatedly. Here are the ones that break production:

1. Session Stickiness Breaks A/B Consistency If you're using a canary to test a new feature that changes backend behaviour, users need to stay on the same version for the duration of their session. Without consistent hashing on a session ID, a user might hit the canary for one request (getting the new feature) and then hit stable for the next (getting the old behaviour). This causes confusing user experience and invalidates your A/B metrics. Solution: use Istio's consistentHash on a cookie or header, or configure your load balancer to use a cookie-based affinity.

2. Database Schema Drift Between Canary and Stable The canary often runs a migration script on startup. If the migration adds a column that the stable version doesn't know about, and the canary writes to that column, the stable version may crash when trying to read or write (depending on column nullability). Solution: always write migrations that are backward-compatible for at least one release (expand-contract pattern). Run the migration only after the canary is promoted to 100%.

3. Metric Lag Causes Premature Promotion Your SLO window says everything is fine, but your Prometheus scrape interval is 15s and latency percentiles are computed over 5-minute windows. A sudden error burst can take up to 5 minutes to show up. If your promotion window is 2 minutes, you'll promote into a disaster. Solution: use a minimum evaluation window of at least 5 minutes for latency-sensitive SLOs, and use a separate 1-minute scrape for error rate.

4. Resource Constraints from Parallel Versions Running two versions of a service means ~double the resource usage during the canary window. In high-traffic systems, this can exhaust CPU or memory on the node. Plan for this: either use cluster autoscaler or schedule canary pods on separate node pools.

5. Canary Promotes But Rollback Fails The most painful scenario: you promote to 100%, then find a bug, but rolling back is impossible because the database schema has already changed. Solution: implement feature flags within the code, so you can disable the feature without rolling back the code. This complements canary releases — canaries test the deployment, feature flags test the behaviour.

6. Metric Aggregation Window Mismatch If your SLO window is 2 minutes but your metric latency percentile is computed over 5 minutes, you will never see spikes in time. Align windows explicitly.

7. Configuration Drift The canary pod might get a different config than intended due to a typo or stale secret. Always verify config checksums or use a diff tool before starting the canary.

8. Observability Overhead Running two versions doubles your logging volume, traces, and metric cardinality. If you're on a pay-per-volume observability platform, expect your bill to spike. Set up a separate canary-specific log stream with lower retention, or use a sampling rate for traces during canary.

9. Network Policies Blocking Cross-Version Traffic In multi-tenant clusters, Kubernetes network policies may accidentally block the canary from reaching required downstream services. Always test network policies before the canary goes live, and include a canary-specific network policy that mirrors the stable policy.

10. Canary as a Retry Amplifier If the canary is slower than stable, client timeouts may trigger retries that hit the stable version, causing double load. This is especially dangerous when the canary is small — the stable version may get overwhelmed. Use retry budgets and circuit breakers between versions to prevent this.

Additional gotcha I've seen repeatedly: Teams forget to update their alerting thresholds when a canary is running. The canary's elevated error rate (expected, since it's under test) can trigger false alarms. Use version-based alert suppression during canary windows.

Failure scenario: A team used the same HPA for both canary and stable. When canary traffic increased, HPA scaled up canary pods, consuming node resources and causing stable pods to be evicted. Solution: use separate HPAs or pin canary replica count. Debugging: Check resource usage per pod with kubectl top pods -l app=myapp. Look for pods from both versions using the same PVC. Performance impact: Running two versions can double log volume. For high-traffic services, that's 10x cost increase on some observability platforms. Use canary-specific log destinations with lower retention.

11. Canary Not Isolated from Stable's Chaos If you run chaos experiments on stable, the canary may get caught in the blast. Ensure canary pods are excluded from chaos experiments during the canary window.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: canary-pdb
spec:
  minAvailable: 1
  selector:
    matchLabels:
      version: canary
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: myapp-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp
  minReplicas: 3
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 80
  - type: Pods
    pods:
      metric:
        name: k8s_pod_error_rate
      target:
        type: AverageValue
        averageValue: 0.01

Automating Canary Releases with Flagger and Argo Rollouts

Manual canary releases don't scale. At a certain traffic volume, you need automation that watches metrics, adjusts traffic weights, and decides promotion or rollback without human intervention. Two popular Kubernetes-native tools provide this: Flagger and Argo Rollouts.

Flagger integrates with Prometheus, Istio, Linkerd, or Nginx ingress. You define a Canary CRD with metric thresholds, traffic steps, and evaluation intervals. Flagger gradually shifts traffic, runs analysis, and either promotes by removing the canary or rolls back by resetting weights to zero.

Argo Rollouts uses a Rollout resource that replaces the standard Deployment. It supports Blue-Green, canary, and progressive delivery. Traffic splitting can be managed via a Service Mesh or ingress controller. Argo Rollouts provides a CLI and dashboard for manual intervention if needed.

Both tools support webhook metrics for business SLOs (e.g., a Prometheus query for conversion rate). They also integrate with GitOps workflows (ArgoCD + Rollouts for declarative progressive delivery).

The key to automation is idempotency: the canary analysis should be repeatable and deterministic. If the metrics breach, the tool must roll back. If they stay green, it promotes. No manual overrides during the window — trust the automation.

Both Flagger and Argo Rollouts require understanding of their custom resources and metric templates. Don't adopt them without a dry run in a staging cluster with simulated traffic. The first automated rollback should be tested with a synthetic fault injection.

A practical note: start with Flagger if you're already on Istio — the integration is seamless. Argo Rollouts is better if you need multi-cluster or advanced blue-green alongside canary.

Important: have a backup plan if the automation fails. For instance, if Flagger's Canary resource becomes stuck due to a bug, you should be able to manually edit the VirtualService to cut traffic. Keep the manual escape hatch open.

Idempotency of analysis templates: Write PromQL queries that are stable over short time windows to avoid false positives from transient metric dips. Use avg_over_time for error rates and histogram_quantile for latency with a sufficient window (5-10 minutes). Test these queries against historical data before using them in production.

Real experience: We once had a Flagger canary that kept rolling back despite the code being fine. The issue: a misconfigured Prometheus query was averaging error rate over 5 minutes, but the canary was only running for 1 minute at 1% weight — the average included zero traffic periods, making the error rate appear high. We fixed it by using rate on a 1-minute window and adding a minimum request count filter.

Failure scenario: If Flagger's analysis template references a metric that doesn't exist (e.g., typo in metric name), the canary will be stuck in 'Progressing' state indefinitely. Always validate metric names with kubectl get prometheusrules. Debugging: Check Flagger logs with kubectl logs -n flagger-system deployment/flagger --tail=50. Look for 'evaluation' lines. Performance impact: Flagger adds ~1 minute to each traffic step due to evaluation interval. For rapid deployments, consider reducing the interval to 30s.

Additional consideration: Both tools allow custom webhooks for metrics not supported natively. If you use Datadog or New Relic, you can create a webhook that queries those backends and returns a pass/fail signal to the canary analysis.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: myapp-rollout
spec:
  replicas: 5
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: myapp
        image: myapp:stable
        ports:
        - containerPort: 8080
  strategy:
    canary:
      steps:
      - setWeight: 10
      - pause: {duration: 5m}
      - setWeight: 50
      - pause: {duration: 10m}
      - setWeight: 100
      analysis:
        templates:
        - templateName: myapp-error-rate-analysis

Observability Requirements for Canary Releases

You can't run a canary release without solid observability. If you can't see what's happening in the canary, you're flying blind — and you'll either promote a broken version or roll back a healthy one. Here's what you actually need:

Metrics (Real-time, low-latency) - Error rate per version (5xx, 4xx) with 1-second resolution if possible. - Latency percentiles (p50, p95, p99) — must be computed on a sliding window, not cumulative. - Request rate to detect sudden drops (could indicate routing errors). - Business metrics: conversion rate, signup rate, revenue per request.

Tracing (End-to-end per request) - Every request must carry a trace ID that identifies which version handled it. - Use distributed tracing (Jaeger, Zipkin, OpenTelemetry) to trace a request across all services. - This helps you attribute errors to the canary even when the failure manifests in a downstream service.

Logs (Structured, searchable) - Include a version label in every log line. - Centralise logs (Elasticsearch, Loki) so you can filter by version. - Log all request/response pairs for the canary during the evaluation window — helps with post-mortem.

Alerting (SLO-based prometheus rules) - Set up Prometheus rules that fire when canary metrics breach SLOs. - Alert should include the current traffic weight, version, and which metric breached. - Don't alert on every spike — use evaluation windows of at least 2 minutes.

Without these four pillars, you're guessing. Invest in observability before you invest in canary automation.

Running canary releases doubles the logging volume, tracing overhead, and metric cardinality. If you're on a pay-per-volume observability platform, expect your bill to temporarily increase. Budget for it and consider dropping low-value logs from canary instances if cost is a concern.

One more thing: make sure your dashboards are version-filtered from day one. If you aggregate metrics across versions, you'll see an average that hides the canary's true health.

Comparison dashboards: Create a dashboard that overlays canary vs stable latency percentiles on the same graph. This makes regressions immediately visible. Staring at two separate panels is slower — the eye catches divergence best when they share an axis.

Canary-specific dashboards: In Grafana, use dashboard variables for the version label so you can toggle between canary, stable, and combined views. This speeds up root cause analysis during incidents.

Real example: A team I worked with had a beautiful Grafana dashboard for their service, but it aggregated all requests into one line. When the canary introduced a 500ms latency spike, it was hidden in the aggregated p99. They didn't notice until a user complained. They now have a dedicated 'Canary View' showing only the canary's metrics overlaid on the stable baseline.

Failure scenario: Without tracing, a canary that causes a downstream service to fail will show up as errors on that downstream service, not on the canary itself. Tracing reveals the actual path. Debugging: Use kubectl port-forward svc/jaeger-query 16686:16686 to access the Jaeger UI and filter by version=canary. Performance impact: Adding distributed tracing adds ~2-5% overhead per request. For high-throughput services, use probabilistic sampling (e.g., 10%) during canary.

Additional insight: Consider using a canary-specific Prometheus recording rule that pre-calculates the delta between canary and stable metrics. This makes dashboards simpler and alerts faster.

groups:
- name: canary-alerts
  rules:
  - alert: CanaryHighErrorRate
    expr: |
      (sum(rate(http_requests_total{version="canary",status=~"5.."}[1m])) 
      / 
      sum(rate(http_requests_total{version="canary"}[1m]))) 
      > 0.01
    for: 2m
    labels:
      severity: critical
    annotations:
      summary: "Canary error rate above 1% for 2 minutes"
      description: "Canary version {{ $labels.version }} has error rate {{ $value | humanizePercentage }}"