CI/CD Silent Failure — Expired Docker Credentials

CI/CD is the backbone of modern software delivery. It replaces the old model of big-bang releases with a continuous flow of small, validated changes. The core idea: every code change goes through an automated pipeline that builds, tests, and — optionally — deploys it. This isn't a luxury; teams that skip CI/CD spend 2x to 3x more time resolving integration conflicts and debugging production failures. Here's the catch: a poorly designed pipeline can be worse than no pipeline — it can give false confidence. This guide covers what CI/CD actually means, how to build one that works, and the production failures you'll face if you don't.

A GitHub Actions CI Pipeline

This GitHub Actions workflow runs on every push to main/develop and on pull requests. It sets up Python, caches pip, installs dependencies, runs ruff linter, runs pytest with coverage, and uploads results. The cache step is critical: without it, each pipeline run downloads all packages from scratch, turning a 2-minute install into a 10-minute one.

# .github/workflows/ci.yml
name: CI

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.12'

      - name: Cache pip packages
        uses: actions/cache@v4
        with:
          path: ~/.cache/pip
          key: ${{ runner.os }}-pip-${{ hashFiles('requirements.txt') }}

      - name: Install dependencies
        run: pip install -r requirements.txt

      - name: Run linter
        run: ruff check .

      - name: Run tests with coverage
        run: pytest --cov=. --cov-report=xml

      - name: Upload coverage
        uses: codecov/codecov-action@v4

Adding CD — Automatic Deployment

The deploy job runs after the test job passes, only on main. It builds a Docker image, tags it with the commit SHA, pushes to a registry, then SSHes to a staging server and redeploys using docker compose. This is the core of Continuous Deployment to staging — a human still gates production deployments?

# Add to the same file — deploy job runs after test passes
  deploy:
    needs: test        # only run if test job passes
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'  # only deploy from main

    steps:
      - uses: actions/checkout@v4

      - name: Build Docker image
        run: |
          docker build -t myapp:${{ github.sha }} .
          docker tag myapp:${{ github.sha }} myapp:latest

      - name: Push to registry
        run: |
          echo ${{ secrets.DOCKER_PASSWORD }} | docker login -u ${{ secrets.DOCKER_USERNAME }} --password-stdin
          docker push myapp:${{ github.sha }}
          docker push myapp:latest

      - name: Deploy to staging
        run: |
          # SSH to server and pull new image
          ssh -o StrictHostKeyChecking=no deploy@${{ secrets.STAGING_HOST }} \
            "docker pull myapp:latest && docker compose up -d"

CI/CD Pipeline Stages and Their Purpose

Every CI/CD pipeline follows a set of stages that gate each other. The typical flow: lint → unit test → build → integration test → deploy → smoke test. Each stage acts as a safety net. Lint catches formatting and logic errors fast. Unit tests validate individual functions. Build produces the artifact. Integration tests confirm the artifact works in a real environment. Deploy publishes it. Smoke test verifies the service is alive.

The order matters — you want the fastest checks first so failures are caught early, without wasting time on slower stages.

name: CI/CD Pipeline

on: [push]

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - run: npm run lint
  test:
    needs: lint
    runs-on: ubuntu-latest
    steps:
      - run: npm test
  build:
    needs: test
    runs-on: ubuntu-latest
    steps:
      - run: docker build -t myapp .
  deploy:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - run: ./deploy.sh

Continuous Delivery vs Continuous Deployment — The Trade-off

Continuous Delivery means every successful build produces a deployable artifact, but a human decides when to push it to production. Continuous Deployment goes all the way — each successful build is automatically deployed to production. The choice depends on your risk tolerance and deployment processes.

Continuous Delivery is safer for regulated industries or when you need a manual QA sign-off. Continuous Deployment is faster for teams with strong automated testing and rollback capability. The real question: can you detect and fix a bad deploy in under 5 minutes? If not, start with Delivery, not Deployment.

CI/CD Best Practices for Production Teams

Over years of building and debugging pipelines, these practices separate effective CI/CD from pipelines that cause more harm than good:

1. Fast feedback — Keep the pipeline under 10 minutes. Long pipelines discourage frequent pushes. Split long tests into separate workflows or parallelise.
2. Idempotent steps — Every step should produce the same result given the same input. Avoid steps that depend on global state or mutable external resources.
3. Secrets management — Never hardcode secrets. Use encrypted environment variables (GitHub Actions secrets, GitLab CI variables, etc.) and rotate them regularly.
4. Health checks after deploy — Deploy is not complete until the new version responds correctly. Add a curl or similar check in the deploy job.
5. Rollback capability — Every deploy should be rollback-able. Tag Docker images with commit SHA so you can redeploy a known-good version.
6. Pipeline as code — Version your pipeline definitions alongside your code. This makes changes reviewable and traceable.

Common CI/CD Pipeline Failures and How to Debug Them

Even well-designed pipelines fail. The most common failures fall into a few categories: - Environment drift: Your pipeline uses a Docker image that is updated upstream, breaking your build. Pin base image versions. - Cache poisoning: An old cache contains corrupt or outdated dependencies. Clear cache periodically. - Flaky tests: Tests that pass locally but fail in CI due to timing or order dependence. Use --reruns or retry strategy. - Secret expiration: Tokens or passwords expire. Automate rotation and alert on failure. - Resource exhaustion: Disk space or memory runs out during build. Add cleanup steps and monitor usage.