Cloud Deploy: Delivery Pipelines, Rollbacks, and Skaffold Integration

You've built a perfect CI pipeline. Code merges, tests pass, artifacts are built. Then your deployment to production silently breaks the checkout flow, and you spend 45 minutes manually reverting a Helm chart. That's the gap Cloud Deploy fills: it's not just about pushing artifacts—it's about orchestrating the rollout with gated promotions, automated rollbacks, and integration with Skaffold to keep your build and deploy in sync. Most teams treat deployment as an afterthought; the ones that don't sleep better. Cloud Deploy forces you to define explicit delivery pipelines with verification steps, so a bad release never reaches production without a fight. This isn't a CI/CD overview—it's a deep dive into how to structure your delivery pipeline for reliability, using Cloud Deploy and Skaffold together.

The Anatomy of a Delivery Pipeline

A delivery pipeline in Cloud Deploy is a series of targets (e.g., dev, staging, prod) connected by promotion rules. Each target can have a verification mechanism—like a Cloud Build test or a custom Skaffold deploy—that must pass before promotion. The pipeline is defined in a YAML file that references Skaffold configurations for each target. This separation of concerns means your build logic stays in Skaffold, while Cloud Deploy handles the orchestration and rollback. A common mistake is to treat the pipeline as a linear sequence; instead, design it with parallel verification steps and approval gates for production. For example, you might have a 'canary' target that runs a subset of traffic before full rollout. The pipeline definition is immutable after creation, so plan your target names and ordering carefully.

apiVersion: deploy.cloud.google.com/v1
kind: DeliveryPipeline
metadata:
  name: my-app-pipeline
serialPipeline:
  stages:
  - targetId: dev
    profiles: []
  - targetId: staging
    profiles: [staging]
  - targetId: prod
    profiles: [prod]
---
apiVersion: deploy.cloud.google.com/v1
kind: Target
metadata:
  name: dev
description: Development cluster
gke:
  cluster: projects/my-project/locations/us-central1/clusters/dev-cluster
---
apiVersion: deploy.cloud.google.com/v1
kind: Target
metadata:
  name: staging
description: Staging cluster
gke:
  cluster: projects/my-project/locations/us-central1/clusters/staging-cluster
---
apiVersion: deploy.cloud.google.com/v1
kind: Target
metadata:
  name: prod
description: Production cluster
requireApproval: true
gke:
  cluster: projects/my-project/locations/us-central1/clusters/prod-cluster

Skaffold as the Deploy Engine

Skaffold is the tool that Cloud Deploy uses to actually run the deployment. It handles building container images, applying Kubernetes manifests, and managing the deploy lifecycle. In the context of Cloud Deploy, Skaffold runs in a Cloud Build step triggered by a release. The Skaffold configuration (skaffold.yaml) defines the build and deploy steps, and Cloud Deploy passes environment-specific profiles (e.g., dev, staging) to customize the deployment. This integration means you can test your deployment locally with Skaffold and then use the same configuration in production. A critical detail: Cloud Deploy uses Skaffold's 'render' and 'apply' commands, so your skaffold.yaml must support those. Avoid using custom build steps that aren't idempotent—Cloud Deploy may retry deployments on failure.

apiVersion: skaffold/v4beta6
kind: Config
build:
  artifacts:
  - image: us-central1-docker.pkg.dev/my-project/my-repo/my-app
    docker:
      dockerfile: Dockerfile
  tagPolicy:
    gitCommit: {}
deploy:
  kubectl:
    manifests:
    - k8s/*.yaml
profiles:
  - name: staging
    patches:
      - op: replace
        path: /deploy/kubectl/manifests
        value: ["k8s/overlays/staging/*.yaml"]
  - name: prod
    patches:
      - op: replace
        path: /deploy/kubectl/manifests
        value: ["k8s/overlays/prod/*.yaml"]

Creating a Release and Promoting

A release in Cloud Deploy is a snapshot of the source code and Skaffold configuration at a point in time. You create a release from a specific commit or tag, and Cloud Deploy automatically renders the manifests and deploys to the first target (e.g., dev). Promotion moves the release to the next target, optionally requiring approval. The promotion can be manual or automated via Cloud Build triggers. A key feature is the ability to specify a 'skaffold version' per release, ensuring reproducibility. When promoting, Cloud Deploy re-renders the manifests using the target's profile, so the same release can produce different manifests for different environments. This is powerful but can lead to surprises if your profiles are not consistent.

gcloud deploy releases create my-release-001 \
  --delivery-pipeline=my-app-pipeline \
  --region=us-central1 \
  --source=./ \
  --skaffold-version=2.0.0 \
  --images=my-app=us-central1-docker.pkg.dev/my-project/my-repo/my-app:latest

Automated Rollbacks on Failure

Cloud Deploy can automatically roll back a deployment if the verification step fails. Verification is defined per target and can be a Cloud Build test, a custom Skaffold deploy, or a Cloud Deploy 'verify' job. When a rollout fails verification, Cloud Deploy automatically rolls back to the previous successful rollout. This is not a simple revert—it re-applies the previous release's manifests. The rollback is atomic and audited. However, automatic rollback is not a substitute for proper canary analysis; it only triggers on explicit verification failures. If your verification is weak (e.g., a simple HTTP check that passes even when the app is broken), the rollback won't save you. Design verification to be as close to production traffic as possible.

apiVersion: deploy.cloud.google.com/v1
kind: Target
metadata:
  name: prod
description: Production cluster
requireApproval: true
gke:
  cluster: projects/my-project/locations/us-central1/clusters/prod-cluster
verification:
  cloudBuild:
    serviceAccount: projects/my-project/serviceAccounts/verify-sa@my-project.iam.gserviceaccount.com
    workerPool: projects/my-project/locations/us-central1/workerPools/verify-pool
    config:
      steps:
      - name: 'gcr.io/cloud-builders/gcloud'
        entrypoint: 'bash'
        args:
        - '-c'
        - |
          gcloud deploy rollouts list --delivery-pipeline=my-app-pipeline --release=my-release-001 --region=us-central1 --filter="state=SUCCEEDED" --format="value(name)" | head -1

Manual Rollbacks and Rollforward

Sometimes automatic rollback isn't enough—you need to manually roll back to a specific release or roll forward to a fix. Cloud Deploy supports manual rollback by creating a new rollout from a previous release. You can also 'roll forward' by creating a new release with the fix and promoting it. The key is that rollbacks are not destructive; they create a new rollout that applies the old manifests. This means you can roll back multiple times. However, if your database schema has changed, a rollback might not be safe. Always test rollbacks in staging first. A common pattern is to keep the last N releases available for quick rollback.

# Roll back to a previous release
gcloud deploy rollouts create rollback-001 \
  --delivery-pipeline=my-app-pipeline \
  --release=my-release-001 \
  --region=us-central1 \
  --target=prod

# Or roll forward with a new release
gcloud deploy releases create my-release-002 \
  --delivery-pipeline=my-app-pipeline \
  --region=us-central1 \
  --source=./ \
  --images=my-app=us-central1-docker.pkg.dev/my-project/my-repo/my-app:fixed

Canary Deployments with Cloud Deploy

Cloud Deploy supports canary deployments by defining multiple targets for the same environment (e.g., prod-canary, prod-stable). You promote to the canary target first, run verification, and then promote to stable. This is not a native canary analysis (like traffic splitting), but a sequential promotion. For true canary analysis, you need to integrate with a service mesh or Cloud Deploy's rollout strategies. Cloud Deploy also supports 'deployment strategies' like 'canary' and 'blue-green' via Skaffold's custom deploy. However, the simplest approach is to have a separate canary cluster and use Cloud Deploy's promotion gates to control traffic. This pattern is common for high-risk deployments.

serialPipeline:
  stages:
  - targetId: dev
  - targetId: staging
  - targetId: prod-canary
  - targetId: prod-stable
---
apiVersion: deploy.cloud.google.com/v1
kind: Target
metadata:
  name: prod-canary
gke:
  cluster: projects/my-project/locations/us-central1/clusters/prod-canary-cluster
---
apiVersion: deploy.cloud.google.com/v1
kind: Target
metadata:
  name: prod-stable
gke:
  cluster: projects/my-project/locations/us-central1/clusters/prod-stable-cluster

Integrating with Cloud Build for CI

Cloud Deploy is often triggered by Cloud Build after a successful CI pipeline. The typical flow: Cloud Build builds the image, runs tests, then creates a Cloud Deploy release and promotes it to dev. This is done using the gcloud deploy commands in a Cloud Build step. You can also use Cloud Build to run verification steps. A common mistake is to put too much logic in Cloud Build; keep it simple: build, test, release. The release creation should be a separate step that can be retried independently. Use Cloud Build's substitution variables to pass the image tag and commit SHA.

steps:
- name: 'gcr.io/cloud-builders/docker'
  args: ['build', '-t', 'us-central1-docker.pkg.dev/$PROJECT_ID/my-repo/my-app:$COMMIT_SHA', '.']
- name: 'gcr.io/cloud-builders/docker'
  args: ['push', 'us-central1-docker.pkg.dev/$PROJECT_ID/my-repo/my-app:$COMMIT_SHA']
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
  entrypoint: 'bash'
  args:
  - '-c'
  - |
    gcloud deploy releases create release-$BUILD_ID \
      --delivery-pipeline=my-app-pipeline \
      --region=us-central1 \
      --source=. \
      --images=my-app=us-central1-docker.pkg.dev/$PROJECT_ID/my-repo/my-app:$COMMIT_SHA
substitutions:
  _REGION: us-central1

Handling Multi-Environment Configurations

Managing environment-specific configurations (e.g., different replica counts, secrets, or feature flags) is a common challenge. Cloud Deploy handles this through Skaffold profiles and Kustomize overlays. Each target can have a different profile that applies patches to the base manifests. For secrets, use Google Cloud Secret Manager and reference them in your manifests via environment variables or volume mounts. Avoid storing secrets in the repository. A robust pattern is to have a base set of manifests and overlay directories for each environment. Cloud Deploy will render the correct manifests based on the target's profile.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
patches:
- path: replica-count.yaml
  target:
    kind: Deployment
    name: my-app
configMapGenerator:
- name: app-config
  literals:
  - ENV=staging
secretGenerator:
- name: app-secrets
  envs:
  - secrets.env

Monitoring and Auditing Deployments

Cloud Deploy integrates with Cloud Logging and Cloud Monitoring to provide visibility into deployments. Each rollout creates log entries that show the state changes, verification results, and any errors. You can set up alerts on rollout failures or rollbacks. For auditing, Cloud Deploy records who created releases and approved promotions. This is essential for compliance. A common gap is not monitoring the verification step itself—if the verification fails silently, you won't know until the rollback happens. Use Cloud Monitoring to track rollout duration and success rate.

gcloud deploy rollouts list \
  --delivery-pipeline=my-app-pipeline \
  --release=my-release-001 \
  --region=us-central1 \
  --format="table(name, state, createTime)"

Advanced: Custom Deploy Strategies with Skaffold

For complex deployments (e.g., blue-green, canary with traffic splitting), you can use Skaffold's custom deploy actions. Cloud Deploy supports custom Skaffold deploy by specifying a 'deploy' section in the skaffold.yaml that runs a script. This script can interact with a service mesh like Istio or use Cloud Deploy's rollout strategies. However, this adds complexity and should be used only when necessary. A simpler approach is to use Cloud Deploy's built-in 'canary' strategy, which is available for GKE targets. This strategy automatically creates a canary deployment and scales it up while monitoring.

deploy:
  custom:
    - name: blue-green
      deployCommand:
        - bash
        - -c
        - |
          kubectl apply -f blue-deployment.yaml
          kubectl apply -f green-deployment.yaml
          # switch service selector
          kubectl patch service my-app -p '{"spec":{"selector":{"version":"blue"}}}'

Canary Deployment Strategy: Phased Rollouts

Cloud Deploy supports canary deployments with phased rollouts. A canary rollout deploys to a percentage of traffic first (e.g., 25%), then advances to 50%, then 100% (stable). Each phase has a deploy job and optional verify job. The phases are named canary-25, canary-50, and stable. On the first deployment to a target (no existing version), the canary phases are skipped and the app deploys directly to stable. After that, canary deployments work as expected. You can configure custom percentages and custom-automated canaries. Production insight: always enable verification on canary phases. We deployed a canary that passed the deploy step but failed the verify step due to elevated error rates. The rollout was automatically halted before reaching 100%. This caught a regression that would have affected all users.

apiVersion: deploy.cloud.google.com/v1
kind: DeliveryPipeline
metadata:
  name: my-app-canary-pipeline
serialPipeline:
  stages:
  - targetId: dev
  - targetId: staging
  - targetId: prod
    strategy:
      canary:
        canaryDeployment:
          percentages: [25, 50, 100]
          verify: true
---
apiVersion: deploy.cloud.google.com/v1
kind: Target
metadata:
  name: prod
gke:
  cluster: projects/my-project/locations/us-central1/clusters/prod-cluster
requireApproval: true

Parallel Deployment: Multi-Target Simultaneous Rollouts

Cloud Deploy supports parallel deployment, allowing you to deploy to multiple targets simultaneously (e.g., deploy to us-central1 and europe-west1 at the same time). This is configured using a multiTarget type or by defining parallel stages in the pipeline. Parallel deployment reduces rollout time for multi-region applications. However, it increases risk—a failure in one region can affect the entire deployment. Use parallel deployment for stateless, region-independent services. For stateful or region-coupled services, use sequential deployment. Production insight: we use parallel deployment for our CDN edge service—it must be updated in all regions simultaneously to avoid cache inconsistency. For the database migration service, we use sequential deployment to avoid split-brain scenarios.

apiVersion: deploy.cloud.google.com/v1
kind: DeliveryPipeline
metadata:
  name: multi-region-pipeline
serialPipeline:
  stages:
  - targetId: staging
  - profiles: []
    strategy:
      standard:
        verify: false
  - targetId: prod-us
  - targetId: prod-eu
---
apiVersion: deploy.cloud.google.com/v1
kind: Target
metadata:
  name: prod-us
gke:
  cluster: projects/my-project/locations/us-central1/clusters/prod-cluster
---
apiVersion: deploy.cloud.google.com/v1
kind: Target
metadata:
  name: prod-eu
gke:
  cluster: projects/my-project/locations/europe-west1/clusters/prod-cluster

Cloud Run Deployer with Skaffold

Cloud Deploy now supports Cloud Run as a target runtime (in addition to GKE and Anthos). This is configured via Skaffold's Cloud Run deployer, which uses the serving.knative.dev/v1 schema. The setup is similar to GKE: you define a delivery pipeline with targets, and Cloud Deploy uses Skaffold's Cloud Run deployer to apply the manifests. This enables progressive delivery (canary) and rollbacks for Cloud Run services. Production insight: Cloud Run's automatic scaling combined with Cloud Deploy's canary deployment creates a powerful combination. We deploy a new revision to 25% of traffic, run verification tests, then promote to 100%. If the verification fails, Cloud Deploy rolls back automatically. This is much simpler than manual traffic splitting.

apiVersion: skaffold/v4beta6
kind: Config
deploy:
  cloudrun:
    projectid: my-project
    region: us-central1
manifests:
  rawYaml:
    - service.yaml
profiles:
  - name: prod
    manifests:
      rawYaml:
        - service-prod.yaml

Troubleshooting Common Failures

Common failures in Cloud Deploy include: permission errors (the Cloud Deploy service account lacks permissions to deploy to the cluster), Skaffold rendering errors (e.g., invalid YAML), and verification timeouts. The first step is to check the rollout logs in Cloud Logging. For Skaffold errors, run skaffold render locally with the same profile. For permission issues, ensure the Cloud Deploy service account has the 'container.developer' role on the cluster. Another common issue is that the release source does not include the skaffold.yaml file—Cloud Deploy requires it at the root of the source directory.

# Get rollout details
gcloud deploy rollouts describe rollout-abc \
  --delivery-pipeline=my-app-pipeline \
  --release=my-release-001 \
  --region=us-central1

# Check logs
kubectl logs -n my-namespace deployment/my-app --tail=50