AWS CodePipeline and CodeBuild: CI/CD on AWS

You've just pushed a hotfix to production. The build passes locally, but the pipeline fails silently because CodeBuild ran out of disk space mid-compilation. Your team scrambles for 45 minutes while customers hit errors. This isn't a tooling problem—it's a design problem. Most teams treat CI/CD as an afterthought, slapping together pipelines that work 90% of the time. The other 10%? That's where production outages live. AWS CodePipeline and CodeBuild are powerful, but they're not magic. Misconfigured artifact policies, oversized build environments, and poorly designed stage transitions are common failure points. This article cuts through the marketing fluff and shows you how to build pipelines that actually survive production. No analogies, no hand-holding—just battle-tested patterns from real AWS environments.

Why CI/CD on AWS Matters

Continuous integration and continuous delivery (CI/CD) are non-negotiable for modern software teams. On AWS, CodePipeline and CodeBuild provide a managed, serverless pipeline that integrates tightly with other AWS services. Unlike self-hosted Jenkins or GitLab runners, you pay only for what you use, and scaling is automatic. The key advantage is the elimination of infrastructure management—no patching build servers, no worrying about disk space. However, this convenience comes with trade-offs: debugging can be harder when you can't SSH into a build host, and pipeline execution costs can surprise you if you're not careful. This article walks through building a production-grade CI/CD pipeline using CodePipeline and CodeBuild, covering everything from basic setup to advanced patterns like approval gates and cross-account deployments.

Core Concepts: Pipelines, Stages, and Actions

A CodePipeline is a workflow that models your release process. It consists of stages (e.g., Source, Build, Test, Deploy) and each stage contains actions. Actions are the actual tasks—like pulling source from GitHub, running a build in CodeBuild, or deploying to ECS. Pipelines are triggered by events (e.g., a push to a branch) or manually. CodeBuild is the build service that compiles code, runs tests, and produces artifacts. It uses buildspec.yml to define commands. Key concepts: artifacts (input/output files passed between stages), environment variables, and service roles. Understanding the difference between pipeline-level and action-level variables is critical for avoiding hardcoded secrets. Always use AWS Secrets Manager or Parameter Store for sensitive values.

version: 0.2

phases:
  install:
    runtime-versions:
      nodejs: 18
    commands:
      - npm ci
  pre_build:
    commands:
      - npm run lint
      - npm test
  build:
    commands:
      - npm run build
  post_build:
    commands:
      - echo Build completed on `date`
artifacts:
  files:
    - '**/*'
  base-directory: dist
cache:
  paths:
    - node_modules/**/*

Setting Up Your First Pipeline

Start by creating a CodePipeline in the AWS Console or via CloudFormation. Choose a source provider: AWS CodeCommit, GitHub (via webhooks), or S3. For GitHub, you'll need to authorize AWS to access your repo. Next, add a build stage using CodeBuild. Create a CodeBuild project with a runtime (e.g., Ubuntu, Amazon Linux 2) and attach your buildspec.yml. Then add a deploy stage—common targets are ECS, Elastic Beanstalk, or Lambda. For this example, we'll deploy to S3 (static site). Important: ensure your IAM roles have least privilege. The CodePipeline service role needs permissions to invoke CodeBuild and read from S3. The CodeBuild role needs permissions to write to S3 and pull from ECR if using custom images. Test the pipeline by pushing a commit. Monitor execution in the console; failed stages show logs.

{
  "pipeline": {
    "name": "my-app-pipeline",
    "roleArn": "arn:aws:iam::123456789012:role/AWS-CodePipeline-Service",
    "artifactStore": {
      "type": "S3",
      "location": "codepipeline-us-east-1-123456789012"
    },
    "stages": [
      {
        "name": "Source",
        "actions": [
          {
            "name": "Source",
            "actionTypeId": {
              "category": "Source",
              "owner": "ThirdParty",
              "provider": "GitHub",
              "version": "1"
            },
            "configuration": {
              "Owner": "myorg",
              "Repo": "my-app",
              "Branch": "main",
              "OAuthToken": "***"
            },
            "outputArtifacts": [
              {
                "name": "source_output"
              }
            ]
          }
        ]
      },
      {
        "name": "Build",
        "actions": [
          {
            "name": "Build",
            "actionTypeId": {
              "category": "Build",
              "owner": "AWS",
              "provider": "CodeBuild",
              "version": "1"
            },
            "configuration": {
              "ProjectName": "my-app-build"
            },
            "inputArtifacts": [
              {
                "name": "source_output"
              }
            ],
            "outputArtifacts": [
              {
                "name": "build_output"
              }
            ]
          }
        ]
      },
      {
        "name": "Deploy",
        "actions": [
          {
            "name": "DeployToS3",
            "actionTypeId": {
              "category": "Deploy",
              "owner": "AWS",
              "provider": "S3",
              "version": "1"
            },
            "configuration": {
              "BucketName": "my-app-bucket",
              "Extract": "true"
            },
            "inputArtifacts": [
              {
                "name": "build_output"
              }
            ]
          }
        ]
      }
    ]
  }
}

Advanced Build Configurations with CodeBuild

CodeBuild supports custom build environments via Docker images. Use a pre-built image from ECR or Docker Hub to speed up builds. For example, a Node.js app can use a public image like node:18-alpine. You can also run multiple build environments in parallel using batch builds. Another powerful feature is local caching: cache directories like node_modules or .m2 to S3 to avoid re-downloading dependencies. Configure this in buildspec.yml under cache.paths. For large monorepos, use the buildspec's phases to conditionally run builds for changed services. Use environment variables to parameterize builds—for example, passing the branch name to set deployment targets. Always set the LOGS_CONFIG to export build logs to CloudWatch for long-term retention.

version: 0.2

phases:
  install:
    runtime-versions:
      java: corretto17
    commands:
      - echo Installing Maven dependencies
  pre_build:
    commands:
      - mvn dependency:resolve
  build:
    commands:
      - mvn package
  post_build:
    commands:
      - echo Build completed
artifacts:
  files:
    - target/*.jar
cache:
  paths:
    - '/root/.m2/**/*'
  type: S3
  location: my-cache-bucket

Managing Secrets and Environment Variables

Never hardcode secrets in buildspec.yml or pipeline definitions. Use AWS Systems Manager Parameter Store or Secrets Manager to store sensitive values like API keys, database passwords, or OAuth tokens. In CodeBuild, reference these as environment variables using the parameter-store or secrets-manager type. For example, in the CodeBuild project configuration, define an environment variable with type PARAMETER_STORE and value /myapp/db_password. CodeBuild will resolve it at build time. For pipeline-level secrets (e.g., GitHub tokens), use the Secret Manager action in the Source stage. Always restrict IAM permissions so that only the CodeBuild role can read specific parameters. Rotate secrets regularly and use CloudTrail to audit access.

{
  "environmentVariables": [
    {
      "name": "DB_PASSWORD",
      "type": "PARAMETER_STORE",
      "value": "/myapp/db_password"
    },
    {
      "name": "API_KEY",
      "type": "SECRETS_MANAGER",
      "value": "arn:aws:secretsmanager:us-east-1:123456789012:secret:myapp/api_key-abc123"
    }
  ]
}

Adding Approval Gates and Manual Interventions

Production deployments often require manual approval. CodePipeline supports approval actions that pause the pipeline until a user approves or rejects. Add an approval stage between Build and Deploy. In the action configuration, specify the SNS topic for notifications and the list of approvers (IAM users/groups). When the pipeline reaches this stage, it sends an email via SNS. Approvers can approve via the console or CLI. For advanced scenarios, use a custom action with Lambda to automate approval based on criteria (e.g., test coverage thresholds). Be careful: approval stages can delay deployments if approvers are slow. Set a timeout (e.g., 7 days) and have a fallback mechanism like auto-reject after timeout.

{
  "name": "Approval",
  "actions": [
    {
      "name": "ApproveOrReject",
      "actionTypeId": {
        "category": "Approval",
        "owner": "AWS",
        "provider": "Manual",
        "version": "1"
      },
      "configuration": {
        "NotificationArn": "arn:aws:sns:us-east-1:123456789012:approval-topic",
        "CustomData": "Deploy to production?",
        "ExternalEntityLink": "https://myapp.com/release-notes"
      },
      "inputArtifacts": []
    }
  ]
}

Cross-Account Deployments and Multi-Region Strategies

For enterprises, separating environments into different AWS accounts is a best practice. CodePipeline supports cross-account actions using resource sharing. For example, the build stage runs in the dev account, and the deploy stage runs in the prod account. To achieve this, create a CodePipeline in the dev account that uses a cross-account role in the prod account. The prod account must trust the dev account's IAM role. In the deploy action, specify the roleArn parameter. For multi-region deployments, you can have parallel actions in the same stage, each deploying to a different region. Use CloudFormation StackSets for consistent deployments across accounts and regions. Be mindful of artifact storage: artifacts must be in a bucket accessible by both accounts.

{
  "name": "DeployToProd",
  "actionTypeId": {
    "category": "Deploy",
    "owner": "AWS",
    "provider": "CloudFormation",
    "version": "1"
  },
  "configuration": {
    "ActionMode": "CREATE_UPDATE",
    "StackName": "my-app-prod",
    "TemplatePath": "build_output::template.yml",
    "RoleArn": "arn:aws:iam::PROD_ACCOUNT:role/CrossAccountDeployRole"
  },
  "inputArtifacts": [
    {
      "name": "build_output"
    }
  ]
}

Monitoring, Logging, and Failure Handling

Pipeline failures happen. CodePipeline integrates with CloudWatch Events to trigger notifications on state changes (e.g., pipeline failed, stage failed). Set up an SNS topic to email the team or post to Slack via AWS Chatbot. For detailed debugging, CodeBuild logs are sent to CloudWatch Logs by default. Enable detailed logging in the CodeBuild project to capture all commands. Use CloudWatch Logs Insights to query logs across multiple builds. For retries, configure the pipeline to automatically retry failed actions (e.g., up to 3 times). For transient failures (e.g., network timeouts), this is effective. For persistent failures, set up a Lambda function to analyze the error and take corrective action, like rolling back a deployment.

{
  "detail-type": ["CodePipeline Pipeline Execution State Change"],
  "source": ["aws.codepipeline"],
  "detail": {
    "state": ["FAILED"]
  }
}

Cost Optimization and Performance Tuning

CodeBuild costs are based on compute time and memory. To optimize, use the smallest compute type that meets your needs (e.g., BUILD_GENERAL1_SMALL for simple builds). Enable local caching to reduce build time and cost. For large projects, use batch builds to run tests in parallel. Monitor build minutes via AWS Cost Explorer. Set a budget alert for unexpected spikes. Another tip: use a custom Docker image with pre-installed dependencies to avoid installing them every build. For pipelines, minimize the number of stages and actions to reduce execution time. Use the pipeline's execution history to identify slow stages. Consider using CodePipeline's parallel actions to run independent tasks concurrently.

version: 0.2

phases:
  install:
    runtime-versions:
      python: 3.11
    commands:
      - pip install -r requirements.txt --no-cache-dir
  build:
    commands:
      - python -m pytest tests/
artifacts:
  files:
    - '**/*'
  base-directory: dist
cache:
  paths:
    - '/root/.cache/pip/**/*'

Security Best Practices for CI/CD Pipelines

Security in CI/CD is critical. Follow the principle of least privilege for IAM roles. The CodePipeline service role should only have permissions to invoke actions. The CodeBuild role should only have permissions to read source, write artifacts, and access specific secrets. Use KMS to encrypt artifacts at rest in S3. Enable encryption in transit by using HTTPS endpoints. For source repositories, require signed commits and use branch protection rules. Scan dependencies for vulnerabilities using tools like Snyk or AWS Inspector in the build phase. Regularly rotate secrets and audit IAM policies. Finally, enable AWS CloudTrail to log all API calls for forensic analysis.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject"
      ],
      "Resource": "arn:aws:s3:::codepipeline-*/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "ssm:GetParameter"
      ],
      "Resource": "arn:aws:ssm:us-east-1:123456789012:parameter/myapp/*"
    }
  ]
}

Testing the Pipeline: Canary Deployments and Rollbacks

Before full production rollout, test your pipeline with canary deployments. Use CodeDeploy with a canary configuration to shift a small percentage of traffic to the new version. Monitor metrics like error rate and latency. If metrics are healthy, the pipeline automatically shifts more traffic. If not, trigger a rollback. CodePipeline can integrate with CodeDeploy for this. Alternatively, use Lambda to implement custom canary logic. For rollbacks, store the previous artifact version and have a rollback stage that redeploys the last known good version. Automate rollback via CloudWatch alarms that trigger a pipeline execution. Always test rollback procedures in a non-production environment first.

{
  "deploymentConfigName": "Canary10Percent5Minutes",
  "minimumHealthyHosts": {
    "type": "HOST_COUNT",
    "value": 1
  },
  "trafficRoutingConfig": {
    "type": "TimeBasedCanary",
    "timeBasedCanary": {
      "canaryPercentage": 10,
      "canaryInterval": 5
    }
  }
}

Putting It All Together: A Production Pipeline Example

Let's assemble a complete production pipeline. Source: GitHub (main branch). Build: CodeBuild with Node.js 18, runs lint, tests, and builds. Artifacts stored in S3. Test: Run integration tests in a separate CodeBuild project. Approval: Manual approval via SNS. Deploy: CodeDeploy to ECS Fargate with canary traffic shifting. Post-deploy: Run smoke tests via Lambda. Rollback: If smoke tests fail, trigger a rollback stage that redeploys the previous artifact. Monitoring: CloudWatch Events send pipeline state changes to Slack. Security: Secrets from Parameter Store, KMS encryption, and IAM least privilege. This pipeline ensures every change is built, tested, approved, and deployed safely. It's fully automated except for the approval gate, which provides a safety net.

AWSTemplateFormatVersion: '2010-09-09'
Resources:
  MyPipeline:
    Type: AWS::CodePipeline::Pipeline
    Properties:
      Name: my-app-pipeline
      RoleArn: !GetAtt PipelineRole.Arn
      ArtifactStore:
        Type: S3
        Location: !Ref ArtifactBucket
      Stages:
        - Name: Source
          Actions:
            - Name: Source
              ActionTypeId:
                Category: Source
                Owner: ThirdParty
                Provider: GitHub
                Version: '1'
              Configuration:
                Owner: myorg
                Repo: my-app
                Branch: main
                OAuthToken: !Ref GitHubToken
              OutputArtifacts:
                - Name: source_output
        - Name: Build
          Actions:
            - Name: Build
              ActionTypeId:
                Category: Build
                Owner: AWS
                Provider: CodeBuild
                Version: '1'
              Configuration:
                ProjectName: !Ref BuildProject
              InputArtifacts:
                - Name: source_output
              OutputArtifacts:
                - Name: build_output
        - Name: Approval
          Actions:
            - Name: Approve
              ActionTypeId:
                Category: Approval
                Owner: AWS
                Provider: Manual
                Version: '1'
              Configuration:
                NotificationArn: !Ref ApprovalTopic
        - Name: Deploy
          Actions:
            - Name: DeployToECS
              ActionTypeId:
                Category: Deploy
                Owner: AWS
                Provider: CodeDeploy
                Version: '1'
              Configuration:
                ApplicationName: my-app
                DeploymentGroupName: prod
              InputArtifacts:
                - Name: build_output