Cosign and Image Signing: Stop Deploying Untrusted Containers

You're deploying 50 containers a day from your CI pipeline. You scan each image for vulnerabilities, run tests, and promote through staging to production. But how do you know the image rolling out to production right now is exactly the one your CI built? How do you prove it hasn't been tampered with in the registry, during transit, or by a compromised registry admin? That's where image signing comes in.

Image signing with Cosign provides cryptographic proof of an image's origin and integrity. A signature on an image proves: (1) who signed it (the signer's identity verified via OIDC), (2) when it was signed (timestamp from Fulcio certificate or Rekor entry), and (3) the image hasn't been altered since signing (the signature covers the image manifest digest). Combined with an admission controller (Kyverno), you can enforce that every image deployed to your cluster has a valid signature from an approved identity.

Cosign is part of the Sigstore project, which has become the de-facto standard for container image signing since 2023. It's used by all major cloud providers, integrated into GitHub Actions and GitLab CI, and supported by all major registries (Docker Hub, GHCR, Harbor, ECR, GCR, ACR). The key innovation is keyless signing: using OIDC and short-lived certificates instead of long-lived private keys, eliminating the key management burden that made previous signing tools (Notary v1, GPG-based signing) impractical.

This guide covers Cosign in production: keyless signing with Fulcio and Rekor, KMS integration for enterprise key control, SLSA provenance attestations, CI/CD integration with GitHub Actions OIDC, a comparison with Notary v2 (Notation), and Kyverno admission policies for enforcing signed images. By the end, you'll have a complete image signing pipeline that ensures every container in your cluster is verifiably from your CI pipeline and unmodified.

Sigstore Ecosystem: Fulcio, Rekor, and Cosign Explained

Sigstore is not just Cosign — it's three components that work together to provide a complete signing infrastructure. Understanding each component's role is essential before deploying production signatures.

Fulcio is the certificate authority (CA) that issues short-lived X.509 certificates for signing. When you run cosign sign, it authenticates you via OIDC (Google, GitHub, Microsoft, or any OIDC provider), requests a certificate from Fulcio, and Fulcio issues a certificate binding your OIDC identity to a temporary key pair. The certificate is valid for 10 minutes. Fulcio's root CA is publicly known, so anyone can verify that a certificate was issued by Sigstore's trusted CA.

Rekor is the transparency log — an append-only, immutable ledger of all signing events. When Cosign signs an image, it creates an entry in Rekor containing the signature, the certificate, and the image digest. This serves as a public record: anyone can query Rekor to verify 'was image X signed by identity Y on date Z?' The Rekor log is verifiable using Merkle tree proofs, similar to Certificate Transparency. Rekor provides the public audit trail that makes keyless signing work — even if the short-lived certificate expires, the Rekor entry proves the signing occurred.

Cosign is the CLI tool that orchestrates the signing workflow. It generates a temporary key pair, gets it signed by Fulcio, runs the actual signing operation (signing the image manifest digest), stores the signature in the registry as an OCI artifact (.sig tag), and submits the signing record to Rekor. On the verification side, Cosign reads the signature from the registry, fetches the certificate from Rekor (or from the signature artifact), and verifies the signature against Fulcio's root CA.

The data flow: (1) cosign sign → (2) OIDC authentication → (3) Fulcio cert issuance → (4) Image digest signature → (5) Signature pushed to registry (.sig) → (6) Entry created in Rekor. Verification: (1) cosign verify → (2) Fetch .sig from registry → (3) Fetch cert from Rekor → (4) Verify cert against Fulcio root → (5) Verify signature against image digest.

// io.thecodeforge — DevOps tutorial

# Install Cosign (macOS)
brew install cosign

# Linux: https://github.com/sigstore/cosign/releases

# Check Sigstore infrastructure status
curl -s -o /dev/null -w "%{http_code}" https://fulcio.sigstore.dev/api/v2/healthCheck
# Should return 200

curl -s -o /dev/null -w "%{http_code}" https://rekor.sigstore.dev/api/v1/log
# Should return 200

# Keyless sign an image (interactive, opens browser for OIDC)
export COSIGN_EXPERIMENTAL=1
cosign sign ghcr.io/myuser/myapp:latest

# Verify the signature
cosign verify ghcr.io/myuser/myapp:latest

# Check Rekor entry for the signing event
cosign verify --rekor-url https://rekor.sigstore.dev ghcr.io/myuser/myapp:latest

# Get the Rekor entry UUID
rekor-cli search --artifact sha256:<image-digest>

# View the Rekor entry details
rekor-cli get --uuid <entry-uuid>
# Shows: signature, certificate, image digest, timestamp, OIDC identity

# For air-gapped: use key-based signing instead
cosign generate-key-pair
cosign sign --key cosign.key ghcr.io/myuser/myapp:latest
cosign verify --key cosign.pub ghcr.io/myuser/myapp:latest

Cosign Sign and Verify: Keyless and Key-Based Workflows

Cosign supports two signing workflows: keyless (recommended, no key management) and key-based (for air-gapped environments or compliance requirements).

Keyless signing: cosign sign <image> triggers an OIDC flow. In interactive mode, it opens a browser. In CI/CD, it reads the OIDC token from the environment. GitHub Actions provides ACTIONS_ID_TOKEN_REQUEST_TOKEN automatically. GitLab CI provides CI_JOB_JWT_V2. The resulting signature is stored in the registry as <image>.sig. Verification: cosign verify <image> reads the signature, fetches the Fulcio certificate from Rekor, and verifies.

Key-based signing: cosign generate-key-pair creates cosign.key (private) and cosign.pub (public). cosign sign --key cosign.key <image> signs with the private key. cosign verify --key cosign.pub <image> verifies with the public key. For CI/CD, store the private key in a KMS (Azure Key Vault, AWS KMS, GCP KMS, HashiCorp Vault) and use --kms flag: cosign sign --key azurekms://mykeyvault/key <image>.

Verification options: cosign verify checks the signature is valid. Add --check-claims to verify OIDC identity claims in the certificate. Add --cert-email <email> to require a specific email. Add --cert-oidc-issuer <issuer> to verify the OIDC issuer. In Kyverno policies, these checks are configured as conditions.

Multiple signatures: an image can have multiple signatures from different signers (CI pipeline, security team, QA). cosign verify checks all signatures by default. Use --signature to check a specific signature tag.

Signature storage: Cosign stores signatures as OCI artifacts named <image>:<digest>.sig. Some registries require the --allow-insecure-registry flag. Harbor, GHCR, Docker Hub, and all major registries support Cosign signatures.

// io.thecodeforge — DevOps tutorial

# === KEYLESS WORKFLOW ===

# Sign (interactive)
export COSIGN_EXPERIMENTAL=1
cosign sign ghcr.io/myuser/myapp:latest

# Sign (GitHub Actions CI)
cosign sign \
  --oidc-issuer https://token.actions.githubusercontent.com \
  --oidc-client-id sigstore \
  ghcr.io/myuser/myapp:latest

# Verify
cosign verify ghcr.io/myuser/myapp:latest

# Verify with identity check
cosign verify \
  --cert-identity https://github.com/myuser/myapp/.github/workflows/ci.yml@refs/heads/main \
  --cert-oidc-issuer https://token.actions.githubusercontent.com \
  ghcr.io/myuser/myapp:latest

# === KEY-BASED WORKFLOW ===

# Generate keys
cosign generate-key-pair

# Sign with key
cosign sign --key cosign.key ghcr.io/myuser/myapp:latest

# Verify with public key
cosign verify --key cosign.pub ghcr.io/myuser/myapp:latest

# === KMS-BASED WORKFLOW ===

# Sign with Azure Key Vault
cosign sign --key azurekms://mykeyvault.vault.azure.net/mykey ghcr.io/myuser/myapp:latest

# Sign with AWS KMS
cosign sign --key awskms:///arn:aws:kms:us-east-1:123456789:key/mykey ghcr.io/myuser/myapp:latest

# Sign with GCP KMS
cosign sign --key gcpkms://projects/myproject/locations/global/keyRings/myring/cryptoKeys/mykey/versions/1 ghcr.io/myuser/myapp:latest

# Sign with HashiCorp Vault
cosign sign --key hashivault://mykey ghcr.io/myuser/myapp:latest

SLSA Provenance Attestations: Proving How Your Images Were Built

Image signing proves the image hasn't been tampered with, but it doesn't tell you HOW the image was built. SLSA (Supply-chain Levels for Software Artifacts) provenance attestations fill this gap by attaching verifiable metadata about the build process: the source repository, build platform, build command, builder identity, and build timestamp.

Cosign supports attaching in-toto attestations to images. The cosign attest command takes a predicate (the attestation payload) and attaches it as an OCI artifact (.att tag). The standard predicate for build provenance is SLSA, stored at slsa.dev/provenance/v1.

A SLSA provenance attestation includes: builder identity (e.g., https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@refs/tags/v2.0.0), source repository URL, source revision (commit SHA), build command, build platform (OS, architecture), and build timestamp. This enables downstream verifiers to check: 'Was this image built from the expected repository at the expected commit by the expected builder?'

Generating SLSA provenance: In GitHub Actions, use the slsa-framework/slsa-github-generator action to generate SLSA provenance for your build artifacts. The output is a signed SLSA provenance attestation. Attach it with cosign attest --predicate <slsa.json> --type slsa.dev/provenance/v1 <image>.

Verifying attestations: cosign verify-attestation --type slsa.dev/provenance/v1 <image> checks that the attestation is valid and signed. Kyverno can verify attestations at admission time: checking the source repository matches your allowed list, the builder matches your CI pipeline, and the build was triggered by an approved event.

SLSA levels: Level 1 (provenance exists), Level 2 (provenance is generated by a hosted build platform), Level 3 (provenance resists tampering with strong cryptographic guarantees). Cosign + GitHub Actions (with OIDC) achieves SLSA Level 3 because the attestation is signed by the OIDC identity, preventing tampering.

# io.thecodeforge — DevOps tutorial

name: Build and Sign with SLSA Provenance
on:
  push:
    branches: [main]

jobs:
  build:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read
      packages: write

    steps:
      - uses: actions/checkout@v4

      - name: Build Docker image
        run: docker build -t ghcr.io/${{ github.repository }}:${{ github.sha }} .

      - name: Push image
        run: docker push ghcr.io/${{ github.repository }}:${{ github.sha }}

      # Generate SLSA provenance using the official generator
      - name: Generate SLSA provenance
        uses: slsa-framework/slsa-github-generator/.github/actions/generate-provenance@v2.0.0
        id: provenance
        with:
          subject-name: ghcr.io/${{ github.repository }}
          subject-digest: sha256:$(docker inspect --format='{{.RepoDigests}}' ghcr.io/${{ github.repository }}:${{ github.sha }} | grep -o 'sha256:[^"]*' | cut -d: -f2)

      - name: Install Cosign
        uses: sigstore/cosign-installer@v3

      - name: Sign image and attach SLSA provenance
        env:
          COSIGN_EXPERIMENTAL: 1
        run: |
          cosign sign ghcr.io/${{ github.repository }}:${{ github.sha }}
          cosign attest \
            --predicate ${{ steps.provenance.outputs.provenance-path }} \
            --type slsa.dev/provenance/v1 \
            ghcr.io/${{ github.repository }}:${{ github.sha }}

      # Tag as latest only after signing and provenance
      - name: Tag image
        run: |
          cosign copy ghcr.io/${{ github.repository }}:${{ github.sha }} \
                    ghcr.io/${{ github.repository }}:latest

KMS Integration: Azure, AWS, GCP, and HashiCorp Vault

For organizations that require centralized key management (compliance, air-gapped environments, or existing KMS infrastructure), Cosign supports signing and verification keys stored in hardware security modules (HSMs) and cloud KMS providers.

Cosign's KMS integration uses the --key <kms-provider>://<key-path> flag. The KMS holds the private key; Cosign requests signing operations from the KMS without the key ever leaving the HSM. This satisfies compliance requirements where private keys must never be stored on disk.

Azure Key Vault: cosign sign --key azurekms://mykeyvault.vault.azure.net/mykey version <image>. Prerequisites: Azure CLI login, key in AKV with sign/verify permissions. Cosign uses the Azure SDK for authentication.

AWS KMS: cosign sign --key awskms:///arn:aws:kms:us-east-1:123456789:key/mykey <image>. Use a KMS key with Sign and Verify usage. Cosign supports asymmetric keys (RSA_2048, ECC_NIST_P256, ECC_SECG_P256K1).

GCP KMS: cosign sign --key gcpkms://projects/myproject/locations/global/keyRings/myring/cryptoKeys/mykey/versions/1 <image>. Requires Cloud KMS API enabled and appropriate IAM permissions.

HashiCorp Vault: cosign sign --key hashivault://mykey <image>. Configure Vault's transit engine with a key named mykey. Cosign uses VAULT_ADDR and VAULT_TOKEN for authentication.

KMS verification: cosign verify --key <kms-provider>://<key-path> <image>. Verification can be done by anyone with access to the KMS public key (no secret required).

Best practice: use KMS for key-based signing in regulated environments. Use automatic key rotation features (AWS KMS automatic rotation, GCP KMS rotation period). Audit all signing operations via KMS CloudTrail (AWS) or Audit Logs (GCP).

// io.thecodeforge — DevOps tutorial

# === Azure Key Vault ===
# Prerequisites: az login, key in AKV
az login

# Create a key in AKV
az keyvault key create --vault-name mykeyvault --name mykey --kty RSA --size 3072

# Sign with Azure KMS
cosign sign --key azurekms://mykeyvault.vault.azure.net/mykey ghcr.io/myuser/myapp:latest

# Verify
cosign verify --key azurekms://mykeyvault.vault.azure.net/mykey ghcr.io/myuser/myapp:latest

# === AWS KMS ===
# Create KMS key (asymmetric, sign/verify)
aws kms create-key --key-usage SIGN_VERIFY --customer-master-key-spec RSA_3072

# Sign with AWS KMS
cosign sign --key awskms:///arn:aws:kms:us-east-1:123456789:key/mykey-id ghcr.io/myuser/myapp:latest

# Verify
cosign verify --key awskms:///arn:aws:kms:us-east-1:123456789:key/mykey-id ghcr.io/myuser/myapp:latest

# === GCP KMS ===
# Create key ring and key
gcloud kms keyrings create myring --location global
gcloud kms keys create mykey --location global --keyring myring --purpose asymmetric-signing --default-algorithm rsa-sign-pkcs1-3072-sha256

# Sign with GCP KMS
cosign sign --key gcpkms://projects/myproject/locations/global/keyRings/myring/cryptoKeys/mykey/versions/1 ghcr.io/myuser/myapp:latest

# Verify
cosign verify --key gcpkms://projects/myproject/locations/global/keyRings/myring/cryptoKeys/mykey/versions/1 ghcr.io/myuser/myapp:latest

# === HashiCorp Vault ===
# Prerequisites: Vault transit engine enabled, key created
# vault secrets enable transit
# vault write -f transit/keys/mykey type=rsa-3072

# Sign with Vault
export VAULT_ADDR=https://vault.example.com:8200
cosign sign --key hashivault://mykey ghcr.io/myuser/myapp:latest

# Verify
cosign verify --key hashivault://mykey ghcr.io/myuser/myapp:latest

Cosign vs Notary v2 (Notation): Choosing the Right Signing Tool

Both Cosign and Notation sign container images, but they differ in architecture, key management, and ecosystem integration. Choosing between them depends on your organization's key management infrastructure and compliance requirements.

Cosign (Sigstore) uses OCI artifact-based signatures (.sig tags). Signatures are stored as separate artifacts in the registry, referenced by the image manifest digest. Supports keyless signing via Fulcio/OIDC and key-based signing with KMS. Has built-in attestation support for vulnerability reports, SLSA provenance, and SBOMs. Integrates natively with Kyverno admission policies. Large ecosystem: all major CI/CD platforms support it, all major registries store its signatures.

Notation (Notary v2) uses OCI 1.1 referrers API to embed signatures directly in the OCI manifest as referrers. Uses X.509 certificates managed via a trust store and trust policy. Designed for enterprise PKI environments where X.509 is already the standard. Requires certificate management (CAs, certificates, CRLs). No keyless mode — requires actual certificates. Integrates with Kyverno (via notations attestors).

Comparison: Cosign is easier to get started with (keyless, no certificate management) and better for cloud-native teams. Notation is better for enterprise PKI environments with existing certificate infrastructure. Cosign's attestation support is more mature. Notation's signature storage (referrers API) is more OCI-native and doesn't create separate tags.

Recommendation: Start with Cosign keyless for most teams. Switch to Notation only if: you need X.509 PKI integration (existing CA), OCI-native signature storage (no .sig tags), or compliance requirements mandate X.509 certificates. Both can be used simultaneously (Harbor and Kyverno support both).

// io.thecodeforge — DevOps tutorial

# === Notation (Notary v2) Setup ===

# Install Notation
# macOS: brew install notation

# Generate test certificate
notation cert generate-test --default "registry.example.com"

# Sign image
notation sign --key "registry.example.com" registry.example.com/myapp:latest

# Verify
notation verify registry.example.com/myapp:latest

# List signatures
notation ls registry.example.com/myapp:latest

# === Kyverno Policy for Notation ===
# apiVersion: kyverno.io/v2beta1
# kind: ClusterPolicy
# metadata:
#   name: require-notation-signature
# spec:
#   rules:
#     - name: verify-notation
#       match:
#         any:
#           - resources:
#               kinds:
#                 - Pod
#       verifyImages:
#         - imageReferences:
#             - "registry.example.com/*"
#           attestors:
#             - entries:
#                 - notations:
#                     certs: "|"  # trust store reference
#                     # Trust policy defines which CAs are trusted

Kyverno Admission Control: Enforcing Signed Images at Deploy Time

An image signature is only useful if you check it before deployment. Kyverno is the Kubernetes admission controller that verifies Cosign signatures and attestations at pod creation time, blocking unsigned or incorrectly-signed images.

Kyverno's verifyImages rule configuration: specify the image reference pattern (e.g., ghcr.io/myrepo/*), the attestor (Cosign public key, KMS key, or keyless), and optional conditions (check OIDC issuer, subject, or certificate attributes). When a pod references an image matching the pattern, Kyverno pulls the image's Cosign signature from the registry, verifies it, and only admits the pod if verification succeeds.

Keyless verification in Kyverno: use verifyImages with attestors: entries: keys: kms: to trigger keyless verification. Kyverno connects to Fulcio and Rekor to verify the signature. Add cert-identity and cert-oidc-issuer to require specific OIDC identities.

Key-based verification: store the public key in a Kubernetes secret and reference it: keys: publicKeys: k8s://namespace/secret-name. Kyverno reads the secret and uses it for verification.

Best practices: (1) Use failureAction: Enforce for production namespaces, Audit for dev. (2) Verify both signature AND identity: check cert-oidc-issuer matches your CI platform. (3) Exclude system images (kube-system, etc.) from verification. (4) Use image reference patterns to apply different policies to different registries. (5) For migration, start with Audit mode: it reports violations without blocking.

# io.thecodeforge — DevOps tutorial

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-image-signature
spec:
  validationFailureAction: Enforce
  rules:
    - name: verify-cosign-keyless
      match:
        any:
          - resources:
              kinds:
                - Pod
      verifyImages:
        - imageReferences:
            - "ghcr.io/myorg/*"
            - "registry.example.com/*"
          attestors:
            - entries:
                - keyless:
                    # Require signing by specific OIDC identity
                    subject: "https://github.com/myorg/myapp/.github/workflows/ci.yml@refs/heads/main"
                    issuer: "https://token.actions.githubusercontent.com"
          # Optional: also verify SLSA provenance attestation
          attestations:
            - predicateType: https://slsa.dev/provenance/v1
              conditions:
                - expression: "predicate.buildDefinition.buildType == 'https://actions.github.io/build'"

---
# Policy for key-based verification (using KMS)
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-image-kms
spec:
  validationFailureAction: Enforce
  rules:
    - name: verify-cosign-kms
      match:
        any:
          - resources:
              kinds:
                - Pod
      verifyImages:
        - imageReferences:
            - "ghcr.io/myorg/*"
          attestors:
            - entries:
                - keys:
                    kms: "azurekms://mykeyvault.vault.azure.net/mykey"