Home DevOps Harbor Registry: The Production-Grade Docker Registry You Should Be Using
Advanced 5 min · July 11, 2026

Harbor Registry: The Production-Grade Docker Registry You Should Be Using

Harbor on Kubernetes via Helm, HA with Patroni/CloudNativePG, Redis Sentinel/Cluster, Trivy integration with --with-trivy, Cosign/Notation content trust, OCI artifact management (Helm charts, OPA bundles, SBOMs), garbage collection, replication, and vulnerability scanning at registry level..

N
Naren Founder & Principal Engineer

20+ years shipping production infrastructure and CI/CD at scale. Drawn from code that ran under real load.

Follow
Production
production tested
July 11, 2026
last updated
1,787
articles · all by Naren
Before you start⏱ 30 min
  • Kubernetes basics (Helm, kubectl)
  • Docker registry concepts (push, pull, tags)
  • PostgreSQL and Redis familiarity
  • Basic understanding of container image signing and vulnerability scanning
 ● Production Incident 🔎 Debug Guide
Quick Answer

Deploy Harbor on Kubernetes with: helm repo add harbor https://helm.goharbor.io && helm install harbor harbor/harbor --set expose.type=loadBalancer --set persistence.enabled=true --set trivy.enabled=true. Access the UI at the LoadBalancer IP, login with admin/Harbor12345, create a project, and push images: docker login && docker tag myapp /library/myapp:latest && docker push /library/myapp:latest. Harbor automatically scans pushed images with Trivy. Enable content trust with Cosign: cosign sign --key cosign.key /library/myapp:latest. Configure replication to mirror images to a secondary Harbor or cloud registry. Set up garbage collection to reclaim storage from deleted images.

✦ Definition~90s read
What is Harbor Registry?

Harbor is an open-source cloud-native container registry that provides security, identity, and management capabilities beyond a basic Docker registry. Originally developed by VMware and now a CNCF-graduated project, Harbor adds vulnerability scanning (Trivy built-in), image signing with Cosign/Notation, OCI artifact support (Helm charts, OPA bundles, SBOMs, any OCI artifact), replication across registries, role-based access control (RBAC), immutable tags, garbage collection, and audit logging.

Think of Harbor as a secure, automated warehouse for your Docker images and software packages.

Harbor is the most popular self-hosted registry for enterprise Kubernetes deployments, replacing the basic Docker Registry v2 with a full-featured platform that serves as the single source of truth for container images and OCI artifacts.

Plain-English First

Think of Harbor as a secure, automated warehouse for your Docker images and software packages. A basic Docker registry is like a simple storage unit — you put boxes (images) in and take them out, but there's no security check, no inventory tracking, and no quality control. Harbor is a full warehouse with security guards (vulnerability scanning), librarians (content trust and signing), inventory managers (replication and garbage collection), and access control (RBAC). It checks every box for pests (vulnerabilities), stamps it with an authenticity seal (Cosign signature), keeps copies in multiple locations (replication), and periodically cleans out expired boxes (GC).

⚙ Browser compatibility
Latest versions — ✓ supported
ChromeFirefoxSafariEdge

You're running 50 microservices on Kubernetes, each deployed multiple times a day. Your images live in Docker Hub or a basic registry. Security asks: 'Are all our images scanned for vulnerabilities?' Compliance asks: 'Can we prove who built each image and when?' Your team asks: 'Why is the registry so slow during deployments?' The answer to all three is Harbor — an enterprise-grade registry that's been the CNCF standard since 2020 and serves as the backbone for production container workflows at thousands of organizations.

Harbor is not just a place to store images. It's an image security platform. It integrates Trivy for automatic vulnerability scanning on push, supports Cosign and Notation for image signing and attestation, manages OCI artifacts beyond containers (Helm charts, OPA policies, SBOMs, signatures), replicates images across clouds and regions for DR, enforces retention policies with tag immutability and garbage collection, and provides a full audit trail of every push, pull, and delete.

By 2026, Harbor has become the default self-hosted registry for production Kubernetes. This guide covers deployment on Kubernetes with Helm, high-availability database configuration (Patroni, CloudNativePG, Redis Sentinel/Cluster), integration with Trivy for scanning, Cosign/Notation for content trust, OCI artifact management, replication strategies, garbage collection, and the production incident patterns that burn teams new to Harbor.

This guide is for platform engineers, DevOps teams, and SREs who need to deploy and operate Harbor in production. By the end, you'll have a production-grade Harbor deployment with HA database, automated scanning, content trust enforcement, and zero-downtime upgrades.

Deploying Harbor on Kubernetes with Helm: HA Configuration

Harbor's Helm chart is the recommended deployment method. It deploys 6+ components: core (API + UI), jobservice (GC, replication, scanning), registry (blob storage, based on Docker Distribution), trivy (vulnerability scanner), portal (web UI), and optional components like Notary, Clair, and ChartMuseum. For HA, you need high-availability for PostgreSQL and Redis — Harbor's stateful dependencies.

The Helm chart supports --set database.type=external to use an external PostgreSQL. For HA, use Patroni (PostgreSQL HA manager), CloudNativePG (Kubernetes-native operator), or a managed cloud database (RDS, Cloud SQL). CloudNativePG is the recommended choice for Kubernetes-native deployments: it manages failover, backups, and replica configuration automatically.

Redis: Harbor uses Redis for session storage (core), job queue (jobservice), and registry cache. For HA, use Redis Sentinel (for smaller deployments) or Redis Cluster (for larger ones). The Helm chart supports --set redis.type=external. Deploy Redis via the bitnami/redis Helm chart with sentinel mode: helm install redis bitnami/redis --set sentinel.enabled=true --set architecture=replication.

Storage: Harbor supports filesystem (PVC), S3, GCS, Azure Blob, and Swift backends. For production, use S3-compatible storage (MinIO, AWS S3, GCS). This decouples blob storage from Kubernetes nodes, enables multi-region replication, and provides durability. The Helm chart configures persistence.imageChartStorage.type: s3.

Ingress: Use a proper ingress controller (nginx-ingress, Traefik, or cloud LB). Harbor needs the ingress to support HTTP/2 for the registry API (it uses the Docker Registry protocol over HTTP/2). Configure TLS termination at the ingress. Set expose.type: ingress and expose.tls.secretName.

Minimal HA configuration: 3x core pods, 3x registry pods, 2x jobservice pods, external PostgreSQL (CloudNativePG cluster with 3 replicas), external Redis (sentinel with 1 primary + 2 replicas), S3 blob storage.

harbor-ha-deploy.shBASH
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
// io.thecodeforge — DevOps tutorial

# Add Harbor Helm repo
helm repo add harbor https://helm.goharbor.io
helm repo update

# Create namespace
kubectl create namespace harbor

# Deploy CloudNativePG PostgreSQL
docker pull ghcr.io/cloudnative-pg/postgresql:16

echo """
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: harbor-db
  namespace: harbor
spec:
  instances: 3
  imageName: ghcr.io/cloudnative-pg/postgresql:16
  storage:
    size: 50Gi
  walStorage:
    size: 10Gi
""" | kubectl apply -f -

# Deploy Redis with Sentinel
helm install redis bitnami/redis -n harbor \
  --set architecture=replication \
  --set sentinel.enabled=true \
  --set auth.enabled=true \
  --set auth.password=redis-password

# Deploy Harbor with HA config
helm install harbor harbor/harbor -n harbor \
  --set expose.type=ingress \
  --set expose.ingress.hosts.core=harbor.example.com \
  --set expose.tls.auto.commonName=harbor.example.com \
  --set database.type=external \
  --set database.external.host=harbor-db-rw \
  --set database.external.port=5432 \
  --set database.external.username=harbor \
  --set database.external.password=harbor-db-password \
  --set database.external.sslmode=require \
  --set redis.type=external \
  --set redis.external.addr=redis-node-0.redis-headless:6379 \
  --set redis.external.sentinelAddrs=redis-node-0.redis-headless:26379 \
  --set redis.external.password=redis-password \
  --set persistence.enabled=true \
  --set persistence.persistentVolumeClaim.registry.size=200Gi \
  --set persistence.imageChartStorage.type=s3 \
  --set persistence.imageChartStorage.s3.region=us-east-1 \
  --set persistence.imageChartStorage.s3.bucket=harbor-blobs \
  --set persistence.imageChartStorage.s3.accesskey=AKIA... \
  --set persistence.imageChartStorage.s3.secretkey=... \
  --set trivy.enabled=true \
  --set trivy.severity=CRITICAL,HIGH \
  --set core.replicas=3 \
  --set registry.replicas=3 \
  --set jobservice.replicas=2
Production Trap: Redis Sentinel Addresses in Helm
When using external Redis with sentinel, the redis.external.sentinelAddrs must include all three sentinel addresses, not just the primary. Harbor uses sentinel to discover the current Redis primary during failover. Without all sentinel addresses, a failover breaks Harbor's connection. Format: redis-node-0.redis-headless:26379,redis-node-1.redis-headless:26379,redis-node-2.redis-headless:26379.
Production Insight
A production Harbor outage was caused by Redis primary failover where Harbor couldn't find the new primary because only one sentinel address was configured. The fix: add all three sentinel addresses and enable redis.external.sentinelMasterName: mymaster. Recovery required restarting all Harbor pods to force Redis reconnection.
Key Takeaway
HA Harbor requires HA PostgreSQL (CloudNativePG or Patroni), HA Redis (Sentinel or Cluster), and external blob storage (S3). Use the Helm chart with external database and redis settings.
docker-harbor-registry THECODEFORGE.IO Harbor Registry Architecture Layered components for secure image management User Interface Harbor Portal | REST API | Docker CLI Security Services Trivy Scanner | Notary Signing | Cosign Verification Core Services Registry | Replication | Garbage Collection Data Storage PostgreSQL | Redis | Object Storage Infrastructure Kubernetes | Helm | Ingress Controller THECODEFORGE.IO
thecodeforge.io
Docker Harbor Registry

Trivy Integration: Automatic Vulnerability Scanning at the Registry Level

Harbor integrates Trivy as its default vulnerability scanner. When configured with --set trivy.enabled=true, Harbor deploys a Trivy scanner pod that automatically scans every image pushed to the registry. Scan results appear in the Harbor UI and are exposed via API. Harbor can enforce policies based on scan results: block pull of images with critical vulnerabilities, block push of images that fail scan, or quarantine images to a separate project.

Trivy in Harbor runs as a separate deployment. It downloads the vulnerability database on startup and updates it periodically. For air-gapped environments, pre-download the Trivy DB and mount it as a volume. The scanner pod needs internet access to update the CVE database, or use a local mirror.

Configuration: trivy.severity sets the severity threshold for blocking (default: CRITICAL). trivy.ignoreUnfixed (default: true) only fails scans on vulnerabilities with a known fix. trivy.gitHubToken sets a GitHub token for API rate limiting when downloading the DB.

Scan-on-push is the default. Schedule scans for existing images with the Harbor jobservice. For large registries, prioritize scanning production-tagged images first. Harbor's scanner supports parallel scanning across multiple scanner pods for throughput.

Policy enforcement: In project configuration, enable 'Prevent vulnerable images from being pulled' and set severity to Critical. Images with any critical CVE will return 403 on pull. This prevents vulnerable images from reaching Kubernetes even if CI/CD scanning missed them.

harbor-scan-policy.yamlYAML
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
# io.thecodeforge — DevOps tutorial

# Harbor project policy: block pull of critical-vulnerability images
# Configured via Harbor API or UI
# This prevents vulnerable images from being deployed

apiVersion: goharbor.io/v1alpha1
kind: HarborProject
metadata:
  name: production
spec:
  vulnerabilityScanPolicy:
    analyzer: trivy
    parameters:
      severity: critical,high
      fixable: true
  enforcement:
    pull:
      enabled: true
      severity: critical
    push:
      enabled: false  # Allow push, block pull
  scanOnPush: true
  retentionPolicy:
    numberOfDays: 90
    immutableTags: true

# API equivalent:
# curl -X PUT "https://harbor.example.com/api/v2.0/projects/production" \
#   -H "Content-Type: application/json" \
#   -d '{"vulnerability_scan_policy": {"analyzer": "trivy", "parameters": {"severity": "critical"}}}'
Interview Gold: Block Pull vs Block Push for Scanning
Block pull (not push) is the recommended enforcement mode. Blocking push frustrates developers during rapid iteration. Blocking pull ensures that Kubernetes can never deploy a vulnerable image — the image exists in the registry but can't be pulled. Production projects should block pull. Dev projects should only scan (no block), with alerts to the team.
Production Insight
The most effective Harbor scanning setup: CI/CD pushes images to a 'dev' project (no scan enforcement), a replication rule copies images to 'staging' (scan enforced, block pull), and a manual promotion copies to 'production' (scan enforced, block pull, immutable tags). This gives developers fast push while ensuring only scanned, signed images reach production.
Key Takeaway
Harbor integrates Trivy for automatic vulnerability scanning on push. Block pull (not push) for production projects. Use a multi-project promotion pipeline for dev-to-prod image progression with increasing scan enforcement.

Cosign and Notation Content Trust: Signing Images in Harbor

Harbor supports image signing and verification through two mechanisms: Cosign (Sigstore) and Notation (Notary v2). Both integrate with Harbor's content trust features to ensure that only signed images can be pulled from specific projects.

Cosign integration: Harbor stores Cosign signatures as OCI artifacts alongside the image. When you cosign sign <harbor-url>/project/image:tag, the signature is stored in Harbor as a separate OCI artifact referenced from the image manifest. Harbor does not verify signatures during push/pull by default — verification happens at the admission controller level (Kyverno) or via cosign verify. However, Harbor can enforce 'immutable tags' to prevent signature overwrite.

Notation (Notary v2) integration: Harbor has built-in support for Notation since version 2.8.0. Signature verification is enforced at the registry level: a project can be configured to require valid Notation signatures before allowing pull. This is the most mature content trust enforcement in Harbor. Set --set notary.enabled=true in the Helm chart to deploy the Notary signer.

Robot accounts: Harbor supports robot accounts for CI/CD automation. Create a robot account with push permissions, and use its credentials in CI pipelines. Robot accounts can be scoped to specific projects and expiration dates.

Best practice: use Cosign for keyless signing (via OIDC) in CI/CD pipelines, and use Notation for enterprise PKI-based signing (if you have an existing certificate authority). Harbor supports both simultaneously.

harbor-signing.shBASH
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
// io.thecodeforge — DevOps tutorial

# 1. Push an image to Harbor
docker pull alpine:latest
docker tag alpine:latest harbor.example.com/library/alpine:1.0
docker push harbor.example.com/library/alpine:1.0

# 2. Sign with Cosign (keyless)
cosign sign harbor.example.com/library/alpine:1.0
# Uses OIDC token from environment (GitHub Actions, etc.)
# Signature stored as OCI artifact in Harbor

# 3. Sign with Cosign (key-based)
cosign generate-key-pair
cosign sign --key cosign.key harbor.example.com/library/alpine:1.0

# 4. List signatures
cosign verify harbor.example.com/library/alpine:1.0 --key cosign.pub

# 5. Sign with Notation (requires Notation installed)
notation cert generate-test --default "harbor.example.com"
notation sign --key "harbor.example.com" harbor.example.com/library/alpine:1.0
notation verify harbor.example.com/library/alpine:1.0

# 6. Enable content trust enforcement in Harbor project
# Harbor UIProjectConfigurationEnable Content Trust
# Then only signed images can be pulled from this project.
Senior Shortcut: Robot Accounts for CI/CD
Don't use admin credentials in CI/CD. Create a robot account per project with push-only permissions. Robot accounts have expiring tokens (max 30 days recommended). Rotate them automatically in your CI/CD secrets manager. Example: harbor robot create --name ci-bot --project production --permission push --expiration 30d.
Production Insight
A compliance audit at a healthcare company required proof that every image in production was signed by a specific CI pipeline. Harbor's immutable tags + Cosign signatures provided the audit trail. The auditor could verify: cosign verify --key cosign.pub image:tag@sha256:... and confirm the signature was created by the CI pipeline's OIDC identity.
Key Takeaway
Harbor supports Cosign (keyless, OIDC) and Notation (X.509 PKI) for image signing. Use robot accounts for CI/CD. Enable content trust enforcement on production projects to require signed images.
Cosign vs Notation for Image Signing Comparing content trust mechanisms in Harbor Cosign Notation Key Management Uses keyless signing with OIDC Requires X.509 certificates Integration with Harbor Native support via Cosign plugin Built-in Notary v2 support Signature Storage Stored as OCI artifacts Stored in Notary trust store Verification Workflow Policy-based with cosign verify Trust policy with notation verify Adoption Widely used in open source Growing with CNCF support THECODEFORGE.IO
thecodeforge.io
Docker Harbor Registry

OCI Artifact Management: Beyond Container Images

Harbor supports OCI artifacts — any OCI-compliant content stored and managed like container images. This includes Helm charts, OPA policies, CNAB bundles, WASM modules, SBOMs, vulnerability reports, and custom artifacts. Harbor provides a unified UI and API for all artifact types.

Helm charts: Store and manage Helm charts directly in Harbor. Harbor's ChartMuseum component (deprecated in favor of OCI-native Helm charts) now supports pushing charts as OCI artifacts: helm push chart.tar.gz oci://harbor.example.com/project. Charts are versioned, searchable, and scanable for vulnerabilities in their dependencies.

OPA bundles: OPA/Gatekeeper policies can be stored as OCI artifacts in Harbor. This enables versioned, signed policy distribution. opa build -t oci . -o policy.tar.gz && oras push harbor.example.com/policies/kubernetes:1.0 policy.tar.gz.

SBOMs: Harbor can store CycloneDX and SPDX SBOMs as OCI artifacts. Attach SBOMs to image artifacts for supply chain transparency. Tools like Syft can generate and push SBOMs directly to Harbor: syft myapp:latest -o cyclonedx-json | oras push harbor.example.com/library/myapp:sbom-latest -.

Cosign signatures and attestations: Harbor stores Cosign signatures (.sig) and attestations (.att) as OCI artifacts referenced by the image manifest. cosign sign pushes to <image>.sig as a referrer. Harbor handles the referrers API automatically (OCI 1.1 spec).

OCI artifact management is configured per-project. Each artifact type can have its own retention policy, access control, and immutability settings.

oci-artifacts.shBASH
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
// io.thecodeforge — DevOps tutorial

# 1. Push a Helm chart as OCI artifact
helm package my-chart/
helm push my-chart-1.0.0.tgz oci://harbor.example.com/helm-charts

# 2. Push an OPA bundle as OCI artifact
docker run --rm -v $(pwd)/policies:/policies openpolicyagent/opa:latest \
  build -t oci -o bundle.tar.gz /policies
oras push harbor.example.com/opa-bundles/k8s:1.0 bundle.tar.gz

# 3. Push an SBOM as OCI artifact
syft harbor.example.com/library/myapp:1.0 -o cyclonedx-json | \
  oras push harbor.example.com/library/myapp:sbom-1.0 -

# 4. List artifacts in a project
curl -u admin:harbor \
  -X GET "https://harbor.example.com/api/v2.0/projects/library/repositories"

# 5. List artifacts of a specific type
curl -u admin:harbor \
  -X GET "https://harbor.example.com/api/v2.0/projects/library/repositories/myapp/artifacts?with_tag=true"

# 6. Download an OCI artifact
oras pull harbor.example.com/opa-bundles/k8s:1.0 -o /tmp/policies
Interview Gold: OCI Referrers API
Harbor supports the OCI 1.1 Referrers API, which allows artifacts to reference other artifacts. For example, a Cosign signature artifact references the image it signs. Harbor's referrers API endpoint: GET /v2/<project>/<repo>/manifests/<digest>/referrers. This is how Harbor associates signatures, SBOMs, and vulnerability reports with their parent images.
Production Insight
A platform team at a large enterprise standardized on Harbor for ALL OCI artifacts: container images (Docker/OCI), Helm charts, OPA policies, WASM modules for their edge computing platform, and CNAB bundles for cloud provisioning. Harbor's unified API and UI replaced 5 separate artifact stores with a single platform. Audit and compliance was simplified to one query.
Key Takeaway
Harbor manages any OCI artifact: container images, Helm charts, OPA bundles, SBOMs, signatures. Use oras or Helm push for OCI-native workflows. Each artifact type gets versioning, access control, and retention policies.

Garbage Collection, Replication, and Retention Policies

Harbor's garbage collection (GC) reclaims storage from blobs that are no longer referenced by any image tag. GC is critical for production registries that see frequent image pushes and deletions. Without GC, deleted images leave orphaned blobs that consume storage indefinitely.

GC configuration: Harbor's jobservice runs GC as a cron job. Default: daily at midnight. Configure via jobservice.gcCron in Helm values. Always run GC with --dry-run first to see what would be deleted. Review the GC report in Harbor UI. Enable --delete-untagged only after validating the dry-run.

Key GC risk: GC can delete blobs that are still referenced by an image tag if the reference count is temporarily inconsistent (e.g., during replication). Solution: schedule GC and replication at different times. Use S3 versioning or blob soft-delete to recover from accidental deletion.

Replication: Harbor replicates images between registries. Use cases: DR (replicate to a secondary Harbor in another region), migration (from Docker Hub to Harbor), multi-cloud (replicate to AWS ECR or GCR for cloud-native deployments), and edge distribution (replicate to edge registries).

Replication modes: push-based (Harbor pushes to remote) and pull-based (remote pulls from Harbor). Filter by repository name, tag pattern, or resource type. Replication respects Harbor's scan and content trust policies — only scan-passing images are replicated.

Retention policies: Harbor can automatically delete images based on rules: number of days since push, number of most recent tags to retain, or tag patterns to exclude. Combine with immutability to protect critical tags (latest, production) from deletion.

harbor-gc-replication.shBASH
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
// io.thecodeforge — DevOps tutorial

# 1. Run GC in dry-run mode
# Create a manual job in Harbor UI or via API:
curl -u admin:harbor -X POST \
  https://harbor.example.com/api/v2.0/system/gc/schedule \
  -H "Content-Type: application/json" \
  -d '{"schedule": {"type": "Manual"}, "parameters": {"dry_run": true}}'

# 2. Check GC results
curl -u admin:harbor \
  https://harbor.example.com/api/v2.0/system/gc/1234

# 3. Run GC for real (only after dry-run review)
curl -u admin:harbor -X POST \
  https://harbor.example.com/api/v2.0/system/gc/schedule \
  -H "Content-Type: application/json" \
  -d '{"schedule": {"type": "Manual"}, "parameters": {"dry_run": false}}'

# 4. Create a replication rule (push to AWS ECR)
# Harbor UIAdministrationReplicationsNew Replication Rule
# Name: replicate-to-ecr
# Source: harbor.example.com/production/*
# Destination: aws_account_id.dkr.ecr.us-east-1.amazonaws.com/production/*
# Trigger: On Push
# Mode: Push-based

# 5. Create retention policy (keep last 30 tags, delete older)
curl -u admin:harbor -X POST \
  https://harbor.example.com/api/v2.0/projects/production/retentions \
  -H "Content-Type: application/json" \
  -d '{
        "algorithm": "or",
        "rules": [{
          "disabled": false,
          "template": "days_since_last_push",
          "params": {"num": 90},
          "tag_selectors": [{"kind": "doublestar", "pattern": "**", "decoration": "matches"}],
          "scope_selectors": {"repository": [{"kind": "doublestar", "pattern": "**", "decoration": "repoMatches"}]}
        }]
      }'
Production Trap: GC + Replication Race Conditions
Never schedule GC and replication at the same time. Replication temporarily updates blob reference counts. If GC scans reference counts during a replication job, it can incorrectly mark blobs as unreferenced and delete them. Always separate GC and replication by at least 2 hours. Enable blob versioning in S3 to recover from accidental deletion.
Production Insight
A large Harbor deployment with 50TB of images saw GC reclaim 12TB after a 6-month backlog. The GC took 18 hours and consumed 100% of the registry pod's CPU. Lessons learned: (1) Run GC at least weekly. (2) Increase registry pod CPU during GC. (3) Use S3 lifecycle policies for blob expiration instead of relying solely on Harbor GC. (4) Monitor GC duration with Harbor metrics.
Key Takeaway
Run Harbor GC weekly with dry-run first. Schedule GC and replication at different times to avoid reference count race conditions. Use S3 lifecycle policies as a secondary cleanup mechanism. Retention policies auto-delete old images per-project.
● Production incidentPOST-MORTEMseverity: high

The Garbage Collection That Ate Our Production Images

Symptom
Kubernetes deployments started failing with 'manifest unknown' errors. Image pulls suddenly returned 404 for tags that existed moments earlier. Three microservices couldn't roll back to previous versions because the tags were gone.
Assumption
Team assumed a replication issue or registry corruption. They checked disk space (plenty), restarted the Harbor pods, and verified PostgreSQL connectivity. The issue persisted and spread to more tags.
Root cause
Harbor's garbage collection ran during an active replication. The replication job updated blob reference counts, causing GC to incorrectly mark blobs as unreferenced. GC skipped the 'dry-run' step because it was configured with --delete-untagged without --dry-run. Over 200 blobs shared by active images were deleted, including blobs referenced by the latest tags of critical production services.
Fix
1) Restored blobs from the S3 bucket backup (Harbor stores blobs separately from metadata). 2) Switched GC to dry-run mode for 2 weeks: harbor gc --dry-run via the jobservice. 3) Changed GC schedule to run during lowest-traffic window (4 AM Sunday) with no replication jobs scheduled concurrently. 4) Enabled blob reference logging to audit future GC runs. 5) Implemented a pre-GC check: verify blob references against active image tags before deletion.
Key lesson
  • Never run Harbor garbage collection without --dry-run first. GC with --delete-untagged can delete blobs shared by active tags if reference counts are temporarily incorrect during replication.
  • Schedule GC and replication jobs at different times. Concurrent GC and replication caused the reference count inconsistency that triggered this incident.
  • Always test GC in a staging environment that mirrors production data size. A 10-image test doesn't reveal reference counting race conditions.
Production debug guideSymptom → Root cause → Fix3 entries
Symptom · 01
Push fails with '413 Request Entity Too Large' or 'denied: exceeded project quota'
Fix
1. Check Harbor project quota: curl -u admin:harbor -X GET https://harbor/api/v2.0/projects/<project>/sum_quota. 2. Increase quota via Harbor UI: Projects → <project> → Configuration → Storage Quota. 3. Check if the registry backend (S3, GCS) has its own quota limits. 4. Run Harbor GC to reclaim space from deleted images.
Symptom · 02
Harbor UI loads slowly or times out on the 'Projects' page
Fix
1. Check PostgreSQL query performance: pg_stat_activity for long-running queries. 2. Ensure PostgreSQL has adequate shared_buffers (25% of RAM). 3. Check Redis connectivity: Harbor uses Redis for session caching and job queue. 4. Scale Harbor's core pods: kubectl scale deployment harbor-core --replicas=3. 5. Consider adding a CDN in front of Harbor for blob distribution.
Symptom · 03
Harbor replication fails with 'no route to host' or TLS errors between registries
Fix
1. Verify network connectivity: kubectl exec <harbor-core> -- curl -k https://remote-registry/v2/. 2. Check TLS certificates on both ends. For self-signed certs, add to Harbor's trust store. 3. Verify credentials: the replication rule uses a robot account or admin credentials. 4. Check the replication log in Harbor UI for specific error messages. 5. Ensure the remote registry supports the OCI distribution spec v1.0+.
FeatureHarborDocker HubAWS ECRDocker Registry v2
Vulnerability scanningBuilt-in (Trivy)Paid (Docker Scout)Built-in (Inspector)None
Image signingCosign + NotationPaidSigner integrationNone
OCI artifactsFull (Helm, OPA, SBOM)Container images onlyContainer images onlyContainer images only
ReplicationPush/Pull, filters, filtersPaidCross-regionNone
RBACProject-level + robot accountsOrganization + teamsIAM policiesNone
Garbage collectionDry-run + scheduledAutomaticAutomatic (lifecycle)Manual
Self-hostedYes (K8s, Docker, VMs)No (SaaS)No (SaaS)Yes
CostFree (open source)Free tier, paid plansPer-GB stored + data transferFree
⚙ Quick Reference
5 commands from this guide
FileCommand / CodePurpose
harbor-ha-deploy.shhelm repo add harbor https://helm.goharbor.ioDeploying Harbor on Kubernetes with Helm
harbor-scan-policy.yamlapiVersion: goharbor.io/v1alpha1Trivy Integration
harbor-signing.shdocker pull alpine:latestCosign and Notation Content Trust
oci-artifacts.shhelm package my-chart/OCI Artifact Management
harbor-gc-replication.shcurl -u admin:harbor -X POST \Garbage Collection, Replication, and Retention Policies

Key takeaways

1
Harbor is the CNCF-standard enterprise container registry with built-in vulnerability scanning (Trivy), image signing (Cosign/Notation), OCI artifact management, and replication.
2
HA deployment requires external PostgreSQL (CloudNativePG) and Redis (Sentinel/Cluster). Use S3-compatible blob storage to decouple storage from Kubernetes nodes.
3
Block pull (not push) for vulnerability enforcement. Use a multi-project pipeline (dev → staging → production) with increasing scan and signing requirements.
4
Run garbage collection weekly with dry-run first. Never schedule GC and replication concurrently
reference count race conditions can cause blob deletion of active images.
5
Harbor manages all OCI artifacts
container images, Helm charts, OPA bundles, SBOMs, signatures. Use robot accounts with expiring tokens for CI/CD automation.

Common mistakes to avoid

4 patterns
×

Using default admin password (Harbor12345) in production

×

Running GC without dry-run and deleting active blobs

×

Using file system storage (PVC) instead of S3 for production

×

Not configuring immutable tags for production images

INTERVIEW PREP · PRACTICE MODE

Interview Questions on This Topic

Q01SENIOR
How does Harbor handle concurrent garbage collection and image pushes? W...
Q02SENIOR
Design a multi-region Harbor deployment with active-active image distrib...
Q03SENIOR
You need to audit every image pull in Harbor for compliance. How would y...
Q04SENIOR
Harbor deployment on Kubernetes is running slow. Walk through the diagno...
Q01 of 04SENIOR

How does Harbor handle concurrent garbage collection and image pushes? What consistency model does it use?

ANSWER
Harbor uses a reference counting system for blob storage. Each image manifest references blobs by digest. When an image is pushed, blob reference counts increment. When a tag is deleted, reference counts decrement. GC scans all blobs and deletes those with zero references. This is eventually consistent due to the time window between tag deletion and GC scan. Harbor prevents GC during active replication to avoid counting inconsistencies. The consistency model is read-after-write for pushes (new image is immediately available) but eventually consistent for GC (deleted blobs may take hours to reclaim). S3 object versioning protects against accidental deletion during GC.
FAQ · 5 QUESTIONS

Frequently Asked Questions

01
Can Harbor replace Docker Hub entirely?
02
How does Harbor handle large-scale concurrent pushes (100+ images simultaneously)?
03
What happens if the Trivy scanner pod crashes? Does it affect registry functionality?
04
How do I migrate from an existing Docker Registry v2 to Harbor?
05
Can Harbor enforce that images are only pushed from specific CI/CD pipelines?
N
Naren Founder & Principal Engineer

20+ years shipping production infrastructure and CI/CD at scale. Drawn from code that ran under real load.

Follow
Verified
production tested
July 11, 2026
last updated
1,787
articles · all by Naren
🔥

That's Docker. Mark it forged?

5 min read · try the examples if you haven't

Previous
Docker Observability with OpenTelemetry
36 / 41 · Docker
Next
Cosign Image Signing and Supply Chain Security