ArgoCD GitOps — Auto-Heal Reverted Scale-Down 3× Mid-Outage

Most teams reach a point where 'kubectl apply -f' starts feeling like playing Jenga blindfolded. One engineer deploys a hotfix directly to production, another runs a Helm upgrade from their laptop, and within a week nobody actually knows what's running in the cluster. Config drift is silent, cumulative, and eventually catastrophic. ArgoCD was built to make that problem structurally impossible — not through discipline or process, but through automation backed by a source of truth that everyone can see and audit.

ArgoCD implements the GitOps operator pattern: it runs inside your cluster, watches a Git repository, and continuously reconciles the live cluster state against the desired state declared in that repo. If the live state drifts — whether from a rogue kubectl command, a failing node replacement, or a mischievous CronJob — ArgoCD detects the divergence and can automatically heal it. This isn't just CI/CD with extra steps; it's a fundamentally different mental model where deployments are a side-effect of merging a pull request, not a separate pipeline stage.

By the end you'll understand ArgoCD's reconciliation engine, how to model complex multi-service deployments with sync waves and hooks, how to harden a production installation with RBAC and SSO, and the non-obvious gotchas that only surface after six months of running it.

The Reconciliation Loop — How ArgoCD Continuously Enforces Git State

ArgoCD runs as a set of Kubernetes controllers inside your cluster. The core component is the Application Controller, which runs a continuous reconciliation loop (default 3 minutes) for each Application resource. It fetches the desired state from Git (via repo server), compares it with the live state from the target cluster (using the Kubernetes API), and if they differ, it marks the Application as 'OutOfSync'. If auto-sync is enabled, it immediately applies the diff.

The key insight is that ArgoCD doesn't just apply YAML. It performs a three-way diff: live state (current cluster), desired state (Git), and last applied state (stored in the Application CRD status). This three-way merge prevents the 'last write wins' problem when resources are updated outside ArgoCD.

When a sync happens, ArgoCD orders resources using sync waves (annotations). Resources with lower wave numbers sync first. By default, all resources are wave 0. Custom Resource Definitions (CRDs) must be wave -1 or lower to be installed before custom resources that depend on them. Hooks (PreSync, Sync, PostSync) run as Jobs at specific stages, allowing you to run database migrations before updating deployments or health checks after.

The repository server caches Git contents. It supports Helm, Kustomize, and plain YAML. For Helm, it runs helm template internally. For Kustomize, it runs kustomize build. The repo server caches rendered manifests to speed up subsequent syncs.

# io.thecodeforge/argocd/application.yaml
# Production ArgoCD Application with sync waves and health checks
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: api-gateway
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io  # Clean up resources when app is deleted
spec:
  project: production
  source:
    repoURL: https://github.com/thecodeforge/infra.git
    targetRevision: HEAD
    path: overlays/production/api-gateway
    helm:
      valueFiles:
        - values.yaml
        - secrets.yaml  # Decrypted via SOPS with age key in argocd-cm
  destination:
    server: https://kubernetes.default.svc
    namespace: api-gateway
  syncPolicy:
    automated:
      prune: true       # Delete resources not in Git
      selfHeal: true    # Revert manual changes (kubectl edit)
      allowEmpty: false # Don't sync if would wipe all resources
    syncOptions:
      - ApplyOutOfSyncOnly=true
      - PrunePropagationPolicy=foreground
      - CreateNamespace=true
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m
  syncWave: "-1"  # Sync before dependent apps
status:
  sync:
    status: Synced
  health:
    status: Healthy

Sync Waves and Hooks — Ordering Complex Deployments

Kubernetes has no built-in ordering for applying resources. If you need to install a CRD before creating a custom resource (like Prometheus CRDs before Prometheus operator), you must order the sync. ArgoCD's sync waves solve this.

Every resource can be annotated with argocd.argoproj.io/sync-wave. Lower numbers sync first. Resources with wave -1 sync before wave 0. Resources in the same wave sync in parallel (order not guaranteed).

CRDs must be in wave -1 or lower. Custom resources (like Prometheus, Istio) must be in wave 0 or higher. If a custom resource references a CRD not yet installed, sync will fail. PreSync hooks run before any wave -1 resources; PostSync hooks run after wave 100.

Hooks are Kubernetes Jobs or Pods that run at specific stages. Example: a database migration container must run before the new app version starts. Use a PreSync hook: the Job runs, ArgoCD waits for it to complete (successfully), then syncs the Deployment. If the hook fails, ArgoCD stops the sync.

Hook types: PreSync (before sync), Sync (during sync, rare), PostSync (after all resources healthy), Skip (don't apply to cluster, just run). Hooks can fail the sync if they return non-zero exit code.

---
# io.thecodeforge/argocd/sync-wave-hooks.yaml
# Example: Install CRDs first (wave -1), then create operator (wave 0), then run migration (PreSync hook)

# CRDs must exist before custom resources
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: gatlingruns.gatling-operator.io
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
spec:
  group: gatling-operator.io
  names:
    kind: GatlingRun
    plural: gatlingruns
  scope: Namespaced
  versions:
    - name: v1
      served: true
      storage: true
---
# Custom resource (depends on CRD above) — sync after wave -1
apiVersion: gatling-operator.io/v1
kind: GatlingRun
metadata:
  name: load-test-run
  annotations:
    argocd.argoproj.io/sync-wave: "0"
spec:
  image: gatling:latest
  replicas: 3
---
# PreSync hook: database migration before app deployment
apiVersion: batch/v1
kind: Job
metadata:
  name: db-migration
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: hook-succeeded
spec:
  template:
    spec:
      containers:
        - name: migrate
          image: migrate/migrate:v4
          command:
            - /bin/sh
            - -c
            - "migrate -database postgres://user:pass@postgres:5432/app?sslmode=disable -path /migrations up"
      restartPolicy: Never
  backoffLimit: 2
---
# PostSync hook: deploy smoke test runner
apiVersion: batch/v1
kind: Job
metadata:
  name: smoke-test
  annotations:
    argocd.argoproj.io/hook: PostSync
    argocd.argoproj.io/hook-delete-policy: hook-succeeded
spec:
  template:
    spec:
      containers:
        - name: tester
          image: curlimages/curl:latest
          command:
            - /bin/sh
            - -c
            - "curl -f https://api.example.com/health || exit 1"
      restartPolicy: Never

Multi-Cluster and Multi-Tenant Patterns — Production Hardening

ArgoCD can manage thousands of clusters from a single control plane. Each target cluster is represented by a secret in the ArgoCD namespace containing the cluster API server endpoint and bearer token. The Application Controller uses these secrets to connect and sync.

For multi-cluster management, organise Applications by cluster and environment: clusters/prod/us-east-1/apps, clusters/staging/eu-west-1/apps. Use ApplicationSets to generate Applications dynamically for each cluster with a templated Git path.

Multi-tenancy within a cluster: use Projects to isolate teams. Each Project defines source repositories, destination clusters/namespaces, and role-based access. For example, the 'team-a' project can only deploy to namespaces prefixed with 'team-a-' and only from Git repos under github.com/team-a.

RBAC in ArgoCD is its own model: policies are defined in argocd-rbac-cm. A policy like p, role:admin, applications, , /, allow gives full access. p, role:viewer, applications, get, team-a/, allow allows read-only access to team-a apps. Map OIDC groups to roles via oidc.config.

For secrets management, ArgoCD integrates with SOPS, SealedSecrets, or Vault. The recommended pattern: commit encrypted secrets to Git, decrypt them in the repo server using a KMS key or age private key stored in Kubernetes secrets. Never put plaintext secrets in Git.

# io/thecodeforge/argocd/argocd-project.yaml
# Multi-tenant RBAC with Projects
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: team-payments
  namespace: argocd
spec:
  description: "Payments team microservices"
  # Restrict source repos
  sourceRepos:
    - 'https://github.com/org/payments-infra.git'
  # Only allow deploying to specific clusters/namespaces
  destinations:
    - namespace: 'payments-*'
      server: https://kubernetes.default.svc
    - namespace: 'payments-*'
      server: https://prod-eu-1.k8s.example.com
  # Deny cluster-scoped resources except specific ones
  clusterResourceWhitelist:
    - group: ''
      kind: Namespace
    - group: rbac.authorization.k8s.io
      kind: ClusterRole
  # Allow only specific resource kinds
  namespaceResourceBlacklist:
    - group: ''
      kind: Secret  # Prevent teams from creating secrets outside vault
  roles:
    - name: developer
      policies:
        - p, proj:team-payments:developer, applications, sync, team-payments/*, allow
        - p, proj:team-payments:developer, applications, get, team-payments/*, allow
      groups:
        - payments-developers

---
# RBAC policy in argocd-rbac-cm (ConfigMap)
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-rbac-cm
  namespace: argocd
data:
  policy.csv: |
    # Admin access to all projects
    p, role:admin, applications, *, */*, allow
    p, role:admin, projects, *, *, allow
    p, role:admin, clusters, *, *, allow
    
    # Team leads can sync their own apps but not change project config
    p, role:team-lead, applications, sync, team-*/production/*, allow
    p, role:team-lead, applications, get, team-*/*, allow
    
    # Read-only access for auditors
    p, role:auditor, applications, get, */*, allow
    
  policy.default: role:readonly  # Default role if not matched
  scopes: '[groups]'