✓Kubernetes cluster (v1.22+), kubectl, Go 1.20+, Kubebuilder v3.7+, Docker, basic understanding of Kubernetes controllers and reconciliation loops.
✦ Definition~90s read
What is Kubernetes CRDs and Operators?
Kubernetes CRDs (Custom Resource Definitions) extend the Kubernetes API by allowing you to define your own resource types, while Operators are controllers that automate the lifecycle of those custom resources. They matter because they enable you to manage complex, stateful applications with Kubernetes-native patterns, turning operational knowledge into code.
★
Think of Kubernetes CRDs and Operators like a tool in your DevOps toolkit — once you understand what it does and when to reach for it, managing Kubernetes clusters becomes second nature.
Use them when you need to automate domain-specific tasks like database backups, cluster upgrades, or application scaling beyond what built-in controllers provide.
Plain-English First
Think of Kubernetes CRDs and Operators like a tool in your DevOps toolkit — once you understand what it does and when to reach for it, managing Kubernetes clusters becomes second nature.
You've deployed a stateful database on Kubernetes. It works—until a node fails, and you realize the pod didn't reattach to the persistent volume. You scramble to restore from a backup, losing hours of data. This is the moment you realize: Kubernetes' built-in controllers don't know how to manage your database. They don't understand replication, failover, or backups. That's where CRDs and Operators come in. They let you encode that operational knowledge into code, turning manual recovery procedures into automated, self-healing logic. CRDs define the 'what'—your custom resource like a Database—and Operators define the 'how'—the reconciliation loop that keeps reality matching your intent. This isn't just about convenience; it's about reliability at scale. Without Operators, you're one failure away from a pager storm. With them, you build systems that survive.
Why CRDs Exist: The Limits of Built-in Resources
Kubernetes ships with Pods, Services, Deployments—great for stateless apps. But stateful workloads like databases, message queues, or monitoring stacks have unique lifecycle needs: backup schedules, failover logic, version upgrades. Built-in resources can't express these. You could hack it with ConfigMaps and init containers, but that's brittle and unmanageable. CRDs solve this by letting you define your own resource schema. For example, a 'PostgresCluster' custom resource can have fields for version, replicas, backup config, and storage class. The API server stores these custom objects just like Pods, so you get the same declarative model, RBAC, and audit logging. The result: your application's domain concepts become first-class Kubernetes citizens.
customresourcedefinition.apiextensions.k8s.io/postgresclusters.example.com created
⚠ Schema Validation Gotcha
CRD schemas are validated at admission time, but complex validation (e.g., cross-field checks) requires a webhook. Don't rely solely on OpenAPI for business logic.
📊 Production Insight
In production, CRD schema changes are tricky: updating a field from required to optional can break existing objects. Always version your CRDs and use conversion webhooks.
🎯 Key Takeaway
CRDs extend the Kubernetes API with your own resource types, enabling domain-specific declarative management.
The Operator Pattern: Controllers That Know Your App
A CRD alone is just a data structure. The magic happens when you pair it with a controller—the Operator. An Operator watches for changes to custom resources and runs a reconciliation loop: it reads the desired state from the custom resource, compares it to the current cluster state, and takes actions to converge them. For a PostgresCluster, that means creating a StatefulSet, configuring a Service, setting up backups, and handling failover. The Operator encodes domain expertise: it knows that PostgreSQL needs a specific init sequence, that WAL archiving must be enabled, and that a primary switch requires updating the Service selector. This is the difference between a generic Helm chart and a production-grade Operator.
Reconciliation loop: creates/updates StatefulSet based on PostgresCluster spec.
💡Start with Kubebuilder or Operator SDK
Writing an Operator from scratch is error-prone. Use Kubebuilder (Go) or Operator SDK (Ansible/Helm) to scaffold the controller, webhooks, and RBAC. They generate the boilerplate and enforce best practices.
📊 Production Insight
A common mistake is not handling deletion gracefully. When a custom resource is deleted, the Operator should clean up all owned resources (e.g., PVCs, Secrets). Use finalizers to block deletion until cleanup completes.
🎯 Key Takeaway
Operators automate the lifecycle of custom resources by running a reconciliation loop that encodes domain-specific operational logic.
thecodeforge.io
Kubernetes Crds Operators
Building Your First Operator: Scaffolding with Kubebuilder
Kubebuilder is the de facto tool for building Go-based Operators. It generates a project structure with controller, API types, webhooks, and RBAC manifests. Start by initializing a project: kubebuilder init --domain example.com --repo github.com/example/postgres-operator. Then create an API: kubebuilder create api --group example --version v1 --kind PostgresCluster --resource --controller. This scaffolds the CRD types and a controller skeleton. The generated types.go defines the Spec and Status structs. The controller's Reconcile method is where you implement your logic. Kubebuilder also sets up a main.go that registers the controller with the manager, which handles leader election, metrics, and health probes. The result is a production-ready Operator skeleton in minutes.
scaffold.shBASH
1
2
3
4
5
6
7
8
9
10
11
12
13
14
# Initialize the project
kubebuilder init --domain example.com --repo github.com/example/postgres-operator
# CreateAPI and controller
kubebuilder create api --group example --version v1 --kind PostgresCluster --resource --controller
# GenerateCRD manifests
make generate
# Build the operator binary
make build
# Deploy to cluster (requires kustomize)
make deploy
Output
Project scaffolded. CRD manifests in config/crd/. Controller code in internal/controller/.
🔥Kubebuilder Versions
Kubebuilder v3+ uses controller-runtime v0.14+ and supports Go 1.20+. Always check compatibility with your Kubernetes version. For older clusters, use Kubebuilder v2.
📊 Production Insight
In production, use make deploy only for dev. For production, build a container image and deploy via Helm or ArgoCD. Ensure your Operator's image is pinned to a specific version and scanned for vulnerabilities.
🎯 Key Takeaway
Kubebuilder automates Operator scaffolding, letting you focus on business logic rather than boilerplate.
CRD Versioning: How to Evolve Your API Without Breaking Users
CRDs support multiple versions via the versions field. You can serve multiple versions simultaneously and convert between them using conversion webhooks. For example, you might have v1alpha1 for early adopters and v1 for stable. When you need to change the schema, you add a new version (e.g., v2) and write a conversion webhook that translates between v1 and v2 objects. The storage version is the version used to persist objects in etcd. Users can submit any served version; the API server converts to the storage version. This allows non-breaking evolution. However, avoid frequent version bumps—each version adds complexity. Prefer adding optional fields to the existing version if possible.
CRD with two versions. v1alpha1 is served but not stored; v1 is the storage version.
⚠ Conversion Webhook Pitfalls
Conversion webhooks must be idempotent and handle all versions. A bug can corrupt data. Test conversions thoroughly with existing objects before rolling out a new version.
📊 Production Insight
In production, never delete a served version that has existing objects. First, migrate all objects to the new version, then remove the old version from the CRD. Use kubectl get crd to check stored versions.
🎯 Key Takeaway
CRD versioning with conversion webhooks allows safe API evolution, but keep versions minimal to reduce maintenance burden.
Reconciliation Loop Deep Dive: The Heart of the Operator
The reconciliation loop is triggered by events: create, update, or delete of the custom resource, or changes to related resources (e.g., Pods). The controller-runtime library handles event filtering and queuing. Your Reconcile method receives a request with the namespaced name of the resource. It must fetch the current state, compute the desired state, and apply diffs. Key patterns: use ctrl.SetControllerReference to set ownership so that garbage collection cleans up child resources when the parent is deleted. Use predicates to filter events—e.g., ignore updates that don't change the spec. Handle errors by returning them; the controller will retry with exponential backoff. For long-running operations, use requeue after a delay. The loop should be idempotent: running it multiple times should produce the same result.
Reconciliation loop with finalizer handling, status update, and periodic requeue.
💡Use RequeueAfter for Periodic Reconciliation
Even without events, requeue periodically (e.g., every 30s) to catch drift from external changes. This ensures your Operator self-heals even if the watch misses an event.
📊 Production Insight
A common production issue is the Operator crashing due to a nil pointer dereference when a child resource is missing. Always check for IsNotFound errors and handle them gracefully.
🎯 Key Takeaway
The reconciliation loop is the core of an Operator; it must be idempotent, handle deletion via finalizers, and update status to reflect reality.
Testing Operators: Unit, Integration, and End-to-End
Testing Operators is non-trivial because they interact with a real Kubernetes API. For unit tests, mock the client using fake.NewClientBuilder() from controller-runtime. This lets you test reconciliation logic without a cluster. For integration tests, use envtest—a local API server and etcd. It starts a real Kubernetes API server in-process, so you can create CRDs and custom resources. Write tests that create a CR, run reconciliation, and assert that child resources are created. For end-to-end tests, deploy the Operator to a real cluster (e.g., kind) and verify the full lifecycle. Use ginkgo for BDD-style tests. Always test edge cases: deletion, invalid spec, network failures. A robust test suite catches regressions before they hit production.
envtest doesn't support all API features (e.g., webhooks, certain controllers). For full validation, run e2e tests on a real cluster. Use kind for local e2e tests.
📊 Production Insight
In production, run integration tests in CI with envtest. They catch 90% of bugs and run in seconds. Reserve e2e tests for pre-release validation.
🎯 Key Takeaway
Test Operators at multiple levels: unit tests with fake clients, integration tests with envtest, and e2e tests on a real cluster.
Advanced Patterns: Finalizers, Status, and Conditions
Finalizers prevent premature deletion of custom resources. When a CR has a finalizer, the deletion timestamp is set but the object isn't removed until the finalizer is cleared. Use finalizers to clean up external resources (e.g., cloud load balancers, DNS records). The status subresource stores observed state. Use conditions (e.g., Ready, Degraded) to communicate the health of the resource. Conditions follow the Kubernetes convention: Type, Status, Reason, Message, LastTransitionTime. Update status only via the status subresource to avoid conflicts. Use r.Status().Update() which uses a separate API endpoint. This prevents the spec update from overwriting status changes.
If your finalizer cleanup fails (e.g., cloud API timeout), the CR will be stuck in Terminating state. Implement retry logic with backoff and consider a manual override via annotation.
📊 Production Insight
In production, monitor stuck finalizers. Set up alerts for resources in Terminating state for more than 5 minutes. This often indicates a bug in the Operator's deletion logic.
🎯 Key Takeaway
Finalizers ensure proper cleanup; status conditions communicate resource health; always update status via the subresource.
thecodeforge.io
Kubernetes Crds Operators
Webhooks: Validating and Mutating Admission Controllers
CRD schema validation is limited. For complex validation (e.g., cross-field checks, uniqueness constraints), use admission webhooks. A validating webhook can reject invalid specs before they are persisted. A mutating webhook can set defaults (e.g., default backup schedule). Webhooks are configured via ValidatingWebhookConfiguration and MutatingWebhookConfiguration. They must be served over HTTPS, so you need a certificate. Kubebooter can generate the webhook code and certificate management. Common use cases: ensure replicas is odd for quorum-based systems, validate that the backup schedule is a valid cron expression, or inject a sidecar container. Webhooks add latency to API requests, so keep them fast and idempotent.
webhook.goGO
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
package webhook
import (
"context""fmt""net/http""sigs.k8s.io/controller-runtime/pkg/webhook/admission"
examplev1 "example.com/api/v1"
)
// +kubebuilder:webhook:path=/validate-example-com-v1-postgrescluster,mutating=false,failurePolicy=fail,sideEffects=None,groups=example.com,resources=postgresclusters,verbs=create;update,versions=v1,name=vpostgrescluster.example.com,admissionReviewVersions=v1
type PostgresClusterValidator struct {
decoder *admission.Decoder
}
func (v *PostgresClusterValidator) Handle(ctx context.Context, req admission.Request) admission.Response {
pg := &examplev1.PostgresCluster{}
if err := v.decoder.Decode(req, pg); err != nil {
return admission.Errored(http.StatusBadRequest, err)
}
// Validate: replicas must be odd for quorum
if pg.Spec.Replicas%2 == 0 {
return admission.Denied(fmt.Sprintf("replicas must be odd, got %d", pg.Spec.Replicas))
}
// Validate: backup schedule must be valid cron
if pg.Spec.Backup.Schedule != "" {
if _, err := cron.ParseStandard(pg.Spec.Backup.Schedule); err != nil {
return admission.Denied(fmt.Sprintf("invalid backup schedule: %v", err))
}
}
return admission.Allowed("valid")
}
Output
Validating webhook that rejects even replica counts and invalid cron schedules.
💡Webhook Timeouts
Webhooks must respond within 30 seconds (default). For slow operations (e.g., checking external systems), use async validation or move logic to the controller.
📊 Production Insight
In production, a misconfigured webhook can block all resource operations. Always set failurePolicy: Ignore for non-critical webhooks, and use sideEffects: None to allow dry-run requests.
🎯 Key Takeaway
Admission webhooks enforce complex validation and defaulting that CRD schemas cannot express.
Operator Lifecycle Manager (OLM): Packaging and Distribution
OLM is a framework for managing Operators on Kubernetes clusters. It provides a catalog of Operators, dependency resolution, and automatic upgrades. To publish your Operator via OLM, you create a ClusterServiceVersion (CSV) that describes the Operator's metadata, CRDs, permissions, and deployment. The CSV is bundled into a package and added to a catalog. Users install your Operator via kubectl apply or OperatorHub. OLM handles upgrades by watching the catalog for new versions. It also manages RBAC: the Operator gets a service account with the permissions declared in the CSV. This is the standard way to distribute Operators for production use.
OLM adds complexity. For internal Operators, manual deployment via Helm or kustomize is simpler. Use OLM when distributing to external users or when you need automatic upgrades.
📊 Production Insight
In production, pin your Operator's OLM channel to a specific version. Auto-upgrades can break if the new version changes CRD schema incompatibly. Test upgrades in a staging cluster first.
🎯 Key Takeaway
OLM standardizes Operator packaging, distribution, and lifecycle management, but adds overhead suitable for public Operators.
Production Pitfalls: Rate Limiting, Leader Election, and Observability
Operators run in production clusters and must be resilient. Rate limiting: the controller-runtime client has built-in rate limiting, but you may need to tune it for high-throughput scenarios. Use MaxConcurrentReconciles to control parallelism. Leader election: if you run multiple replicas for high availability, only one should reconcile at a time. Controller-runtime supports leader election via ConfigMap or Lease objects. Observability: expose metrics via Prometheus (controller-runtime automatically serves metrics on port 8080). Add structured logging with logr. Create dashboards for reconciliation latency, error rates, and queue depth. Without observability, you're flying blind.
main.goGO
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
package main
import (
"flag""os""go.uber.org/zap/zapcore""sigs.k8s.io/controller-runtime/pkg/log/zap""sigs.k8s.io/controller-runtime/pkg/manager""sigs.k8s.io/controller-runtime/pkg/metrics/server"
examplev1 "example.com/api/v1""example.com/postgres-operator/internal/controller"
)
func main() {
var metricsAddr string
var enableLeaderElection bool
var probeAddr string
flag.StringVar(&metricsAddr, "metrics-bind-address", ":8080", "The address the metric endpoint binds to.")
flag.StringVar(&probeAddr, "health-probe-bind-address", ":8081", "The address the probe endpoint binds to.")
flag.BoolVar(&enableLeaderElection, "leader-elect", false,
"Enable leader election for controller manager. "+
"Enabling this will ensure there is only one active controller manager.")
flag.Parse()
ctrl.SetLogger(zap.New(zap.UseFlagOptions(&zap.Options{
Development: true,
TimeEncoder: zapcore.ISO8601TimeEncoder,
})))
mgr, err := manager.New(ctrl.GetConfigOrDie(), manager.Options{
Metrics: server.Options{
BindAddress: metricsAddr,
},
HealthProbeBindAddress: probeAddr,
LeaderElection: enableLeaderElection,
LeaderElectionID: "postgres-operator.example.com",
})
if err != nil {
setupLog.Error(err, "unable to start manager")
os.Exit(1)
}
if err = (&controller.PostgresClusterReconciler{
Client: mgr.GetClient(),
Scheme: mgr.GetScheme(),
}).SetupWithManager(mgr); err != nil {
setupLog.Error(err, "unable to create controller", "controller", "PostgresCluster")
os.Exit(1)
}
if err := mgr.AddHealthzCheck("healthz", mgr.GetWebhookServer().StartedChecker); err != nil {
setupLog.Error(err, "unable to set up health check")
os.Exit(1)
}
if err := mgr.AddReadyzCheck("readyz", mgr.GetWebhookServer().StartedChecker); err != nil {
setupLog.Error(err, "unable to set up ready check")
os.Exit(1)
}
setupLog.Info("starting manager")
if err := mgr.Start(ctrl.SetupSignalHandler()); err != nil {
setupLog.Error(err, "problem running manager")
os.Exit(1)
}
}
Output
Main function with leader election, metrics, and health probes.
⚠ Leader Election in Small Clusters
Leader election uses a Lease object. If the leader crashes, it takes ~15 seconds for a new leader to take over. For critical Operators, consider running a single replica to avoid failover delay.
📊 Production Insight
A common production incident: Operator OOM due to unbounded memory usage from caching all custom resources. Set CacheSyncTimeout and use selective caching with client.ObjectList filters.
🎯 Key Takeaway
Production Operators need rate limiting, leader election, and observability (metrics, logs, health probes) to operate reliably.
Real-World Example: Building a Backup Operator
Let's build a simple BackupOperator that manages scheduled backups for any resource with a backup annotation. The CRD defines a BackupSchedule with fields for schedule, retention, and target resource selector. The Operator creates CronJobs that run backup scripts. It also handles cleanup of old backups. This pattern is common: many Operators manage scheduled tasks (e.g., cert renewal, snapshotting). The key is to use Kubernetes CronJobs for scheduling and let the Operator manage their lifecycle. The Operator watches BackupSchedule resources and ensures the corresponding CronJob exists with the correct schedule. It also updates status with the last backup time and number of retained backups.
Don't reinvent backup logic. Integrate with tools like Velero or pg_dump. Your Operator should orchestrate, not implement low-level backup code.
📊 Production Insight
In production, backup Operators must handle concurrent backups. Use a mutex or check that no other backup is running before starting a new one. Otherwise, you risk data corruption.
🎯 Key Takeaway
Operators can manage scheduled tasks by creating CronJobs, demonstrating the pattern of using Kubernetes primitives to implement custom logic.
Built-in Resources vs CRDsComparing flexibility, lifecycle, and operational complexityBuilt-in ResourcesCustom Resources (CRDs)Schema DefinitionFixed by Kubernetes coreUser-defined via OpenAPI v3Controller LogicBuilt-in controllers onlyCustom operator controllersVersioningStable API versionsMultiple versions with conversionLifecycleManaged by KubernetesUser-managed CRD and operatorExtensibilityLimited to core typesUnlimited custom resourcesOperational BurdenLow (no extra components)High (operator deployment, monitoring)THECODEFORGE.IO
thecodeforge.io
Kubernetes Crds Operators
When Not to Use an Operator: Alternatives and Trade-offs
Operators are powerful but overkill for many scenarios. If you just need to deploy a simple app with configurable parameters, use Helm charts. If you need to run a one-time job, use a CronJob. If you need to react to events (e.g., scale based on metrics), use the Horizontal Pod Autoscaler. Operators shine when you need to manage the full lifecycle of a complex, stateful application with domain-specific logic. They require significant development effort and ongoing maintenance. Consider the cost: writing an Operator takes weeks, not days. For internal tools, a Python script with kubectl might suffice. Always ask: can this be done with existing Kubernetes resources? If yes, don't build an Operator.
Helm is simpler for basic deployments; Operator adds automation for complex operations.
🔥Operator Maturity Model
Level I: Basic install. Level II: Seamless upgrades. Level III: Full lifecycle (backup, recovery, scaling). Level IV: Deep insights (metrics, alerts). Level V: Auto-pilot. Most Operators never go beyond Level III.
📊 Production Insight
In production, I've seen teams build Operators for every little thing, creating a maintenance nightmare. Be ruthless: if a Helm chart with a post-install hook works, use that.
🎯 Key Takeaway
Operators are not a silver bullet. Use them only when the complexity of the application's lifecycle justifies the development cost.
They let you define custom resource types that are stored in etcd and managed via kubectl, enabling domain-specific declarative management.
2
Operators automate operational knowledge
A controller watches custom resources and runs a reconciliation loop to converge the cluster state to the desired state, encoding expertise like backup, failover, and upgrades.
3
Version CRDs carefully
Use multiple versions with conversion webhooks to evolve your API without breaking existing users. Never delete a version that has stored objects.
4
Test at multiple levels
Unit tests with fake clients, integration tests with envtest, and e2e tests on real clusters. Without thorough testing, Operators become a source of production incidents.
INTERVIEW PREP · PRACTICE MODE
Interview Questions on This Topic
Q01JUNIOR
What is the difference between a CRD and an Operator?
Q02JUNIOR
Can I write an Operator in Python or Ansible?
Q03JUNIOR
How do I handle upgrades of my CRD without breaking existing objects?
Q01 of 03JUNIOR
What is the difference between a CRD and an Operator?
ANSWER
A CRD defines a new resource type in Kubernetes (like a custom API object). An Operator is a controller that watches those custom resources and takes actions to manage their lifecycle. You need both:
Q02 of 03JUNIOR
Can I write an Operator in Python or Ansible?
ANSWER
Yes, the Operator SDK supports Ansible and Helm-based Operators. However, Go-based Operators are more performant and have better ecosystem support. For production, Go is recommended.
Q03 of 03JUNIOR
How do I handle upgrades of my CRD without breaking existing objects?
ANSWER
Use CRD versioning with conversion webhooks. Add a new version (e.g., v2) and write a webhook to convert between v1 and v2. Keep the old version served until all objects are migrated. Never delete a v
01
What is the difference between a CRD and an Operator?
JUNIOR
02
Can I write an Operator in Python or Ansible?
JUNIOR
03
How do I handle upgrades of my CRD without breaking existing objects?
JUNIOR
FAQ · 6 QUESTIONS
Frequently Asked Questions
01
What is the difference between a CRD and an Operator?
A CRD defines a new resource type in Kubernetes (like a custom API object). An Operator is a controller that watches those custom resources and takes actions to manage their lifecycle. You need both: the CRD for the data model, the Operator for the automation logic.
Was this helpful?
02
Can I write an Operator in Python or Ansible?
Yes, the Operator SDK supports Ansible and Helm-based Operators. However, Go-based Operators are more performant and have better ecosystem support. For production, Go is recommended.
Was this helpful?
03
How do I handle upgrades of my CRD without breaking existing objects?
Use CRD versioning with conversion webhooks. Add a new version (e.g., v2) and write a webhook to convert between v1 and v2. Keep the old version served until all objects are migrated. Never delete a version that has stored objects.
Was this helpful?
04
What is a finalizer and why do I need it?
A finalizer is a string on a Kubernetes object that prevents deletion until the finalizer is removed. Operators use finalizers to ensure cleanup of external resources (e.g., cloud resources, PVCs) before the custom resource is deleted. Without finalizers, deletion can leave orphaned resources.
Was this helpful?
05
How do I test my Operator locally?
Use envtest (part of controller-runtime) to run a local API server and etcd. Write integration tests that create CRs and verify reconciliation. For quick iteration, use make run which starts the Operator locally against your current kubeconfig.
Was this helpful?
06
What are the most common production issues with Operators?
Top issues: (1) Operator crashes due to nil pointer dereference when a child resource is missing. (2) Stuck finalizers preventing deletion. (3) Webhook misconfiguration blocking all API requests. (4) Memory leaks from unbounded caching. (5) Race conditions in reconciliation logic.