Operators & CRDs

CRD + controller = operator

Built-in controllers (Deployment, StatefulSet) already follow this model: you set spec.replicas: 3; the Deployment controller creates ReplicaSets and Pods until reality matches. Operators extend the same reconciliation loop to application domains Kubernetes does not natively understand.

Production operator examples

$ kubectl get crd | grep -E 'postgres|kafka|cert-manager|monitoring'
$ kubectl api-resources --api-group=postgresql.cnpg.io
$ kubectl get postgrescluster -A
$ kubectl get certificate -A
$ kubectl get servicemonitor -n monitoring$ oc get crd | grep -E 'postgres|kafka|cert-manager|monitoring'
$ oc get csv -A | grep -i postgres
$ oc get kafkas.kafka.strimzi.io -A
$ oc get clusterissuer
$ oc get packagemanifest -n openshift-marketplace | grep -i cert

Extending the Kubernetes API

Every API resource has a GroupVersionKind (GVK). A CRD declares the group (e.g. cache.example.com), version(s), scope (Namespaced or Cluster), and schema. The API server stores CR instances in etcd alongside Pods and Deployments.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: caches.cache.example.com   # <plural>.<group>
spec:
  group: cache.example.com
  scope: Namespaced
  names:
    plural: caches
    singular: cache
    kind: Cache
    shortNames:
      - ch
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              required: [replicas, memory]
              properties:
                replicas:
                  type: integer
                  minimum: 1
                  maximum: 10
                memory:
                  type: string
                  pattern: '^[0-9]+(Mi|Gi)$'
            status:
              type: object
              properties:
                readyReplicas:
                  type: integer
                conditions:
                  type: array
                  items:
                    type: object
                    properties:
                      type:
                        type: string
                      status:
                        type: string
                        enum: ["True", "False", "Unknown"]
                      reason:
                        type: string
                      message:
                        type: string
      subresources:
        status: {}
      additionalPrinterColumns:
        - name: Replicas
          type: integer
          jsonPath: .spec.replicas
        - name: Ready
          type: integer
          jsonPath: .status.readyReplicas
        - name: Age
          type: date
          jsonPath: .metadata.creationTimestamp

OpenAPI schema validation

Since Kubernetes 1.16+, structural schemas are required for apiextensions.k8s.io/v1 CRDs. The API server rejects CRs that violate the schema at admission time—before they reach etcd. Use kubectl explain cache.spec to introspect fields (works like built-in resources).

Version lifecycle: v1alpha1 → v1

CRD versions follow the same maturity path as core APIs. Start with v1alpha1 (experimental, may break), graduate to v1beta1 (more stable), then v1 (GA). Multiple versions can be served: true simultaneously; exactly one is storage: true (the version persisted in etcd).

Conversion webhooks

When multiple versions are served, the API server may need to convert between storage and requested versions. A conversion webhook (spec.conversion.strategy: Webhook) calls your service to translate v1beta1 ↔ v1 fields. Without conversion, only one version can be served.

Status subresource

Splitting spec (user intent) from status (observed state) is a Kubernetes best practice. Enabling subresources.status: {} lets controllers update status via /status without triggering full-object reconciliation loops or conflicting with user spec edits.

Finalizers

metadata.finalizers block CR deletion until the controller removes them—ensuring cleanup (delete cloud volumes, revoke certs, drain Kafka partitions) completes before the object disappears. A stuck finalizer leaves the CR in Terminating state indefinitely.

$ kubectl apply -f cache-crd.yaml
$ kubectl get crd caches.cache.example.com
$ kubectl explain cache.spec
$ kubectl get caches -o wide
$ kubectl describe cache my-cache
→ check Events for schema validation failures
$ kubectl patch cache my-cache --type=merge -p '{"metadata":{"finalizers":[]}}'
→ last resort: remove stuck finalizer (data loss risk)$ oc apply -f cache-crd.yaml
$ oc get crd caches.cache.example.com -o yaml
$ oc explain cache.spec
$ oc get caches -o wide
$ oc adm inspect crd/caches.cache.example.com --dest-dir=/tmp/inspect

Operator SDK workflows

controller-runtime reconcile loop

The core pattern: watch resources → enqueue reconcile key → reconcile until desired state matches actual state → requeue on error or periodic resync. Idempotency is mandatory—the same reconcile may run many times for one spec change.

flowchart TB
  W["Informer watches\nCR + owned resources"] --> Q["Work queue"]
  Q --> R["Reconcile(req)"]
  R --> G{"Get CR\nfrom API"}
  G -->|NotFound| D["Done — object deleted"]
  G -->|Found| C{"Finalizer\non delete?"}
  C -->|Deleting| CL["Run cleanup\nremove finalizer"]
  C -->|Active| S["Read spec\ncompare to actual"]
  S --> A["Create/Update/Patch\noperands (STS, SVC, CM)"]
  A --> U["Update status\nconditions + generation"]
  U --> OK{"Error?"}
  OK -->|No| Q2["Requeue after interval\nor done"]
  OK -->|Yes| E["Requeue with backoff"]
  CL --> D
  E --> Q
  Q2 --> Q

Owner references

Set ownerReferences on child objects (StatefulSet, Service, ConfigMap) pointing to the parent CR. Benefits: garbage collection deletes children when the CR is deleted; secondary watches re-enqueue the parent when a child changes. Use controllerutil.SetControllerReference() in Go.

Conditions

Report observable state in status.conditions—mirroring built-in resources like Node and Deployment. Standard pattern: Ready, Progressing, Degraded with type, status (True/False/Unknown), reason, message, and lastTransitionTime.

Generation tracking

metadata.generation increments on every spec change. Store status.observedGeneration in your CR status. When observedGeneration < generation, reconciliation is stale—useful for "RollingUpgradeInProgress" conditions and avoiding false Ready signals during spec updates.

func (r *CacheReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    var cache cachev1.Cache
    if err := r.Get(ctx, req.NamespacedName, &cache); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    // Handle deletion + finalizer
    if !cache.DeletionTimestamp.IsZero() {
        if controllerutil.ContainsFinalizer(&cache, finalizerName) {
            if err := r.cleanup(ctx, &cache); err != nil {
                return ctrl.Result{}, err
            }
            controllerutil.RemoveFinalizer(&cache, finalizerName)
            return ctrl.Result{}, r.Update(ctx, &cache)
        }
        return ctrl.Result{}, nil
    }

    // Ensure finalizer on create
    if !controllerutil.ContainsFinalizer(&cache, finalizerName) {
        controllerutil.AddFinalizer(&cache, finalizerName)
        return ctrl.Result{}, r.Update(ctx, &cache)
    }

    // Reconcile operands
    sts := r.statefulSetFor(&cache)
    if err := controllerutil.SetControllerReference(&cache, sts, r.Scheme); err != nil {
        return ctrl.Result{}, err
    }
    if err := r.createOrUpdate(ctx, sts); err != nil {
        return ctrl.Result{}, err
    }

    // Update status
    cache.Status.ReadyReplicas = *sts.Spec.Replicas
    cache.Status.ObservedGeneration = cache.Generation
    meta.SetStatusCondition(&cache.Status.Conditions, metav1.Condition{
        Type:   "Ready",
        Status: metav1.ConditionTrue,
        Reason: "Reconciled",
    })
    return ctrl.Result{RequeueAfter: 5 * time.Minute}, r.Status().Update(ctx, &cache)
}

$ operator-sdk init --domain example.com --repo github.com/you/cache-operator
$ operator-sdk create api --group cache --version v1 --kind Cache --resource --controller
$ make install   # apply CRDs to cluster
$ make run       # run controller locally against cluster
$ make docker-build docker-push IMG=quay.io/you/cache-operator:v0.1.0
$ make deploy IMG=quay.io/you/cache-operator:v0.1.0
$ kubectl logs -n cache-operator-system deployment/cache-operator-controller-manager -f$ operator-sdk init --domain example.com --repo github.com/you/cache-operator
$ make install && make run
$ oc new-project cache-operator
$ make deploy IMG=image-registry.openshift-image-registry.svc:5000/cache-operator/controller:v0.1.0
$ oc logs -n cache-operator deployment/cache-operator-controller-manager -f
$ oc adm policy add-scc-to-user anyuid -z cache-operator-controller-manager
→ only if operator image requires non-default UID

Evaluating operators for procurement

1. Ask the vendor which level they claim—and which CRs prove it (backup CR? upgrade CR?)
2. Level 3+ is typical minimum for production stateful services (databases, messaging)
3. Level 5 is rare; treat marketing claims skeptically until demonstrated in your failure scenarios
4. Red Hat Certified Operators must document supported upgrade paths—often Level 2–3 minimum

ClusterServiceVersion (CSV)

A ClusterServiceVersion is the operator's install manifest bundle: Deployment spec, owned CRD definitions, required RBAC, webhook configurations, dependency constraints, and maturity metadata. OLM transitions CSV phases: Pending → Installing → Succeeded (or Failed).

OperatorHub

OperatorHub aggregates PackageManifest entries from CatalogSource objects. On OpenShift, default catalogs include redhat-operators, redhat-marketplace, and community-operators. Platform teams can publish private catalogs via index images.

Subscription channels

Each operator package exposes channels (e.g. stable, fast, candidate) mapping to CSV version lines. A Subscription pins the desired channel; OLM upgrades the installed CSV when a newer version appears in that channel—unless blocked by startingCSV or manual approval.

flowchart LR
  OH["OperatorHub\n(console browse)"] --> CS["CatalogSource\n(gRPC index)"]
  CS --> PM["PackageManifest"]
  SUB["Subscription\nchannel + source"] --> IP["InstallPlan"]
  IP -->|Manual approval| AP["Admin approves"]
  AP --> CSV["CSV applied\nCRDs + RBAC + Deployment"]
  IP -->|Automatic| CSV
  CSV --> OP["Operator pod running"]
  OP --> CR["User creates CRs"]

InstallPlan manual approval

Set installPlanApproval: Manual on the Subscription for change-controlled environments. OLM creates an InstallPlan in Pending state; a cluster admin reviews the CSV and dependency list, then approves: spec.approved: true. Automatic approval applies immediately—faster but riskier during platform upgrades.

apiVersion: operators.coreos.com/v1alpha1
kind: OperatorGroup
metadata:
  name: postgres-operators
  namespace: operators
spec:
  targetNamespaces:
    - databases
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: crunchy-postgres-operator
  namespace: operators
spec:
  channel: v5
  name: crunchy-postgres-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace
  installPlanApproval: Manual
  startingCSV: postgresoperator.v5.5.0

$ # OLM on vanilla K8s — install operator-framework first
$ kubectl get csv -A
$ kubectl get subscription,installplan -A
$ kubectl describe installplan <name> -n operators
→ approve: kubectl patch installplan <name> -n operators --type merge -p '{"spec":{"approved":true}}'
$ operator-sdk run bundle quay.io/operator/cache-operator-bundle:v0.1.0$ oc get packagemanifest -n openshift-marketplace | grep postgres
$ oc apply -f operatorgroup.yaml -f subscription.yaml
$ oc get installplan -n operators
$ oc patch installplan <name> -n operators --type merge -p '{"spec":{"approved":true}}'
$ oc get csv -n operators
→ PHASE: Succeeded required before creating application CRs
$ oc describe csv postgresoperator.v5.5.0 -n operators
$ oc get subscription crunchy-postgres-operator -n operators -o yaml
$ oc get catalogsource -n openshift-marketplace