kubernetes-operator

Kubernetes Operator

End-to-end Kubernetes operator design and construction. Covers the operator pattern (control loops for stateful workloads), CRD design (schema, validation, conversion, status), the reconciliation loop (idempotency, convergence, level-triggered vs edge-triggered), framework selection (controller-runtime / Kubebuilder / operator-SDK / metacontroller), and the operational concerns (finalizers, leader election, RBAC scoping, status subresource, observability).

This skill works with Go-based operators (the dominant ecosystem) and notes when alternative languages or frameworks (Python / KOPF, Java / JOSDK) apply.

When to use this skill

| Situation | Skill applies | |-----------|---------------| | Building a new operator for an internal platform primitive | Yes — start with the operator pattern decision | | Auditing an existing operator for production-readiness | Yes — use anti-patterns + scripts/reconciliation_audit.py | | Designing CRDs for a custom resource | Yes — use CRD design + scripts/crd_validator.py | | Deciding "operator vs Helm chart vs plain manifests" | Yes — use the decision matrix below | | Scaffolding a new operator project | Yes — scripts/operator_scaffold.py | | Debugging a controller that "isn't reconciling" | Yes — use reconciliation troubleshooting | | Just running someone else's operator (Postgres, Kafka, etc.) | Partially — useful for understanding what it does and how to monitor it |

When NOT to write an operator

Operators are extensions of Kubernetes. They cost ongoing maintenance, RBAC review, security audit, version-skew handling, and (often) a dedicated SRE rotation. Avoid if:

• Your resource is stateless and fits CronJob + Deployment + ConfigMap (no extra control loop needed)
• You can model the workload as a Helm chart with values overrides
• An existing community operator (CNCF / vendor) covers your needs — adopt it instead
• The lifecycle has < 5 transitions (then a Job + readiness probe is simpler)
• You'd be the only consumer (consider a less heavyweight extension point — admission webhook, Lua/JSONNet, GitOps controller config)

A useful rule: write an operator only when the resource has at least 2-3 non-trivial state transitions AND existing primitives can't model them cleanly.

The operator pattern in one paragraph

An operator is a controller (Kubernetes control-loop program) that watches CustomResources (CRs) representing application-specific state, and drives the cluster toward the desired state declared in the CR's spec, recording observed state in the CR's status. The pattern: a domain expert encodes operational knowledge as software — "when CR X is created, do A; when X.spec.replicas changes, do B; when underlying Pod fails, do C" — so users get a declarative API instead of a wiki of runbooks.

Three building blocks:

1. CRD (CustomResourceDefinition) — schema for the new resource type
2. Controller — control loop that reconciles spec → state
3. Custom Resource (CR) — instances users create to ask for things

Operator vs alternatives — decision matrix

| Need | Use | |------|-----| | Run a stateless app | Deployment + Service | | Run a stateful app with simple lifecycle | StatefulSet + headless Service | | Manage configuration drift | Argo CD / Flux (GitOps controllers) | | Add validation/mutation to existing K8s API | Admission webhook (no operator) | | Manage external resources from inside cluster | Operator (if non-trivial), OR Crossplane (if matches its model) | | Automate complex stateful workloads with domain expertise | Operator | | Multi-cluster federation | Operator + push to other clusters (e.g., Karmada, KubeFed) | | Simple "do X on YAML change" | Metacontroller (declarative composition, no Go) |

CRD design — the foundation

A bad CRD = perpetual operator pain. Spend time here.

Required design decisions

| Decision | Options | Default | |----------|---------|---------| | Scope | Namespaced / Cluster | Namespaced (unless cluster-wide makes no sense without it) | | Versioning | v1alpha1 / v1beta1 / v1 | Start v1alpha1, graduate per Kubernetes API conventions | | Conversion | None / Webhook / None-with-storage-version | Webhook once you have > 1 served version | | Subresources | status / scale / both | Always enable status; scale if user-controlled replicas | | Validation | OpenAPI schema / admission webhook / both | OpenAPI for shape; webhook for cross-field | | Printer columns | Yes (recommended) | Always — kubectl get is much nicer | | Categories | Optional | Add for grouping (e.g., kubectl get all,databases) | | Short names | Optional | Add (e.g., db for Database) |

Spec / Status separation

Spec = what the user wants (desired state) Status = what the operator observes (actual state)

The user writes Spec. The operator writes Status. Never the other way round.

apiVersion: example.com/v1
kind: Database
metadata:
  name: prod-orders
spec:                          # user-owned
  version: "14.10"
  storage: 100Gi
  replicas: 3
  backup:
    schedule: "0 2 * * *"
status:                        # operator-owned
  phase: Running
  observedGeneration: 5
  conditions:
    - type: Available
      status: "True"
      reason: AllReplicasReady
      lastTransitionTime: "2026-05-27T08:00:00Z"
  endpoints:
    primary: prod-orders-primary.default.svc.cluster.local
    replicas: ["prod-orders-replica-0.default.svc.cluster.local"]
  observedVersion: "14.10"

CRD schema rules

• Use OpenAPI v3 schema, structural (no x-kubernetes-preserve-unknown-fields unless you really need it)
• Validate at the API edge — required fields, enum values, format, regex pattern, min/max
• Use defaults sparingly — every default is something users can't tell the operator "I don't care, you decide"
• Add descriptions — they show up in kubectl explain, which is how users learn your API
• Use x-kubernetes-validations (CEL, K8s 1.25+) for cross-field validation that doesn't need a webhook
• Version your API thoughtfully — v1alpha1 can break; v1 is forever

See [references/operator-pattern-and-crds.md](references/operator-pattern-and-crds.md) for the full schema design guide including conversion webhooks, status subresource semantics, and the printer-column DSL.

The reconciliation loop

The control loop is the heart of an operator. It runs whenever a watched resource changes (or periodically).

loop {
  obj = fetch(CR_key)
  if (obj is being deleted) {
    handle_finalizer(obj)
    return
  }
  desired = compute_desired_state(obj.spec)
  actual = observe_actual_state(cluster)
  if (desired != actual) {
    apply_diff(desired, actual)
  }
  update_status(obj, actual)
  if (transient_error) requeue_after(backoff)
}

Five rules of reconciliation

1. Idempotent. Running reconcile twice with the same inputs produces the same effect. No "create or fail" — always "ensure exists."
2. Level-triggered, not edge-triggered. Reconcile from current state, not from the diff of what changed. Don't say "user changed X, so increment Y" — say "Y should equal f(X), check and set if needed."
3. Converge. Each reconcile gets closer to desired state, or stays there. Doesn't oscillate.
4. Fail safe. If something errors, requeue with exponential backoff. Don't crash; don't loop tight.
5. Observed generation tracking. Status.observedGeneration tells users "yes, I saw your latest spec change."

Common reconciliation patterns

| Pattern | When | |---------|------| | Direct apply | Stateless dependent resources (Deployments, Services, ConfigMaps owned by this CR) | | Phase machine | Multi-step lifecycle with explicit phases (Pending → Bootstrapping → Running → Updating → Terminating) | | Sub-controllers | One controller per logical concern (e.g., one for the StatefulSet, one for backups, one for network policies) | |