Helm, Kustomize & Package Management

Chart structure

A standard chart follows the Helm project layout. Templates live under templates/; default configuration is in values.yaml. Files beginning with _ are not rendered as manifests.

my-app/
├── Chart.yaml          # chart metadata (name, version, dependencies)
├── values.yaml         # default values — overridden at install/upgrade
├── values.schema.json  # optional JSON Schema validation for values
├── charts/             # subchart dependencies (vendored or from helm dependency)
├── templates/
│   ├── _helpers.tpl    # named template definitions (include/define)
│   ├── deployment.yaml
│   ├── service.yaml
│   ├── ingress.yaml
│   ├── hpa.yaml
│   ├── pdb.yaml
│   ├── networkpolicy.yaml
│   ├── rbac.yaml
│   ├── NOTES.txt       # post-install instructions (helm install output)
│   └── tests/          # helm test hook pods
└── .helmignore

apiVersion: v2
name: payments-api
description: Payments API — production Helm chart
type: application          # application | library
version: 2.4.1             # chart version (semver) — bump on template/values changes
appVersion: "1.18.3"       # app version label — informational, not used for upgrades
kubeVersion: ">=1.25.0-0"
dependencies:
  - name: postgresql
    version: "12.x.x"
    repository: https://charts.bitnami.com/bitnami
    condition: postgresql.enabled
maintainers:
  - name: platform-team
    email: platform@example.com

Go templates

Helm renders templates with Go's text/template plus Sprig functions. Values are accessed as .Values.replicaCount; chart metadata as .Chart.Name and .Release.Name. Use {{ include "my-app.fullname" . }} from _helpers.tpl for consistent naming.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "payments-api.fullname" . }}
  labels:
    {{- include "payments-api.labels" . | nindent 4 }}
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      {{- include "payments-api.selectorLabels" . | nindent 6 }}
  template:
    metadata:
      annotations:
        checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
      labels:
        {{- include "payments-api.selectorLabels" . | nindent 8 }}
    spec:
      serviceAccountName: {{ include "payments-api.serviceAccountName" . }}
      containers:
        - name: {{ .Chart.Name }}
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
          resources:
            {{- toYaml .Values.resources | nindent 12 }}

Install, upgrade, rollback

helm install creates a release (revision 1). helm upgrade applies a new chart/values set and increments revision. helm rollback reverts to a prior revision without re-running CI—useful for fast incident recovery. Use --atomic to auto-rollback on failure and --wait to block until resources are ready.

stateDiagram-v2
  [*] --> Rev1: helm install
  Rev1 --> Rev2: helm upgrade (new image tag)
  Rev2 --> Rev3: helm upgrade (values change)
  Rev3 --> Rev2: helm rollback 2
  Rev2 --> [*]: helm uninstall

$ helm repo add bitnami https://charts.bitnami.com/bitnami
$ helm repo update
$ helm install payments ./charts/payments-api \
  --namespace payments --create-namespace \
  -f values/prod.yaml --set image.tag=1.18.3
$ helm upgrade payments ./charts/payments-api \
  -f values/prod.yaml --atomic --wait --timeout 5m
$ helm history payments -n payments
$ helm rollback payments 3 -n payments
$ helm template payments ./charts/payments-api -f values/prod.yaml | kubectl apply -f - --dry-run=server
$ kubectl get secret -n payments -l owner=helm$ helm install payments ./charts/payments-api \
  --namespace payments --create-namespace \
  -f values/ocp-prod.yaml
→ ocp-prod.yaml may set route.enabled, securityContextConstraints
$ oc get route -n payments
$ helm upgrade payments ./charts/payments-api --atomic --wait
$ oc adm policy scc-to-user anyuid -z payments-api -n payments
→ only when chart requires non-default UID; prefer restricted SCC
$ helm history payments -n payments

Releases & values override

Release names are unique per namespace. Values merge in order: chart defaults → parent values → -f files (last wins) → --set / --set-string. For GitOps, render with helm template and commit manifests, or use ArgoCD's native Helm source.

Chart repositories

helm repo add registers an HTTP index (index.yaml). OCI registries (Harbor, ECR, GHCR) are first-class in Helm 3.8+: helm install my-app oci://registry.example.com/charts/my-app --version 1.2.0. Pin chart versions in CI; helm search repo lists available versions.

Helm Secrets & SOPS

Never commit plaintext Secrets to git. Patterns:

• helm-secrets plugin — decrypts SOPS-encrypted value files at render time (helm secrets install -f secrets.prod.yaml)
• External Secrets Operator / Sealed Secrets — chart references ExternalSecret CR; cluster controller materializes Secrets
• Mozilla SOPS — encrypt YAML/JSON with KMS (AWS, GCP, Azure) or age/PGP keys; .sops.yaml defines creation rules

# secrets.prod.yaml — encrypted with sops; safe in git
database:
  password: ENC[AES256_GCM,data:...,iv:...,tag:...,type:str]
sops:
  kms:
    - arn: arn:aws:kms:us-east-1:123456789:key/abc
      created_at: "2026-01-15T10:00:00Z"
      enc: AQICAHh...base64...
  lastmodified: "2026-01-15T10:00:00Z"
  version: 3.9.0

Hooks

Annotate resources with helm.sh/hook to run at specific lifecycle points. Hook pods/jobs run outside the normal upgrade order—critical for DB migrations and cache warmers.

Set helm.sh/hook-weight for ordering and helm.sh/hook-delete-policy (hook-succeeded, before-hook-creation) to control hook resource cleanup.

Library charts

Charts with type: library in Chart.yaml provide shared templates only—no installable release. Parent charts import them via dependencies and call {{ include "common.labels" . }}. Bitnami's common chart is the canonical example: standard labels, affinities, image helpers, and compatibility snippets across dozens of application charts.

Helmfile

Helmfile declaratively orchestrates multiple Helm releases—repos, environments, value file layering, and diff/apply in one manifest. Ideal for platform teams managing 20+ third-party charts (ingress, cert-manager, prometheus, external-dns) with per-environment overrides.

repositories:
  - name: ingress-nginx
    url: https://kubernetes.github.io/ingress-nginx
  - name: jetstack
    url: https://charts.jetstack.io

environments:
  prod:
    values:
      - environments/prod.yaml

releases:
  - name: ingress-nginx
    namespace: ingress
    chart: ingress-nginx/ingress-nginx
    version: 4.10.0
    values:
      - values/ingress-nginx.yaml
  - name: cert-manager
    namespace: cert-manager
    chart: jetstack/cert-manager
    version: v1.14.4
    needs:
      - ingress/ingress-nginx
    values:
      - values/cert-manager.yaml

$ helmfile -e prod diff
→ preview changes before apply (requires helm-diff plugin)
$ helmfile -e prod apply
$ helmfile -l name=cert-manager destroy$ helmfile -e prod -f helmfile-ocp.yaml apply
→ ocp overlay may swap Ingress for Route templates
$ oc get co ingress

Requests & limits

Always expose resources.requests and resources.limits in values with sensible defaults. Requests drive scheduling and HPA/VPA calculations; limits prevent noisy-neighbor OOM cascades. Document expected CPU/memory profiles in values.yaml comments.

resources:
  requests:
    cpu: 250m
    memory: 512Mi
  limits:
    cpu: "1"
    memory: 1Gi

# Allow disabling limits for JVM workloads that need burstable memory
resourcesPolicy:
  enforceLimits: true

HPA, PDB, NetworkPolicy, RBAC in chart

Ship optional templates gated by values flags. Consumers enable what their cluster supports without editing raw YAML.

{{- if .Values.autoscaling.enabled }}
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: {{ include "payments-api.fullname" . }}
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: {{ include "payments-api.fullname" . }}
  minReplicas: {{ .Values.autoscaling.minReplicas }}
  maxReplicas: {{ .Values.autoscaling.maxReplicas }}
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: {{ .Values.autoscaling.targetCPUUtilizationPercentage }}
{{- end }}

NOTES.txt

templates/NOTES.txt prints after helm install—the first thing an operator sees. Include: how to reach the app (URL/port-forward), how to check health, where logs live, and links to runbooks.

1. Get the application URL:
{{- if .Values.ingress.enabled }}
  https://{{ .Values.ingress.host }}
{{- else }}
  export POD=$(kubectl get pods -l app.kubernetes.io/name={{ include "payments-api.name" . }} -o jsonpath='{.items[0].metadata.name}')
  kubectl port-forward $POD 8080:8080
  echo "Visit http://127.0.0.1:8080"
{{- end }}

2. Check rollout status:
  kubectl rollout status deployment/{{ include "payments-api.fullname" . }} -n {{ .Release.Namespace }}

_helpers.tpl

Centralize naming and labels in templates/_helpers.tpl to satisfy recommended labels and avoid 63-character DNS label truncation bugs.

{{- define "payments-api.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
{{- end }}

{{- define "payments-api.fullname" -}}
{{- if .Values.fullnameOverride }}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- $name := default .Chart.Name .Values.nameOverride }}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
{{- end }}
{{- end }}

{{- define "payments-api.labels" -}}
helm.sh/chart: {{ include "payments-api.chart" . }}
{{ include "payments-api.selectorLabels" . }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end }}

Semver: chart version vs app version

version in Chart.yaml is the chart semver—bump on any template, default values, or dependency change. appVersion is informational (container image tag label)—it does not trigger upgrades by itself. CI should bump version on every chart publish; tie appVersion to the application release tag.

kustomization.yaml fields

The kustomization.yaml file at each directory root declares how to build a set of resources.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: payments-prod

resources:
  - ../../base
  - ingress.yaml

commonLabels:
  env: prod
  team: payments

commonAnnotations:
  app.kubernetes.io/managed-by: kustomize

namePrefix: prod-

images:
  - name: registry.example.com/payments-api
    newTag: v1.18.3

configMapGenerator:
  - name: payments-config
    behavior: merge
    literals:
      - LOG_LEVEL=info
      - FEATURE_FLAGS=ledger-v2

secretGenerator:
  - name: payments-tls
    files:
      - tls.crt
      - tls.key
    type: kubernetes.io/tls

patches:
  - path: patch-replicas.yaml
  - target:
      kind: Deployment
      name: payments-api
    patch: |-
      - op: replace
        path: /spec/template/spec/containers/0/resources/limits/memory
        value: 2Gi

replicas:
  - name: payments-api
    count: 6

Base + overlays

The canonical layout keeps a base with environment-agnostic manifests and overlays per environment (dev, staging, prod). Overlays reference the base and apply deltas only—DRY without a templating language.

flowchart TB
  subgraph base["base/"]
    B1["deployment.yaml"]
    B2["service.yaml"]
    B3["kustomization.yaml"]
  end
  subgraph dev["overlays/dev/"]
    D1["kustomization.yaml"]
    D2["patch-resources.yaml"]
  end
  subgraph prod["overlays/prod/"]
    P1["kustomization.yaml"]
    P2["patch-replicas.yaml"]
    P3["ingress.yaml"]
  end
  dev --> base
  prod --> base
  prod -->|"kubectl apply -k overlays/prod"| API["API server"]

deploy/
├── base/
│   ├── kustomization.yaml
│   ├── deployment.yaml
│   ├── service.yaml
│   └── kustomization.yaml   # resources: [deployment.yaml, service.yaml]
└── overlays/
    ├── dev/
    │   ├── kustomization.yaml
    │   └── patch-replicas.yaml
    └── prod/
        ├── kustomization.yaml
        ├── patch-replicas.yaml
        └── ingress.yaml

Patches: strategic merge vs JSON6902

Strategic merge patches use partial resource YAML—Kustomize merges by apiVersion/kind/name using the OpenAPI schema's merge strategy (e.g. append to lists with $patch: replace). JSON6902 (RFC 6902) patches target precise JSON paths—better for surgical edits (replace one env var, one limit field).

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments-api
spec:
  replicas: 6
  template:
    spec:
      containers:
        - name: payments-api
          env:
            - name: LOG_LEVEL
              value: warn

# in kustomization.yaml patches:
- target:
    group: apps
    version: v1
    kind: Deployment
    name: payments-api
  patch: |-
    - op: add
      path: /spec/template/spec/containers/0/env/-
      value:
        name: OTEL_EXPORTER_OTLP_ENDPOINT
        value: http://otel-collector.monitoring:4317

Images transformer

The images field rewrites container image references across all resources—CI can inject newTag or digest without sed scripts. Pair with Kustomize's replacements or ArgoCD Image Updater for automated tag promotion.

Generators & hash suffix

configMapGenerator and secretGenerator create ConfigMaps/Secrets from files or literals. By default, Kustomize appends a content hash suffix to names (e.g. payments-config-7f3k2m) so Pod specs change when content changes—triggering rolling restarts. Disable with generatorOptions.disableNameSuffixHash: true only when you accept manual rollout triggers.

kubectl apply -k

Build and apply in one step, or build to stdout for GitOps commit pipelines:

$ kubectl kustomize deploy/overlays/prod > /tmp/rendered.yaml
$ kubectl apply -k deploy/overlays/prod
$ kubectl diff -k deploy/overlays/prod
→ requires diff plugin; preview before apply
$ kustomize build deploy/overlays/prod | kubeconform -summary
$ kubectl get deploy -n payments-prod -l env=prod$ oc apply -k deploy/overlays/ocp-prod
→ ocp-prod overlay adds Route, SCC-compatible securityContext
$ oc kustomize deploy/overlays/ocp-prod | oc apply -f - --dry-run=server
$ oc get route payments-api -n payments-prod

When to use each

• Helm for third-party — You don't own the upstream manifests. Consume versioned charts from Artifact Hub with values overrides. Upgrades are helm upgrade with changelog review.
• Kustomize for in-house — Your services are already plain YAML in git. Overlays express env deltas without forking a chart. Developers read real Deployment specs, not {{ .Values.foo }}.
• Helm when you need hooks & packaging — DB migration Jobs on upgrade, chart dependencies, OCI distribution to customers.
• Kustomize when GitOps purity matters — Flux/ArgoCD render exactly what's in git; no cluster-side release state beyond applied resources.

helm template | kustomize pattern

Combine both: render third-party Helm charts to static YAML, then Kustomize-patch for cluster-specific labels, NetworkPolicies, or OCP Routes. Commit the rendered output to git for auditable GitOps, or render in CI and push to a manifests repo.

flowchart LR
  HC["Helm chart\n(third-party)"] --> HT["helm template"]
  HT --> RAW["rendered/"]
  RAW --> KU["kustomize overlay\n(cluster patches)"]
  KU --> GIT["git commit"]
  GIT --> ARGO["ArgoCD / Flux"]
  ARGO --> CLUSTER["Cluster"]

$ helm template prometheus prometheus-community/kube-prometheus-stack \
  -f values/prometheus.yaml \
  --namespace monitoring > platform/prometheus-rendered.yaml
$ # Move into kustomize resources/ then overlay cluster labels
$ kubectl kustomize platform/overlays/prod | kubectl apply -f - --dry-run=server
$ kubectl apply -k platform/overlays/prod$ helm template prometheus prometheus-community/kube-prometheus-stack \
  -f values/prometheus-ocp.yaml > platform/prometheus-rendered.yaml
$ oc apply -k platform/overlays/ocp-prod
→ ocp overlay adds cluster-monitoring label requirements

oc process — legacy Templates

Template is an OpenShift-specific API (template.openshift.io/v1). Parameters substitute into embedded objects. Still found in quick-start examples and legacy CI; new work should prefer Helm or Kustomize.

apiVersion: template.openshift.io/v1
kind: Template
metadata:
  name: payments-api
parameters:
  - name: IMAGE_TAG
    description: Container image tag
    required: true
  - name: REPLICAS
    value: "2"
objects:
  - apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: payments-api
    spec:
      replicas: ${{REPLICAS}}
      template:
        spec:
          containers:
            - name: api
              image: image-registry.openshift-image-registry.svc:5000/payments/api:${{IMAGE_TAG}}

$ # Templates are OpenShift-specific — not available on vanilla K8s
$ kubectl api-resources | grep -i template
→ no template.openshift.io on vanilla clusters$ oc process -f template.yaml -p IMAGE_TAG=v1.18.3 -p REPLICAS=3 | oc apply -f -
$ oc get template -n openshift
$ oc new-app --template=payments-api -p IMAGE_TAG=v1.18.3
→ shorthand wrapper around process + apply

OperatorHub

OperatorHub is the OpenShift console and CLI marketplace for operators. Sources include: Red Hat-operated catalogs (platform + certified partners), community operators, and custom catalogs you publish. Each operator is distributed as a ClusterServiceVersion (CSV) bundle in an index image.

OLM: CatalogSource, Subscription, InstallPlan, CSV

OLM reconciles operator installs through a pipeline of CRs. Understanding this chain is essential for debugging "operator stuck in InstallPlanPending" incidents.

flowchart TB
  CS["CatalogSource\n(index image gRPC)"] --> PKG["Package manifest"]
  SUB["Subscription\n(channel, source)"] --> IP["InstallPlan\n(approved)"]
  IP --> CSV["ClusterServiceVersion\n(operator + CRDs + RBAC)"]
  CSV --> DEP["Operator Deployment"]
  DEP --> CR["Custom Resources\n(user intent)"]
  CR --> DEP

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: postgres-operator
  namespace: operators
spec:
  channel: stable
  name: crunchy-postgres-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace
  installPlanApproval: Automatic
  startingCSV: postgresoperator.v5.5.0

Install flow

1. Cluster admin creates OperatorGroup in target namespace(s)
2. Create Subscription referencing catalog source and channel
3. OLM resolves CSV from CatalogSource gRPC; creates InstallPlan
4. InstallPlan approved (Automatic or Manual) → OLM applies CSV, CRDs, RBAC, operator Deployment
5. CSV phase transitions: Installing → Succeeded
6. User creates application CRs (e.g. PostgresCluster) → operator reconciles operands

$ # OLM is OpenShift / Operator Framework — limited on vanilla K8s
$ kubectl get csv -A 2>/dev/null || echo "Install OLM via operator-framework"
$ operator-sdk run bundle quay.io/operator/bundle:v1.0.0$ oc get packagemanifest -n openshift-marketplace | grep postgres
$ oc apply -f operatorgroup.yaml -f subscription.yaml
$ oc get installplan -n operators
$ oc get csv -n operators
→ PHASE must be Succeeded; check STATUS for requirement failures
$ oc describe csv postgresoperator.v5.5.0 -n operators
$ oc get catalogsource -n openshift-marketplace
$ oc get subscription postgres-operator -n operators -o yaml

Red Hat certified operators

Red Hat OpenShift Certified operators pass interoperability, security, and support lifecycle tests. They appear in the redhat-operators CatalogSource with predictable upgrade paths tied to OCP versions. Certified operators include Crunchy Postgres, AMQ Streams, OpenShift GitOps (ArgoCD), OpenShift Pipelines (Tekton), and ODF (storage).