GitOps & CI/CD on Kubernetes

flowchart LR
  DEV["Developer\nPR merge"]
  GIT["Git repo\nsource of truth"]
  CI["CI pipeline\nbuild + test"]
  REG["Container registry\nimage tag"]
  GITOPS["GitOps controller\nArgoCD / Flux"]
  K8S["Kubernetes API\netcd desired state"]
  RUN["Running workloads\npods, services, ingress"]

  DEV --> GIT
  GIT --> CI
  CI --> REG
  CI -->|"update image tag\nin git"| GIT
  GIT --> GITOPS
  GITOPS -->|"apply / prune"| K8S
  K8S --> RUN
  RUN -.->|"drift detected"| GITOPS
  GITOPS -.->|"self-heal"| K8S

Core principles

No kubectl in production

Production changes flow through git merge → reconciler apply. Direct kubectl apply or oc apply from CI or laptops bypasses review, breaks audit trails, and fights the reconciler (self-heal will revert manual edits—or worse, you disable self-heal and drift accumulates).

• Allowed: read-only debugging (kubectl get/describe/logs), emergency break-glass with documented procedure
• CI job writes to git: bump image tag, open PR, merge after checks—never push manifests directly to API
• Break-glass: time-limited RBAC, incident ticket, follow-up PR to restore git truth

Architecture components

• API Server — gRPC/REST API, Web UI, RBAC enforcement, webhook receiver
• Repository Server — clones git, generates manifests (plain YAML, Helm, Kustomize, Jsonnet, plugins)
• Application Controller — compares live vs desired state, drives sync, tracks health
• Redis — caching layer for generated manifests and session state
• Dex (optional) — OIDC/LDAP/SAML SSO connector
• ApplicationSet Controller — generates many Application resources from generators
• Notifications Controller — Slack, email, GitHub status via Notification CRs

flowchart TB
  subgraph git["Git / Helm repos"]
    REPO["Git repository\nmanifests + values"]
  end
  subgraph argo["Argo CD namespace"]
    API["API Server\nUI + webhooks"]
    RS["Repo Server\nhelm template / kustomize"]
    AC["Application Controller\ncompare + sync"]
    AS["ApplicationSet Controller"]
    NC["Notifications Controller"]
    RD["Redis cache"]
  end
  subgraph cluster["Target cluster(s)"]
    K8S["Kubernetes API"]
    WL["Workloads"]
  end
  REPO --> RS
  API --> AC
  RS --> AC
  AC --> RD
  AS -->|"generates Application CRs"| AC
  AC -->|"apply / prune / hook"| K8S
  K8S --> WL
  AC --> NC

Application CR

An Application binds a source (git path, Helm chart, OCI) to a destination (cluster + namespace). ArgoCD continuously reconciles that binding. Store Application CRs in the argocd namespace (or allow them in app namespaces with project RBAC).

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: payments-api
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: team-payments          # AppProject RBAC scope
  source:
    repoURL: https://github.com/acme/gitops-config.git
    targetRevision: main          # branch, tag, or commit SHA
    path: apps/payments/overlays/prod
    kustomize:
      images:
        - payments-api=registry.acme.io/payments:v2.4.1
  destination:
    server: https://kubernetes.default.svc
    namespace: payments-prod
  syncPolicy:
    automated:
      prune: true                 # delete resources removed from git
      selfHeal: true              # revert manual cluster edits
      allowEmpty: false
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m
  revisionHistoryLimit: 10

Sync policy

• Manual sync — operator clicks Sync or runs argocd app sync; good for regulated change windows
• Automated sync — applies on every git change; pair with prune and selfHeal carefully
• Sync options — CreateNamespace, ServerSideApply, ApplyOutOfSyncOnly, PruneLast
• Sync waves & hooks — ordering via argocd.argoproj.io/sync-wave annotation; PreSync/PostSync/Skip hooks for jobs and migrations

Sync & health status

App of Apps

Bootstrap pattern: a root Application points at a directory of child Application manifests. Deploying the root app creates the entire fleet. Common for platform teams managing dozens of microservices and cluster add-ons.

ApplicationSet

Generates Application CRs from generators—git directory discovery, cluster list, SCM provider (GitHub/GitLab apps), matrix, merge. Ideal for multi-cluster (same app to dev/stage/prod) and monorepo folder-per-service layouts.

• Git generator — one Application per subdirectory under apps/*
• Cluster generator — fan-out to registered cluster secrets
• SCM provider — discover repos in an org automatically

Hooks & health checks

ArgoCD respects Helm-style hooks and sync-wave annotations. Use PreSync for DB migrations, PostSync for smoke tests, Sync for main resources. Custom health checks in argocd-cm ConfigMap extend Lua logic for CRDs (e.g. cert-manager Certificate, Crossplane claims).

Notifications

The notifications controller watches Application events and sends alerts on sync failed, health degraded, deployed. Configure triggers and templates in argocd-notifications-cm; subscribe per-app via annotations (notifications.argoproj.io/subscribe.on-sync-succeeded.slack: channel).

RBAC & Projects

AppProject restricts which repos, clusters, namespaces, and resource kinds an Application may target. ArgoCD RBAC (Casbin policies in argocd-rbac-cm) maps OIDC groups to roles: role:team-payments → get/sync apps in project team-payments only.

OpenShift GitOps operator

Red Hat ships ArgoCD as the OpenShift GitOps operator (OLM). Install via OperatorHub; it creates an ArgoCD instance in openshift-gitops namespace, integrates with OpenShift OAuth, and is supported on OCP subscriptions. Use ArgoCD CR to configure HA, resource limits, and repo credentials.

$ kubectl get applications -n argocd
$ kubectl describe application payments-api -n argocd
→ Sync Status: Synced | Health: Healthy
$ argocd app diff payments-api --revision main
$ argocd app sync payments-api --prune
$ kubectl logs -n argocd deploy/argocd-application-controller -f$ oc get argocd -n openshift-gitops
$ oc get applications -n openshift-gitops
$ oc route -n openshift-gitops
→ openshift-gitops-server URL for Web UI (OAuth login)
$ oc adm policy add-cluster-role-to-user cluster-admin developer -z openshift-gitops-argocd-application-controller -n openshift-gitops
# ↑ only for break-glass cross-namespace debugging — prefer AppProject RBAC

Modular controllers

• Source Controller — watches GitRepository, HelmRepository, Bucket, OCIRepository
• Kustomize Controller — applies Kustomization CRs (built-in Kustomize + health checks)
• Helm Controller — manages HelmRelease CRs (install/upgrade/test/rollback)
• Notification Controller — dispatches events to Slack, Teams, Git commit status
• Image Reflector / Automation — scans registries, updates git with new tags

GitRepository

Declares a git source and revision to track. The source controller clones (or pulls) on interval and exposes an artifact in the status (.status.artifact) that downstream CRs reference.

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: platform-config
  namespace: flux-system
spec:
  interval: 1m
  url: https://github.com/acme/platform-gitops.git
  ref:
    branch: main
  secretRef:
    name: github-deploy-key
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: payments-prod
  namespace: flux-system
spec:
  interval: 5m
  targetNamespace: payments-prod
  sourceRef:
    kind: GitRepository
    name: platform-config
  path: ./apps/payments/overlays/prod
  prune: true
  wait: true                          # block until resources healthy
  timeout: 5m
  healthChecks:
    - apiVersion: apps/v1
      kind: Deployment
      name: payments-api
      namespace: payments-prod
  postBuild:
    substitute:
      cluster_env: production
  dependsOn:
    - name: cluster-addons          # ordering: addons before apps

HelmRelease

Flux-native Helm operator. Declares chart source (HelmRepository or git), values, upgrade remediation (rollback on failure), and test hooks. Replaces helm upgrade --install in CI.

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: ingress-nginx
  namespace: flux-system
spec:
  interval: 10m
  targetNamespace: ingress-nginx
  chart:
    spec:
      chart: ingress-nginx
      version: "4.10.x"
      sourceRef:
        kind: HelmRepository
        name: ingress-nginx
        namespace: flux-system
  install:
    remediation:
      retries: 3
  upgrade:
    remediation:
      remediateLastFailure: true
  values:
    controller:
      replicaCount: 2

Flux vs ArgoCD

Image Automation

Flux can close the image tag loop without custom CI scripts:

1. ImageRepository — scans container registry on interval
2. ImagePolicy — semver range or alphabetical filter (1.0.x)
3. ImageUpdateAutomation — commits updated tags to git via GitRepository push credentials

Kustomize images field or Helm values markers (# {"$imagepolicy": ...}) tell the automation which fields to rewrite.

$ flux check
$ flux get sources git -A
$ flux get kustomizations -A
→ READY=True means last reconcile succeeded
$ flux reconcile kustomization payments-prod --with-source
$ kubectl describe kustomization payments-prod -n flux-system$ flux install --namespace=flux-system
$ oc get pods -n flux-system
$ flux get helmreleases -A

GitOps pipeline steps

1. Developer merges to app repo — feature branch → main; triggers CI on application source code
2. CI builds & tests — unit tests, SAST, container image build, push to registry with immutable tag (git SHA or semver)
3. CI updates GitOps repo — bot opens PR (or commits) bumping image tag in overlay/kustomization.yaml or Helm values
4. GitOps reconciler deploys — ArgoCD/Flux detects new commit, syncs cluster; health checks gate promotion

flowchart TB
  subgraph app["Application repo"]
    CODE["Source code\nDockerfile / tests"]
    CIP["CI pipeline\nGitHub Actions / Tekton"]
  end
  subgraph reg["Registry"]
    IMG["payments-api:abc123f"]
  end
  subgraph gitops["GitOps repo"]
    MAN["manifests/overlays/prod\nimage tag abc123f"]
    PR["Pull request\nplatform review"]
  end
  subgraph deliver["Delivery"]
    REC["ArgoCD / Flux"]
    PROD["Production cluster"]
  end
  CODE --> CIP
  CIP --> IMG
  CIP -->|"PR: bump tag"| MAN
  MAN --> PR
  PR -->|"merge"| REC
  REC --> PROD
  IMG -.->|"pulled by kubelet"| PROD

CI vs GitOps repo separation

Keep application repos (code, Dockerfile, unit tests) separate from GitOps repos (K8s manifests, environment config). Reasons:

• Different access models — developers push code; only platform bots merge to prod gitops
• Different promotion paths — same image tag promoted dev → stage → prod via overlay PRs
• Blast radius — compromised app CI cannot directly mutate cluster RBAC or cluster-scoped resources
• Audit — gitops repo history is the deployment ledger

Monorepo variant: /src and /deploy directories with path-filtered CI workflows—works for smaller teams but weakens access separation.

Never kubectl in CI — anti-pattern

# Jenkins / GitLab CI stage — IMPERATIVE DEPLOY
docker build -t payments:$GIT_SHA .
docker push registry.acme.io/payments:$GIT_SHA
kubectl set image deployment/payments-api \
  payments=registry.acme.io/payments:$GIT_SHA -n prod
kubectl rollout status deployment/payments-api -n prod
# Problems: no git audit, fights GitOps self-heal, kubeconfig in CI is cluster-admin bait

# CI ends at registry + git commit to GitOps repo
docker build -t registry.acme.io/payments:${GIT_SHA} .
docker push registry.acme.io/payments:${GIT_SHA}
yq -i '.images[0].newTag = strenv(GIT_SHA)' deploy/overlays/prod/kustomization.yaml
git commit -am "deploy payments ${GIT_SHA}"
git push origin HEAD  # or open PR via gh pr create
# ArgoCD/Flux reconciles — CI holds zero cluster credentials

Core CRDs

Workspaces

Workspaces pass data between tasks: source code (PVC, emptyDir, git clone), cache volumes, Docker config secrets, and optional results. PipelineRun binds concrete volumes to Pipeline workspace declarations.

• volumeClaimTemplate — dynamic PVC per PipelineRun (CI scratch space)
• persistentVolumeClaim — shared cache (Maven, npm) across runs
• secret — .dockerconfigjson for registry push

apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
  generateName: payments-build-
  namespace: team-payments
spec:
  pipelineRef:
    name: build-and-push
  taskRunTemplate:
    podTemplate:
      securityContext:
        fsGroup: 65532
  params:
    - name: git-url
      value: https://github.com/acme/payments.git
    - name: git-revision
      value: main
    - name: image
      value: image-registry.openshift-image-registry.svc:5000/team-payments/payments-api
  workspaces:
    - name: source
      volumeClaimTemplate:
        spec:
          accessModes: [ReadWriteOnce]
          resources:
            requests:
              storage: 1Gi
    - name: docker-credentials
      secret:
        secretName: push-secret
  timeouts:
    pipeline: 30m

OpenShift Pipelines operator

Install from OperatorHub (redhat-openshift-pipelines). Ships Tekton Pipelines, Triggers, Chains (SLSA attestations), and Hub integration. Console adds a Pipelines perspective; tkn CLI is the primary UX.

Tekton Hub

Catalog of community Tasks and Pipelines (git-clone, buildah, kaniko, helm-upgrade). Reference via resolver: hub.tekton.dev/… in taskRef or bundle OCI artifacts for air-gapped installs.

Triggers & EventListener

Event-driven CI: TriggerBinding extracts payload fields (git SHA, repo URL), TriggerTemplate renders PipelineRun YAML, EventListener exposes HTTP endpoint (Knative or K8s Service) for GitHub/GitLab webhooks.

$ tkn pipeline list -n team-payments
$ tkn pipelinerun logs -f -n team-payments
$ kubectl get taskrun -n team-payments
$ tkn pipelinerun describe payments-build-x7k2p -n team-payments$ oc get tektonconfig
$ oc get pods -n openshift-pipelines
$ tkn hub install task git-clone
$ oc get eventlistener -n team-payments
→ expose Route or Ingress for webhook URL

BuildConfig strategies

apiVersion: build.openshift.io/v1
kind: BuildConfig
metadata:
  name: payments-api
  namespace: team-payments
spec:
  source:
    type: Git
    git:
      uri: https://github.com/acme/payments.git
      ref: main
    contextDir: /services/payments-api
  strategy:
    type: Docker
    dockerStrategy:
      dockerfilePath: Dockerfile
  output:
    to:
      kind: ImageStreamTag
      name: payments-api:latest
  triggers:
    - type: ConfigChange
    - type: ImageChange
      imageChange:
        from:
          kind: ImageStreamTag
          name: ubi9-openjdk-21:latest
    - type: GitHub
      github:
        secretReference:
          name: github-webhook-secret

Buildah

OpenShift builds increasingly use Buildah (rootless OCI image build) inside build pods instead of the Docker daemon. Tekton's buildah task is the portable equivalent for pipeline-based builds on OCP and vanilla K8s (with privileged or custom SCC).

oc start-build

Manually queue a build from CLI—useful for debugging, hotfix binary uploads, or triggering from scripts. Does not replace GitOps for deploy; only produces a new image tag.

$ # BuildConfig is OpenShift-specific — use Tekton/Kaniko on vanilla K8s
$ kubectl get builds 2>/dev/null || echo "Install OpenShift or use CI pipelines"$ oc start-build payments-api --from-dir=. --follow
$ oc logs -f build/payments-api-42
$ oc get imagestream payments-api -o yaml
→ status.tags lists pushed digests
$ oc describe bc payments-api
$ oc import-image ubi9/openjdk-21:latest --confirm

ImageStream triggers

When a build pushes a new image to an ImageStreamTag, OpenShift can automatically roll out dependent workloads:

• DeploymentConfig (legacy) — native ImageChange trigger on DeploymentConfig
• Deployment — annotate with image.openshift.io/triggers JSON referencing ImageStreamTag
• Build chaining — base image ImageChange trigger rebuilds downstream apps when platform base updates

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments-api
  namespace: team-payments
  annotations:
    image.openshift.io/triggers: |
      [{"from":{"kind":"ImageStreamTag","name":"payments-api:latest"},"fieldPath":"spec.template.spec.containers[0].image"}]
spec:
  template:
    spec:
      containers:
        - name: payments-api
          image: image-registry.openshift-image-registry.svc:5000/team-payments/payments-api:latest

sequenceDiagram
  participant GH as GitHub webhook
  participant BC as BuildConfig
  participant BP as Build pod\nBuildah/S2I
  participant IS as ImageStream
  participant DEP as Deployment
  participant GIT as GitOps repo
  participant AR as ArgoCD

  GH->>BC: Git push trigger
  BC->>BP: Start build pod
  BP->>IS: Push new image tag
  IS->>DEP: ImageChange rollout (optional)
  Note over GIT,AR: Preferred production path
  BP->>GIT: CI bumps image tag in git
  GIT->>AR: Sync deployment manifest
  AR->>DEP: Reconcile desired state