Storage, Volumes & PersistentVolumes

emptyDir

Created when the pod is scheduled to a node. Initially empty; all containers in the pod can read/write the same path. Deleted when the pod is removed from the node. Use medium: Memory for tmpfs-style RAM disks (bounded by node memory; contents count against container memory limits if set).

configMap and secret volumes

Mount keys as individual files under a directory. Updates propagate on kubelet sync interval (not instant)— apps should watch files or use subPath carefully. Secrets are base64-encoded in etcd (encryption at rest is a separate concern; see Configuration Management).

hostPath

Maps a path on the host into the container. Ties data to a specific node; if the pod reschedules elsewhere, the data does not follow. Dangerous with privileged paths—restricted by Pod Security Admission on hardened clusters.

downwardAPI

Projects pod fields (name, namespace, labels, annotations, resource requests/limits) into files or env vars. Useful for observability sidecars that tag logs with pod metadata without talking to the API server.

projected volumes

Combines multiple volume sources into one mount—commonly service account tokens (with audience/expiry), configMaps, secrets, and downwardAPI. Replaces older separate mounts and supports automatic token rotation.

┌─── POD ─────────────────────────────────────────────────────┐
│  /data/scratch  ← emptyDir          [ephemeral, node-local]   │
│  /etc/config    ← configMap         [API-backed files]        │
│  /etc/secrets   ← secret            [API-backed, sensitive]   │
│  /var/lib/data  ← PVC → PV          [durable, CSI/block]      │
│  /meta          ← downwardAPI       [pod labels → files]      │
├─────────────────────────────────────────────────────────────┤
│  container A          container B (sidecar reads emptyDir)    │
└─────────────────────────────────────────────────────────────┘

apiVersion: v1
kind: Pod
metadata:
  name: volume-demo
spec:
  containers:
    - name: app
      image: nginx:1.27
      volumeMounts:
        - name: cache
          mountPath: /cache
        - name: cfg
          mountPath: /etc/app
          readOnly: true
        - name: meta
          mountPath: /etc/podinfo
  volumes:
    - name: cache
      emptyDir:
        medium: Memory
        sizeLimit: 256Mi
    - name: cfg
      configMap:
        name: app-config
    - name: meta
      downwardAPI:
        items:
          - path: labels
            fieldRef:
              fieldPath: metadata.labels
    - name: sa-token
      projected:
        sources:
          - serviceAccountToken:
              path: token
              expirationSeconds: 3600
              audience: api

Core objects

Dynamic provisioning flow

sequenceDiagram
  participant Dev as Developer
  participant API as API Server
  participant Ctrl as PVC Controller
  participant Prov as CSI Provisioner
  participant Back as Storage Backend

  Dev->>API: Create PVC (storageClassName, size)
  API->>Ctrl: Watch PVC Pending
  Ctrl->>Prov: CreateVolume (CSI CreateVolume)
  Prov->>Back: Allocate disk / export
  Back-->>Prov: Volume ID
  Prov->>API: Create PV (Bound to PVC)
  API-->>Dev: PVC status Bound

1. User creates a PVC referencing a storageClassName and resources.requests.storage.
2. PVC controller finds a matching StorageClass and calls the external provisioner (CSI CreateVolume).
3. Provisioner creates storage in the backend and a PV object with a claimRef pointing to the PVC.
4. PVC status becomes Bound; pods can reference the PVC in volumes.persistentVolumeClaim.
5. On pod schedule, kubelet calls CSI NodePublishVolume to mount into the container.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pg-data
  namespace: payments
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: gp3-encrypted
  resources:
    requests:
      storage: 100Gi
  volumeMode: Block
---
apiVersion: v1
kind: Pod
metadata:
  name: postgres-0
spec:
  containers:
    - name: postgres
      image: postgres:16
      volumeDevices:
        - name: data
          devicePath: /dev/xvdb
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: pg-data

# Inspect PVC binding and events
kubectl get pvc -n payments
kubectl describe pvc pg-data -n payments

# List PVs and reclaim status
kubectl get pv

# OpenShift equivalents
oc get pvc -n payments
oc describe pvc pg-data
oc get storageclass

Common provisioners

Key StorageClass fields

• provisioner — CSI driver name that handles create/delete.
• parameters — Driver-specific key/value pairs (e.g. type: gp3, encrypted: "true", fsType: xfs).
• reclaimPolicy — Delete (default for dynamic) or Retain (PV object kept; backend volume manual cleanup).
• volumeBindingMode — Immediate (provision on PVC create) vs WaitForFirstConsumer (delay until pod is scheduled—critical for topology-aware provisioning).
• allowVolumeExpansion — Enables PVC resize via kubectl patch when the CSI driver supports it.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3-encrypted
  annotations:
    storageclass.kubernetes.io/is-default-class: "false"
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  encrypted: "true"
  fsType: ext4
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
mountOptions:
  - nouuid

# Expand a PVC (requires allowVolumeExpansion + CSI support)
kubectl patch pvc pg-data -n payments -p '{"spec":{"resources":{"requests":{"storage":"200Gi"}}}}'

# Verify expansion status
kubectl get pvc pg-data -n payments -o jsonpath='{.status.conditions}'

# Set default StorageClass
kubectl annotate storageclass gp3-encrypted storageclass.kubernetes.io/is-default-class=true --overwrite

oc patch pvc pg-data -n payments -p '{"spec":{"resources":{"requests":{"storage":"200Gi"}}}}'

Cloud block storage limitations

AWS EBS, GCE PD, and Azure Managed Disks attach to one node at a time—they support RWO (and RWOP), not RWX. Running two replicas of a Deployment against the same RWO PVC fails: the second pod stays Pending with volume multi-attach errors. For shared files use NFS, EFS, CephFS, or Azure Files—not block StorageClasses.

RWO (block)                    RWX (shared file)
─────────────                  ───────────────────
 Node A                         Node A ──┐
  └─ Pod 1 ✓                      Pod 1  ├── NFS export
 Node B                         Node B ──┤   (shared)
  └─ Pod 2 ✗ (blocked)            Pod 2  ┘

Static vs dynamic provisioning

PVC / PV phases

stateDiagram-v2
  [*] --> Pending: PVC created
  Pending --> Bound: PV matched or provisioned
  Bound --> [*]: PVC deleted
  note right of Bound
    Pod mounts via kubelet + CSI
  end note
  [*] --> Available: Static PV ready
  Available --> Bound: claimRef set
  Bound --> Released: PVC deleted
  Released --> Available: claimRef cleared (manual)
  Released --> [*]: Reclaim Delete or admin cleanup

1. Pending — PVC waiting for matching PV or provisioner. Check events: wrong StorageClass, insufficient quota, provisioner down, or WaitForFirstConsumer waiting for a schedulable pod.
2. Bound — PVC linked to PV; pods can mount. PV claimRef prevents other PVCs from stealing it.
3. Released — PVC deleted but PV retained (reclaimPolicy: Retain). PV still holds stale claimRef; must be cleared before reuse.
4. Reclaim — With Delete, external-attacher deletes backend volume and PV object. With Retain, admin snapshots backend and deletes PV manually.

# PVC stuck Pending — check events and StorageClass
kubectl describe pvc my-claim -n app
kubectl get events -n app --field-selector involvedObject.name=my-claim

# Released PV — clear claimRef to reuse (static workflows)
kubectl patch pv pv-001 -p '{"spec":{"claimRef": null}}'

# Find orphaned PVs (Released phase)
kubectl get pv | grep Released

# Force-delete stuck PVC (last resort — understand data impact)
kubectl patch pvc my-claim -n app -p '{"metadata":{"finalizers":null}}' --type=merge

CSI components in Kubernetes

• CSI driver controller — CreateVolume, DeleteVolume, CreateSnapshot, controller publish.
• CSI driver node — DaemonSet on every node: NodeStageVolume, NodePublishVolume.
• external-provisioner — Watches PVCs, creates PVs.
• external-attacher — Watches VolumeAttachment objects for multi-node attach.
• external-snapshotter — Watches VolumeSnapshot CRs.
• external-resizer — Handles PVC expansion.

Volume snapshots

Snapshots are cluster-scoped API objects (snapshot.storage.k8s.io/v1). A VolumeSnapshotClass names the driver and deletion policy; a VolumeSnapshot references a PVC or specifies a volume handle. Restore by creating a new PVC with dataSource pointing to the VolumeSnapshot.

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: csi-aws-snapclass
driver: ebs.csi.aws.com
deletionPolicy: Retain
---
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: pg-data-snap-20250605
  namespace: payments
spec:
  volumeSnapshotClassName: csi-aws-snapclass
  source:
    persistentVolumeClaimName: pg-data
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pg-data-restored
  namespace: payments
spec:
  storageClassName: gp3-encrypted
  dataSource:
    name: pg-data-snap-20250605
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi

# Verify CSI driver and snapshot CRDs
kubectl get csidrivers
kubectl api-resources | grep snapshot

# Create and inspect snapshot
kubectl apply -f pg-snapshot.yaml
kubectl get volumesnapshot -n payments
kubectl describe volumesnapshot pg-data-snap-20250605 -n payments

# List CSI node pods
kubectl get pods -n kube-system -l app=csi-node

oc get volumesnapshot -n payments

In-tree deprecation

Legacy in-tree volume plugins (kubernetes.io/aws-ebs, GCE PD, Azure Disk) are removed. Clusters must migrate to CSI drivers—often via cloud provider migration tools that detach in-tree volumes and reattach via CSI without data loss. New clusters should install only CSI drivers from day one.

StatefulSet + PVC templates

Each pod name-N gets its own PVC data-name-N from the template. Pods are created ordinally; PVCs persist after StatefulSet scale-down—by design. Use podManagementPolicy: Parallel only when ordering does not matter.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis-cluster
spec:
  serviceName: redis
  replicas: 3
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      containers:
        - name: redis
          image: redis:7.2
          ports:
            - containerPort: 6379
          volumeMounts:
            - name: data
              mountPath: /data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: gp3-encrypted
        resources:
          requests:
            storage: 20Gi

Shared RWX pattern

Content farms, CI artifact caches, and legacy apps expecting NFS use a single RWX PVC mounted by multiple Deployment replicas. Requires CephFS, NFS, EFS, or Azure Files—not block StorageClasses. Watch file-lock semantics and performance under concurrent writers.

Database on Kubernetes

• Single-primary RWO — Postgres/MySQL with one replica and PVC; backups via VolumeSnapshot or logical dump (pg_dump).
• Managed DB often wins — RDS/Cloud SQL handles patching, failover, and IOPS planning; K8s runs the app tier.
• When in-cluster — Use operators (CloudNativePG, Percona, Strimzi for Kafka) for failover, backup CRDs, and version upgrades.
• Topology — WaitForFirstConsumer + pod anti-affinity keeps data in the same AZ as the volume.

Operators

Operators encode day-2 operations: backup schedules, point-in-time recovery, rolling credential rotation, and rebalancing. They reconcile custom resources (e.g. PostgresCluster) into StatefulSets, Services, PVCs, and CronJobs— see Operators & CRDs.

ODF / Rook-Ceph

OpenShift Data Foundation (formerly OCS) deploys Rook-managed Ceph via the StorageCluster CR (ocs-storagecluster in openshift-storage namespace). Provides block (RBD), file (CephFS), and object (S3-compatible) from the same cluster—ideal for on-prem and hybrid when cloud block is unavailable.

NFS provisioner

For clusters without ODF, OpenShift supports cluster NFS provisioners (legacy in-tree or CSI NFS). Mount an enterprise NAS export once; StorageClass provisions per-PVC subdirectories. Simpler than Ceph but single point of failure at the NAS head.

Local Storage Operator

Local Storage Operator discovers raw disks on specific nodes and creates LocalVolume / LocalVolumeSet CRs. PVs have node affinity—pods must schedule where the disk lives. High performance for Kafka, etcd, or Ceph OSDs; no cross-node resilience unless the app replicates (e.g. Ceph, Kafka RF3).

Platform defaults

Managed OpenShift on AWS/GCP/Azure installs cloud CSI drivers as default StorageClasses. Bare-metal IPI/UPI installs often prompt for ODF or require admin to set a default class. Always inspect oc get sc before deploying stateful workloads—assumptions from vanilla K8s docs may not match the cluster.

# OpenShift storage inspection
oc get storagecluster -n openshift-storage
oc get storageclass
oc describe storageclass ocs-storagecluster-ceph-rbd

# ODF health
oc get cephcluster -n openshift-storage
oc adm top pvc -n payments

# Local Storage Operator
oc get localvolumeset -n openshift-local-storage
oc get pv -l localvolume=