Advanced Scheduling & Node Management

Taint syntax: key=value:effect

A taint is a triple: key, optional value, and effect. The value is optional but commonly set for clarity.

Common use cases

• GPU nodes — Taint nvidia.com/gpu=present:NoSchedule; only ML workloads with matching toleration land there.
• Spot / preemptible — Taint lifecycle=spot:PreferNoSchedule or NoSchedule; fault-tolerant jobs tolerate interruption.
• Master / control-plane — Auto-taint node-role.kubernetes.io/control-plane:NoSchedule prevents user workloads on etcd/API nodes.

# Applied to node (or via MachineSet / node pool config):
# kubectl taint nodes gpu-node-1 nvidia.com/gpu=present:NoSchedule

apiVersion: v1
kind: Pod
metadata:
  name: training-job
spec:
  tolerations:
    - key: "nvidia.com/gpu"
      operator: "Equal"
      value: "present"
      effect: "NoSchedule"
    - key: "lifecycle"
      operator: "Equal"
      value: "spot"
      effect: "NoSchedule"
  nodeSelector:
    accelerator: nvidia-a100
  containers:
    - name: trainer
      image: pytorch/pytorch:2.2-cuda12.1-cudnn8-runtime
      resources:
        limits:
          nvidia.com/gpu: "1"

tolerationSeconds

For taints with effect NoExecute, tolerationSeconds controls how long a pod may remain after the taint appears before eviction. The kubelet automatically adds node.kubernetes.io/not-ready:NoExecute after a node becomes NotReady for 5 minutes (300s default)—pods without a toleration for that taint are evicted. Setting tolerationSeconds: 3600 on a toleration gives a one-hour grace period during brief network blips.

tolerations:
  - key: "node.kubernetes.io/not-ready"
    operator: "Exists"
    effect: "NoExecute"
    tolerationSeconds: 300
  - key: "node.kubernetes.io/unreachable"
    operator: "Exists"
    effect: "NoExecute"
    tolerationSeconds: 300

$ kubectl taint nodes worker-gpu-1 nvidia.com/gpu=present:NoSchedule
$ kubectl taint nodes worker-gpu-1 nvidia.com/gpu=present:NoSchedule-
→ trailing minus removes the taint
$ kubectl describe node worker-gpu-1 | grep -A3 Taints$ oc adm taint nodes worker-gpu-1 lifecycle=spot:PreferNoSchedule
$ oc describe node worker-gpu-1 | grep Taints

NodeSelector (legacy simple form)

Exact key/value match on node labels. All expressions must match (AND). No soft preferences—if no node matches, pod stays Pending.

spec:
  nodeSelector:
    disktype: ssd
    topology.kubernetes.io/zone: us-east-1a

NodeAffinity — required vs preferred

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
        tier: backend
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: node.kubernetes.io/instance-type
                    operator: In
                    values: ["m6i.xlarge", "m6i.2xlarge"]
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 80
              preference:
                matchExpressions:
                  - key: topology.kubernetes.io/zone
                    operator: In
                    values: ["us-east-1a", "us-east-1b"]
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchLabels:
                  app: api
              topologyKey: kubernetes.io/hostname
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app: cache
                topologyKey: kubernetes.io/hostname

Pod Affinity / Anti-Affinity

Pod affinity schedules near pods matching a label selector within a topology domain. Pod anti-affinity spreads replicas across domains—classic HA pattern: one replica per node or per zone. topologyKey defines the boundary: kubernetes.io/hostname for node-level spread, topology.kubernetes.io/zone for AZ-level spread.

flowchart TB
  subgraph zone_a["zone us-east-1a"]
    N1[node-1]
    N2[node-2]
  end
  subgraph zone_b["zone us-east-1b"]
    N3[node-3]
  end
  P1[api pod replica-1] --> N1
  P2[api pod replica-2] --> N2
  P3[api pod replica-3] --> N3
  AA["podAntiAffinity topologyKey=zone"]

Key fields

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 6
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: web
          matchLabelKeys:
            - pod-template-hash
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app: web

Replacing anti-affinity for zones

Instead of required pod anti-affinity on zone (which fails when zones are asymmetric), use: maxSkew: 1, topologyKey: topology.kubernetes.io/zone, whenUnsatisfiable: DoNotSchedule. With 3 zones and 6 replicas, you get 2+2+2—not 4+2+0. For node-level spread, swap topologyKey to kubernetes.io/hostname.

Cordon, drain, uncordon

$ kubectl cordon worker-3
node/worker-3 cordoned
$ kubectl drain worker-3 \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --grace-period=120 \
  --timeout=10m
evicting pod default/web-7d4f8b9c-xk2lm
node/worker-3 drained
$ kubectl uncordon worker-3
$ kubectl get nodes -o wide
$ kubectl describe node worker-3 | grep -A10 Conditions$ oc adm cordon worker-3
$ oc adm drain worker-3 \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --force \
  --grace-period=120
→ --force evicts pods not managed by RC/RS/Job (use cautiously)
$ oc adm uncordon worker-3
$ oc get nodes
$ oc describe node worker-3

Node conditions

The kubelet reports conditions on each node. The scheduler and controllers react to unhealthy signals.

Kubelet eviction thresholds

When node resources drop below eviction thresholds, the kubelet ranks pods by QoS and usage, then evicts BestEffort first, then Burstable exceeding requests, then Guaranteed last. Thresholds are configured via kubelet flags or KubeletConfiguration: memory.available, nodefs.available, imagefs.available, pid.available.

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
evictionHard:
  memory.available: "500Mi"
  nodefs.available: "10%"
  imagefs.available: "15%"
evictionSoft:
  memory.available: "1Gi"
evictionSoftGracePeriod:
  memory.available: "1m30s"
evictionMinimumReclaim:
  memory.available: "0Mi"

NotReady taint (5 minutes)

When a node transitions to NotReady, the control plane waits 5 minutes (default) before adding node.kubernetes.io/not-ready:NoExecute. Pods without a matching toleration are evicted so workloads reschedule elsewhere. The same applies to node.kubernetes.io/unreachable when the node controller loses contact.

OpenShift: Machine Config Pool

On OCP, nodes are grouped into Machine Config Pools (MCP)—master, worker, and custom pools (e.g. gpu, infra). MachineConfig changes roll out pool-by-pool: nodes cordon, drain, reboot with new config, uncordon. Always drain one MCP segment at a time; watch oc get mcp for UPDATED / UPDATING status.

Requests vs limits

CPU throttling & memory OOMKill

• CPU — Container exceeds its CPU limit → throttled (CFS bandwidth). Symptoms: latency spikes without pod restart. Fix: raise limit or lower load; consider omitting CPU limit for latency-sensitive apps (keep request).
• Memory — Container exceeds memory limit → Linux OOM killer terminates the container (exit 137). Unlike CPU, memory is not compressible—always set limits ≤ node capacity with headroom.

QoS classes

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: tier-1-production
value: 1000000
globalDefault: false
description: "Critical payment path — preempts lower tiers"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments-api
spec:
  replicas: 3
  template:
    spec:
      priorityClassName: tier-1-production
      containers:
        - name: api
          image: payments:2.4.0
          resources:
            requests:
              cpu: "500m"
              memory: "512Mi"
            limits:
              cpu: "2000m"
              memory: "1Gi"
          # QoS: Burstable (limits != requests)
---
# Guaranteed QoS example — limits == requests for all containers
apiVersion: v1
kind: Pod
metadata:
  name: guaranteed-example
spec:
  containers:
    - name: app
      image: nginx:1.25
      resources:
        requests:
          cpu: "250m"
          memory: "256Mi"
        limits:
          cpu: "250m"
          memory: "256Mi"

Production recommendations

• Always set memory requests ≈ expected working set; limits at 1.2–1.5× request for Java/Go heap growth.
• Set CPU requests from p95 usage (VPA or metrics); avoid CPU limits on latency-critical services or set limit ≥ 2× request.
• Run tier-1 workloads as Guaranteed or high-priority Burstable with accurate requests.
• Never run production apps as BestEffort—first evicted under node pressure.
• Use PriorityClass so platform components and payment paths preempt batch jobs.

ResourceQuota

Caps aggregate resource consumption per namespace—prevents one team from consuming the entire cluster. Quota is enforced at admission time (create/update pod).

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-payments-quota
  namespace: payments
spec:
  hard:
    requests.cpu: "20"
    requests.memory: 40Gi
    limits.cpu: "40"
    limits.memory: 80Gi
    pods: "50"
    persistentvolumeclaims: "10"
---
apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
  namespace: payments
spec:
  limits:
    - type: Container
      default:
        cpu: "500m"
        memory: "512Mi"
      defaultRequest:
        cpu: "100m"
        memory: "128Mi"
      max:
        cpu: "4"
        memory: 8Gi
      min:
        cpu: "50m"
        memory: 64Mi

$ kubectl describe quota -n payments
$ kubectl describe limitrange -n payments
$ kubectl get pods -n payments -o custom-columns=NAME:.metadata.name,QOS:.status.qosClass
$ kubectl top pods -n payments$ oc describe quota -n payments
$ oc adm quota -n payments --list
$ oc get pods -n payments -o custom-columns=NAME:.metadata.name,QOS:.status.qosClass
$ oc adm top pods -n payments