Container Runtime & Lifecycle

State machine

Docker tracks each container through a well-defined lifecycle. The writable container layer (the "upperdir" in OverlayFS) persists until the container is removed—even after stop. Paused containers freeze all processes via cgroup freezer; stopped containers have exited but retain filesystem and metadata.

stateDiagram-v2
  [*] --> created: docker create
  created --> running: docker start / docker run
  running --> paused: docker pause
  paused --> running: docker unpause
  running --> stopped: exit / docker stop / docker kill
  paused --> stopped: docker kill
  stopped --> running: docker start
  stopped --> removed: docker rm
  running --> removed: docker rm -f
  created --> removed: docker rm
  removed --> [*]

docker create vs docker run

docker create builds the OCI bundle—namespaces, cgroups, mounts, env, network endpoint—but does not call runc start. The container sits in created state until docker start. This split is useful for pre-provisioning containers, attaching to custom networks before start, or orchestrators that separate "define" from "launch."

docker run is create + start in one command. With -d it returns immediately after backgrounding; without -d it attaches to stdout/stderr and blocks until the container exits.

# Create without starting — inspect config before launch
docker create --name api-cache -e REDIS_PASSWORD=secret redis:7-alpine
docker inspect api-cache --format '{{.State.Status}}'   # created

# Start when ready
docker start api-cache

# Equivalent one-liner
docker run -d --name api-cache -e REDIS_PASSWORD=secret redis:7-alpine

Start, stop, and kill

Pause and unpause

docker pause freezes every process in the container using the cgroup freezer controller. Memory stays allocated; network connections may time out on peers. Rare in app deploys, but useful for consistent filesystem snapshots or debugging race conditions without tearing down the process tree.

Remove (docker rm)

Removing a stopped container deletes its writable layer and frees the name. Volumes declared with -v on docker rm also delete anonymous volumes attached to that container. Named volumes survive unless explicitly removed with docker volume rm. docker rm -f kills a running container first—dangerous in production without orchestrator coordination.

Restart policies (--restart)

Restart policies tell the Docker daemon whether to automatically restart a container when it exits or when the daemon itself restarts. Set at docker run or docker update; stored in container config.

# Restart up to 3 times on failure only
docker run -d --restart on-failure:3 --name worker myapp:latest

# Survive host reboot — won't restart if you docker stop it manually
docker run -d --restart unless-stopped --name edge-proxy nginx:alpine

# Check restart count after crash loop
docker inspect --format '{{.RestartCount}}' worker

docker run flags reference

Production-shaped example

$ docker run -d \
  --name payments-api \
  --restart unless-stopped \
  --network backend \
  --network-alias payments \
  -p 127.0.0.1:8080:8080 \
  -e SPRING_PROFILES_ACTIVE=prod \
  --env-file /etc/payments.env \
  --user 10001:10001 \
  --read-only \
  --tmpfs /tmp:rw,noexec,nosuid,size=128m \
  --security-opt no-new-privileges \
  --cap-drop ALL \
  --memory 1g \
  --cpus 2 \
  --pids-limit 512 \
  --init \
  myregistry/payments-api:sha256-abc123
f3a8c2d91e04b7...

Why PID 1 is special

• Signal handling — PID 1 ignores SIGTERM/SIGINT unless the process explicitly installs handlers. Shells and JVMs behave differently.
• Zombie reaping — Orphaned children are reparented to PID 1. If PID 1 doesn't call wait(), zombies accumulate (defunct in ps).
• Shutdown path — docker stop → SIGTERM to PID 1 → grace period → SIGKILL. Your app must handle SIGTERM to drain connections and flush state.

Shell form trap

Dockerfile shell form (CMD ./start.sh) wraps your command in /bin/sh -c. The shell becomes PID 1—not your app. The shell often does not forward signals to child processes, so Java, Node, or nginx never see SIGTERM.

# BAD — shell is PID 1; SIGTERM may not reach java
FROM eclipse-temurin:21-jre
COPY app.jar /app.jar
CMD java -jar /app.jar

# BETTER — exec form: java is PID 1
CMD ["java", "-jar", "/app.jar"]

# BEST for shell scripts — exec replaces shell with script's final exec
CMD ["/start.sh"]

In a shell script used as entrypoint, end with exec "$@" or exec java -jar ... so the application binary replaces the shell and becomes PID 1.

Exec form fix

Exec form (CMD ["java", "-jar", "app.jar"]) invokes the binary directly—no shell wrapper. Signals go straight to your application. This is the minimum bar for production JVM, Node, and Go images.

tini and docker run --init

When you must run a shell script or a process that spawns children (nginx master/worker, supervisord), use a minimal init system. tini (bundled as docker-init) becomes PID 1: it forwards signals and reaps zombies. Enable with docker run --init or Dockerfile ENTRYPOINT ["/sbin/tini", "--"].

# Dockerfile with tini
RUN apt-get update && apt-get install -y tini && rm -rf /var/lib/apt/lists/*
ENTRYPOINT ["/usr/bin/tini", "--"]
CMD ["nginx", "-g", "daemon off;"]

# Or at runtime without image change
# docker run --init nginx

Spring Boot SIGTERM

Spring Boot registers a shutdown hook on SIGTERM when running as PID 1 with exec form. It stops accepting new requests, completes in-flight work (within timeout), closes the application context, and exits. Configure grace period alignment:

• server.shutdown=graceful + spring.lifecycle.timeout-per-shutdown-phase=30s
• docker stop -t 35 <container> — stop timeout must exceed Spring's shutdown phase
• Kubernetes terminationGracePeriodSeconds must exceed both

sequenceDiagram
  participant D as docker stop
  participant C as containerd
  participant I as init (tini / app)
  participant A as app process
  D->>C: Stop container (timeout T)
  C->>I: SIGTERM to PID 1
  alt exec form / tini
    I->>A: Forward SIGTERM
    A->>A: Drain requests, flush state
    A->>C: exit 0
  else shell form without exec
    I->>I: Shell exits or ignores
    Note over C: Wait T seconds
    C->>I: SIGKILL
  end

Command reference

Logs — first stop for runtime errors

# Stream logs with timestamps
docker logs -f -t --tail 200 payments-api

# Logs since last deploy (approximate)
docker logs --since 2026-06-05T10:00:00 payments-api

# Exit code of stopped container
docker inspect --format '{{.State.ExitCode}}' payments-api

Logs are stored by the logging driver (default json-file on disk under /var/lib/docker/containers/<id>/). They survive container stop but not docker rm unless forwarded to a centralized driver (fluentd, awslogs, etc.).

Exec — interactive debugging

$ docker exec -it payments-api sh
/app $ ps aux
PID   USER     COMMAND
    1 10001    java -jar app.jar
   47 10001    sh

/app $ curl -s localhost:8080/actuator/health
{"status":"UP"}

Inspect — the source of truth

# High-value inspect templates
docker inspect payments-api --format 'Status={{.State.Status}} OOM={{.State.OOMKilled}} Exit={{.State.ExitCode}}'
docker inspect payments-api --format '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}'
docker inspect payments-api --format '{{json .HostConfig.Memory}}'

# Pretty-print full state
docker inspect payments-api | jq '.[0].State'

Stats and top — live resource view

docker stats reads cgroup v2 counters—memory usage includes cache; compare against --memory limit. docker top lists processes with host PIDs, bridging container and host views for signal debugging.

Diff and cp — filesystem forensics

docker diff shows what the writable layer changed—useful when an app writes config at runtime or malware is suspected. docker cp extracts heap dumps, thread dumps, or config files without installing tools in production images.

Host-level debugging: nsenter and /proc/<pid>/root

When docker exec isn't enough, pivot to the host. Every container process has a host PID (from docker inspect --format '{{.State.Pid}}'). From there:

• /proc/<pid>/root — view the container's filesystem root from the host
• nsenter -t <pid> -m -u -i -n -p sh — enter mount, UTS, IPC, net, and PID namespaces
• /proc/<pid>/ns/ — compare namespace inodes with other containers or host

PID=$(docker inspect --format '{{.State.Pid}}' payments-api)

# Browse container rootfs from host
sudo ls /proc/$PID/root/app

# Enter all namespaces (requires root on host)
sudo nsenter -t $PID -m -u -i -n -p -- ps aux

# Compare network namespace with host
readlink /proc/$PID/ns/net
readlink /proc/1/ns/net

Memory limits

--memory sets the hard ceiling (memory.max in cgroup v2). When the container's memory usage (anon + mapped file cache charged to cgroup) exceeds the limit, the kernel OOM killer selects a process in the cgroup—usually your app—and sends SIGKILL.

CPU limits

--cpus=1.5 sets CFS quota: 150% of one core (or spread across cores). --cpu-shares (default 1024) only matters when CPUs are contended—it is relative weight, not a guarantee. For latency-sensitive services, prefer explicit --cpus over shares alone.

# Verify cgroup limits from host (v2)
CID=$(docker ps -qf name=payments-api)
CGROUP=$(docker inspect --format '{{.Id}}' $CID)
cat /sys/fs/cgroup/system.slice/docker-${CGROUP}.scope/memory.max
cat /sys/fs/cgroup/system.slice/docker-${CGROUP}.scope/cpu.max

# Live usage
docker stats --no-stream payments-api

OOMKilled — diagnosis

$ docker inspect payments-api --format 'OOM={{.State.OOMKilled}} Exit={{.State.ExitCode}}'
OOM=true Exit=137

$ dmesg | tail -5 | grep -i oom
Memory cgroup out of memory: Killed process 91234 (java) total-vm:2147484kB, anon-rss:524288kB

$ docker logs --tail 20 payments-api
# Often no graceful message — SIGKILL is not catchable

Exit code 137 = 128 + 9 (SIGKILL)—strong hint for OOM or docker kill. Java without container-aware heap sizing sets -Xmx from host RAM, not cgroup limit—use -XX:+UseContainerSupport (default Java 10+) and prefer -XX:MaxRAMPercentage over hard-coded -Xmx.

Danger of no limits