Cheat Sheets

Delivery semantics

# Durable producer (at-least-once on wire)
acks=all
enable.idempotence=true
retries=2147483647
max.in.flight.requests.per.connection=5

Producer config

// Async — never .get() per record in hot loop
kafkaTemplate.send("orders", key, event);  // CompletableFuture in Spring Kafka 3.x

Consumer config

Spring Kafka

@KafkaListener(topics = "orders", groupId = "processor")
public void onOrder(@Payload OrderEvent o, Acknowledgment ack) {
  process(o);
  ack.acknowledge();
}

spring.kafka.listener.ack-mode: record
spring.kafka.listener.concurrency: 3

Partitioning & keys

CLI quick ref

kafka-console-producer.sh --bootstrap-server kafka:9092 --topic orders
kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic orders --from-beginning --group debug

kafka-topics.sh --describe --topic orders --bootstrap-server kafka:9092
kafka-consumer-groups.sh --describe --group order-processor --bootstrap-server kafka:9092

Kafka Streams DSL

streams.builder()
  .stream("orders")
  .groupByKey()
  .windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(5)))
  .count()
  .toStream()
  .to("order-counts");

# EOS
processing.guarantee=exactly_once_v2
isolation.level=read_committed

Connect & Debezium

# Debezium envelope fields
op: c|u|d|r   before / after   source.ts_ms   ts_ms

# Outbox SMT
transforms=outbox
transforms.outbox.type=io.debezium.transforms.outbox.EventRouter

# Error handling
errors.tolerance=all
errors.deadletterqueue.topic.name=connect.dlq

Schema Registry

Wire: 0x00 + schema_id (4 bytes BE) + payload

Subject: orders-value  (TopicNameStrategy)
auto.register.schemas=false   # CI registers schemas

Architecture patterns

Performance tuning matrix

partitions ≈ target_MB/s ÷ per_partition_MB/s  (measure!)
Per partition: ~10-100 MB/s depending on disk/NIC

kafka-producer-perf-test.sh --topic perf --record-size 1024 --throughput -1 ...
kafka-consumer-perf-test.sh --topic perf --messages 5000000 ...

Reliability checklist

Topic administration

# Create (size partitions up-front — cannot reduce)
kafka-topics.sh --create --topic orders --partitions 24 --replication-factor 3 \
  --config min.insync.replicas=2 --config retention.ms=604800000

kafka-topics.sh --alter --topic orders --partitions 36   # increase only
kafka-configs.sh --alter --entity-type topics --entity-name orders \
  --add-config retention.ms=1209600000

kafka-topics.sh --delete --topic orders-staging   # needs delete.topic.enable

Consumer group ops

kafka-consumer-groups.sh --list --bootstrap-server kafka:9092
kafka-consumer-groups.sh --describe --group order-processor

# Reset (stop consumers first; dry-run then --execute)
kafka-consumer-groups.sh --group order-processor --topic orders \
  --reset-offsets --to-datetime 2026-06-01T00:00:00.000 --dry-run
# --to-earliest | --to-latest | --to-offset | --by-duration PT2H

Critical metrics — page these

Stack: JMX → Prometheus JMX Exporter → Grafana (Confluent dashboards)
Lag SLA: Burrow (OK/WARN/ERR) or records-lag-max growth rate

Security

kafka-acls.sh --add --allow-principal User:order-service \
  --operation Write --topic orders

kafka-acls.sh --add --allow-principal User:processor \
  --operation Read --topic orders
kafka-acls.sh --add --allow-principal User:processor \
  --operation Read --group order-processor

Broker & OS tuning

Partition reassignment

kafka-reassign-partitions.sh --broker-list 3 --generate
kafka-reassign-partitions.sh --reassignment-json-file plan.json --execute
kafka-reassign-partitions.sh --reassignment-json-file plan.json --verify

# Throttle before execute; remove throttle after verify
leader.replication.throttled.rate=52428800

Capacity formulas

disk ≈ retention_sec × peak_bytes_in × RF × 1.2

network ≈ peak_bytes_in × (RF - 1 + consumer_fan_out)

partitions/broker ceiling: ~2000–4000 (rebalance + metadata cost)

KRaft: dedicated controller quorum; monitor metadata apply latency

KRaft vs ZooKeeper (legacy)