Kafka Operations & Administration

kafka-topics.sh essentials

# Create
kafka-topics.sh --bootstrap-server kafka:9092 \
  --create --topic orders \
  --partitions 24 --replication-factor 3 \
  --config min.insync.replicas=2 \
  --config retention.ms=604800000

# Describe (configs + partition leaders)
kafka-topics.sh --bootstrap-server kafka:9092 --describe --topic orders

# Alter partitions (increase only) or configs
kafka-topics.sh --bootstrap-server kafka:9092 \
  --alter --topic orders --partitions 36

kafka-configs.sh --bootstrap-server kafka:9092 \
  --entity-type topics --entity-name orders \
  --alter --add-config retention.ms=1209600000

# Delete (disabled by default: delete.topic.enable=true on broker)
kafka-topics.sh --bootstrap-server kafka:9092 --delete --topic orders-staging

Partition count

Partitions bound maximum consumer parallelism (one consumer per partition per group) and spread write load across brokers. You can increase partitions; you cannot decrease without recreating the topic and migrating data.

Rule of thumb:

partitions ≈ target_throughput ÷ per_partition_throughput

Example:
  Target: 120 MB/s ingest
  Per partition sustainable: ~5 MB/s (measure on your hardware)
  → 120 ÷ 5 = 24 partitions (round up, add headroom)

• Start higher than you think for high-throughput topics—adding partitions later does not reorder existing keys
• Keyed producers: more partitions = better key spread; too many = metadata overhead and slow rebalances
• Align with downstream task count (Flink parallelism, Connect tasks.max)

Replication factor

Always ≥ 3 in production. RF=3 tolerates one broker loss with min.insync.replicas=2 and acks=all. RF=2 is acceptable only in dev or cost-constrained DR tiers with explicit risk acceptance.

Compaction vs retention

Retention and segment configs

kafka-topics.sh --bootstrap-server kafka:9092 \
  --create --topic customer-profiles \
  --partitions 12 --replication-factor 3 \
  --config cleanup.policy=compact \
  --config min.compaction.lag.ms=3600000 \
  --config segment.bytes=268435456 \
  --config min.insync.replicas=2

kafka-consumer-groups.sh

# List all groups
kafka-consumer-groups.sh --bootstrap-server kafka:9092 --list

# Describe members, assignments, lag
kafka-consumer-groups.sh --bootstrap-server kafka:9092 \
  --describe --group order-processor

# Delete empty group (no active members)
kafka-consumer-groups.sh --bootstrap-server kafka:9092 \
  --delete --group order-processor-stale

--describe output columns:

• CURRENT-OFFSET — last committed offset per partition
• LOG-END-OFFSET — high watermark on broker
• LAG — LOG-END − CURRENT (records behind)
• CONSUMER-ID / HOST — member owning partition

Reset offsets

Group must be stopped (no active members) or use --execute with dry-run first. Always use --dry-run before --execute.

kafka-consumer-groups.sh --bootstrap-server kafka:9092 \
  --group order-processor \
  --topic orders \
  --reset-offsets \
  --to-datetime 2026-06-01T14:30:00.000 \
  --dry-run

# If output looks correct:
kafka-consumer-groups.sh --bootstrap-server kafka:9092 \
  --group order-processor \
  --topic orders \
  --reset-offsets \
  --to-datetime 2026-06-01T14:30:00.000 \
  --execute

When to reset

• Replay after processing bug — fix code, reset to incident timestamp, reprocess with idempotent sinks
• Bootstrap new service — new consumer group reads from earliest or specific datetime for historical backfill
• Skip poison backlog — to-latest after DLT quarantine (last resort)

Lag monitoring

--describe is the CLI lag view. For alerting, export records-lag-max via JMX/Prometheus or use Burrow (see Monitoring section). Sustained lag growth means consumers cannot keep pace—investigate processing time, partition count, or broker I/O.

Key broker defaults

Network tuning

Replication tuning

• replica.lag.time.max.ms — follower out of ISR if no fetch within this window (default 30s)
• replica.fetch.max.bytes — max bytes per follower fetch; raise for large messages (align with message.max.bytes)
• num.replica.fetchers — parallel fetchers per broker; 1–4 typical

Unclean leader election

unclean.leader.election.enable=false in production. When true, a non-ISR replica can become leader after full ISR loss—data loss for un-replicated writes. With false, partition goes offline until ISR replica returns—availability hit, but no silent truncation.

JVM heap and GC

Kafka brokers rely on the OS page cache for hot reads—not a giant heap. ~6 GB heap is the common ceiling; more heap steals RAM from cache and can hurt throughput.

export KAFKA_HEAP_OPTS="-Xms6g -Xmx6g"
export KAFKA_JVM_PERFORMANCE_OPTS="-XX:+UseG1GC \
  -XX:MaxGCPauseMillis=20 \
  -XX:InitiatingHeapOccupancyPercent=35 \
  -XX:+ExplicitGCInvokesConcurrent \
  -Djava.awt.headless=true"

num.partitions=12
default.replication.factor=3
min.insync.replicas=2
unclean.leader.election.enable=false
log.retention.hours=168
log.segment.bytes=1073741824
num.network.threads=8
num.io.threads=16
replica.lag.time.max.ms=30000
replica.fetch.max.bytes=1048576
auto.create.topics.enable=false

kafka-reassign-partitions.sh

Workflow: generate JSON plan → execute → verify → (optional) throttle → remove throttle.

# Generate plan: move all partitions off broker 3
kafka-reassign-partitions.sh --bootstrap-server kafka:9092 \
  --broker-list 3 --generate

# Save proposed assignment to reassignment.json, edit if needed, then:
kafka-reassign-partitions.sh --bootstrap-server kafka:9092 \
  --reassignment-json-file reassignment.json --execute

# Monitor progress
kafka-reassign-partitions.sh --bootstrap-server kafka:9092 \
  --reassignment-json-file reassignment.json --verify

When reassignment is needed

• Broker added — existing partitions never auto-migrate; rebalance to use new capacity
• Broker decommission — evacuate all partitions before shutdown
• Uneven distribution — one broker holds disproportionate leaders or disk usage
• Rack awareness fix — after correcting broker.rack assignments

Throttle reassignment

Unthrottled replication can peg network and spike produce latency. Set per-broker throttle before execute:

kafka-configs.sh --bootstrap-server kafka:9092 \
  --alter --add-config "follower.replication.throttled.replicas=0:0:0-1000,1:0:0-1000" \
  --entity-type brokers --entity-name 0

kafka-configs.sh --bootstrap-server kafka:9092 \
  --alter --add-config "leader.replication.throttled.replicas=0:0:0-1000" \
  --entity-type brokers --entity-name 1

# Set rate (bytes/sec) — also replica.alter.log.dirs.io.max.bytes.per.second
kafka-configs.sh --bootstrap-server kafka:9092 \
  --alter --add-config "leader.replication.throttled.rate=52428800" \
  --entity-type brokers --entity-name 0

# After --verify completes: remove throttle configs

flowchart LR
  G[--generate plan]
  T[Apply throttle]
  E[--execute]
  V[--verify complete]
  R[Remove throttle]

  G --> T --> E --> V --> R

Cruise Control

LinkedIn’s Cruise Control automates partition rebalancing from broker metrics (CPU, disk, network, leader count). Goals: rack-aware distribution, even load, broker capacity limits. Supports anomaly detection and self-healing proposals— the operational upgrade from manual JSON plans at scale.

flowchart LR
  B[Kafka brokers\nJMX :9999]
  P[Prometheus JMX Exporter\n/metrics]
  PR[Prometheus]
  G[Grafana dashboards]
  A[Alertmanager / PagerDuty]
  BU[Burrow\nlag SLA]

  B --> P --> PR --> G
  PR --> A
  B --> BU --> A

Broker metrics (critical)

Producer metrics

Consumer metrics

JMX and Prometheus

Kafka brokers expose MBeans under kafka.server, kafka.network, etc. Run JMX Exporter as a Java agent or sidecar; it maps MBean patterns to Prometheus gauges/counters on /metrics.

rules:
  - pattern: kafka.server<>Value
    name: kafka_under_replicated_partitions
    type: GAUGE
  - pattern: kafka.controller<>Value
    name: kafka_active_controller_count
    type: GAUGE

Grafana dashboards

Confluent publishes official Kafka Grafana dashboards (broker overview, topic, consumer lag). Import by dashboard ID or JSON; wire Prometheus datasource. Panel highlights: URP, offline partitions, bytes in/out, request handler idle.

Burrow

LinkedIn Burrow evaluates consumer lag trends—not just absolute lag—and emits status (OK, WARN, ERR, STOP). Better than static “lag > 10000” alerts that false-positive during deploys. Pair with SLA definitions per consumer group.

flowchart TB
  C[Client]
  TLS[SSL/TLS\nencrypt + broker identity]
  SASL[SASL authentication\nwho are you]
  ACL[ACL authorization\nwhat can you do]
  B[Broker]

  C --> TLS --> SASL --> ACL --> B

Authentication mechanisms

listeners=SASL_SSL://0.0.0.0:9093
advertised.listeners=SASL_SSL://broker1.example.com:9093
security.inter.broker.protocol=SASL_SSL
sasl.mechanism.inter.broker.protocol=SCRAM-SHA-512
sasl.enabled.mechanisms=SCRAM-SHA-512
ssl.keystore.location=/var/ssl/kafka.server.keystore.jks
ssl.keystore.password=${KEYSTORE_PASSWORD}
ssl.key.password=${KEY_PASSWORD}
authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizer
super.users=User:admin

Authorization — ACLs

KRaft clusters use StandardAuthorizer (or legacy AclAuthorizer). ACLs bind principal → operation → resource. Deny rules override allow.

Operations: Read, Write, Create, Delete, Alter, Describe, ClusterAction, IdempotentWrite, …

# Producer: write to orders topic
kafka-acls.sh --bootstrap-server kafka:9093 --command-config admin.properties \
  --add --allow-principal User:order-service \
  --operation Write --topic orders

# Consumer: read topic + join group
kafka-acls.sh --bootstrap-server kafka:9093 --command-config admin.properties \
  --add --allow-principal User:order-processor \
  --operation Read --topic orders

kafka-acls.sh --bootstrap-server kafka:9093 --command-config admin.properties \
  --add --allow-principal User:order-processor \
  --operation Read --group order-processor

# List ACLs for topic
kafka-acls.sh --bootstrap-server kafka:9093 --command-config admin.properties \
  --list --topic orders

Encryption at rest

Not built into Kafka. Options: LUKS/dm-crypt on broker volumes, cloud EBS encryption (AWS KMS, GCP CMEK), or encrypted filesystems. Inter-broker and client-broker TLS covers in-flight; at-rest protects stolen disks.

Audit logging

Open-source Kafka has limited built-in audit trails. Confluent Platform adds audit logs (authentication, ACL changes, admin actions). Alternatives: broker access logs, Kubernetes audit, SIEM ingestion of ACL CLI changes via GitOps PR history.

Storage

disk ≈ retention_seconds × peak_bytes_in_per_sec × replication_factor × 1.2

Example:
  retention = 7 days = 604,800 s
  peak ingest = 50 MB/s
  RF = 3
  → 604800 × 50 × 3 × 1.2 ≈ 108 TB raw cluster (spread across brokers)

Add: compacted topics, mirror-maker duplication, segment overhead

The 1.2 factor covers compaction lag, segment rounding, and headroom. Monitor actual disk per broker; alert at 70% sustained—reassignment needs free space.

Network

peak_egress ≈ peak_bytes_in × (replication_factor - 1 + consumer_fan_out)

Example:
  50 MB/s in, RF=3, 2 consumer clusters reading full stream
  replication traffic: 50 × (3-1) = 100 MB/s internal
  consumer traffic: 50 × 2 = 100 MB/s
  → broker NIC planning ~200+ MB/s sustained per ingress broker (not burst-only)

Cross-AZ replication doubles cloud egress bills—prefer rack-aware same-AZ replicas when AZ failure tolerance allows, or accept cost for true AZ isolation.

CPU

Typically not the first bottleneck except with heavy compression (zstd level high), SSL everywhere, or excessive metric cardinality scraping. Profile before buying larger CPUs—add brokers for I/O parallelism first.

Partition ceiling per broker

Operational limit: ~2,000–4,000 partitions per broker (cluster-dependent). Beyond this:

• Controller metadata grows—slower leader election and topic creation
• Rebalance and recovery times stretch to hours
• Each partition = file handles and metadata memory

flowchart LR
  D[Disk I/O\npage cache]
  N[Network\nRF + fan-out]
  M[Metadata\npartitions / broker]
  C[CPU\ncompression / SSL]

  D -->|usually first| N --> M --> C