Kafka Performance Tuning

Benchmark baseline: kafka-producer-perf-test.sh

Establish MB/s and records/s before changing configs. Run from a host with network proximity to brokers:

kafka-producer-perf-test.sh \
  --topic perf-test \
  --num-records 5000000 \
  --record-size 1024 \
  --throughput -1 \
  --producer-props \
    bootstrap.servers=kafka:9092 \
    acks=1 \
    batch.size=131072 \
    linger.ms=20 \
    compression.type=lz4 \
    buffer.memory=134217728

# Output: records/sec, MB/sec, avg latency, max latency, 50th/95th/99th/999th percentiles

Vary one knob per run. Save results with topic partition count and broker version noted— perf tests on empty clusters overstate real-world numbers.

Throughput-oriented producer config

batch.size=131072
linger.ms=20
compression.type=lz4
buffer.memory=134217728
acks=1
retries=3
max.in.flight.requests.per.connection=5

Async send — never block in a loop

producer.send(record).get() in a tight loop serializes on broker RTT—throughput collapses to ~1 request at a time. Fire async sends and optionally flush at interval or use callback batching.

// BAD — synchronous, ~hundreds msg/s
for (Event e : events) {
  producer.send(new ProducerRecord<>("topic", e.getKey(), e)).get();
}

// GOOD — async pipeline, flush periodically
List<CompletableFuture<RecordMetadata>> futures = new ArrayList<>();
for (Event e : events) {
  futures.add(producer.send(new ProducerRecord<>("topic", e.getKey(), e))
      .thenApply(r -> r.recordMetadata()));
}
CompletableFuture.allOf(futures.toArray(CompletableFuture[]::new)).join();
// or: producer.flush() every N records in streaming loop

Partition count and producer parallelism

Producer maintains a separate batch per partition. More partitions = more parallel in-flight requests to brokers— up to broker and network limits. Diminishing returns past ~100–200 partitions per topic without more brokers. Align with target consumer parallelism (see Partition guidelines).

flowchart LR
  P[Producer]
  B0[Batch partition 0]
  B1[Batch partition 1]
  B2[Batch partition 2]
  BR[Broker leaders]

  P --> B0
  P --> B1
  P --> B2
  B0 --> BR
  B1 --> BR
  B2 --> BR

Benchmark: kafka-consumer-perf-test.sh

# First, load topic with producer perf test, then:
kafka-consumer-perf-test.sh \
  --bootstrap-server kafka:9092 \
  --topic perf-test \
  --messages 5000000 \
  --threads 1 \
  --consumer.config consumer-perf.properties

# consumer-perf.properties:
# fetch.min.bytes=100000
# fetch.max.wait.ms=500
# max.poll.records=1000

Throughput-oriented consumer config

fetch.min.bytes=100000
fetch.max.wait.ms=500
max.poll.records=1000
max.poll.interval.ms=300000
session.timeout.ms=45000
enable.auto.commit=false

Parallel processing

Two levers—often combined:

• Increase partition count — more buckets for consumer group to assign
• Increase concurrency — Spring concurrency or multiple consumer instances in same group

Effective parallelism = min(partitions, consumer_threads). Adding 20 threads to 6 partitions wastes 14 threads.

Avoid slow work in the poll loop

If processing exceeds max.poll.interval.ms, coordinator revokes partitions—rebalance storm. Patterns for heavy handlers:

1. Delegate to thread pool — poll thread hands off records; commit after worker completes (careful with ordering)
2. pause / resume — pause partition while backlog processes, resume when caught up
3. Batch + bulk insert — DB bulk write per poll batch instead of per-record round trips

@KafkaListener(id = "heavy-listener", topics = "events")
public void onMessage(ConsumerRecord<String, Event> record,
    Consumer<String, Event> consumer,
    Acknowledgment ack) {
  TopicPartition tp = new TopicPartition(record.topic(), record.partition());
  consumer.pause(List.of(tp));
  workerPool.submit(() -> {
    try {
      heavyProcess(record.value());
      ack.acknowledge();
    } finally {
      consumer.resume(List.of(tp));
    }
  });
}

Filesystem and mount options

• XFS or ext4 — standard choices; XFS common on large sequential volumes
• Separate disks — log dirs on dedicated NVMe/SSD; avoid sharing with OS or ZooKeeper (legacy)
• noatime — disable access-time updates on log mount; reduces metadata writes
• RAID10 / JBOD — JBOD with one log dir per disk often beats RAID5 for write latency

/dev/nvme0n1  /var/kafka-logs  xfs  defaults,noatime,nodiratime  0  0

Network sysctl

Increase socket buffer defaults so high-throughput TCP streams do not stall on small kernel buffers:

net.core.rmem_default = 1048576
net.core.wmem_default = 1048576
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 1048576 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.core.netdev_max_backlog = 250000

Align with broker socket.send.buffer.bytes / socket.receive.buffer.bytes (often 100 KB–1 MB).

VM and page cache

JVM heap

4–6 GB max heap; set -Xms equal to -Xmx to avoid resize pauses. Remaining RAM for OS cache. G1GC with low pause target.

export KAFKA_HEAP_OPTS="-Xms6g -Xmx6g"
export KAFKA_JVM_PERFORMANCE_OPTS="-XX:+UseG1GC \
  -XX:MaxGCPauseMillis=20 \
  -XX:InitiatingHeapOccupancyPercent=35 \
  -XX:+DisableExplicitGC"

Broker thread pools

num.network.threads=8
num.io.threads=16
socket.send.buffer.bytes=1048576
socket.receive.buffer.bytes=1048576
replica.fetch.max.bytes=1048576
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000

Producer — low latency

linger.ms=0
batch.size=16384
compression.type=none
acks=1
buffer.memory=67108864

Each record ships immediately (per partition batch may still be small). For sub-10ms paths, co-locate producers with brokers (same AZ). acks=all adds replica round-trip—required for durability, costs latency.

Consumer — low latency

fetch.min.bytes=1
fetch.max.wait.ms=0
max.poll.records=100

Consumer returns as soon as any data is available—more polls, higher CPU, lower end-to-end delay.

Broker — replication fetch

• num.replica.fetchers — more parallel follower fetch threads
• replica.fetch.min.bytes=1 — followers do not wait for large fetch batches
• Keep ISR healthy — out-of-sync replicas force leader-only reads or delayed commits

Avoid on latency-critical paths

• Compression — CPU cycles add milliseconds; use only if network is the proven bottleneck
• Large batches / high linger — intentional delay to fill batches
• TLS without hardware acceleration — software AES/GCM at very high RPS adds up; use AES-NI capable instances
• Oversized max.poll.records — long processing before next fetch

flowchart TB
  subgraph budget["Typical p99 budget (same AZ)"]
    L[linger 0–5ms]
    N[Network RTT 1–3ms]
    B[Broker append 1–5ms]
    R[Replication if acks=all 2–10ms]
    C[Consumer poll 1–50ms]
  end
  L --> N --> B --> R --> C

Too few partitions

• Single consumer thread per topic caps throughput
• Hot partition on skewed key distribution
• Cannot scale consumer fleet past partition count

Too many partitions

• More leader elections and metadata on controller
• Slower consumer group rebalance (more assignments)
• More file handles and segment files per broker (~2–4 per partition minimum)
• Producer batches spread thin—smaller batches if key space is sparse

Sizing formula

partitions ≈ target_throughput ÷ single_partition_throughput

single_partition_throughput (rule of thumb):
  HDD brokers:     ~10–30 MB/s
  SSD brokers:     ~30–80 MB/s
  NVMe + 10 Gbps:  ~50–100+ MB/s  (measure with perf test!)

Example:
  Target 600 MB/s ingest, measured 40 MB/s per partition → 600 ÷ 40 = 15 → use 18–24 with headroom

Start with 12–24 for medium workloads; revisit when BytesInPerSec per broker or consumer lag growth indicates saturation. See Topic management—partitions cannot be reduced.

flowchart LR
  Few[Too few\nconsumer bound]
  Sweet[Sweet spot\n12-48 typical topic]
  Many[Too many\nrebalance + metadata cost]

  Few -->|add partitions| Sweet
  Sweet -->|uncapped growth| Many