How Kafka Works Under the Hood

Pick any high-throughput data pipeline at a tech company — clickstream analytics at 500k events/second, real-time fraud detection on payment transactions, CDC from a Postgres database feeding a dozen downstream services — and there's a good chance Kafka is sitting in the middle of it. Most teams treat it as a black box: messages go in, messages come out, distributed durability happens somehow.

That "somehow" is worth understanding. The internal design explains almost every operational decision you'll ever have to make about Kafka, from partition counts to acks settings to why your consumers keep lagging after deployments.

The Mental Model

Kafka is a distributed commit log, not a message queue. The distinction matters. A traditional message queue removes messages after they're consumed. Kafka retains everything for a configurable window (hours, days, weeks) and lets consumers track their own read position in that log. Multiple consumer groups can read the same data independently, at different speeds, without affecting each other.

Every topic is split into partitions. Each partition lives on one broker as the leader, with copies on other brokers as followers. Producers always write to the leader.

Consumers read from the leader by default, and that "by default" is load-bearing. Since Kafka 2.4 (KIP-392) a broker can redirect a consumer to a nearby replica through a pluggable ReplicaSelector. LeaderSelector is the default and preserves the old behaviour; swap in RackAwareReplicaSelector via replica.selector.class, give your consumers a client.rack, and they'll fetch from a same-AZ follower instead. On a cloud bill where cross-AZ transfer is metered per gigabyte, that is often the single largest Kafka saving available, and it means "followers exist purely for durability" has been out of date for years.

The Commit Log: Segments on Disk

Kafka doesn't store data in a database or use any intermediate format. It writes directly to the filesystem in an append-only structure. Each partition maps to a directory containing a sequence of segments:

/var/kafka/data/orders-0/
  00000000000000000000.log       # message data, base offset = 0
  00000000000000000000.index     # offset → byte position (sparse)
  00000000000000000000.timeindex # timestamp → offset
  00000000000001048576.log       # next segment, base offset = 1,048,576
  00000000000001048576.index
  00000000000001048576.timeindex

The filename prefix is the base offset of the first message in that segment — a message count, not a byte count, so the gap between two filenames tells you how many records fit in a segment, not how big it was. When the active segment hits log.segment.bytes (default 1073741824, i.e. 1 GB) or log.roll.hours (default 168, i.e. 7 days), whichever comes first, Kafka closes it read-only and opens a new one. All new writes always go to the single active segment — sequential, append-only.

To look up a message at offset 5000, Kafka does three steps:

1. Finds which .log contains offset 5000, by a floor lookup over the sorted segment base offsets
2. Binary-searches that segment's .index for the nearest lower entry — the index is sparse, one entry per log.index.interval.bytes (default 4096) of log
3. Scans forward in the .log from that byte position

The sparse index keeps the index file small enough to fit in memory while still giving O(log n) lookup. Kafka memory-maps the index files, so lookups are effectively in-RAM reads.

Sequential disk writes are the foundation of Kafka's throughput. The number that matters isn't the peak sequential bandwidth of your disk — measure that yourself with fio if you care, because it varies by an order of magnitude between a network EBS volume and a local NVMe. The structural point is that Kafka never seeks on write. On spinning disks that's the difference between fast and unusable; on SSDs it's the difference between a well-behaved write pattern and write amplification. The entire write path is: receive batch from producer → append to the active .log → done. The OS page cache handles the flush asynchronously.

Replication: Leaders, Followers, and the ISR

Fault tolerance comes from replication. Each partition has a replication factor (typically 3): one leader and two followers. The followers replicate from the leader by sending fetch requests, just like a consumer does, and appending the received batches to their own local logs.

The In-Sync Replica List

Kafka tracks the In-Sync Replica (ISR) list. The rule is replica.lag.time.max.ms, default 30000, and Kafka's own config doc states it precisely: "If a follower hasn't sent any fetch requests or hasn't consumed up to the leader's log end offset for at least this time, the leader will remove the follower from ISR." Note it's either condition — a follower that is fetching busily but never catching up gets dropped too. When it catches up again, it's re-added.

This is what makes producer acks meaningful:

Pair acks=all with min.insync.replicas=2 and you get a hard guarantee: if fewer than 2 replicas are in-sync, the produce request fails loudly with NotEnoughReplicasException rather than silently accepting data that could be lost.

When a leader crashes, the cluster controller elects a new leader — but only from the current ISR. Every ISR member is guaranteed to have every committed message, so no data loss occurs on failover. There's an escape hatch: unclean.leader.election.enable=true allows election of an out-of-sync replica, which sacrifices durability for availability. Don't enable this unless you know exactly what you're losing.

The Controller: Cluster Metadata in KRaft

Someone has to decide which broker is the leader for each partition, detect broker failures, and trigger leader elections. In Kafka 4.0+, this is handled by a controller quorum running the Raft consensus protocol — no ZooKeeper required.

A small set of brokers (3 or 5) form the quorum. One is elected the active controller via Raft. All cluster metadata — partition assignments, leader changes, broker registrations — is appended as events to a special internal topic called __cluster_metadata. Other controllers replicate this log via Raft. Regular data brokers subscribe to it and maintain their own local cache of cluster state.

__cluster_metadata topic structure (simplified):
  offset 0: BrokerRegistration(brokerId=1, host=kafka1, port=9092)
  offset 1: BrokerRegistration(brokerId=2, host=kafka2, port=9092)
  offset 2: TopicRecord(topicId=uuid, name=orders)
  offset 3: PartitionRecord(topicId=uuid, partitionId=0, leader=1, isr=[1,2,3])
  offset 4: PartitionChangeRecord(partitionId=0, leader=2, isr=[2,3])  # broker 1 failed

The reason this is faster than the ZooKeeper design is structural, and Confluent's own write-up on the change puts it plainly. Under ZooKeeper, "the first thing the new controller will do is to fetch metadata from ZooKeeper, including all of the topic partition information across all of the" brokers, and then propagate it "to all of the other brokers, one at a time" — work that, in their words, "could take seconds or even more." Under KRaft, "the newly elected leader would already have replicated all of the committed records up to the new epoch and thus wouldn't need any time to bootstrap from the metadata log."

Confluent published a comparison on a cluster with two million topic partitions and reported that "for both controlled shutdown and uncontrolled failover, the latency was largely reduced with the Quorum Controller." I am deliberately not putting a millisecond figure on it here: the numbers depend entirely on partition count, and the ones that circulate get detached from the cluster they were measured on. The mechanism is the durable claim — a new controller with the log already in memory has nothing to load.

This also removed Kafka's external dependency on ZooKeeper, which was its own complex distributed system to operate.

Consumer Groups and Partition Assignment

Consumers don't subscribe to topics in the abstract — they get assigned specific partitions. Within a consumer group, each partition is consumed by exactly one consumer. This is the constraint that makes Kafka scale reads linearly with partition count: more partitions means more consumers working in parallel.

The Group Coordination Protocol

Every consumer group has a group coordinator — a specific broker determined by hashing the groupId against the __consumer_offsets topic partitions. The coordinator manages the group's lifecycle.

When a consumer starts (or when any member joins or leaves), a rebalance occurs:

The partition assignment is computed by the group leader (one of the consumers), not the coordinator. This keeps the coordinator's logic simple and lets you swap in custom assignment strategies.

Two of those strategies get conflated, and the version numbers are the giveaway. StickyAssignor came from KIP-54 and has been around since Kafka 0.11 — it minimises partition movement, but still follows the eager protocol, so every member revokes everything before the new assignment lands. CooperativeStickyAssignor is the Kafka 2.4 one (KIP-429): same sticky algorithm, but incremental revocation, so consumers keep processing the partitions that aren't moving while the rebalance runs. If you're carrying partition-local state, cooperative is the one you want, and Kafka's own javadoc agrees: "Users should prefer this assignor for newer clusters." Note the upgrade path — you cannot mix protocols freely across a group mid-flight.

Offset Commits

After processing a batch, consumers commit their position to __consumer_offsets — a compacted internal topic where the key is (groupId, topic, partition) and the value is the latest committed offset. On restart, the consumer reads its last committed offset from this topic and resumes from there.

// Process first, then commit — this is at-least-once
while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
    process(records);
    consumer.commitSync(); // blocks until the coordinator persists the offset
}

The ordering of those two lines is the delivery guarantee, and there is nothing else to it:

• process, then commit (above): a crash between them replays the batch. At-least-once. Your handler must be idempotent.
• commit, then process: a crash between them loses the batch. At-most-once. Almost never what you want, and easy to write by accident if you enable enable.auto.commit, because the auto-committer fires from inside poll() — before your handler has necessarily finished with the previous batch.

Exactly-once delivery is possible but requires Kafka transactions: the producer and consumer operate within a transaction that atomically commits the output messages and the input offset together.

Zero-Copy Reads: Why Kafka Can Move Data Fast

When a consumer fetches messages, Kafka uses the sendfile() system call to transfer data directly from the OS page cache to the network socket. The application layer never touches the data.

Without zero-copy, the path is:

Disk → Kernel page cache → User space buffer → Kernel socket buffer → NIC

With sendfile():

Disk → Kernel page cache → NIC

Two fewer copies, and no CPU cycles spent moving bytes through user space. Combine that with the fact that active data is almost always warm in the page cache, and most consumer fetches become pure memory-to-network transfers. Your OS is doing the work, not your JVM heap.

Practical Implications

These internals translate directly into operational decisions:

Partitions are your parallelism ceiling. You cannot have more active consumers in a group than partitions. A topic with 4 partitions and 8 consumers means 4 consumers sit idle. Plan partition count based on your target consumer throughput, not current load — you can increase partitions but never decrease them without recreating the topic.

Producer batching is free throughput. Kafka's producer accumulates messages into batches before sending (linger.ms, batch.size). Larger batches mean fewer network round-trips, better compression ratios, and fewer disk writes. For high-throughput producers, tune linger.ms=5 and batch.size=64KB before reaching for more partitions.

Rebalances hurt. Every join/leave event pauses consumption for all members of the group during the rebalance window. In production, use session.timeout.ms and heartbeat.interval.ms carefully (session timeout should be 3x heartbeat interval). For long-lived consumers with stable partition state, group.instance.id enables static membership — the consumer reconnects to its previous assignment without triggering a full rebalance.

Consumer lag is messages, not time. records-lag-max (the metric exposed by the consumer) tells you how many messages behind you are. On a bursty topic, 100k message lag might clear in seconds. On a slow topic, 100k message lag might be hours. Always correlate lag with topic throughput to understand real urgency.

The commit log model is the thing to hold onto. Everything else — ISR replication, KRaft controller quorum, consumer group protocol, zero-copy sendfile — is engineering in service of one idea: an append-only, retention-bounded log where producers are decoupled from consumers by an offset they each maintain independently. Once that model is solid in your head, every Kafka config option becomes obvious.